Context

AI systems applied to clinical settings face requirements of reliability, explainability and regulatory compliance that most general-purpose AI architectures do not cover. A model that achieves good results on a retrospective dataset in the lab requires, to be used on a ward, a full system around it: data acquisition, traceability, integration with hospital information systems, prospective clinical validation, regulatory documentation. The gap between the two levels is mostly architectural, not algorithmic.

Challenges specific to healthcare

Healthcare systems operate with sensitive data, high-risk decisions and regulated workflows. AI must integrate into these contexts without compromising their safety and reliability:

  • Patient data protection (PII, clinical data)
  • Explainability of decisions for clinical staff
  • Integration with existing hospital systems (HIS, LIS, electronic health record)
  • Regulatory compliance (GDPR, MDR, EU AI Act)
  • Clinical validation of AI outputs through prospective studies

Digital twin in paediatrics

In the Short Bowel Syndrome (SBS) project with Meyer Children’s Hospital in Florence, we worked on digital twins of paediatric patients with a rare disease — computational models that replicate relevant physiological parameters (parenteral nutrition, fluid balance, growth) to support therapeutic decision-making.

Applications of this kind require much more than just model accuracy. The clinician must be able to understand why the system suggests a dosage or therapeutic adjustment, verify the data the decision is based on, and have guarantees about patient data protection at every stage of the process.

An architectural approach

Trustworthy AI for healthcare requires a system architecture designed for transparency, monitoring and control. Every AI component must be auditable, every decision traceable, every piece of data protected.

In practice, this means designing systems with:

  • Data pipelines with a full audit trail — every transformation documented and reversible
  • Models with explainable output — not just the result but the reasoning that produced it
  • Native anonymisation layers — personal data must never reach the model in clear text
  • Integration with existing clinical standards (HL7 FHIR, DICOM) to ensure interoperability
  • Continuous monitoring of model performance in production with drift detection

The European regulatory framework

The EU AI Act classifies healthcare AI systems as “high-risk”, imposing stringent requirements on documentation, transparency and human oversight. The Medical Device Regulation (MDR) adds further constraints for systems classified as medical devices. Designing regulatory compliance as part of the architecture — and not as an after-the-fact addition — is the only sustainable approach.

Operational preconditions

Clinicians’ trust in an AI system is built through verifiable system properties. Data auditability, decision explainability, evidence traceability, native regulatory compliance — the same criteria OISG formalises as adequacy requirements for autonomous AI systems. In healthcare these properties are operational preconditions.