Context

AI systems applied in clinical settings are subject to reliability, explainability and regulatory requirements that most general-purpose AI architectures do not cover. A model that performs well on a retrospective dataset in the lab needs an entire system around it to be usable in a ward: data acquisition, traceability, integration with hospital information systems, prospective clinical validation, regulatory documentation. The distance between these two levels is mostly architectural, not algorithmic.

Healthcare-Specific Challenges

Healthcare systems operate with sensitive data, high-stakes decisions, and regulated workflows. AI must integrate into these contexts without compromising their safety and reliability:

  • Patient data protection (PII, clinical data)
  • Decision explainability for clinical staff
  • Integration with existing hospital systems (HIS, LIS, electronic health records)
  • Regulatory compliance (GDPR, MDR, EU AI Act)
  • Clinical validation of AI results with prospective studies

Digital Twins in Paediatrics

In the Short Bowel Syndrome (SBS) project with Meyer Children’s Hospital in Florence, we worked on digital twins of paediatric patients with rare disease — computational models that replicate relevant physiological parameters (parenteral nutrition, fluid balance, growth) to support therapeutic decisions.

These applications require far more than model accuracy. The clinician must be able to understand why the system suggests a specific dosage or therapeutic modification, verify the underlying data, and have guarantees on patient data protection at every stage of the process.

An Architectural Approach

Trustworthy AI for healthcare is not achieved through better models alone — it requires a system architecture designed for transparency, monitoring, and control. Every AI component must be auditable, every decision traceable, every piece of data protected.

In practice, this means designing systems with:

  • Data pipelines with complete audit trails — every transformation documented and reversible
  • Models with explainable output — not just the result, but the reasoning that produced it
  • Native anonymisation layers — personal data must never reach the model in plaintext
  • Integration with existing clinical standards (HL7 FHIR, DICOM) to ensure interoperability
  • Continuous monitoring of model performance in production with drift detection

The European Regulatory Framework

The EU AI Act classifies healthcare AI systems as “high risk,” imposing stringent requirements for documentation, transparency, and human oversight. The Medical Device Regulation (MDR) adds further constraints for systems classified as medical devices. Designing regulatory compliance as part of the architecture — not as an afterthought — is the only sustainable approach.

Conclusion

Clinical trust in an AI system is built on verifiable system properties, not on its predictive capabilities. These properties — data auditability, decision explainability, evidence traceability, native regulatory compliance — are the same criteria that OISG formalises as adequacy requirements for autonomous AI systems. In healthcare they are not optional: they are operational preconditions.