A language model assisting a clinician over a patient’s data inherits two regulatory regimes at once: the EU AI Act, which classifies it as high-risk when it is a medical device or a safety component of one, and the GDPR, which places health data among the special categories of Article 9. The two regimes do not substitute for one another: they stack. It is this constraint, more than the choice of model, that dictates how to build an architecture putting an LLM (Large Language Model) in contact with clinical records, reports, and diagnostic images.

Context and perimeter

Clinical data has a property ordinary data lacks: it cannot leave the perimeter in which it was collected without an explicit legal basis. Article 9 GDPR generally prohibits the processing of health data, with derogations only on exhaustive conditions — explicit consent, care or public-health purposes with a basis in law, research with a basis in law. Article 32 asks for security measures proportionate to the sensitivity of the data, naming encryption and pseudonymisation explicitly.

The practical consequence is that inference cannot be delegated to a remote endpoint that ships the prompt — and with it the clinical content — outside the controlled infrastructure. A call to an external managed service transfers the data to a third-party processor, reopens the legal-basis assessment and, if the provider sits outside the EU, the international-transfer chapter too. This is why systems working over clinical data gravitate towards locally executed models: inference stays where the data has the right to be.

Architecture

The recurring pattern brings together three planes. The first is structured-data access: HL7 FHIR R4 for clinical records, DICOM for diagnostic images. FHIR exposes resources queryable over a REST API; DICOM handles radiology studies and the associated metadata. A system retrieving context for the model reads from a FHIR server and, for the imaging part, from normalised DICOM metadata — FHIR resources exist precisely to represent imaging metadata and carry it back into the clinical repository.

The second plane is the RAG (Retrieval-Augmented Generation) pipeline: the model is not retrained on patient data but, at runtime, receives the relevant fragments pulled from a vector index built over guidelines, protocols, and case-specific data. The exposure surface is reduced — the data never lands in the model’s weights — and the provenance of what the model saw becomes traceable. Work such as FHIR-RAG-MEDS describes exactly this coupling: patient-specific retrieval from a FHIR server plus clinical guidelines from a vector database.

The third plane is the model executed on local infrastructure, behind the organisation’s network. This is where the sovereignty cost concentrates: on-premise GPUs, model lifecycle management, monitoring. In exchange, no fragment of a clinical prompt crosses the infrastructure boundary.

The critical point: governance in both directions, and logging

What separates a prototype from a system you can put into production is the governance plane applied both on the way in and on the way out. On the way in, PII redaction (Personally Identifiable Information) must happen before the text reaches the index or the model: direct identifiers — name, national ID, contact details — must be detected and neutralised upstream, because once they enter the model’s context or the logs they become an additional processing operation to justify. On the way out, the output must be inspected before it is shown, because a model can reconstruct or surface information that should not appear.

With logging, the EU AI Act becomes binding in a measurable way. Article 12 requires high-risk systems to perform the automatic recording of events across the whole lifecycle, with retention of at least six months (aligned to the longer period where other obligations, such as GDPR ones, demand it). Article 9 requires a continuous, iterative risk-management system, not a one-off assessment. In architectural terms this means every governance decision — what was redacted, what was blocked, which fragment was retrieved — must leave a retained, verifiable trace. A hash chain over the audit records makes the trace’s integrity checkable after the fact, which a mutable log does not guarantee.

Implications: risk classification is not optional

A system that takes part in a clinical decision is rarely low-risk under the EU AI Act. Classification follows Article 6: if the AI is itself a medical device, or a safety component of a device regulated under MDR (Medical Device Regulation) or IVDR (In Vitro Diagnostic Regulation), it falls automatically into the high-risk category of Article 6(1). The timeline here is precise: for systems that are high-risk because embedded in the regulated products of Annex I, the obligations apply from 2 August 2027, whereas for Annex III systems not subject to a notified-body conformity assessment the date is 2 August 2026.

This means risk classification belongs in the design phase, not downstream of deployment, because it decides which documentation and risk-management obligations apply. A classifier that maps the system against Articles 9–15 and flags the gaps serves more as a design tool than as a final formality: it tells you what is missing while the architecture can still be changed.

Limits

Keeping model and data inside the perimeter does not close the compliance question on its own. Upstream pseudonymisation reduces but does not eliminate the re-identification risk, especially when the retrieved context is rich and the clinical case is rare. Immutable logging sits in tension with the right to erasure: an audit hash chain, by definition, cannot be touched, and must be designed to keep governance metadata separate from content subject to data-subject rights. Risk assessment under MDR requires clinical competence that does not reduce to software. And the locally executed model remains exposed to the same errors as a remote one: inference inside the perimeter protects the data, not the correctness of the answer, which in a clinical setting must be validated by whoever holds responsibility for the decision.

A concrete translation of these constraints into a clinical-support suite — RAG over FHIR/DICOM data, on-premise architecture, an MDR path — is described by noze in the insight on AgenticHealth: https://www.noze.it/en/insights/open-intelligence-secure-governance/.

Cover image: A radiologist seated at a clinical workstation interprets brain MRI images displayed on a medical-grade monitor — photo by The Medical Futurist editors, CC BY 4.0 — https://commons.wikimedia.org/wiki/File:Radiologist_interpreting_MRI.jpg