AI in Hospitals: From Concept to Safe, Tested Operation

How a Governance Framework Based on Values, Clear Roles, and Reliable Processes Enables the Responsible Use of AI in Clinical Practice

Authors: Prof. Dr. Thomas Neumuth, Prof. Dr. Toralf Kirsten, Leipzig University Medical Center, July 27, 2026

Artificial intelligence is making its way into everyday clinical practice, from diagnostic decision support to automated documentation. At the same time, the European AI Act, the Medical Device Regulation (MDR), and the General Data Protection Regulation (GDPR) establish a dense regulatory framework that poses a practical question for hospitals: How can AI be introduced in a way that allows for innovation while ensuring safety, ethics, and legal compliance? The guideline, developed as part of the creation of an AI policy for a German university hospital, translates these requirements into a practical governance model [1].

The guideline comprises five binding core statements. First and foremost is a principle that also shapes the debate on medical responsibility: AI supports, but does not replace. Overall responsibility—particularly medical responsibility—always remains with humans. This gives rise to the following provisions: Every application is reviewed, ethically evaluated, and clearly explained; responsibility is clearly distributed among users, developers, and operators; every AI system undergoes a centralized, documented implementation process; and violations result in consequences under labor law, civil law, and, where applicable, criminal law.

This is underpinned by a value framework consisting of three guiding principles and seven core principles, ranging from “support rather than replacement” to fairness, accountability, and explainability. They reflect the core requirements of the EU High-Level Expert Group on Trustworthy AI [2] and can be operationalized using the ALTAI self-assessment checklist [3].

Classifying an AI system into one of the five risk categories of the AI Act is the central control mechanism of governance. It determines the scope of risk analysis, documentation, transparency requirements, and approval. The spectrum ranges from prohibited practices such as social scoring (Category I, completely banned) to high-risk AI—such as medical diagnostics—which requires systematic risk analysis and integration with the MDR, all the way to GPAI models, such as large language models, which have their own documentation requirements. Important for the hospital context: AI systems that support diagnostic or therapeutic decisions are generally considered medical devices and must be approved as such.

Verification requirements apply before any system is put into service. These include a trial run in the actual operational environment, verification of the CE marking and the declaration of conformity, and—in cases of incomplete documentation—independent validation. In addition, every application is documented in an in-house AI registry. This serves as a single source of truth for transparency, audits, and vigilance.

Responsibility is divided among three operational roles: users, developers or manufacturers, and specifically designated operators for each AI system. In addition, there are three central units: an AI Office serving as a visible point of contact with its own mandate; an interdisciplinary AI Regulatory Working Group comprising experts from quality management, compliance, data protection, IT security, medical technology, and the legal department; and the Project Management Office, which includes the Data Protection and Information Security Officers. This framework can be structurally embedded in a management system compliant with ISO/IEC 42001 [4].

Every AI project follows a standardized eight-step process. This ranges from the initial idea (1) through the initial consultation with the AI Office (2), the regulatory checklist for classification as a medical device, risk classification, and ALTAI assessment (3), registration (4), evaluation by the Regulatory Working Group (5), and mandatory consultation for applications in clinical care and administration (6), all the way to the PMO application—including a cost-benefit decision (7)—and ongoing operation with continuous vigilance (8). Ideas are thus channeled early on, assessed from a regulatory perspective, and fully documented before any investment decision is made.

The process is accompanied by a clear prerequisite for use. The right to use the system is granted only after training, documented proof of competence, and written approval from a supervisor. This directly implements the obligation to ensure AI competence in accordance with Article 4 of the AI Regulation. The framework of sanctions demonstrates that these rules are not mere declarations of intent. The AI Regulation provides for fines of up to 35 million euros or seven percent of global annual revenue; the GDPR additionally provides for fines of up to 20 million euros or four percent.

The guideline shows that the introduction of AI in a hospital is not purely a technical project, but above all an organizational one. Those who combine a value framework, risk classification, clear responsibilities, and a standardized implementation process lay the foundation for AI use that enables innovation while simultaneously ensuring patient safety, medical accountability, and regulatory compliance. Ten recommendations for action—ranging from an established mission statement to transparent reporting channels—translate the framework into everyday hospital practice.