AI Is Not Magic

How Credible AI Plays a Crucial Role – Especially in Medicine

Author: Dr Marika Kaden, Mittweida University of Applied Sciences, 21 March 2026

Artificial Intelligence (AI) is becoming increasingly embedded in our everyday lives, from self-driving cars to personalised recommendations. The possibilities seem almost limitless. Yet behind these impressive applications lies no magic, but rather complex mathematics and data analysis. Particularly in critical fields such as medicine, it is essential to understand how AI works and to trust it – and that is only possible through credible, interpretable AI.

There are different approaches to AI. In most cases, these involve Machine Learning (ML). Models are trained on large amounts of data to identify patterns and make predictions. A special case is Deep Learning, which is based on artificial neural networks inspired by the functioning of the human brain and is often highly complex.

The potential applications of ML in medicine are extensive. They range from the early detection of diseases and personalised treatment approaches to assistance during surgical procedures. ML systems can, for example, analyse X-ray images for signs of lung cancer that may even escape experienced radiologists. They can also analyse genetic data to predict disease risks or generate personalised medication plans tailored to individual patients. All of this is based on analysing large volumes of data in relation to a specific question and training a complex model accordingly.

However, in medicine – where lives and health are at stake – trust is essential. This trust is often challenged by the fact that many ML systems are considered black boxes. While the output of the model, such as a diagnosis, is visible, the path leading to that result is often difficult or impossible to interpret meaningfully.

Explainable AI: Looking Inside the Black Box

To shed light on these decision-making processes, researchers use methods from Explainable AI (XAI). XAI aims to make existing ML models more transparent and understandable. These methods are typically applied post hoc and seek to explain not only what an algorithm decides, but also why it reaches a particular decision.

Common approaches include:

1. Feature Importance
This method highlights which features – such as blood test values – had the greatest influence on a model’s decision. Two widely used techniques are:

• Local Interpretable Model-agnostic Explanations (LIME): explains the prediction of a single data point by building a simplified, interpretable model around it.
• SHapley Additive exPlanations (SHAP): uses concepts from game theory to quantify the contribution of each feature to a prediction.

2. Visualisation
Heatmaps and other graphical representations can help illustrate how an algorithm arrives at its conclusions.

3. Counterfactual Explanations
These methods demonstrate what would have needed to change in the input data to produce a different outcome. This allows clinicians to assess the robustness of AI decisions and understand which factors were truly decisive.

Models explained through such approaches are often referred to as grey-box models. In general, however, explanations are obtained by simplifying highly complex ML models. This can occasionally lead to misunderstandings or misinterpretations.

For this reason, a growing field of research focuses on interpretable models by design, where transparency and comprehensibility are built into AI systems from the outset.

Interpretable? It Depends.

But what exactly does “interpretable” mean? The answer is: it depends. Different aspects contribute to understanding, trust and acceptance:

• Relevance Learning: Understanding which input features influence decisions and which information is essential for the model.
• Smart and Robust Decisions (Low Data Budget & Model Certainty): Models should remain appropriately sized and reach reliable conclusions with efficient use of resources.
• Knowledge-Informed AI: Incorporating expert knowledge into model design improves trust and leads to more intelligent ML systems.
• Validation of Decision Pathways (Causal Learning): Verifying the logic and reasoning behind decisions to ensure consistency with domain expertise.
• Communication of Model Limitations and Rejection Strategies: AI systems should indicate uncertainty and reject inputs that differ substantially from training data.
• Bias Detection and Mitigation: Identifying and reducing biases in training data, such as gender or demographic bias.

No single interpretable ML model fulfils all of these requirements simultaneously. Moreover, not every application requires every aspect. Nevertheless, many of these requirements can be addressed directly through interpretable models, whereas black-box approaches often require additional methods or architectural modifications and therefore become grey-box systems.

Selecting the right ML model requires careful analysis of both the problem and the available data. A common mistake is to focus immediately on the latest algorithms without first clarifying the underlying challenge that needs to be solved. This is where interdisciplinary collaboration becomes essential.

Domain experts are needed to define the problem precisely and identify relevant data sources. Data scientists can then analyse, prepare and assess these data with regard to quality and availability. This is exactly where the KIMed network can provide support by bringing together different areas of expertise and enabling the development and training of problem-specific ML models that support well-founded and trustworthy decisions.

Conclusion

There is no single Machine Learning model or AI system capable of solving every problem. ML experts and clinical professionals must work together to define realistic challenges and pursue appropriate solutions. Only through this collaboration can trustworthy and effective AI applications be developed for medicine.