Towards trustworthy medical AI at scale: training with reports and explanations

Artificial intelligence (AI) can help radiologists analyze medical images and detect cancers earlier, but broad clinical adoption requires AI accuracy and trustworthiness. A key threat is bias: irrelevant image features (e.g., text in X-rays) spuriously correlated with diagnostic labels. AI may base decisions on bias, failing in real-world clinical applications. We first introduce ISNet, a classifier trained to resist bias in image backgrounds, by minimizing background relevance in Layer-wise Relevance Propagation (LRP) explanation heatmaps. On chest X-ray datasets, ISNet ignored background bias, improving generalization to unseen hospitals and surpassing 7 state-of-the-art methods in classifying COVID-19, tuberculosis, and pneumonia. We then propose TEA, which combats background and foreground bias by training a student classifier to match a larger teacher’s explanation heatmaps, inheriting the teacher’s rationale and bias resistance. Across 6 datasets, including X-ray classification, TEA outperformed 14 state-of-the-art methods. Next, we address tumor detection and segmentation. Segmenters outline tumors, allowing radiologists to verify and trust AI outputs. However, segmenters traditionally train on radiologist-drawn tumor masks–scarce and completely unavailable for many tumor types. In contrast, radiology reports are abundant and cover all tumor types. We present R-Super, which uses reports to supplement or substitute masks in training. It teaches AI to segment tumors matching descriptions in reports, training on CT-Report pairs or CT-Report plus CT-Mask pairs. Trained on 101,654 reports, R-Super segments 7 tumor types not detected by previous public AI. R-Super enables tumor segmentation without masks and scales CT-Mask datasets via abundant CT-Report pairs. We also introduce RadGPT, a segmentation-based report generation framework that surpasses prior methods for tumor reporting. RadGPT helped radiologists create AbdomenAtlas 3.0, the first public dataset with CT, masks, and reports. Finally, we led Touchstone, a large-scale segmentation benchmark with 14 teams, evaluating AI at an unseen hospital and assessing fairness across patient groups.