TY  - JOUR
AU  - Denner, Stefan
AU  - Zimmerer, David
AU  - Bounias, Dimitrios
AU  - Bujotzek, Markus
AU  - Xiao, Shuhan
AU  - Stock, Raphael
AU  - Kausch, Lisa
AU  - Schader, Philipp
AU  - Penzkofer, Tobias
AU  - Jäger, Paul
AU  - Maier-Hein, Klaus
TI  - Leveraging foundation models for content-based image retrieval in radiology.
JO  - Computers in biology and medicine
VL  - 196
IS  - Pt A
SN  - 0010-4825
CY  - Amsterdam [u.a.]
PB  - Elsevier Science
M1  - DKFZ-2025-01429
SP  - 110640
PY  - 2025
N1  - #EA:E230#LA:E230#
AB  - Content-based image retrieval (CBIR) has the potential to significantly improve diagnostic aid and medical research in radiology. However, current CBIR systems face limitations due to their specialization to certain pathologies, limiting their utility. On the other hand, several vision foundation models have been shown to produce general-purpose visual features. Therefore, in this work, we propose using vision foundation models as powerful and versatile off-the-shelf feature extractors for content-based image retrieval. Our contributions include: (1) benchmarking a diverse set of vision foundation models on an extensive dataset comprising 1.6 million 2D radiological images across four modalities and 161 pathologies; (2) identifying weakly-supervised models, particularly BiomedCLIP, as highly effective, achieving a P@1 of up to 0.594 (P@3: 0.590, P@5: 0.588, P@10: 0.583), comparable to specialized CBIR systems but without additional training; (3) conducting an in-depth analysis of the impact of index size on retrieval performance; (4) evaluating the quality of embedding spaces generated by different models; and (5) investigating specific challenges associated with retrieving anatomical versus pathological structures. Despite these challenges, our research underscores the vast potential of foundation models for CBIR in radiology, proposing a shift towards versatile, general-purpose medical image retrieval systems that do not require specific tuning. Our code, dataset splits and embeddings are publicly available here.
KW  - Content-based image retrieval (Other)
KW  - Foundation models (Other)
KW  - Medical imaging (Other)
KW  - Self-supervised learning (Other)
KW  - Supervised learning (Other)
KW  - Weakly-supervised learning (Other)
LB  - PUB:(DE-HGF)16
C6  - pmid:40639009
DO  - DOI:10.1016/j.compbiomed.2025.110640
UR  - https://inrepo02.dkfz.de/record/302889
ER  -