arXiv 2510.232242026-09-15cs.CVcs.IR

基于注意力视觉-语言对齐的准确且可扩展的多模态病理检索

Accurate and Scalable Multimodal Pathology Retrieval via Attentive Vision-Language Alignment

Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科学与技术大学计算机科学与工程系）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology（华中科技大学同济医学院附属同济医院放射科）；Department of Pathology, Sir Run Run Shaw Hospital, School of Medicine, Zhejiang University（浙江大学医学院附属邵氏医院病理科）；Department of Pathology, The Central Hospital of Wuhan, Tongji Medical College, Huazhong University of Science and Technology（华中科技大学同济医学院附属武汉中心医院病理科）；Department of Pathology, The First Affiliated Hospital, School of Medicine, Zhejiang University（浙江大学医学院附属第一医院病理科）；Department of Chemical and Biological Engineering, The Hong Kong University of Science and Technology（香港科学与技术大学化学与生物工程系）；Division of Life Science, The Hong Kong University of Science and Technology（香港科学与技术大学生命科学系）

Hongyi Wang, Zhengjie Zhu, Junlin Hou, Jiabo Ma, Fang Wang, Yue Shi, Qiuyu Cai, Jili Wang, Bo Luo, Zhizhong Chai, Zhengyu Zhang, Li Liang, Xiuming Zhang, Yen-We… 展开作者

Hongyi Wang, Zhengjie Zhu, Junlin Hou, Jiabo Ma, Fang Wang, Yue Shi, Qiuyu Cai, Jili Wang, Bo Luo, Zhizhong Chai, Zhengyu Zhang, Li Liang, Xiuming Zhang, Yen-Wei Chen, Lanfen Lin, Hao Chen

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

PathSearch通过注意力马赛克和视觉-语言对比学习，实现准确且可扩展的多模态病理切片检索，在八个任务中优于现有方法。

摘要：

组织病理学切片的快速数字化为临床和研究工作流程中的计算工具开辟了新的机遇。基于内容的切片检索可以帮助病理学家识别形态学和语义上相关的先例病例，支持专家诊断和基于实例的教育。然而，全切片图像（WSIs）的有效检索仍然具有挑战性，因为千兆像素级别的切片包含大量无关内容、局灶性诊断模式和切片级语义信息，这些必须以可行的搜索成本来表示。在此，我们提出了PathSearch，一个检索框架，它将细粒度的注意力马赛克与通过视觉-语言对比学习对齐的切片级嵌入相结合。PathSearch在6,926对切片-报告对上训练，能够捕捉细粒度的形态学线索和高层语义模式，从而实现准确且灵活的检索。该框架支持两个关键功能：（1）基于马赛克的图像到图像（I2I）检索，确保准确且高效的切片搜索；（2）多模态检索，其中文本查询可以直接检索相关切片。PathSearch在包含5,021张评估切片的八个任务上进行了评估，涵盖冷冻切片和苏木精-伊红（H&E）染色切片的恶性程度评估、淋巴结转移检测、肿瘤亚型分类、混合图库罕见癌症检索以及肝细胞癌（HCC）风险分层。内部和外部实验结果表明，PathSearch在不影响多模态准确性的情况下，始终优于现有的最强方法。一项多中心读者研究进一步表明，在PathSearch的支持下，任务级平均诊断准确性、置信度和观察者间一致性均有所提高。总之，这些结果支持了PathSearch在多样化检索任务和评估设置中的有效性。

英文摘要：

The rapid digitization of histopathology slides has opened new opportunities for computational tools in clinical and research workflows. Content-based slide retrieval can help pathologists identify morphologically and semantically related precedent cases, supporting expert diagnosis and example-based education. Effective retrieval of whole-slide images (WSIs), however, remains challenging because gigapixel slides contain abundant irrelevant content, focal diagnostic patterns and slide-level semantic information that must be represented at a practicable search cost. Here we present PathSearch, a retrieval framework that combines fine-grained attentive mosaics with slide-level embeddings aligned through vision-language contrastive learning. Trained on 6,926 slide-report pairs, PathSearch captures both fine-grained morphological cues and high-level semantic patterns to enable accurate and flexible retrieval. The framework supports two key functionalities: (1) mosaic-based image-to-image (I2I) retrieval, ensuring accurate and efficient slide search; and (2) multimodal retrieval, where text queries can directly retrieve relevant slides. PathSearch was evaluated on eight tasks comprising 5,021 evaluation slides, spanning malignancy assessment on frozen and hematoxylin and eosin (H\&E)-stained slides, lymph-node metastasis detection, tumor subtyping, mixed-gallery rare-cancer retrieval, and hepatocellular carcinoma (HCC) risk stratification. Internal and external experimental results demonstrate that PathSearch consistently outperforms the strongest existing methods without compromising multimodal accuracy. A multi-center reader study further demonstrated increases in task-level mean diagnostic accuracy, confidence, and inter-observer agreement with PathSearch's support. Together, these results support the effectiveness of PathSearch across diverse retrieval tasks and evaluation settings.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。