arXiv 2506.100062025-07-31cs.MMcs.AIcs.CVcs.LG

基于动态双向重建的支持灵活多模态输入的HER2表达预测

HER2 Expression Prediction with Flexible Multi-Modal Inputs via Dynamic Bidirectional Reconstruction

School of Computer Science and Technology, University of the Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；Institute for Clarity in Documentation（清晰文档研究所）；Inria Paris-Rocquencourt（巴黎- Rocquencourt 国家信息与自动化所）；Rajiv Gandhi University（拉贾·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕勒实验室）

Jie Qin, Wei Yang, Yan Su, Yiran Zhu, Weizhen Li, Yunyue Pan, Chengchang Pan, Honggang Qi

备注：: 8 pages,6 figures,3 tables,accepted by the 33rd ACM International Conference on Multimedia(ACM MM 2025)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对乳腺癌HER2评估中双模态图像获取受限的问题，提出含动态分支选择器和CM-GAN的自适应双模态预测框架，支持灵活输入，大幅提升单模态准确率，为资源有限地区提供高性价比方案。

摘要：

在乳腺癌HER2评估中，临床评估依赖H&E和IHC图像的结合，但获取两种模态的图像往往受临床条件和成本限制。我们提出了一种自适应双模态预测框架，通过两项核心创新灵活支持单模态或双模态输入：动态分支选择器根据输入可用性激活模态补全或联合推理，以及跨模态GAN（CM-GAN，即Cross-Modal Generative Adversarial Network）实现缺失模态的特征空间重建。该设计将仅使用H&E图像的准确率从71.44%大幅提升至94.25%，使用完整双模态输入时准确率达到95.09%，在单模态条件下仍保持90.28%的可靠性。这种“双模态优先、单模态兼容”的架构无需强制同步采集即可达到接近双模态的准确率，为资源有限地区提供了高性价比的解决方案，显著提升了HER2评估的可及性。

英文摘要：

In breast cancer HER2 assessment, clinical evaluation relies on combined H&E and IHC images, yet acquiring both modalities is often hindered by clinical constraints and cost. We propose an adaptive bimodal prediction framework that flexibly supports single- or dual-modality inputs through two core innovations: a dynamic branch selector activating modality completion or joint inference based on input availability, and a cross-modal GAN (CM-GAN) enabling feature-space reconstruction of missing modalities. This design dramatically improves H&E-only accuracy from 71.44% to 94.25%, achieves 95.09% with full dual-modality inputs, and maintains 90.28% reliability under single-modality conditions. The "dual-modality preferred, single-modality compatible" architecture delivers near-dual-modality accuracy without mandatory synchronized acquisition, offering a cost-effective solution for resource-limited regions and significantly improving HER2 assessment accessibility.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。