LILE：先看深后看远--基于Transformers的双注意力网络在历史学档案跨模态信息检索中的应用
LILE: Look In-Depth before Looking Elsewhere -- A Dual Attention Network using Transformers for Cross-Modal Information Retrieval in Histopathology Archives
摘要：
近年来，在许多应用程序中，可用数据量急剧增长。此外，分别使用多种模式的网络时代实际上已经结束。因此，使双向跨模态数据检索能够处理已成为许多领域和学科的研究的要求。在医学领域尤其如此，因为数据有多种类型，包括各种类型的图像和报告以及分子数据。大多数当代作品运用交叉注意力来突出图像或文本与其他形式的关系的基本要素，并试图将它们匹配在一起。然而，不管它们在各自模态中的重要性如何，这些方法通常平等地考虑每种模态的特征。在本研究中，自我注意作为一个额外的损失术语将被提出来丰富的内部表征提供到交叉注意模块。这项工作提出了一种新的架构与一个新的损失项，以帮助代表图像和文本的联合潜在空间。在MS-COCO数据集和MS-COCO数据集上的实验结果表明了该方法的有效性。
英文摘要：
The volume of available data has grown dramatically in recent years in many applications. Furthermore, the age of networks that used multiple modalities separately has practically ended. Therefore, enabling bidirectional cross-modality data retrieval capable of processing has become a requirement for many domains and disciplines of research. This is especially true in the medical field, as data comes in a multitude of types, including various types of images and reports as well as molecular data. Most contemporary works apply cross attention to highlight the essential elements of an image or text in relation to the other modalities and try to match them together. However, regardless of their importance in their own modality, these approaches usually consider features of each modality equally. In this study, self-attention as an additional loss term will be proposed to enrich the internal representation provided into the cross attention module. This work suggests a novel architecture with a new loss term to help represent images and texts in the joint latent space. Experiment results on two benchmark datasets, i.e. MS-COCO and ARCH, show the effectiveness of the proposed method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
