弥合模态与知识转移：增强多模态理解和识别
Bridging Modalities and Transferring Knowledge: Enhanced Multimodal Understanding and Recognition
AI总结：
本文提出多模态对齐、翻译、融合和转移方法，提升复杂输入的理解与识别能力，涵盖空间语言、医学文本、知识图谱和动作识别等多个领域。
摘要：
本文探讨了多模态对齐、翻译、融合和转移，以增强机器对复杂输入的理解。我们将工作分为五章，每章都针对多模态机器学习中的独特挑战。第三章介绍了Spatial-Reasoning Bert，用于将基于文本的空间关系翻译成剪贴画之间的2D布局。这使文本空间语言能够有效地解码为视觉表示，为与人类空间理解一致的自动场景生成铺平道路。第四章提出了一种将医学文本翻译成解剖学大典中特定3D位置的方法。我们引入了一个利用医学术语的空间共现的损失函数，创建了可解释的映射，显著提高了医学文本的可导航性。第五章解决将结构化文本转换为知识图谱中的规范事实。我们开发了一个基准，用于将自然语言链接到实体和谓词，解决文本提取中的歧义，以提供更清晰、可操作的见解。第六章探讨了多模态融合方法用于组合动作识别。我们提出了一种融合视频帧和对象检测表示的方法，提高了识别的鲁棒性和准确性。第七章研究了多模态知识转移用于中心视角动作识别。我们展示了多模态知识蒸馏如何使RGB-only模型模仿基于多模态融合的能力，从而减少计算需求，同时保持性能。这些贡献推进了空间语言理解、医学文本解释、知识图谱丰富和动作识别的方法，增强了计算系统处理复杂多模态输入的能力，应用于各种应用。
英文摘要：
This manuscript explores multimodal alignment, translation, fusion, and transference to enhance machine understanding of complex inputs. We organize the work into five chapters, each addressing unique challenges in multimodal machine learning. Chapter 3 introduces Spatial-Reasoning Bert for translating text-based spatial relations into 2D arrangements between clip-arts. This enables effective decoding of spatial language into visual representations, paving the way for automated scene generation aligned with human spatial understanding. Chapter 4 presents a method for translating medical texts into specific 3D locations within an anatomical atlas. We introduce a loss function leveraging spatial co-occurrences of medical terms to create interpretable mappings, significantly enhancing medical text navigability. Chapter 5 tackles translating structured text into canonical facts within knowledge graphs. We develop a benchmark for linking natural language to entities and predicates, addressing ambiguities in text extraction to provide clearer, actionable insights. Chapter 6 explores multimodal fusion methods for compositional action recognition. We propose a method fusing video frames and object detection representations, improving recognition robustness and accuracy. Chapter 7 investigates multimodal knowledge transference for egocentric action recognition. We demonstrate how multimodal knowledge distillation enables RGB-only models to mimic multimodal fusion-based capabilities, reducing computational requirements while maintaining performance. These contributions advance methodologies for spatial language understanding, medical text interpretation, knowledge graph enrichment, and action recognition, enhancing computational systems' ability to process complex, multimodal inputs across diverse applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
