HoloLLM：面向语言基础的人感知与推理的多感官基础模型
HoloLLM: Multisensory Foundation Model for Language-Grounded Human Sensing and Reasoning
AI总结：
HoloLLM通过整合LiDAR、红外、毫米波雷达和Wi-Fi等多感官数据，提升语言基础的人感知与推理能力，实验表明其在真实场景中的性能提升达30%。
摘要：
在智能家庭中运作的具身代理必须通过多样化的感官输入来理解人类行为，并通过自然语言进行交流。尽管视觉-语言模型（VLMs）已经实现了令人印象深刻的语言基础感知能力，但其对视觉数据的依赖限制了在遮挡、光照差或隐私约束等现实场景中的鲁棒性。在本文中，我们引入了HoloLLM，一种多模态大语言模型（MLLM），它整合了不常见但强大的传感模态，如激光雷达、红外、毫米波雷达和Wi-Fi，以在异构环境中实现无缝的人感知和推理。我们解决了两个关键挑战：（1）稀有传感器的对齐模态-文本数据稀缺，以及（2）它们的物理信号表示的异质性。为了解决这些问题，我们设计了通用模态注入投影器（UMIP），通过粗到细的交叉注意力机制，利用定制编码器生成的细粒度、文本对齐的特征，增强预对齐的模态嵌入，而无需引入显著的对齐开销。我们进一步引入了人-VLM协作数据整理流程，以生成传感数据集的配对文本注释。在两个新构建的基准测试中，HoloLLM在两个新构建的基准测试中显著优于现有MLLMs，将语言基础的人感知准确性提高了高达30%。这项工作为现实世界中的语言引导的多感官具身智能建立了新的基础。
英文摘要：
Embodied agents operating in smart homes must understand human behavior through diverse sensory inputs and communicate via natural language. While Vision-Language Models (VLMs) have enabled impressive language-grounded perception, their reliance on visual data limits robustness in real-world scenarios with occlusions, poor lighting, or privacy constraints. In this paper, we introduce HoloLLM, a Multimodal Large Language Model (MLLM) that integrates uncommon but powerful sensing modalities, such as LiDAR, infrared, mmWave radar, and WiFi, to enable seamless human perception and reasoning across heterogeneous environments. We address two key challenges: (1) the scarcity of aligned modality-text data for rare sensors, and (2) the heterogeneity of their physical signal representations. To overcome these, we design a Universal Modality-Injection Projector (UMIP) that enhances pre-aligned modality embeddings with fine-grained, text-aligned features from tailored encoders via coarse-to-fine cross-attention without introducing significant alignment overhead. We further introduce a human-VLM collaborative data curation pipeline to generate paired textual annotations for sensing datasets. Extensive experiments on two newly constructed benchmarks show that HoloLLM significantly outperforms existing MLLMs, improving language-grounded human sensing accuracy by up to 30%. This work establishes a new foundation for real-world, language-informed multisensory embodied intelligence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
