迈向视觉-声音-语言-动作范式：用于以声音为中心的操作的HEAR框架
Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation
AI总结：
本文提出HEAR框架，通过整合声音、视觉、语言和本体感知，解决实时声音中心操作中的关键声音遗漏问题，强调因果持续性和显式时间学习的重要性。
摘要：
尽管最近的视觉-语言-动作（VLA）模型开始整合音频，但通常将声音视为静态预执行提示或仅专注于人类语音。这在需要实时、以声音为中心的操作中留下显著空白，其中短暂的环境声学提供任务执行中的关键状态验证。因此，由于低频更新或系统延迟，关键声音容易被遗漏。此问题在开放式循环执行中进一步加剧，创建了盲执行间隔，其中声音事件在离散音频观察窗口之间丢失。认识到持续听觉意识的必要性，我们正式化视觉-声音-语言-动作（VSLA）为一种基于视觉、流式音频、语言和本体感知的连续控制范式，受延迟决策循环条件。作为实例，我们引入HEAR，一个整合四个组件的VSLA框架：（i）一个流式Historizer，用于在执行间隙中维持紧凑的因果音频上下文；（ii）一个从omni基础模型改编的Envisioner，用于推理多感官输入；（iii）一个形式化为音频世界模型的Advancer，通过预测近未来音频代码学习时间动态；以及（iv）一个流匹配的Realizer策略，用于生成平滑的动作块。为了解决VSLA的预训练数据和评估的稀缺性，我们构建了OpenX-Sound用于预训练，以及HEAR-Bench，第一个以声音为中心的操作基准，具有严格的时间因果规则。我们的结果表明，稳健的声音中心操作需要因果持续性和显式时间学习。此框架为多感官基础模型提供了实际步骤，使机器人能够感知和与动态环境互动。代码和视频可在https://hear.irmv.top获取。
英文摘要：
While recent Vision-Language-Action (VLA) models have begun to incorporate audio, they typically treat sound as static pre-execution prompts or focus exclusively on human speech. This leaves a significant gap in real-time, sound-centric manipulation where fleeting environmental acoustics provide critical state verification during task execution. Consequently, key sounds are easily missed due to low-frequency updates or system latency. This problem is exacerbated by action chunking with open-loop execution, which creates a Blind Execution Interval where acoustic events are lost between discrete audio observation windows. Recognizing the necessity of continuous auditory awareness, we formalize Vision-Sound-Language-Action (VSLA) as a continuous control paradigm conditioned on vision, streaming audio, language, and proprioception under delayed decision loops. As an instantiation, we introduce HEAR, a VSLA framework integrating four components: (i) a streaming Historizer to maintain a compact, causal audio context across execution gaps; (ii) an Envisioner adapted from omni foundation models to reason over multi-sensory inputs; (iii) an Advancer, formulated as an audio world model, to learn temporal dynamics by predicting near-future audio codes; and (iv) a flow-matching Realizer policy to generate smooth action chunks. To address the scarcity of pretraining data and evaluations for VSLA, we construct OpenX-Sound for pretraining, alongside HEAR-Bench, the first sound-centric manipulation benchmark with strict causal timing rules. Our results suggest that robust sound-centric manipulation necessitates causal persistence and explicit temporal learning. This framework provides a practical step toward multi-sensory foundation models for embodied agents, enabling robots to perceive and interact with dynamic environments. Code and videos are available at https://hear.irmv.top
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
