VITA-E：具备并发视觉、听觉、语音与行动能力的自然具身交互
VITA-E: Natural Embodied Interaction with Concurrent Seeing, Hearing, Speaking, and Acting
AI总结：
针对VLA模型缺乏并发交互与实时打断能力的问题，提出双模型架构的VITA-E框架，实现看听说行动并行，并在物理人形平台上验证了高成功率，推动自然具身交互发展。
摘要：
当前的视觉-语言-动作（VLA）模型往往受限于僵化、静态的交互范式，缺乏并发地看、听、说和行动的能力，也无法动态处理用户的实时打断。这阻碍了无缝的具身协作，导致用户体验不灵活且响应迟钝。为解决这些局限，我们提出了VITA-E，一种新颖的具身交互框架，专为行为并发性和近实时打断而设计。我们方法的核心是一种双模型架构，其中两个并行的VLA实例分别作为“主动模型”和“待命模型”运行，使具身智能体能够同时且可打断地观察环境、聆听用户语音、提供口头回应并执行动作，模拟人类多任务处理能力。我们进一步提出了一种“模型即控制器”范式，通过微调视觉语言模型（VLM）生成作为直接系统级命令的特殊标记，将模型的推理与系统的行为耦合。在物理人形平台上进行的实验表明，VITA-E能够可靠地处理复杂的交互场景。我们的框架兼容多种双系统VLA模型，在紧急停止和语音打断方面实现了极高的成功率，同时还能成功执行并发语音与动作。这标志着向更自然、能力更强的具身助手迈出了重要一步。
英文摘要：
Current Vision-Language-Action (VLA) models are often constrained by a rigid, static interaction paradigm, which lacks the ability to see, hear, speak, and act concurrently as well as handle real-time user interruptions dynamically. This hinders seamless embodied collaboration, resulting in an inflexible and unresponsive user experience. To address these limitations, we introduce VITA-E, a novel embodied interaction framework designed for both behavioral concurrency and nearly real-time interruption. The core of our approach is a dual-model architecture where two parallel VLA instances operate as an ``Active Model'' and a ``Standby Model'', allowing the embodied agent to observe its environment, listen to user speech, provide verbal responses, and execute actions, all concurrently and interruptibly, mimicking human-like multitasking capabilities. We further propose a ``model-as-controller'' paradigm, where we fine-tune the VLM to generate special tokens that serve as direct system-level commands, coupling the model's reasoning with the system's behavior. Experiments conducted on a physical humanoid platform demonstrate that VITA-E can reliably handle complex interactive scenarios. Our framework is compatible with various dual-system VLA models, achieving an extremely high success rate on emergency stops and speech interruptions while also successfully performing concurrent speech and action. This represents a significant step towards more natural and capable embodied assistants.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
