超越独白：基于对话音频上下文感知核的交互式谈话-倾听虚拟角色生成
Beyond Monologue: Interactive Talking-Listening Avatar Generation with Conversational Audio Context-Aware Kernels
AI总结：
本文提出基于对话音频上下文感知核的交互式虚拟角色生成方法，通过引入多头高斯核解决谈话与倾听行为的时间尺度差异问题，构建了能同时处理双流音频输入的全双工虚拟代理，并在VoxHear数据集上验证了其在生成自然响应的全双工数字人类方面的优越性。
摘要：
基于音频驱动的人类视频生成在独白场景中取得了显著成功，主要得益于强大视频生成基础模型的进步。超越独白，真实的人类交流本质上是全双工交互过程，要求虚拟代理不仅能够表达自己的言语，还必须自然响应输入的对话音频。大多数现有方法仅将传统音频驱动范式扩展到倾听场景。然而，依赖严格的帧到帧对齐使模型对长程对话动态的响应变得僵化，而直接引入全局注意力会严重破坏嘴唇同步。认识到谈话与倾听行为之间独特的时序尺度差异，我们引入多头高斯核，将这种物理直觉显式注入模型作为渐进的时间归纳偏置。在此基础上，我们构建了一个能够同时处理谈话和倾听双流音频输入的全双工虚拟代理。此外，我们引入了一个严格清洗的谈话-倾听数据集VoxHear，包含完美解耦的语音和背景音频轨道。广泛实验表明，我们的方法成功融合了强时间对齐与深层语义上下文，为生成高度自然和响应的全双工交互数字人类设定了新的状态-of-the-art。项目页面可在https://warmcongee.github.io/beyond-monologue/上找到。
英文摘要：
Audio-driven human video generation has achieved remarkable success in monologue scenarios, largely driven by advancements in powerful video generation foundation models. Moving beyond monologues, authentic human communication is inherently a full-duplex interactive process, requiring virtual agents not only to articulate their own speech but also to react naturally to incoming conversational audio. Most existing methods simply extend conventional audio-driven paradigms to listening scenarios. However, relying on strict frame-to-frame alignment renders the model's response to long-range conversational dynamics rigid, whereas directly introducing global attention catastrophically degrades lip synchronization. Recognizing the unique temporal Scale Discrepancy between talking and listening behaviors, we introduce a multi-head Gaussian kernel to explicitly inject this physical intuition into the model as a progressive temporal inductive bias. Building upon this, we construct a full-duplex interactive virtual agent capable of simultaneously processing dual-stream audio inputs for both talking and listening. Furthermore, we introduce a rigorously cleaned Talking-Listening dataset VoxHear featuring perfectly decoupled speech and background audio tracks. Extensive experiments demonstrate that our approach successfully fuses strong temporal alignment with deep contextual semantics, setting a new state-of-the-art for generating highly natural and responsive full-duplex interactive digital humans. The project page is available at https://warmcongee.github.io/beyond-monologue/ .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
