EchoFoley: 面向视频接地创意声音生成的事件中心层次化控制
EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generation
AI总结：
提出EchoFoley任务，通过事件级局部控制和层次化语义控制实现视频接地声音生成，构建EchoFoley-6k基准并设计EchoVidia框架，在可控性和感知质量上分别提升40.7%和12.5%。
摘要：
音效构建了多模态叙事的重要层次，塑造了视频的情感氛围和叙事语义。尽管视频-文本到音频（VT2A）近期取得进展，当前公式面临三个关键限制：首先，视觉和文本条件之间的不平衡导致视觉主导；其次，缺乏细粒度可控生成的具体定义；第三，指令理解与跟随能力弱，因为现有数据集依赖简短的类别标签。为解决这些限制，我们引入EchoFoley，这是一个新任务，旨在实现具有事件级局部控制和层次化语义控制的视频接地声音生成。我们的发声事件符号表示指定了每个声音在视频或指令中何时、何地以及如何产生，实现了细粒度控制，如声音生成、插入和编辑。为支持该任务，我们构建了EchoFoley-6k，一个大规模、专家策划的基准，包含超过6,000个视频-指令-注释三元组。在此基础上，我们提出EchoVidia，一个以发声事件为中心的智能体生成框架，采用慢-快思考策略。实验表明，EchoVidia在可控性上超越近期VT2A模型40.7%，在感知质量上超越12.5%。
英文摘要：
Sound effects build an essential layer of multimodal storytelling, shaping the emotional atmosphere and the narrative semantics of videos. Despite recent advancement in video-text-to-audio (VT2A), the current formulation faces three key limitations: First, an imbalance between visual and textual conditioning that leads to visual dominance; Second, the absence of a concrete definition for fine-grained controllable generation; Third, weak instruction understanding and following, as existing datasets rely on brief categorical tags. To address these limitations, we introduce EchoFoley, a new task designed for video-grounded sound generation with both event level local control and hierarchical semantic control. Our symbolic representation for sounding events specifies when, what, and how each sound is produced within a video or instruction, enabling fine-grained controls like sound generation, insertion, and editing. To support this task, we construct EchoFoley-6k, a large-scale, expert-curated benchmark containing over 6,000 video-instruction-annotation triplets. Building upon this foundation, we propose EchoVidia a sounding-event-centric agentic generation framework with slow-fast thinking strategy. Experiments show that EchoVidia surpasses recent VT2A models by 40.7% in controllability and 12.5% in perceptual quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
