ControlFoley: 一种统一且可控的视频到音频生成方法，具有跨模态冲突处理
ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling
AI总结：
ControlFoley通过联合视觉编码和时域-音域解耦提升视频到音频生成的可控性与同步性，在多种任务中表现优异。
摘要：
近期视频到音频（V2A）生成的进步使高质量音频合成成为可能，但实现稳健且细粒度的可控性仍具挑战。现有方法在视觉-文本冲突下的文本可控性弱，且因参考音频中交织的时序和音色信息导致风格控制不精确。此外，缺乏标准化基准限制了系统性评估。我们提出ControlFoley，一种统一的多模态V2A框架，能够精确控制视频、文本和参考音频。我们引入联合视觉编码范式，整合CLIP与时空音频-视觉编码器以提升对齐和文本可控性。我们进一步提出时域-音域解耦以抑制冗余时序线索同时保留判别音色特征。此外，我们设计了模态鲁棒训练方案，采用统一多模态表示对齐（REPA）和随机模态丢弃。我们还提出了VGGSound-TVC，用于评估在不同视觉-文本冲突程度下的文本可控性。大量实验显示ControlFoley在多种V2A任务中均达到最先进的性能，包括文本引导、文本控制和音频控制生成。ControlFoley在跨模态冲突下实现优越的可控性，同时保持强同步性和音频质量，并在与工业V2A系统相比时表现竞争或更优。代码、模型、数据集和演示可在：https://yjx-research.github.io/ControlFoley/。
英文摘要：
Recent advances in video-to-audio (V2A) generation enable high-quality audio synthesis from visual content, yet achieving robust and fine-grained controllability remains challenging. Existing methods suffer from weak textual controllability under visual-text conflict and imprecise stylistic control due to entangled temporal and timbre information in reference audio. Moreover, the lack of standardized benchmarks limits systematic evaluation. We propose ControlFoley, a unified multimodal V2A framework that enables precise control over video, text, and reference audio. We introduce a joint visual encoding paradigm that integrates CLIP with a spatio-temporal audio-visual encoder to improve alignment and textual controllability. We further propose temporal-timbre decoupling to suppress redundant temporal cues while preserving discriminative timbre features. In addition, we design a modality-robust training scheme with unified multimodal representation alignment (REPA) and random modality dropout. We also present VGGSound-TVC, a benchmark for evaluating textual controllability under varying degrees of visual-text conflict. Extensive experiments demonstrate state-of-the-art performance across multiple V2A tasks, including text-guided, text-controlled, and audio-controlled generation. ControlFoley achieves superior controllability under cross-modal conflict while maintaining strong synchronization and audio quality, and shows competitive or better performance compared to an industrial V2A system. Code, models, datasets, and demos are available at: https://github.com/xiaomi-research/controlfoley.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
