FoleyDirector: 通过结构化脚本实现视频到音频生成的细粒度时间控制
FoleyDirector: Fine-Grained Temporal Steering for Video-to-Audio Generation via Structured Scripts
AI总结：
本文提出FoleyDirector框架，通过结构化时间脚本提升视频到音频生成的细粒度时间控制能力，同时保持音频质量并支持无缝切换生成与受控合成。
摘要：
近期视频到音频（V2A）方法取得了显著进展，能够合成逼真高质量的音频。然而，它们在多事件场景或多视觉线索不足时，如小区域、屏幕外声音或遮挡或部分可见物体时，难以实现细粒度时间控制。本文提出FoleyDirector框架，首次在DiT基础上实现精确时间引导的V2A生成，同时保持基模型的音频质量和无缝切换V2A生成与受控合成能力。FoleyDirector引入结构化时间脚本（STS），一组对应短时间段的标题，以提供更丰富的时序信息。这些特征通过脚本引导的时间融合模块进行整合，该模块利用时间脚本注意力来融合STS特征。为了处理复杂的多事件场景，我们进一步提出双帧声音合成，实现并行的帧内和帧外音频生成，提高可控性。为了支持训练和评估，我们构建了DirectorSound数据集，并引入VGGSoundDirector和DirectorBench。实验表明，FoleyDirector显著增强了时间可控性，同时保持高质量音频，使用户能够作为Foley导演，推动V2A向更表达性和可控的生成发展。
英文摘要：
Recent Video-to-Audio (V2A) methods have achieved remarkable progress, enabling the synthesis of realistic, high-quality audio. However, they struggle with fine-grained temporal control in multi-event scenarios or when visual cues are insufficient, such as small regions, off-screen sounds, or occluded or partially visible objects. In this paper, we propose FoleyDirector, a framework that, for the first time, enables precise temporal guidance in DiT-based V2A generation while preserving the base model's audio quality and allowing seamless switching between V2A generation and temporally controlled synthesis. FoleyDirector introduces Structured Temporal Scripts (STS), a set of captions corresponding to short temporal segments, to provide richer temporal information. These features are integrated via the Script-Guided Temporal Fusion Module, which employs Temporal Script Attention to fuse STS features coherently. To handle complex multi-event scenarios, we further propose Bi-Frame Sound Synthesis, enabling parallel in-frame and out-of-frame audio generation and improving controllability. To support training and evaluation, we construct the DirectorSound dataset and introduce VGGSoundDirector and DirectorBench. Experiments demonstrate that FoleyDirector substantially enhances temporal controllability while maintaining high audio fidelity, empowering users to act as Foley directors and advancing V2A toward more expressive and controllable generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
