SpongeBob：同步感知的和谐视听生成式编辑
SpongeBob: Sync-Aware Harmonious Audio-Visual Generative Editing
AI总结：
提出首个端到端视听联合编辑框架SpongeBob，通过双向跨模态交互的同步感知机制和上下文感知模块，解决视频编辑中的音画不同步和语义冲突问题。
摘要：
物理世界中的视觉和声学事件本质上是耦合的，然而现有的视频编辑方法通常采用解耦的流水线，缺乏双向模态交互。这导致两个关键限制：(i) 视听不同步和(ii) 生成的音频与保留内容之间的上下文冲突。为了解决这些问题，我们提出了SpongeBob，这是第一个具有双向跨模态交互的端到端视听联合编辑框架。对于同步，同步感知机制通过双向注意力、时间对齐和空间约束将视觉编辑与声音事件对齐。对于上下文一致性，上下文感知模块利用声学和视觉上下文注意力来防止语义冲突。此外，我们引入了同步保持训练和指导（SPTG），以在不降低质量的情况下增强对齐。由于配对数据的稀缺，我们构建了一个可扩展的数据流水线和一个大规模的主题级数据集。我们还提出了SpongeBob-Bench用于系统评估。实验表明，SpongeBob显著优于现有基线，将Sync-C提高了30%，Ctx-F1提高了12.5%。我们的项目页面位于：https://hy-spongebob.github.io/。
英文摘要：
Visual and acoustic events in the physical world are inherently coupled, yet existing video editing methods typically adopt decoupled pipelines, lacking bidirectional modality interaction. This results in two key limitations: (i) audio-visual desynchronization and (ii) contextual conflicts between generated audio and preserved content. To address these, we propose SpongeBob, the first end-to-end audio-visual joint editing framework featuring bidirectional cross-modal interaction. For synchronization, a Sync-Aware Mechanism aligns visual edits with sound events via bidirectional attention, temporal alignment, and spatial constraints. For contextual consistency, a Context-Aware Module leverages acoustic and visual context attention to prevent semantic clashes. Additionally, we introduce Sync-Preserving Training and Guidance (SPTG) to enhance alignment without degrading quality. Due to the scarcity of paired data, we construct a scalable data pipeline and a large-scale subject-level dataset. We also propose SpongeBob-Bench for systematic evaluation. Experiments show SpongeBob significantly outperforms existing baselines, improving Sync-C by 30% and Ctx-F1 by 12.5%. Our project page is available at: https://hy-spongebob.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
