AVControl: 用于音频-视觉控制训练的高效框架
AVControl: Efficient Framework for Training Audio-Visual Controls
AI总结：
AVControl通过轻量级框架实现多模态控制，无需架构改动，支持多种独立训练的模态，如深度、姿态、边缘、相机轨迹等，并在多个基准测试中表现优异。
摘要：
控制视频和音频生成需要多种模态，如深度、姿态、相机轨迹和音频转换，但现有方法要么训练单一模型，要么为新模态引入高成本架构。我们提出AVControl，基于LTX-2联合音频-视觉基础模型，每个控制模态作为独立LoRA在并行画布上训练，无需架构改动。我们证明仅将基于图像的上下文方法扩展到视频失败于结构控制，而并行画布方法解决了这一问题。在VACE基准测试中，我们在深度和姿态引导生成、修复和扩展中优于所有评估基线，并在相机控制和音频-视觉基准中表现竞争。我们的框架支持多种独立训练的模态：空间对齐的控制如深度、姿态和边缘，具有内参的相机轨迹，稀疏运动控制，视频编辑，以及我们所知的第一个模块化音频-视觉控制用于联合生成模型。我们的方法在计算和数据效率方面都很高效：每个模态只需小数据集，训练几步到几千步即可收敛，远低于单一模型的预算。我们公开发布代码和训练的LoRA检查点。
英文摘要：
Controlling video and audio generation requires diverse modalities, from depth and pose to camera trajectories and audio transformations, yet existing approaches either train a single monolithic model for a fixed set of controls or introduce costly architectural changes for each new modality. We introduce AVControl, a lightweight, extendable framework built on LTX-2, a joint audio-visual foundation model, where each control modality is trained as a separate LoRA on a parallel canvas that provides the reference signal as additional tokens in the attention layers, requiring no architectural changes beyond the LoRA adapters themselves. We show that simply extending image-based in-context methods to video fails for structural control, and that our parallel canvas approach resolves this. On the VACE Benchmark, we outperform all evaluated baselines on depth- and pose-guided generation, inpainting, and outpainting, and show competitive results on camera control and audio-visual benchmarks. Our framework supports a diverse set of independently trained modalities: spatially-aligned controls such as depth, pose, and edges, camera trajectory with intrinsics, sparse motion control, video editing, and, to our knowledge, the first modular audio-visual controls for a joint generation model. Our method is both compute- and data-efficient: each modality requires only a small dataset and converges within a few hundred to a few thousand training steps, a fraction of the budget of monolithic alternatives. We publicly release our code and trained LoRA checkpoints.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
