SMRABooth: 主体和运动表示对齐用于定制视频生成
SMRABooth: Subject and Motion Representation Alignment for Customized Video Generation
AI总结：
SMRABooth通过自监督和光流编码器对齐主体和运动表示，提升定制视频生成中主体外观和运动模式的一致性。
摘要：
定制视频生成旨在生成视频，这些视频忠实保留参考图像中主体的外观，同时保持参考视频中时间一致的运动。现有方法难以确保主体外观相似性和运动模式一致性，因为缺乏对主体和运动的物体级指导。为了解决这个问题，我们提出了SMRABooth，它利用自监督编码器和光流编码器来提供物体级的主体和运动表示。这些表示在LoRA微调过程中与模型对齐。我们的方法分为三个核心阶段：（1）我们利用自监督编码器获取主体表示，以引导主体对齐，使模型能够捕捉主体的整体结构并增强高层语义一致性。（2）我们利用光流编码器获取的运动表示来捕捉结构上一致且独立于外观的物体级运动轨迹。（3）我们提出了一种主体-运动关联解耦策略，该策略在位置和时间上应用稀疏LoRAs注入，有效减少主体和运动LoRAs之间的干扰。广泛的实验表明，SMRABooth在主体和运动定制方面表现出色，保持一致的主体外观和运动模式，证明了其在可控文本到视频生成中的有效性。
英文摘要：
Customized video generation aims to produce videos that faithfully preserve the subject's appearance from reference images while maintaining temporally consistent motion from reference videos. Existing methods struggle to ensure both subject appearance similarity and motion pattern consistency due to the lack of object-level guidance for subject and motion. To address this, we propose SMRABooth, which leverages the self-supervised encoder and optical flow encoder to provide object-level subject and motion representations. These representations are aligned with the model during the LoRA fine-tuning process. Our approach is structured in three core stages: (1) We exploit subject representations via a self-supervised encoder to guide subject alignment, enabling the model to capture overall structure of subject and enhance high-level semantic consistency. (2) We utilize motion representations from an optical flow encoder to capture structurally coherent and object-level motion trajectories independent of appearance. (3) We propose a subject-motion association decoupling strategy that applies sparse LoRAs injection across both locations and timing, effectively reducing interference between subject and motion LoRAs. Extensive experiments show that SMRABooth excels in subject and motion customization, maintaining consistent subject appearance and motion patterns, proving its effectiveness in controllable text-to-video generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
