CustomTTT：基于测试时训练的运动与外观定制化视频生成
CustomTTT: Motion and Appearance Customized Video Generation via Test-Time Training
AI总结：
针对多概念LoRA组合产生伪影的问题，提出CustomTTT方法，通过分析提示词影响定位特定层，并引入测试时训练技术联合定制视频外观与运动，在定性与定量评估中优于现有方法。
摘要：
得益于文本-视频对的大规模预训练，当前的文本到视频（T2V）扩散模型能够根据文本描述生成高质量视频。此外，给定一些参考图像或视频，参数高效的微调方法，即LoRA，能够生成高质量的定制化概念，例如来自参考视频的特定主体或运动。然而，将来自不同参考的已训练多个概念组合到单个网络中会表现出明显的伪影。为此，我们提出CustomTTT，在其中我们可以轻松地联合定制给定视频的外观和运动。具体而言，我们首先分析了当前视频扩散模型中的提示词影响，并发现外观和运动定制仅需针对特定层使用LoRA。此外，由于每个LoRA是单独训练的，我们提出一种新颖的测试时训练技术，利用已训练的定制模型在组合后更新参数。我们进行了详细的实验以验证所提方法的有效性。我们的方法在定性和定量评估中均优于几项最先进的工作。
英文摘要：
Benefiting from large-scale pre-training of text-video pairs, current text-to-video (T2V) diffusion models can generate high-quality videos from the text description. Besides, given some reference images or videos, the parameter-efficient fine-tuning method, i.e. LoRA, can generate high-quality customized concepts, e.g., the specific subject or the motions from a reference video. However, combining the trained multiple concepts from different references into a single network shows obvious artifacts. To this end, we propose CustomTTT, where we can joint custom the appearance and the motion of the given video easily. In detail, we first analyze the prompt influence in the current video diffusion model and find the LoRAs are only needed for the specific layers for appearance and motion customization. Besides, since each LoRA is trained individually, we propose a novel test-time training technique to update parameters after combination utilizing the trained customized models. We conduct detailed experiments to verify the effectiveness of the proposed methods. Our method outperforms several state-of-the-art works in both qualitative and quantitative evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
