ID-LoRA：基于身份的音频视频个性化与上下文LoRA
ID-LoRA: Identity-Driven Audio-Video Personalization with In-Context LoRA
AI总结：
ID-LoRA通过联合生成音频和视频实现身份驱动的个性化，利用上下文LoRA技术在单次生成过程中提升语音与视觉的一致性。
摘要：
现有视频个性化方法在保持视觉相似性的同时，将视频和音频分开处理。在没有视觉场景的情况下，音频模型无法将声音与屏幕上的动作同步；而且由于经典语音克隆模型仅根据参考录音进行条件化，文本提示无法改变说话风格或声音环境。我们提出了ID-LoRA（身份驱动的上下文LoRA），它在一个模型中联合生成主体的外观和声音，使文本提示、参考图像和短音频片段共同控制两种模态。ID-LoRA通过参数高效的上下文LoRA调整LTX-2联合音频视频扩散主干网络，并且据我们所知，这是首个在单次生成过程中实现视觉外观和声音个性化的 方法。两个挑战出现。参考和生成的token共享相同的位移编码空间，使它们难以区分；我们通过负时间位置来解决这个问题，将参考token放置在不重叠的RoPE区域，同时保持其内部时间结构。说话特征在去噪过程中也容易被稀释；我们引入了身份指导，这是一种分类自由指导变体，通过将预测与无参考信号的预测进行对比来放大说话特定的特征。在人类偏好研究中，ID-LoRA在语音相似性方面被73%的标注者首选，说话风格方面被65%的标注者首选。在跨环境设置中，说话相似性比Kling提高了24%，随着条件的分歧，差距进一步扩大。初步的用户研究进一步表明，联合生成为物理基础的声音合成提供了有用的归纳偏差。ID-LoRA仅使用约3000对训练数据在单个GPU上实现了这些结果。代码、模型和数据将被发布。
英文摘要：
Existing video personalization methods preserve visual likeness but treat video and audio separately. Without access to the visual scene, audio models cannot synchronize sounds with on-screen actions; and because classical voice-cloning models condition only on a reference recording, a text prompt cannot redirect speaking style or acoustic environment. We propose ID-LoRA (Identity-Driven In-Context LoRA), which jointly generates a subject's appearance and voice in a single model, letting a text prompt, a reference image, and a short audio clip govern both modalities together. ID-LoRA adapts the LTX-2 joint audio-video diffusion backbone via parameter-efficient In-Context LoRA and, to our knowledge, is the first method to personalize visual appearance and voice in a single generative pass. Two challenges arise. Reference and generation tokens share the same positional-encoding space, making them hard to distinguish; we address this with negative temporal positions, placing reference tokens in a disjoint RoPE region while preserving their internal temporal structure. Speaker characteristics also tend to be diluted during denoising; we introduce identity guidance, a classifier-free guidance variant that amplifies speaker-specific features by contrasting predictions with and without the reference signal. In human preference studies, ID-LoRA is preferred over Kling 2.6 Pro by 73% of annotators for voice similarity and 65% for speaking style. On cross-environment settings, speaker similarity improves by 24% over Kling, with the gap widening as conditions diverge. A preliminary user study further suggests that joint generation provides a useful inductive bias for physically grounded sound synthesis. ID-LoRA achieves these results with only ~3K training pairs on a single GPU. Code, models, and data will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
