SocialDirector: 无训练的社会互动控制多人物视频生成
SocialDirector: Training-Free Social Interaction Control for Multi-Person Video Generation
AI总结:
SocialDirector通过调节交叉注意力图提升多人物视频生成的社会互动准确性,解决演员动作不匹配和社交动态紊乱问题,实验表明其能显著提升互动真实度。
摘要:
视频生成技术已迅速发展,能够从文本或图像提示生成逼真视频。然而,电影制作和社会机器人日益需要包含对话、手势和协调动作的多人物视频。现有模型无法显式控制互动,如谁执行哪个动作、何时发生以及指向谁。这导致演员-动作不匹配、社交动态紊乱和动作目标错误。为解决这些问题,我们提出了SocialDirector,一种无训练的交互控制器,通过调节交叉注意力图增强生成模型。SocialDirector包含两个模块:社会演员掩码和方向重新加权。社会演员掩码通过时空掩码约束每个人视觉token仅关注自身文本描述,避免演员-动作不匹配和社交动态紊乱。方向重新加权放大对方向词(如“左”、“右”)的注意力,使动作指向其预期目标。为了评估生成的社会互动,我们对现有数据集进行标注并构建完全自动化的评估流程,利用开源VLMs。在不同视频生成模型上的实验表明,SocialDirector显著提高了互动保真度,并接近由真实视频设定的上限。
英文摘要:
Video generation has advanced rapidly, producing photorealistic videos from text or image prompts. Meanwhile, film production and social robotics increasingly demand multi-person videos with rich social interactions, including conversations, gestures, and coordinated actions. However, existing models offer no explicit control over interactions, such as who performs which action, when it occurs, and toward whom it is directed. This often results in wrong person performing unintended actions (actor-action mismatch), disordered social dynamics, and wrong action targets. To address these challenges, we present SocialDirector, a training-free interaction controller that enhances the generation model by modulating cross-attention maps. SocialDirector contains two modules: Social Actor Masking and Directional Reweighting. Social Actor Masking constrains each person's visual tokens to attend only to their own textual descriptions via a spatiotemporal mask, avoiding actor-action mismatch and disordered social dynamics. Directional Reweighting amplifies attention to directional words (e.g., "leftward", "right"), leading each action towards its intended target. To evaluate generated social interactions, we annotate existing datasets with interaction descriptions and build a fully automated evaluation pipeline powered by open-source VLMs. Experiments on different video generation models show that SocialDirector significantly improves interaction fidelity and approaches the upper bound set by real videos.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。