发表机构
The University of Tokyo; Alaya Lab（东京大学; 阿莱亚实验室）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究提出HelloWorld视频世界模型，通过自蒸馏流水线和训练模块实现角色与用户的社交互动，构建含400样本的基准，其互动质量优于基线且保持顶尖图像美学。
AI 中文摘要
尽管近期视频世界模型已取得显著进展，但这些世界中用户与角色之间的社交互动仍未得到支持。为填补这一空白，我们提出了HelloWorld，这是一种能够实现与世界内角色进行社交互动的视频世界模型。用户只需按下单个按钮，即可提示屏幕上的角色对镜头做出回应，例如转向观看者、挥手、点头或说出简短问候语。为使这些互动自然，我们提出了一种自蒸馏流水线，该流水线利用自身合成的数据对视频生成模型进行微调。每个合成片段均包含社交互动和相机运动，使模型能够学习相机姿态条件，同时不降低互动质量。在推理阶段，我们进一步引入了无需训练的模块，用于确定互动发生的时机。当按下按钮时，该模块会调制DiT的交叉注意力掩码，使与互动相关的文本提示仅关注按下窗口内的帧，从而在时间上定位角色的回应。我们还构建了HelloWorldBench，这是一个包含400个样本的基准，配备三种社交互动指标和三种常规指标，用于评估。实验表明，HelloWorld在互动质量方面优于多种基线方法，同时保持了最先进的图像美学和相机姿态跟随能力。项目页面：this https URL
英文摘要
Despite the remarkable recent progress of video world models, social interaction between users and the characters within these worlds remains unsupported. To fill this gap, we present HelloWorld, a video world model that enables social interaction with in-world characters. With a single button press, users can prompt the on-screen character to respond toward the camera, e.g., turning to the viewer, waving, nodding, or speaking a short greeting. To make these interactions natural, we propose a self-distillation pipeline that finetunes the video generation model on data synthesized by itself. Each synthesized clip contains both social interactions and camera motion, allowing the model to learn camera-pose conditioning without degrading interaction quality. At inference, we further introduce a training-free module that determines when the interaction occurs. Upon a button press, it modulates the cross-attention masks of the DiT so that the interaction-related text prompt attends only to the frames within the press window, temporally localizing the character's response. We further build HelloWorldBench, a 400-sample benchmark with three social interaction metrics alongside three conventional metrics, for evaluation. Experiments demonstrate that HelloWorld surpasses a variety of baselines in interaction quality, while maintaining state-of-the-art picture aesthetics and camera-pose following. Project page: https://github.com/AlayaLab/HelloWorld
CommentsProject page: https://github.com/AlayaLab/HelloWorld
