MuSteerNet：通过观察-反应相互引导生成人类反应
NOUS: Video-Driven 3D Human Reaction Generation via Observation-Reaction Mutual Steering
AI总结：
本文提出MuSteerNet，通过观察-反应相互引导机制生成3D人类动作，解决视频输入对反应生成的引导不足问题，提升反应质量。
摘要：
视频驱动的人类反应生成旨在合成直接响应观察视频序列的3D人类动作，对构建类人交互AI系统至关重要。然而，现有方法难以有效利用视频输入引导人类反应生成，导致生成的反应动作与视频内容不匹配。本文发现这一限制源于视觉观察与反应类型之间严重的关系失真。为此，我们提出MuSteerNet，一个简单有效的框架，通过观察-反应相互引导生成3D人类反应。具体而言，我们首先提出原型反馈引导机制，通过门控delta修正模块和关系边际约束，利用从人类反应中学习的原型向量来细化视觉观察，以缓解关系失真。然后引入双耦合反应细化机制，充分利用修正后的视觉线索进一步引导生成反应动作的细化，从而有效提升反应质量，使MuSteerNet达到竞争性性能。大量实验和消融研究验证了方法的有效性。代码即将发布：https://github.com/zhouyuan888888/MuSteerNet。
英文摘要：
Video-driven 3D human reaction generation aims to synthesize 3D human motion in response to the action observed in a video, playing an important role in interactive multimedia systems and embodied agents. Yet reaction motions generated by current methods often fail to match what the observed video calls for. We observe that one factor behind this failure is relational distortion in the correspondence between visual observations and reactions: videos lying close in the visual space may correspond to entirely different motions in the reaction space, which misleads the model into generating reactions inconsistent with the conditioning video. This motivates us to propose a new observatioN-reactiOn mUtual Steering (\texttt{NOUS}) framework that enables mutual steering between the video and motion modalities. It first performs Motion Feedback Steering (MFS), equipping the frozen pretrained video encoder with a lightweight rectification modulator and training the modulator with a relational margin loss that pulls each video embedding toward the motion prototype of its own category and away from those of other categories. In this way, the misaligned correspondence between visual observations and reactions can be calibrated. \texttt{NOUS} then applies Observation-Guided Refinement (OGR), which in turn exploits the rectified observations to further refine the generated reactions and enhance their quality. The results on the ViMo dataset demonstrate that \texttt{NOUS} improves the quality of reaction motion while incurring negligible computational overhead at inference. Also, \texttt{NOUS} yields consistent gains across four pretrained video encoders, showing its good compatibility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
