arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.28056cs.CV
收藏

CogPortrait: 通过分层智能体规划实现肖像动画中的细粒度眼部区域控制

CogPortrait: Fine-Grained Eye-Region Control in Portrait Animation via Hierarchical Agent Planning

Harbin Institute of Technology（哈尔滨工业大学）；University of New South Wales（新南威尔士大学）

He Feng, Yongjia Ma, Donglin Di, Lei Fan, Tonghua Su

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出CogPortrait两阶段框架，利用多模态大语言模型智能体从高层标签生成关键点，再通过DiT视频生成骨干合成动画，实现细粒度眼部控制，并引入EMH基准评估。

摘要：

肖像动画方法已实现显著的视觉质量和唇形同步，但眼部区域的细粒度操控仍面临输入粒度与运动精度之间的权衡。现有方法使用情感标签或粗略文本提示不足以描述细微的眼部动态，而基于动作单元或驱动视频的方法以更高的输入负担为代价提供更高的保真度。这些限制对于超越情感状态（例如思考）和困倦状态仍然具有局限性。鉴于此，我们提出CogPortrait，一个从高层标签生成肖像动画的两阶段框架。在第一阶段，三个思维链多模态大语言模型（MLLMs）智能体通过时间事件规划、原型检索和从真实行为库中组合以及语义-生理约束执行，将高层标签编译为面部关键点。在第二阶段，基于DiT的视频生成骨干以关键点、参考肖像、音频和文本提示为条件合成最终动画，并通过动态无分类器引导策略（具有眼部区域感知重新加权和基于KTO的边界情况细化）增强。我们进一步引入了EMH基准，涵盖多样化的情感和超越情感类别，并带有两个AU级指标用于评估细粒度眼部区域和头部运动控制。在HDTF和EMH基准上的大量实验表明，CogPortrait在保持优越视觉质量和身份一致性的同时，实现了比现有方法更精确的眼部区域控制。

英文摘要：

Portrait animation methods have achieved substantial visual quality and lip synchronization, but fine-grained manipulation of the eye region still faces a trade-off between input granularity and motion accuracy. Existing methods using emotion labels or coarse text prompts are insufficient for describing subtle ocular dynamics, whereas approaches based on Action Units or driving videos provide higher fidelity at the cost of a heavier input burden. These limitations are still restrictive for beyond-emotion states (e.g., thinking) and drowsiness. In light of the above, we propose CogPortrait, a two-stage framework that generates portrait animations from high-level labels. In the first stage, three chain-of-thought Multimodal Large Language Models (MLLMs) agents compile high-level labels into facial keypoints through temporal event planning, prototype retrieval, and composition from a real-behavior library, and semantic-physiological constraint enforcement. In the second stage, a DiT-based video generation backbone synthesizes the final animation conditioned on the keypoints, reference portrait, audio, and text prompt, enhanced by a dynamic classifier-free guidance strategy with eye-region-aware reweighting and KTO-based refinement for boundary cases. We further introduce the EMH benchmark covering diverse emotions and beyond-emotion categories with two AU-level metrics for evaluating fine-grained eye-region and head-motion control. Extensive experiments on HDTF and the EMH benchmark demonstrate that CogPortrait achieves more precise eye-region control than existing methods while maintaining supe- rior visual quality and identity consistency

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑