UniEmo: 利用可学习专家查询统一情感理解与生成
UniEmo: Unifying Emotional Understanding and Generation with Learnable Expert Queries
AI总结：
提出UniEmo统一框架，通过可学习专家查询的分层情感理解链提取多尺度情感特征，并融合情感表示引导扩散模型生成情感图像，同时利用生成驱动的双重反馈提升理解能力，在情感理解和生成任务上超越现有方法。
摘要：
情感理解和生成通常被视为独立的任务，然而它们本质上是互补的，可以相互增强。在本文中，我们提出UniEmo，一个无缝集成这两个任务的统一框架。关键挑战在于情感的抽象性质，需要提取对两个任务都有益的视觉表示。为此，我们提出一个带有可学习专家查询的分层情感理解链，逐步提取多尺度情感特征，从而作为统一的基础步骤。同时，我们融合这些专家查询和情感表示，以指导扩散模型生成引发情感反应的图像。为了增强生成情感图像的多样性和保真度，我们进一步在融合过程中引入情感相关系数和情感条件损失。这一步骤促进了由理解引导的情感生成的融合与对齐。反过来，我们证明联合训练允许生成部分向理解部分提供隐式反馈。此外，我们提出一种新颖的数据过滤算法，以选择由训练良好的模型生成的高质量和多样化的情感图像，这些图像显式地反馈到理解部分。这些生成驱动的双重反馈过程共同增强了模型的理解能力。大量实验表明，UniEmo在情感理解和生成任务上均显著优于现有方法。所提出方法的代码可在 https://github.com/JiuTian-VL/UniEmo 获取。
英文摘要：
Emotional understanding and generation are often treated as separate tasks, yet they are inherently complementary and can mutually enhance each other. In this paper, we propose the UniEmo, a unified framework that seamlessly integrates these two tasks. The key challenge lies in the abstract nature of emotions, necessitating the extraction of visual representations beneficial for both tasks. To address this, we propose a hierarchical emotional understanding chain with learnable expert queries that progressively extracts multi-scale emotional features, thereby serving as a foundational step for unification. Simultaneously, we fuse these expert queries and emotional representations to guide the diffusion model in generating emotion-evoking images. To enhance the diversity and fidelity of the generated emotional images, we further introduce the emotional correlation coefficient and emotional condition loss into the fusion process. This step facilitates fusion and alignment for emotional generation guided by the understanding. In turn, we demonstrate that joint training allows the generation component to provide implicit feedback to the understanding part. Furthermore, we propose a novel data filtering algorithm to select high-quality and diverse emotional images generated by the well-trained model, which explicitly feedback into the understanding part. Together, these generation-driven dual feedback processes enhance the model's understanding capacity. Extensive experiments show that UniEmo significantly outperforms state-of-the-art methods in both emotional understanding and generation tasks. The code for the proposed method is available at https://github.com/JiuTian-VL/UniEmo.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
