AlignCap：使语音情感描述对齐人类偏好
AlignCap: Aligning Speech Emotion Captioning to Human Preferences
AI总结：
针对语音情感描述中的幻觉与泛化问题，本文提出基于 LLM 的 AlignCap，通过语音-文本蒸馏对齐和人类偏好优化提升零样本描述性能。
摘要：
语音情感描述（Speech Emotion Captioning，SEC）已逐渐成为一项活跃的研究任务。人类语音所传达的情感内容通常较为复杂，将其划分为固定类别可能不足以完整捕捉语音情感。通过自然语言描述语音情感或许是一种更有效的方法。然而，现有 SEC 方法往往会产生幻觉，并在未见过的语音上丧失泛化能力。为克服这些问题，我们提出 AlignCap，它基于大语言模型（LLM）使语音情感描述对齐人类偏好，并具备两项特性：1）语音-文本对齐，利用知识蒸馏（KD）正则化，最小化 LLM 对语音输入和文本输入的响应预测分布之间的差异。2）人类偏好对齐，我们设计偏好优化（PO）正则化，以消除事实性和忠实性幻觉。我们还在 KD 正则化下提取情感线索作为提示，用于丰富细粒度信息。实验表明，在 Zero-shot SEC 任务上，AlignCap 相比其他最先进方法表现出更强的性能。
英文摘要：
Speech Emotion Captioning (SEC) has gradually become an active research task. The emotional content conveyed through human speech are often complex, and classifying them into fixed categories may not be enough to fully capture speech emotions. Describing speech emotions through natural language may be a more effective approach. However, existing SEC methods often produce hallucinations and lose generalization on unseen speech. To overcome these problems, we propose AlignCap, which Aligning Speech Emotion Captioning to Human Preferences based on large language model (LLM) with two properties: 1) Speech-Text Alignment, which minimizing the divergence between the LLM's response prediction distributions for speech and text inputs using knowledge distillation (KD) Regularization. 2) Human Preference Alignment, where we design Preference Optimization (PO) Regularization to eliminate factuality and faithfulness hallucinations. We also extract emotional clues as a prompt for enriching fine-grained information under KD-Regularization. Experiments demonstrate that AlignCap presents stronger performance to other state-of-the-art methods on Zero-shot SEC task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
