OmniVox：利用全模态大语言模型进行零样本情绪识别
OmniVox: Zero-Shot Emotion Recognition with Omni-LLMs
AI总结：
本文提出OmniVox，首次系统评估四种全模态大语言模型在零样本情绪识别任务上的表现，并引入声学提示策略，在IEMOCAP和MELD基准上验证其优于或媲美微调音频模型。
摘要：
全模态大语言模型（接受任何模态输入的大型语言模型）在涉及语音的多模态认知状态任务中的应用研究尚不充分。我们提出了OmniVox，这是首个对四种全模态大语言模型在零样本情绪识别任务上进行系统性评估的工作。我们在两个广泛使用的多模态情绪基准（IEMOCAP和MELD）上进行了评估，发现零样本全模态大语言模型的表现优于或与微调后的音频模型相当。除了仅音频评估外，我们还评估了全模态大语言模型在仅文本以及文本和音频输入上的表现。我们提出了声学提示（acoustic prompting），一种针对全模态大语言模型的音频特定提示策略，该策略侧重于声学特征分析、对话上下文分析和逐步推理。我们将声学提示与最小提示和完整思维链提示技术进行了比较。我们在IEMOCAP和MELD上进行了上下文窗口分析，发现使用上下文有助于提升性能，尤其是在IEMOCAP上。最后，我们对全模态大语言模型生成的声学推理输出进行了错误分析。
英文摘要：
The use of omni-LLMs (large language models that accept any modality as input), particularly for multimodal cognitive state tasks involving speech, is understudied. We present OmniVox, the first systematic evaluation of four omni-LLMs on the zero-shot emotion recognition task. We evaluate on two widely used multimodal emotion benchmarks: IEMOCAP and MELD, and find zero-shot omni-LLMs outperform or are competitive with fine-tuned audio models. Alongside our audio-only evaluation, we also evaluate omni-LLMs on text only and text and audio. We present acoustic prompting, an audio-specific prompting strategy for omni-LLMs which focuses on acoustic feature analysis, conversation context analysis, and step-by-step reasoning. We compare our acoustic prompting to minimal prompting and full chain-of-thought prompting techniques. We perform a context window analysis on IEMOCAP and MELD, and find that using context helps, especially on IEMOCAP. We conclude with an error analysis on the generated acoustic reasoning outputs from the omni-LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
