大语言模型能理解对话中强调句的隐含意义吗？
Can LLMs Understand the Implication of Emphasized Sentences in Dialogue?
AI总结：
本文提出强调标注对话基准 Emphasized-Talk，评估多种大语言模型理解强调隐含意义的能力，并设计基于 GPT-4 的自动评估流程，发现商业模型虽表现更优但仍存在明显改进空间。
摘要：
强调是人类交流中的关键组成部分，它在对话中指示了说话者超越纯文本的意图和隐含意义。尽管大语言模型（LLMs）已经彻底变革了自然语言处理，但它们理解对话中强调的能力仍不明确。本文引入了 Emphasized-Talk，一个带有强调标注的对话样本基准，用于捕捉强调的隐含意义。我们评估了多种大语言模型，包括开源和商业模型，以衡量它们在理解强调方面的表现。此外，我们提出了一个使用 GPT-4 的自动评估流程，该流程与人类评分取得了高度相关性。我们的研究结果表明，尽管商业大语言模型通常表现更好，但在理解强调句方面仍有显著的改进空间。
英文摘要：
Emphasis is a crucial component in human communication, which indicates the speaker's intention and implication beyond pure text in dialogue. While Large Language Models (LLMs) have revolutionized natural language processing, their ability to understand emphasis in dialogue remains unclear. This paper introduces Emphasized-Talk, a benchmark with emphasis-annotated dialogue samples capturing the implications of emphasis. We evaluate various LLMs, both open-source and commercial, to measure their performance in understanding emphasis. Additionally, we propose an automatic evaluation pipeline using GPT-4, which achieves a high correlation with human rating. Our findings reveal that although commercial LLMs generally perform better, there is still significant room for improvement in comprehending emphasized sentences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
