C3：探索复杂对话挑战的口语对话模型双语基准
C3: A Bilingual Benchmark for Spoken Dialogue Models Exploring Challenges in Complex Conversations
AI总结：
针对口语对话模型（SDM）缺乏全面性能基准的问题，该研究提出包含1079条中英文实例的双语基准C3，搭配与人类判断对齐的LLM评估方法，用于探究SDM应对口语歧义、语境依赖等复杂对话挑战的能力。
摘要：
口语对话模型（Spoken Dialogue Models，SDMs）凭借可直接针对用户语音查询生成语音回复的能力，近期受到广泛关注。尽管其应用日益普及，但在全面理解这类模型理解与模拟人类对话的实际效用方面，相关研究仍存在空白，与受益于大量基准测试的文本型大语言模型（Large Language Models，LLMs）相比尤为明显。由于口语对话的独有特性，人类语音交互本质上比文本更复杂：歧义是一大挑战，既来自多义性等语义因素，也源于同形异义词、同形异音异义词及重音模式等语音层面的问题；此外，省略、指代、多轮交互等语境依赖特性进一步增加了人类对话动态的复杂度。为阐明SDM的发展现状并应对上述挑战，本文提出一个包含1079条中英文实例的基准数据集，搭配一种与人类判断高度对齐的基于LLM的评估方法，可助力全面探究SDM应对这些实际挑战的性能表现。
英文摘要：
Spoken Dialogue Models (SDMs) have recently attracted significant attention for their ability to generate voice responses directly to users' spoken queries. Despite their increasing popularity, there exists a gap in research focused on comprehensively understanding their practical effectiveness in comprehending and emulating human conversations. This is especially true compared to text-based Large Language Models (LLMs), which benefit from extensive benchmarking. Human voice interactions are inherently more complex than text due to characteristics unique to spoken dialogue. Ambiguity poses one challenge, stemming from semantic factors like polysemy, as well as phonological aspects such as heterograph, heteronyms, and stress patterns. Additionally, context-dependency, like omission, coreference, and multi-turn interaction, adds further complexity to human conversational dynamics. To illuminate the current state of SDM development and to address these challenges, we present a benchmark dataset in this paper, which comprises 1,079 instances in English and Chinese. Accompanied by an LLM-based evaluation method that closely aligns with human judgment, this dataset facilitates a comprehensive exploration of the performance of SDMs in tackling these practical challenges.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
