Conversations with LMs involve two participants: a human user leading the conversation, and an LM assistant responding to the user's request. To satisfy this specific role, LMs are post-trained to be helpful assistants -- optimized to produce exhaustive and well-structured responses, free of ambiguity and grammar errors. User utterances, on the other hand, are rarely perfected, with each user phrasing requests in unique ways, sometimes putting in partial effort at each turn and refining on the fly. To evaluate LM performance in realistic settings, prior work simulated users in multi-turn conversations, often by prompting an LM originally trained to be a helpful assistant to act as a user. However, we show that assistant LMs make for poor user simulators, with the surprising finding that better assistants yield worse simulators. Instead, we introduce purpose-built User Language Models (User LMs) - models post-trained to simulate human users in multi-turn conversations. Through various evaluations, we show how User LMs align better with human behavior and achieve better simulation robustness than existing simulation methods. When leveraging User LMs to simulate coding and math conversations, the performance of a strong assistant (GPT-4o) drops from 74.6% to 57.4%, confirming that more realistic simulation environments lead to assistant struggles as they fail to cope with the nuances of users in multi-turn setups.
PARADISE: A Framework for Evaluating Spoken Dialogue Agents
本文介绍了Paradise(PARAdigm for DIMENSIONAL System Evaluation),一个评价口语对话代理的通用框架。该框架从代理人的对话行为中提取任务要求,支持对话策略之间的比较,能够计算子对话和整个对话的性能,指定各种因素对性能的相对贡献,并通过规范化任务复杂性来比较执行不同任务的代理人。
Re-evaluating ADEM: A Deeper Look at Scoring Dialogue Responses
Indian Institute of Technology, Madras(印度理工学院马德拉斯分校);Robert Bosch Center for Data Sciences and AI (RBC-DSAI), Indian Institute of Technology, Madras(罗伯特博世数据科学与人工智能中心(RBC-DSAI),印度理工学院马德拉斯分校);Microsoft, India(微软印度公司);NVIDIA, India(英伟达印度公司)
自动评估非结构化领域的对话响应的质量是一个具有挑战性的问题。ADEM(Lowe et al. 2017)将对话系统的自动评估表述为一个学习问题,并表明这种模型能够预测与人类判断显著相关的反应,无论是在话语还是系统层面。他们的系统被证明击败了单词重叠指标,如BLEU,具有很大的利润率。我们从对手是否能利用ADEM模型的问题开始。针对基于神经网络的ADEM评估系统设计了一系列有针对性的攻击,表明对话系统的自动评估还有很长的路要走。ADEM可能会与文本中的词序颠倒这样简单的变体混淆!我们报告几个这样的对抗性的情况下,得出违反直觉的分数的对话反应的实验。我们采取了系统的看ADEM提出的评分功能,并将其连接到线性系统理论来预测系统中明显的缺点。我们还设计了一种攻击,可以欺骗这样的系统,以评价响应生成系统作为有利的。最后,我们提到未来的研究方向,使用对抗攻击,设计一个真正的自动化对话评估系统。
What Do Humans Hear When Interacting? Experiments on Selective Listening for Evaluating ASR of Spoken Dialogue Systems
Nara Institute of Science and Technology(奈良科学技术研究所);RIKEN(理化学研究所);Japan National Institute of Informatics(日本信息处理学会);Tokyo Institute of Technology(东京技术大学)
Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration
National University of Singapore(新加坡国立大学);Singapore Management University(新加坡管理大学);The Chinese University of Hong Kong(香港中文大学);Sichuan University(四川大学)
StanceFlip: A Comprehensive Multi-Dimensional Benchmark for Multimodal Conversational Stance Flipping Forecasting
College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机科学与软件学院);Harbin Institute of Technology Shenzhen(哈尔滨工业大学(深圳));City University of Macau(澳门城市大学)
SDARE-Bench: Evaluating Large Language Models on Conversational Stigma Detection and Response in Dyadic and Group Dialogue
Monash University(莫纳什大学);University of Liverpool(利物浦大学);University of Edinburgh(爱丁堡大学);Federation University(联邦大学);Orygen, The University of Melbourne(墨尔本大学下属Orygen机构)
PersuasiveToM: A Benchmark for Evaluating Machine Theory of Mind in Persuasive Dialogues
National Key Laboratory for Novel Software Technology, Nanjing University, China(南京大学新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University, China(南京大学人工智能学院);Department of Computer Science and Engineering, Shanghai Jiao Tong University(上海交通大学计算机科学与工程系);University of California, San Diego, CA, USA(美国加州大学圣地亚哥分校)
clem:todd: A Framework for the Systematic Benchmarking of LLM-Based Task-Oriented Dialogue System Realisations
Computational Linguistics, Department of Linguistics University of Potsdam(普腾多夫大学语言学系);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心)
CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference
The Hong Kong Polytechnic University(香港理工大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Research Centre for Data Science & Artificial Intelligence(数据科学与人工智能研究中心)