大型语言模型是否受多方对话的困扰？一种诊断方法用于对话中的收件人识别和回应选择
Do LLMs suffer from Multi-Party Hangover? A Diagnostic Approach to Addressee Recognition and Response Selection in Conversations
AI总结：
本文提出了一种方法论流程，用于研究对话特定结构属性下的模型性能，通过诊断子数据集分析回应选择和收件人识别任务，揭示模型弱点。
摘要：
评估系统对多方对话(MPC)分类性能具有挑战性，因为对话的语言和结构特征相互关联。传统评估方法常忽略模型行为在交互图不同结构复杂性层面的差异。本文提出了一种方法论流程，通过从大规模开放在线MPC语料库中提取代表性诊断子数据集，固定用户数量并保证结构多样性，以研究模型性能。我们进一步在数据最小化框架下进行，避免使用原始用户名以保护隐私，并提出替代原始文本消息的方法。结果表明，回应选择更依赖对话文本内容，而收件人识别需要捕捉其结构维度。使用零样本设置的LLM，进一步强调了对提示变化的敏感性依赖于任务。
英文摘要：
Assessing the performance of systems to classify Multi-Party Conversations (MPC) is challenging due to the interconnection between linguistic and structural characteristics of conversations. Conventional evaluation methods often overlook variances in model behavior across different levels of structural complexity on interaction graphs. In this work, we propose a methodological pipeline to investigate model performance across specific structural attributes of conversations. As a proof of concept we focus on Response Selection and Addressee Recognition tasks, to diagnose model weaknesses. To this end, we extract representative diagnostic subdatasets with a fixed number of users and a good structural variety from a large and open corpus of online MPCs. We further frame our work in terms of data minimization, avoiding the use of original usernames to preserve privacy, and propose alternatives to using original text messages. Results show that response selection relies more on the textual content of conversations, while addressee recognition requires capturing their structural dimension. Using an LLM in a zero-shot setting, we further highlight how sensitivity to prompt variations is task-dependent.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
