从事实到判断：探讨任务框架对对话系统中LLM确信度的影响
From Fact to Judgment: Investigating the Impact of Task Framing on LLM Conviction in Dialogue Systems
AI总结：
研究探讨了任务框架对LLM在对话系统中判断确信度的影响，发现对话上下文显著改变模型判断，强调了对话框架在评估中的重要性。
摘要：
大型语言模型（LLMs）正在越来越多地被用作各种任务中的裁判，包括涉及日常社交互动的任务。然而，尚不清楚这些LLM裁判能否可靠地评估需要社交或对话判断的任务。我们研究了当任务从直接的事实查询重新框架化为对话判断任务时，LLM的确信度如何变化。我们的评估框架将模型在直接事实查询上的表现与在最小对话中呈现相同信息时对说话者正确性的评估进行对比，从而将查询从“这个陈述是否正确？”转变为“这个说话者是否正确？”。此外，我们以简单的反驳（“之前的回答是错误的”）的形式对两种条件施加压力。这种扰动使我们能够测量模型在对话压力下保持立场的坚定程度。我们的发现表明，尽管一些模型如GPT-4o-mini在社交框架任务中表现出趋炎附势的倾向，而其他如Llama-8B-Instruct则变得过于批判。我们观察到所有模型的平均表现变化为9.24%，证明即使是最小的对话上下文也能显著改变模型的判断，突显了对话框架作为LLM评估中的关键因素。所提出的框架提供了一种可重复的方法来诊断模型的确信度，并有助于开发更加可信的对话系统。
英文摘要：
LLMs are increasingly employed as judges across a variety of tasks, including those involving everyday social interactions. Yet, it remains unclear whether such LLM-judges can reliably assess tasks that require social or conversational judgment. We investigate how an LLM's conviction is changed when a task is reframed from a direct factual query to a Conversational Judgment Task. Our evaluation framework contrasts the model's performance on direct factual queries with its assessment of a speaker's correctness when the same information is presented within a minimal dialogue, effectively shifting the query from "Is this statement correct?" to "Is this speaker correct?". Furthermore, we apply pressure in the form of a simple rebuttal ("The previous answer is incorrect.") to both conditions. This perturbation allows us to measure how firmly the model maintains its position under conversational pressure. Our findings show that while some models like GPT-4o-mini reveal sycophantic tendencies under social framing tasks, others like Llama-8B-Instruct become overly-critical. We observe an average performance change of 9.24% across all models, demonstrating that even minimal dialogue context can significantly alter model judgment, underscoring conversational framing as a key factor in LLM-based evaluation. The proposed framework offers a reproducible methodology for diagnosing model conviction and contributes to the development of more trustworthy dialogue systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
