BeliefShift：评估LLM代理中时间信念一致性和意见漂移的基准测试
BeliefShift: Benchmarking Temporal Belief Consistency and Opinion Drift in LLM Agents
AI总结：
BeliefShift提出一个长期基准测试，评估多会话LLM交互中的信念动态，涵盖时间信念一致性、矛盾检测和证据驱动修正三个赛道，通过2400个标注交互轨迹验证模型在零样本和RAG设置下的表现。
摘要：
LLMs日益被用作长期对话代理，但现有评估其记忆的基准将用户信息视为静态事实存储和检索，这不正确。人们会改变主意，长时间交互中意见漂移、过度一致性和确认偏见变得重要。BeliefShift引入了一个专门评估多会话LLM交互中信念动态的长期基准测试，涵盖三个赛道：时间信念一致性、矛盾检测和证据驱动修正。数据集包含2400个由人类标注的多会话交互轨迹，涵盖健康、政治、个人价值观和产品偏好。我们评估了包括GPT-4o、Claude 3.5 Sonnet、Gemini 1.5 Pro、LLaMA-3和Mistral-Large在内的七种模型，在零样本和检索增强生成（RAG）设置下。结果揭示了明显的权衡：过度个性化模型对漂移的抵抗能力差，而事实基础模型会错过合法的信念更新。我们进一步引入了四个新的评估指标：信念修正准确率（BRA）、漂移一致性得分（DCS）、矛盾解决率（CRR）和证据敏感度指数（ESI）。
英文摘要：
LLMs are increasingly used as long-running conversational agents, yet every major benchmark evaluating their memory treats user information as static facts to be stored and retrieved. That's the wrong model. People change their minds, and over extended interactions, phenomena like opinion drift, over-alignment, and confirmation bias start to matter a lot. BeliefShift introduces a longitudinal benchmark designed specifically to evaluate belief dynamics in multi-session LLM interactions. It covers three tracks: Temporal Belief Consistency, Contradiction Detection, and Evidence-Driven Revision. The dataset includes 2,400 human-annotated multi-session interaction trajectories spanning health, politics, personal values, and product preferences. We evaluate seven models including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, LLaMA-3, and Mistral-Large under zero-shot and retrieval-augmented generation (RAG) settings. Results reveal a clear trade-off: models that personalize aggressively resist drift poorly, while factually grounded models miss legitimate belief updates. We further introduce four novel evaluation metrics: Belief Revision Accuracy (BRA), Drift Coherence Score (DCS), Contradiction Resolution Rate (CRR), and Evidence Sensitivity Index (ESI).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
