当个人记忆没有唯一答案时：评估大语言模型智能体在不可约冲突下的表现

中文摘要 AI 辅助

大语言模型智能体越来越多地在多轮交互会话中维护个人记忆，但这些记忆可能存在冲突。智能体的偏好依赖于上下文，行为会随时间演变，不同记忆来源也可能相互冲突。当查询缺乏足够的上下文、时间信息或来源权威性来解释冲突时，将某一条记忆视为确定答案，会将未解决的冲突转化为无依据、过度自信的行动。现有基准测试会从冲突证据中恢复出唯一答案，却忽略了智能体是否能认识到信息不足、保留替代方案、寻求缺失信息并选择合适的行动。我们提出了用于测试智能体应对真实、潜在及纠缠记忆冲突的基准测试 TANGLE（Testing Agents' Navigation of Genuine, Latent, and Entangled Memory Conflicts），该基准针对真正不可解决的记忆冲突，包含40个角色、三类共541个实例：上下文划分冲突（CPC）、行为振荡冲突（BOC）和来源矛盾冲突（SCC）。我们在两个轨道上评估了五项指标：包含精心整理记忆的神谕轨道，以及从多轮对话中提取记忆的流水线轨道，评估维度为冲突感知、因果推理、置信度校准、澄清请求和记忆忠实度。实验揭示了流水线存在的挑战：在神谕轨道中，模型识别冲突的可靠性高于校准行动或寻求针对性澄清的能力；在端到端流水线记忆中，提取过程无法保留下游推理所需的冲突关联。策略对比显示，当行动必须反映冲突时，固定规则并不适用。这些发现推动了冲突感知行动策略（CAAP）的提出，该策略会利用现有证据为每种冲突调整行动。TANGLE将冲突处理定义为认识到信息不足、保留冲突证据以及在不强制给出确定答案的情况下采取行动。

英文摘要

LLM agents increasingly maintain personal memory across sessions, but it can conflict. Preferences depend on context, behavior evolves, and sources can conflict. When a query lacks context, time, or source authority to interpret conflict, treating one memory as definitive converts unresolved conflict into an unjustified, overconfident action. Existing benchmarks recover one answer from conflicting evidence, overlooking whether agents recognize underdetermination, preserve alternatives, seek missing information, and choose appropriate actions. We introduce \underline{T}esting \underline{A}gents' \underline{N}avigation of \underline{G}enuine, \underline{L}atent, and \underline{E}ntangled Memory Conflicts (\textsc{TANGLE}), a benchmark for genuinely unresolvable memory conflicts. It comprises 541 instances across 40 personas and three types: Context-Partitioned Conflict (CPC), Behavior-Oscillation Conflict (BOC), and Source-Contradiction Conflict (SCC). We evaluate two tracks---an oracle track with curated memory and a pipeline track that extracts memory from multi-session dialogues---on five dimensions: conflict perception, causal reasoning, confidence calibration, clarification seeking, and memory faithfulness. Experiments reveal pipeline challenges. With curated memory, models recognize conflicts more reliably than they calibrate actions or seek targeted clarification. With end-to-end pipeline memory, extraction fails to preserve conflict-bearing relations needed for downstream reasoning. Policy comparisons show fixed rules are insufficient when actions must reflect conflict. These findings motivate Conflict-Aware Action Policy (CAAP), which adapts actions to each conflict using available evidence. \textsc{TANGLE} frames conflict handling as recognizing underdetermination, retaining conflicting evidence, and acting without forcing a definitive answer.

发表机构

• Ant Group（蚂蚁集团）

机构由 AI 辅助整理，请以论文原文为准。