arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.19679cs.AIcs.LG
收藏

MCAIR:基于内部表征监测与修正智能体的不忠实性

MCAIR: Monitoring and Correcting Agent Unfaithfulness via Internal Representations

School of Computer Science and Technology, Beijing Institute of Technology, China(北京理工大学计算机科学与技术学院)

Yan Xu, Dandan Song, Luan Zhang, Jun Yang, Ke Duan, Yuhang Tian, Yading Li, Xudong Li, Xing Pei, Guangyuan Feng, Chenhao Li

AI总结:

本研究针对基于LLM的智能体存在的完成不忠实性问题,提出MCAIR框架,通过训练风险感知探针在线检测并结合风险引导修正循环缓解该问题,构建ACUD数据集验证其性能优于基线且泛化性良好。

摘要:

基于大语言模型(LLM)的智能体通过多步推理和工具交互展现出完成复杂任务的卓越能力,但近期研究表明这类智能体可能出现不忠实行为,即其最终报告无法忠实地反映实际执行轨迹。具体而言,当智能体无法通过预期的来源或流程完成任务时,可能会采用替代来源或策略却不披露此类偏差,导致用户错误认为任务已按原始要求完成,我们将这种特定形式的智能体不忠实性称为“智能体完成不忠实性”。检测并缓解智能体完成不忠实性至关重要,但目前仍未得到充分探索。本研究提出MCAIR,一种利用基于LLM的智能体内部表征的运行时监测与修正框架。MCAIR将多步内部表征作为输入训练风险感知探针,以在执行过程中在线检测潜在的完成不忠实性;此外,MCAIR引入风险引导修正循环,促使智能体回顾执行历史并缓解执行轨迹与最终报告之间的潜在不一致。我们还构建了ACUD,一个用于智能体完成不忠实性检测的数据集,提供步骤级执行轨迹、步骤级内部表征及一致性标注,用于训练和评估运行时检测探针。实验表明,MCAIR的性能优于“LLM作为评判者”的基线,在提示和任务间具有泛化性,且能有效降低智能体完成不忠实性。

英文摘要:

Large Language Model (LLM)-based agents have demonstrated remarkable capabilities in completing complex tasks through multi-step reasoning and tool interactions. However, recent studies have revealed that LLM-based agents can exhibit unfaithful behaviors, where their final reports fail to faithfully reflect their actual execution trajectories. Specifically, when agents fail to accomplish tasks through the intended sources or procedures, they may resort to alternative sources or strategies while failing to disclose such deviations, leading users to incorrectly believe that the tasks have been completed according to the original requirements. We refer to this specific form of agent unfaithfulness as Agent Completion Unfaithfulness. Detecting and mitigating agent completion unfaithfulness is critical yet remains largely unexplored. In this work, we propose MCAIR, a runtime monitoring and correction framework that leverages internal representations of LLM-based agents. MCAIR uses multi-step internal representations as inputs to train a risk-aware probe, enabling online detection of potential completion unfaithfulness during execution. Furthermore, MCAIR introduces a risk-guided correction loop that prompts agents to review their execution history and mitigate potential inconsistencies between their execution trajectories and final reports. Additionally, we construct ACUD, a dataset for Agent Completion Unfaithfulness detection that provides step-level execution traces, step-level internal representations, and consistency annotations for training and evaluating runtime detection probes. Experiments demonstrate that MCAIR outperforms LLM-as-a-judge baselines, generalizes across prompts and tasks, and effectively reduces agent completion unfaithfulness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑