Watson：一个面向LLM驱动智能体推理的认知可观测性框架
Watson: A Cognitive Observability Framework for the Reasoning of LLM-Powered Agents
AI总结：
Watson框架通过提示归因技术追溯推断LLM智能体的隐式推理过程，在不改变其行为的前提下实现认知可观测性，并在多个基准和智能体上验证了其对提升Agentware系统透明度和可靠性的实用价值。
摘要：
大语言模型（LLMs）正越来越多地被集成到自主系统中，催生了一类被称为Agentware的新型软件。在这类软件中，由LLM驱动的智能体在软件工程、客户服务和数据分析等领域执行复杂、开放式的任务。然而，它们的高度自主性和不透明的推理过程给传统的软件可观测性方法带来了重大挑战。为了解决这一问题，我们引入了认知可观测性的概念——即恢复并检查智能体决策背后隐式推理的能力。我们提出了Watson，这是一个通用框架，用于在不改变快速思考型LLM智能体行为的情况下观测其推理过程。Watson利用提示归因技术追溯性地推断推理轨迹。我们在MMLU基准测试以及SWE-bench-lite数据集上的AutoCodeRover和OpenHands智能体上，分别在手动调试和自动修正场景中对Watson进行了评估。在静态和动态设置下，Watson都能揭示可操作的推理见解，并支持有针对性的干预，证明了其在提高Agentware系统透明度和可靠性方面的实际效用。
英文摘要：
Large language models (LLMs) are increasingly integrated into autonomous systems, giving rise to a new class of software known as Agentware, where LLM-powered agents perform complex, open-ended tasks in domains such as software engineering, customer service, and data analysis. However, their high autonomy and opaque reasoning processes pose significant challenges for traditional software observability methods. To address this, we introduce the concept of cognitive observability - the ability to recover and inspect the implicit reasoning behind agent decisions. We present Watson, a general-purpose framework for observing the reasoning processes of fast-thinking LLM agents without altering their behavior. Watson retroactively infers reasoning traces using prompt attribution techniques. We evaluate Watson in both manual debugging and automated correction scenarios across the MMLU benchmark and the AutoCodeRover and OpenHands agents on the SWE-bench-lite dataset. In both static and dynamic settings, Watson surfaces actionable reasoning insights and supports targeted interventions, demonstrating its practical utility for improving transparency and reliability in Agentware systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
