CryptoAnalystBench: 多工具长文本LLM分析中的失败
CryptoAnalystBench: Failures in Multi-Tool Long-Form LLM Analysis
AI总结：
本文提出CryptoAnalystBench，通过198个加密和DeFi查询评估多工具LLM在复杂输入中的表现，揭示七类高阶错误类型，并提供评估框架和缓解策略。
摘要：
现代分析师代理必须处理复杂的高token输入，包括数十份检索文档、工具输出和时效数据。尽管已有工具调用基准和知识增强系统的事实性研究，但较少探讨其交集：LLM必须整合大量动态、结构化和非结构化多工具输出的场景。我们通过加密领域研究LLM在该模式下的失败模式，引入CryptoAnalystBench，一个包含198个生产加密和DeFi查询的分析师对齐基准；一个配备相关加密和DeFi工具的代理框架，用于生成多个前沿LLM的响应；以及一个包含引用验证和LLM作为评判标准的评估流程，涵盖四个用户定义的成功维度：相关性、时间相关性、深度和数据一致性。通过人工标注，我们开发了七种高阶错误类型分类，这些类型无法可靠地通过事实性检查或LLM基于质量评分捕获。我们发现这些失败在最先进的系统中仍然存在，并可能影响高风险决策。基于此分类，我们优化了评判标准以更好地捕捉这些错误。虽然评判标准在不同迭代中与人工标注者在精确评分上不一致，但其能可靠地识别关键失败模式，为开发者和研究分析师风格代理的研究提供可扩展的反馈。我们发布了CryptoAnalystBench，包含标注查询、评估流程、评判标准和错误分类，并概述了评估长文本、多工具增强系统的方法论和开放挑战。
英文摘要：
Modern analyst agents must reason over complex, high token inputs, including dozens of retrieved documents, tool outputs, and time sensitive data. While prior work has produced tool calling benchmarks and examined factuality in knowledge augmented systems, relatively little work studies their intersection: settings where LLMs must integrate large volumes of dynamic, structured and unstructured multi tool outputs. We investigate LLM failure modes in this regime using crypto as a representative high data density domain. We introduce (1) CryptoAnalystBench, an analyst aligned benchmark of 198 production crypto and DeFi queries spanning 11 categories; (2) an agentic harness equipped with relevant crypto and DeFi tools to generate responses across multiple frontier LLMs; and (3) an evaluation pipeline with citation verification and an LLM as a judge rubric spanning four user defined success dimensions: relevance, temporal relevance, depth, and data consistency. Using human annotation, we develop a taxonomy of seven higher order error types that are not reliably captured by factuality checks or LLM based quality scoring. We find that these failures persist even in state of the art systems and can compromise high stakes decisions. Based on this taxonomy, we refine the judge rubric to better capture these errors. While the judge does not align with human annotators on precise scoring across rubric iterations, it reliably identifies critical failure modes, enabling scalable feedback for developers and researchers studying analyst style agents. We release CryptoAnalystBench with annotated queries, the evaluation pipeline, judge rubrics, and the error taxonomy, and outline mitigation strategies and open challenges in evaluating long form, multi tool augmented systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
