arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.26346cs.AI
收藏

工具增强的LLM智能体在真实世界能源分析任务中的表现如何?

How Do Tool-Augmented LLM Agents Perform on Real-World Energy Analytics Tasks?

Independent Researcher(独立研究员);Tume AI

David Akinpelu, Akintonde Abbas, Rereloluwa Alimi, Ayodeji Lana

AI总结:

针对能源领域缺乏智能体评估基准的问题,构建了包含243个专家策划问题的测试环境,评估工具增强的LLM智能体在数据检索、知识解释和定量建模三类任务上的表现。

摘要:

通用和特定领域的智能体基准测试已出现在金融、编程、法律和药物发现等领域,但能源领域的评估仍主要局限于静态知识回忆。对于需要实时数据检索、专业监管和市场知识以及多步定量推理的行业来说,这是一个关键缺口。我们提出了对工具增强的LLM智能体在真实世界能源市场分析任务上的实证研究。我们的评估环境包含243个专家策划的问题,分为三类:(1) 市场数据检索与分析,(2) 知识检索与解释,(3) 高级定量建模与决策分析。任务包括价格和需求分析、关税影响建模、资产收入和回报估算、对冲策略分析以及优化建模,问题涵盖多个难度级别。智能体配备了一套可配置的领域工具,包括美国主要ISO的实时电力市场API、监管案卷搜索、公用事业关税数据库、资产优化模型以及基于检索增强生成的能源市场文档。我们使用多维评估协议评估智能体的响应,该协议对方法正确性、答案准确性、属性对齐和来源有效性进行评分,并通过类别感知路由将评分标准与问题类型匹配。我们评估了闭源和开源LLM,提供了模型能力与领域工具在高风险专业领域如何相互作用的比较分析。关键工件已公开发布,以支持可重复性和未来研究。

英文摘要:

Agentic benchmarks have emerged across general-purpose and domain-specific settings, including finance, coding, law, and drug discovery, yet energy-domain evaluations remain largely limited to static knowledge recall. This is a critical gap for a sector that requires live data retrieval, specialized regulatory and market knowledge, and multi-step quantitative reasoning under real-world constraints. We present an empirical study of tool-augmented LLM agents on real-world energy market analytics tasks. Our evaluation environment includes 243 expert-curated problems across three categories: (1) Market Data Retrieval and Analysis, (2) Knowledge Retrieval and Interpretation, and (3) Advanced Quantitative Modeling and Decision Analytics. Tasks include price and demand analysis, tariff impact modeling, asset revenue and returns estimation, hedging strategy analysis, and optimization modeling, with problems spanning multiple difficulty levels. Agents are equipped with a configurable suite of domain tools, including live electricity market APIs for major U.S. ISOs, regulatory docket search, utility tariff databases, asset optimization models, and retrieval-augmented generation over energy market documents. We assess agent responses using a multi-dimensional evaluation protocol that scores approach correctness, answer accuracy, attribute alignment, and source validity, with category-aware routing to match scoring criteria to question type. We evaluate both closed-source and open-source LLMs, providing a comparative analysis of how model capability and domain tooling interact in a high-stakes professional domain. Key artifacts are publicly released to support reproducibility and future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑