Agentic benchmarks have emerged across general-purpose and domain-specific settings, including finance, coding, law, and drug discovery, yet energy-domain evaluations remain largely limited to static knowledge recall. This is a critical gap for a sector that requires live data retrieval, specialized regulatory and market knowledge, and multi-step quantitative reasoning under real-world constraints.
We present an empirical study of tool-augmented LLM agents on real-world energy market analytics tasks. Our evaluation environment includes 243 expert-curated problems across three categories: (1) Market Data Retrieval and Analysis, (2) Knowledge Retrieval and Interpretation, and (3) Advanced Quantitative Modeling and Decision Analytics. Tasks include price and demand analysis, tariff impact modeling, asset revenue and returns estimation, hedging strategy analysis, and optimization modeling, with problems spanning multiple difficulty levels.
Agents are equipped with a configurable suite of domain tools, including live electricity market APIs for major U.S. ISOs, regulatory docket search, utility tariff databases, asset optimization models, and retrieval-augmented generation over energy market documents. We assess agent responses using a multi-dimensional evaluation protocol that scores approach correctness, answer accuracy, attribute alignment, and source validity, with category-aware routing to match scoring criteria to question type. We evaluate both closed-source and open-source LLMs, providing a comparative analysis of how model capability and domain tooling interact in a high-stakes professional domain. Key artifacts are publicly released to support reproducibility and future research.
ToolGate: Token-Efficient Pre-Call Control for Tool-Augmented Vision-Language Agents
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));University College London(伦敦大学学院);AI Lab, The Yangtze River Delta(长江三角洲人工智能实验室)
VitalAgent: A Tool-Augmented Agent for Reactive and Proactive Physiological Monitoring over Wearable Health Data
The University of Melbourne, Australia(墨尔本大学);Dartmouth College, US(达特茅斯学院);University of Auckland, New Zealand(奥克兰大学);Eindhoven University of Technology, Netherlands(埃因霍温理工大学)
Exploring Multi-Modal Data with Tool-Augmented LLM Agents for Precise Causal Discovery
University of Houston(德克萨斯大学休斯顿分校);NEC Laboratories America(NEC美国实验室);Florida International University(佛罗里达国际大学);North Carolina State University(北卡罗来纳州立大学)
Learning to reason about rare diseases through retrieval-augmented agents
Technical University of Munich(慕尼黑技术大学);GE HealthCare(GE医疗);Munich Center for Machine Learning(慕尼黑机器学习中心);Klinikum Rechts der Isar(莱布尼茨医院);Helmholtz Munich(亥姆霍兹慕尼黑研究中心);King's College London(伦敦国王学院)
When Tool Outputs Become Commands: Separating Action Induction from Runtime Authorization in Tool-Augmented LLM Agents
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
Large Language Models (LLMs) as Agents for Augmented Democracy
University of Toulouse(图卢兹大学);Corvinus University of Budapest(布达佩斯考文纽斯大学);IRIT, Université Toulouse Capitole(图卢兹 Capitole 大学图卢兹计算机研究所);Toulouse School of Economics(图卢兹经济学院);Université Toulouse Capitole(图卢兹 Capitole 大学)
Leveraging Chemistry Foundation Models to Facilitate Structure Focused Retrieval Augmented Generation in Multi-Agent Workflows for Catalyst and Materials Design