Large Language Models (LLMs) excel in diverse applications including generation of code snippets, but often struggle with generating code for complex Machine Learning (ML) tasks. Although existing LLM single-agent based systems give varying performance depending on the task complexity, they purely rely on larger and expensive models such as GPT-4. Our investigation reveals that no-cost and low-cost models such as Gemini-Pro, Mixtral and CodeLlama perform far worse than GPT-4 in a single-agent setting. With the motivation of developing a cost-efficient LLM based solution for solving ML tasks, we propose an LLM Multi-Agent based system which leverages combination of experts using profiling, efficient retrieval of past observations, LLM cascades, and ask-the-expert calls. Through empirical analysis on ML engineering tasks in the MLAgentBench benchmark, we demonstrate the effectiveness of our system, using no-cost models, namely Gemini as the base LLM, paired with GPT-4 in cascade and expert to serve occasional ask-the-expert calls for planning. With 94.2\% reduction in the cost (from \$0.931 per run cost averaged over all tasks for GPT-4 single agent system to \$0.054), our system is able to yield better average success rate of 32.95\% as compared to GPT-4 single-agent system yielding 22.72\% success rate averaged over all the tasks of MLAgentBench.
Automating Skill Acquisition through Large-Scale Mining of Open-Source Agentic Repositories: A Framework for Multi-Agent Procedural Knowledge Extraction
Can LLMs Help Create Grammar?: Automating Grammar Creation for Endangered Languages with In-Context Learning
Language and Information Technology Programme, Faculty of Arts, CU(朱拉隆功大学文学院语言与信息技术项目);Center for Information and Language Processing (CIS), LMU Munich(慕尼黑大学信息与语言处理中心)
From Intention To Implementation: Automating Biomedical Research via LLMs
School of Informatics, National Institute for Data Science in Health and Medicine, Xiamen University, Xiamen(信息学院、医学数据科学国家研究院、厦门大学、厦门);School of Medicine, Xiamen University, Xiamen(医学院、厦门大学、厦门);Microsoft Research Asia, Beijing(微软亚洲研究院、北京);College of Computing, Georgia Institute of Technology, Atlanta(计算学院、佐治亚理工学院、亚特兰大);Zhongguancun Academy, Beijing(中关村学院、北京)
AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment
Department of Computer Science, Sapienza University of Rome(计算机科学系,罗马萨皮恩扎大学);Department of Computer, Control, and Management Engineering, Sapienza University of Rome(计算机、控制与管理工程系,罗马萨皮恩扎大学);eZecute S.R.L.(eZecute公司)
DeepWisdom(深度智慧);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Renmin University of China(中国人民大学);Nanjing University(南京大学);Fudan University(复旦大学);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);Mila(米拉研究所);The Hong Kong University of Science and Technology(香港科技大学)
Automating Legal Interpretation with LLMs: Retrieval, Generation, and Evaluation
Wangxuan Institute of Computer Technology, Peking University(北京大学王轩计算机技术研究所);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究所);Peking University Law School(北京大学法学院)
SciAgents: Automating scientific discovery through multi-agent intelligent graph reasoning
Massachusetts Institute of Technology(麻省理工学院);Schwarzman College of Computing(计算学院);Center for Computational Science and Engineering(计算科学与工程中心);Laboratory for Atomistic and Molecular Mechanics (LAMM)(原子与分子力学实验室)
Automating Thematic Analysis: How LLMs Analyse Controversial Topics
School of Social Science, The University of Queensland(昆士兰大学社会科学学院);School of Computing Technologies, RMIT University(皇家墨尔本理工大学计算技术学院);Monash University(莫纳什大学);Queensland University of Technology(昆士兰科技大学);Microsoft(微软公司);School of Humanities and Communications Arts, Western Sydney University(西悉尼大学人文与传播艺术学院)
LLMs-as-Instructors: Learning from Errors Toward Automating Model Improvement
Singapore Management University(新加坡管理大学);Skywork AI(天工AI);Fudan University(复旦大学);University of Science and Technology of China(中国科学技术大学);Beijing Institute of Technology(北京理工大学)
Automating Expert-Level Medical Reasoning Evaluation of Large Language Models
University of Minnesota(明尼苏达大学);University of Georgia(佐治亚大学);University of Virginia(弗吉尼亚大学);Columbia University(哥伦比亚大学);University of California San Francisco(加利福尼亚大学旧金山分校);University of Colorado Anschutz Medical Campus(科罗拉多大学安舒茨医学中心)
Fine-tuning a Large Language Model for Automating Computational Fluid Dynamics Simulations
State Key Laboratory for Turbulence and Complex Systems, College of Engineering, Peking University, Beijing 100871, China(湍流与复杂系统国家重点实验室,工程学院,北京大学,北京100871,中国);HEDPS-CAPT, Peking University, Beijing 100871, China(HEDPS-CAPT,北京大学,北京100871,中国)
AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decomposition
University of Chicago(芝加哥大学);Peking University(北京大学);University of California, Berkeley(加州大学伯克利分校);Carnegie Mellon University(卡内基梅隆大学);UNC-Chapel Hill(北卡罗来纳大学教堂山分校)
Foam-Agent: A Large Language Model-Based Multi-Agent Framework for Automating Computational Fluid Dynamics Workflows
Department of Computer Science, Rensselaer Polytechnic Institute(里士满理工学院计算机科学系);Department of Mechanical, Aerospace, and Nuclear Engineering, Rensselaer Polytechnic Institute(里士满理工学院机械、航空航天与核工程系);Department of Computer Science and Engineering, University of California San Diego(加州大学圣地亚哥分校计算机科学与工程系);College of Education, Zhejiang Normal University(浙江师范大学教育学院);School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院)