arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.07464cs.MAcs.AIcs.CLcs.LG
收藏

BudgetMLAgent:一种用于自动化机器学习任务的高性价比 LLM 多智能体系统

BudgetMLAgent: A Cost-Effective LLM Multi-Agent system for Automating Machine Learning Tasks

TCS Research(TCS研究院)

Shubham Gandhi, Manasi Patwardhan, Lovekesh Vig, Gautam Shroff

AI总结:

针对 LLM 完成复杂 ML 任务成本高、低成本单智能体表现差的问题,提出 BudgetMLAgent 多智能体系统,结合画像、经验检索、LLM 级联与专家调用,在 MLAgentBench 上以降低 94.2% 的成本将平均成功率提升至 32.95%。

摘要:

大语言模型(LLM)在包括代码片段生成在内的多种应用中表现出色,但在为复杂的机器学习(ML)任务生成代码时往往遇到困难。尽管现有的基于 LLM 单智能体的系统会根据任务复杂度呈现出不同性能,但它们完全依赖 GPT-4 等更庞大且昂贵的模型。我们的调查表明,在单智能体设置下,Gemini-Pro、Mixtral 和 CodeLlama 等零成本与低成本模型的表现远差于 GPT-4。出于开发一种具有成本效益的、基于 LLM 的解决方案以解决 ML 任务的动机,我们提出了一种基于 LLM 多智能体的系统,该系统通过画像、对过往观察的高效检索、LLM 级联以及询问专家调用来利用专家组合。通过对 MLAgentBench 基准中的 ML 工程任务进行实证分析,我们证明了系统的有效性:该系统使用零成本模型(即 Gemini)作为基础 LLM,并与级联及专家角色中的 GPT-4 配对,以在规划时服务于偶尔的询问专家调用。凭借成本降低 94.2%(从 GPT-4 单智能体系统在所有任务上平均每次运行 0.931 美元的成本降至 0.054 美元),我们的系统能够取得 32.95% 的更优平均成功率,而 GPT-4 单智能体系统在 MLAgentBench 全部任务上的平均成功率为 22.72%。

英文摘要:

Large Language Models (LLMs) excel in diverse applications including generation of code snippets, but often struggle with generating code for complex Machine Learning (ML) tasks. Although existing LLM single-agent based systems give varying performance depending on the task complexity, they purely rely on larger and expensive models such as GPT-4. Our investigation reveals that no-cost and low-cost models such as Gemini-Pro, Mixtral and CodeLlama perform far worse than GPT-4 in a single-agent setting. With the motivation of developing a cost-efficient LLM based solution for solving ML tasks, we propose an LLM Multi-Agent based system which leverages combination of experts using profiling, efficient retrieval of past observations, LLM cascades, and ask-the-expert calls. Through empirical analysis on ML engineering tasks in the MLAgentBench benchmark, we demonstrate the effectiveness of our system, using no-cost models, namely Gemini as the base LLM, paired with GPT-4 in cascade and expert to serve occasional ask-the-expert calls for planning. With 94.2\% reduction in the cost (from \$0.931 per run cost averaged over all tasks for GPT-4 single agent system to \$0.054), our system is able to yield better average success rate of 32.95\% as compared to GPT-4 single-agent system yielding 22.72\% success rate averaged over all the tasks of MLAgentBench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑