arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.04494cs.CL
收藏

MAG-V：一种用于合成数据生成与验证的多智能体框架

MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verification

Splunk Inc（Splunk公司）

Saptarshi Sengupta, Harsh Vashistha, Kristal Curtis, Akshay Mallipeddi, Abhinav Mathur, Joseph Ross, Liang Gou

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对工业界智能体测试数据稀缺、工具调用轨迹易引发错误的问题，提出多智能体框架MAG-V，可生成模拟客户查询的合成数据并验证轨迹，性能优于GPT-4o基准、与GPT-4相当。

摘要：

大语言模型（LLM）通过具备环境交互功能的工具拓展能力，推动了智能体范式的兴起。在工业界，由于领域数据稀缺、专有客户数据存在法律限制、业务需求快速变化以及需要快速搭建新助手原型等问题，训练LLM往往并不可行。智能体依托底层LLM的零样本推理能力，借助工具对客户数据进行探索与推理并响应用户请求，为上述问题提供了巧妙的解决方案。但这一方案存在两个问题：（1）获取大规模用于智能体测试的客户查询十分耗时；（2）智能体响应用户查询时高度依赖工具调用序列（或称轨迹），可能引发意外或错误行为。\n为解决这些问题，我们提出MAG-V——一种多智能体框架，该框架首先生成模拟客户查询的问题数据集，其次从响应中反向推导备选问题，用于轨迹验证。初步结果表明，我们的合成数据能够提升智能体在真实客户查询上的表现。此外，受远程监督启发、采用传统机器学习（ML）模型的轨迹验证方法，在我们构建的数据集上，准确率比GPT-4o评判基准高出11%，且性能与GPT-4评判相当。总体而言，我们的方法是将各类任务智能体整合到统一框架以实现对齐目标的一步进展。

英文摘要：

Extending the capabilities of Large Language Models (LLMs) with functions or tools for environment interaction has led to the emergence of the agent paradigm. In industry, training an LLM is not always feasible because of the scarcity of domain data, legal holds on proprietary customer data, rapidly changing business requirements, and the need to prototype new assistants. Agents provide an elegant solution to the above by relying on the zero-shot reasoning abilities of the underlying LLM and utilizing tools to explore and reason over customer data and respond to user requests. However, there are two concerns here: (I) acquiring large scale customer queries for agent testing is time-consuming, and (II) high reliance on the tool call sequence (or trajectory) followed by the agent to respond to user queries may lead to unexpected or incorrect behavior. To address this, we propose MAG-V, a multi-agent framework to first generate a dataset of questions that mimic customer queries; and second, reverse-engineer alternate questions from the responses for trajectory verification. Initial results indicate that our synthetic data can improve agent performance on actual customer queries. Furthermore, our trajectory verification methodology, inspired by distant supervision and using traditional machine learning (ML) models, outperforms a GPT-4o judge baseline by 11% accuracy and matches the performance of a GPT-4 judge on our constructed dataset. Overall, our approach is a step towards unifying diverse task agents into a cohesive framework for achieving an aligned objective.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑