arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.07367cs.SD
收藏

FOCAL:多模态代理的新型基准测试技术

FOCAL: A Novel Benchmarking Technique for Multi-modal Agents

Anupam Purwar, Aditya Choudhary

AI总结:

FOCAL提出了一种新型基准测试技术,用于评估多模态代理的端到端推理能力、误差传播及对话效果。

摘要:

随着推理能力的最新进展,使用MCP服务器和音频语言模型(ALMs)进行工具调用,多模态代理(支持语音和文本)的开发和整合已成为行业前沿。尽管如此,语音代理的级联管道仍因LLMs带来的强大推理能力而处于行业核心地位。然而,级联管道往往会导致管道中的错误传播。我们提出了一种名为FOCAL的框架,用于基准测试端到端推理、按组件误差传播和误差分析,以自动化和人工辅助测试多模态代理(语音到语音+文本输入)。我们还分享了两个新的指标,即推理得分和语义得分,用于评估代理在语音模式下进行有意义对话的有效性。

英文摘要:

With the recent advancements in reasoning capabilities, tool calling using MCP servers and Audio Language Models (ALMs), development and integration of multi-modal agents (with voice and text support) has come to the industry forefront. Cascading pipelines for voice agents still play a central role in the industry owing to their superior reasoning capabilities facilitated by LLMs. Although, cascading pipelines often present error propagation through the pipeline. We propose a framework, FOCAL to benchmark end-to-end reasoning, component-wise error propagation and error analysis for automated as well as human-assisted testing of multi-modal agents (voice to voice + text input). We also share two novel metrics viz. Reasoning and Semantic scores to evaluate efficacy of the agent in having meaningful conversations in voice mode.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑