基于跨代理语义注意力和深度残差合成的混合代理增强
Attention-MoA: Enhancing Mixture-of-Agents via Inter-Agent Semantic Attention and Deep Residual Synthesis
AI总结：
Attention-MoA通过跨代理语义注意力和深度残差合成，提升混合代理系统的性能，优于现有基线。
摘要：
随着大语言模型（LLMs）的发展从参数扩展转向推理时协作，混合代理（MoA）框架已成为一种通用范式，通过分层多样化模型来利用集体智能。尽管最近的MoA变种引入了动态路由和残差连接来提高效率，但这些方法往往无法促进代理之间的深度语义交互，限制了系统主动纠正幻觉和优化逻辑的能力。在本文中，我们引入了Attention-MoA，一种基于MoA的新框架，通过跨代理语义注意力重新定义协作。辅以具有自适应提前停止机制的跨层残差模块，我们的架构在抑制深层信息退化的同时提高了计算效率。在AlpacaEval 2.0、MT-Bench和FLASK上的广泛评估显示，Attention-MoA显著优于最先进的基线，实现了在AlpacaEval 2.0上的91.15%长度控制胜率，并在FLASK的12项能力中占10项。值得注意的是，Attention-MoA使一组小型开源模型能够超越像Claude-4.5-Sonnet和GPT-4.1这样的大规模专有模型，实现了MT-Bench得分为8.83和AlpacaEval 2.0长度控制胜率为77.36%。
英文摘要：
As the development of Large Language Models (LLMs) shifts from parameter scaling to inference-time collaboration, the Mixture-of-Agents (MoA) framework has emerged as a general paradigm to harness collective intelligence by layering diverse models. While recent MoA variants have introduced dynamic routing and residual connections to improve efficiency, these methods often fail to facilitate deep semantic interaction between agents, limiting the system's ability to actively correct hallucinations and refine logic. In this paper, we introduce Attention-MoA, a novel MoA-based framework that redefines collaboration through Inter-agent Semantic Attention. Complemented by an Inter-layer Residual Module with Adaptive Early Stopping Mechanism, our architecture mitigates information degradation in deep layers while improving computational efficiency. Extensive evaluations across AlpacaEval 2.0, MT-Bench, and FLASK demonstrate that Attention-MoA significantly outperforms state-of-the-art baselines, achieving a 91.15% Length-Controlled Win Rate on AlpacaEval 2.0 and dominating in 10 out of 12 capabilities on FLASK. Notably, Attention-MoA enables an ensemble of small open-source models to outperform massive proprietary models like Claude-4.5-Sonnet and GPT-4.1, achieving an MT-Bench score of 8.83 and an AlpacaEval 2.0 LC Win Rate of 77.36%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
