重新思考模型效率：基于大模型的多智能体推断
Rethinking Model Efficiency: Multi-Agent Inference with Large Models
AI总结：
本文通过分析不同视觉-语言模型的延迟，发现大模型通过减少输出token数量可提升效率，并提出多智能体推断框架以复用小模型的推理token，从而接近大模型性能。
摘要：
大多数视觉-语言模型（VLMs）将大型语言模型（LLM）作为解码器，通过自回归生成响应token，导致输出token数量成为端到端延迟的瓶颈。然而，不同模型为达到相似性能可能需要截然不同的输出token数量。本文对VLMs各组件的延迟进行了全面分析，实验表明，具有较少输出token的大模型比具有长输出序列的小模型更高效。在多样化的现实基准测试中，观察到大模型可实现与显著较少输出token的小模型相当的性能。为利用大模型的效率，我们提出了一种多智能体推断框架，该框架在需要时将关键推理token从小模型转移至大模型，从而复用小模型的推理token以接近大模型自身推理的性能，验证了我们方法的有效性。
英文摘要：
Most vision-language models (VLMs) apply a large language model (LLM) as the decoder, where the response tokens are generated sequentially through autoregression. Therefore, the number of output tokens can be the bottleneck of the end-to-end latency. However, different models may require vastly different numbers of output tokens to achieve comparable performance. In this work, we conduct a comprehensive analysis of the latency across different components of VLMs on simulated data. The experiment shows that a large model with fewer output tokens can be more efficient than a small model with a long output sequence. The empirical study on diverse real-world benchmarks confirms the observation that a large model can achieve better or comparable performance as a small model with significantly fewer output tokens. To leverage the efficiency of large models, we propose a multi-agent inference framework that keeps large models with short responses but transfers the key reasoning tokens from the small model when necessary. The comparison on benchmark tasks demonstrates that by reusing the reasoning tokens from small models, it can help approach the performance of a large model with its own reasoning, which confirms the effectiveness of our proposal.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
