视觉多智能体系统：通过视觉流缓解幻觉雪球效应
Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow
AI总结：
本文提出ViF方法，通过视觉流和注意力重新分配缓解多智能体系统中的视觉幻觉雪球效应，提升性能。
摘要：
多智能体系统（MAS）借助视觉语言模型（VLMs）能够完成具有挑战性的任务，但存在一种新的失败因素，即多智能体视觉幻觉雪球效应，其中幻觉在单个智能体中被种子，并由于过度依赖文本流来传递视觉信息而被后续智能体放大。通过轮次、层和token级的注意力分析，我们提供了关于幻觉雪球效应本质的深入见解，涉及视觉注意力分配的减少。这使我们识别出一组在中间层具有单模注意力峰值的视觉token，它们最能保留视觉证据，但随着更深层的智能体轮次逐渐减弱，导致MAS中的视觉幻觉雪球效应。因此，我们提出了ViF，一种轻量且即插即用的缓解范式，通过所选视觉中继token驱动的视觉流传递智能体间信息，并应用注意力重新分配来放大此模式。实验结果表明，我们的方法显著减少了幻觉雪球效应，在基于四种常见MAS结构和十个基础模型的八个基准测试中，始终提高了性能。源代码可在：https://github.com/YU-deep/ViF.git 公开获取。
英文摘要：
Multi-Agent System (MAS) powered by Visual Language Models (VLMs) enables challenging tasks but suffers from a novel failure term, multi-agent visual hallucination snowballing, where hallucinations are seeded in a single agent and amplified by following ones due to the over-reliance on textual flow to relay visual information. Through turn-, layer-, and token-wise attention analyses, we provide detailed insights into the essence of hallucination snowballing regarding the reduction of visual attention allocation. It leads us to identify a subset of vision tokens with a unimodal attention peak in middle layers that best preserve visual evidence but gradually diminish in deeper agent turns, resulting in the visual hallucination snowballing in MAS. Thus, we propose ViF, a lightweight, plug-and-play mitigation paradigm that relays inter-agent messages with Visual Flow powered by the selected visual relay tokens and applies attention reallocation to amplify this pattern. The experiment results demonstrate that our method markedly reduces hallucination snowballing, consistently improving the performance across eight benchmarks based on four common MAS structures and ten base models. The source code is publicly available at: https://github.com/YU-deep/ViF.git.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
