arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.06453cs.AI
收藏

ConSensus:面向多模态感知的多智能体协作

ConSensus: Multi-Agent Collaboration for Multimodal Sensing

KAIST(韩国科学技术院);Nokia Bell Labs(诺基亚贝尔实验室);University of Glasgow(格拉斯哥大学)

Hyungjun Yoon, Mohammad Malekzadeh, Sung-Ju Lee, Fahim Kawsar, Lorena Qendro

AI总结:

提出ConSensus,一种无需训练的多智能体协作框架,通过将多模态感知任务分解为专用智能体并采用混合融合机制,在五个基准上平均准确率提升7.1%,融合token成本降低12.7倍。

摘要:

大型语言模型(LLMs)越来越多地基于传感器数据来感知和推理人类生理及物理世界。然而,准确解释异构多模态传感器数据仍然是一个基本挑战。我们表明,单一的整体LLM通常无法跨模态进行连贯推理,导致解释不完整和先验知识偏差。我们引入了ConSensus,一种无需训练的多智能体协作框架,将多模态感知任务分解为专门的、模态感知的智能体。为了聚合智能体级别的解释,我们提出了一种混合融合机制,该机制平衡了语义聚合(实现跨模态推理和上下文理解)与统计共识(通过跨模态一致性提供鲁棒性)。虽然每种方法都有互补的失败模式,但它们的组合能够在传感器噪声和缺失数据下实现可靠推理。我们在五个不同的多模态感知基准上评估了ConSensus,与单智能体基线相比,平均准确率提高了7.1%。此外,ConSensus匹配或超过了迭代多智能体辩论方法的性能,同时通过单轮混合融合协议将平均融合token成本降低了12.7倍,为现实世界的多模态感知任务提供了鲁棒且高效的解决方案。源代码可在https://github.com/nokia/multi-agent-collaboration-for-multimodal-sensing获取。

英文摘要:

Large language models (LLMs) are increasingly grounded in sensor data to perceive and reason about human physiology and the physical world. However, accurately interpreting heterogeneous multimodal sensor data remains a fundamental challenge. We show that a single monolithic LLM often fails to reason coherently across modalities, leading to incomplete interpretations and prior-knowledge bias. We introduce ConSensus, a training-free multi-agent collaboration framework that decomposes multimodal sensing tasks into specialized, modality-aware agents. To aggregate agent-level interpretations, we propose a hybrid fusion mechanism that balances semantic aggregation, which enables cross-modal reasoning and contextual understanding, with statistical consensus, which provides robustness through agreement across modalities. While each approach has complementary failure modes, their combination enables reliable inference under sensor noise and missing data. We evaluate ConSensus on five diverse multimodal sensing benchmarks, demonstrating an average accuracy improvement of 7.1% over the single-agent baseline. Furthermore, ConSensus matches or exceeds the performance of iterative multi-agent debate methods while achieving a 12.7 times reduction in average fusion token cost through a single-round hybrid fusion protocol, yielding a robust and efficient solution for real-world multimodal sensing tasks. The source code is available at https://github.com/nokia/multi-agent-collaboration-for-multimodal-sensing.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑