arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.06426cs.CLcs.AIcs.MA
收藏

DebUnc:利用不确定性指标改进大语言模型智能体通信

DebUnc: Improving Large Language Model Agent Communication With Uncertainty Metrics

University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校)

Luke Yoffe, Alfonso Amayuelas, William Yang Wang

AI总结:

针对大语言模型多智能体辩论中错误自信回答易误导同伴的问题,提出DebUnc框架,通过不确定性指标评估置信度并经改进注意力机制或文本提示传递,显著提升性能。

摘要:

多智能体辩论机制已被用于提升大语言模型(LLMs)的准确率,具体方式是让多个智能体经过多轮辩论讨论问题的解决方案。然而,模型往往会生成听起来很有把握的错误回答,进而误导其他智能体。该问题的部分成因是智能体不会考量同伴的置信程度。为解决这一问题,我们提出DebUnc——一种利用不确定性指标评估智能体置信度的辩论框架。置信度随后通过调整token权重的改进注意力机制,或是通过文本提示进行传递。在多个基准上的评估结果显示,基于注意力的方法效果尤为突出,且随着不确定性估计变得更可靠,模型性能会持续提升。代码可在https://github.com/lukeyoffe/debunc获取。

英文摘要:

Multi-agent debates have been introduced to improve the accuracy of Large Language Models (LLMs) by having multiple agents discuss solutions to a problem over several rounds of debate. However, models often generate incorrect yet confident-sounding responses, which can mislead others. This issue arises partly because agents do not consider how confident their peers are. To address this, we propose DebUnc, a debate framework that uses uncertainty metrics to assess agent confidence. Confidence is then conveyed through a modified attention mechanism that adjusts token weights, or through textual prompts. Evaluations across benchmarks show that attention-based methods are particularly effective and that performance continues to improve as uncertainty estimation becomes more reliable. The code is available at https://github.com/lukeyoffe/debunc.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑