Multi-agent debates have been introduced to improve the accuracy of Large Language Models (LLMs) by having multiple agents discuss solutions to a problem over several rounds of debate. However, models often generate incorrect yet confident-sounding responses, which can mislead others. This issue arises partly because agents do not consider how confident their peers are. To address this, we propose DebUnc, a debate framework that uses uncertainty metrics to assess agent confidence. Confidence is then conveyed through a modified attention mechanism that adjusts token weights, or through textual prompts. Evaluations across benchmarks show that attention-based methods are particularly effective and that performance continues to improve as uncertainty estimation becomes more reliable. The code is available at https://github.com/lukeyoffe/debunc.
Improving Uncertainty Estimation through Semantically Diverse Language Generation
ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz(林茨ELLIS单元和LIT AI实验室,机器学习研究所,林茨约瑟夫·胡纳克大学);NXAI GmbH(NXAI公司)
ALIEN: Aligned Entropy Head for Improving Uncertainty Estimation of LLMs
Applied AI Institute, Moscow, Russia(应用人工智能研究所,莫斯科,俄罗斯);SB AI Lab, Moscow, Russia(SB AI实验室,莫斯科,俄罗斯);Intellectual data analysis and predictive modeling Institute, Moscow, Russia(智能数据分析与预测建模研究所,莫斯科,俄罗斯)
Improving Back-Translation with Uncertainty-based Confidence Estimation
ESTATES PTE LTD(六境私人有限公司);Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Beijing Advanced Innovation Center for Language Resources(北京语言资源高精尖创新中心)
Improving the Robustness of Distantly-Supervised Named Entity Recognition via Uncertainty-Aware Teacher Learning and Student-Student Collaborative Learning
National Key Laboratory for Multimedia Information Processing, Peking University(北京大学多媒体信息处理国家重点实验室);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);Tencent Inc.(腾讯公司);Jiangsu Collaborative Innovation Center for Language Ability, Jiangsu Normal University(江苏省语言能力协同创新中心,江苏师范大学)
Improving Sampling for Masked Diffusion Models via Information Gain
Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Massachusetts Institute of Technology(麻省理工学院);Shanghai Jiao Tong University(上海交通大学);Beihang University(北航);College of AI, Tsinghua University(清华大学人工智能学院)