多智能体辩论用于具有自适应稳定性检测的LLM裁判
Multi-Agent Debate for LLM Judges with Adaptive Stability Detection
AI总结：
提出多智能体辩论裁判框架，通过数学形式化证明辩论优于静态集成，并引入基于时变Beta-Binomial混合和KS检验的自适应稳定性检测机制，在多个基准上提升评判准确性且保持计算效率。
摘要：
随着推理能力的进步，大型语言模型（LLMs）越来越多地被用于自动化评判任务。虽然LLM作为裁判在自动化评估方面展现出前景，但当前的方法往往依赖于简单的聚合方法（例如多数投票），即使在个别智能体提供正确答案时也可能失败。为解决这一问题，我们提出了一种多智能体辩论裁判框架，其中智能体协作推理并迭代优化其响应。我们从数学上形式化了辩论过程，分析了智能体之间的交互，并证明了与静态集成相比，辩论能放大正确性。为了提高效率，我们引入了一种稳定性检测机制，该机制通过时变Beta-Binomial混合模型来建模裁判共识动态，并基于分布相似性（Kolmogorov-Smirnov检验）进行自适应停止。该机制使用时变Beta-Binomial分布混合模型来建模裁判集体正确率动态，并采用基于分布相似性（Kolmogorov-Smirnov统计量）的自适应停止准则。在多个基准和模型上的实验表明，我们的框架在保持计算效率的同时，相较于多数投票提高了评判准确性。
英文摘要：
With advancements in reasoning capabilities, Large Language Models (LLMs) are increasingly employed for automated judgment tasks. While LLMs-as-Judges offer promise in automating evaluations, current approaches often rely on simplistic aggregation methods (e.g., majority voting), which can fail even when individual agents provide correct answers. To address this, we propose a multi-agent debate judge framework where agents collaboratively reason and iteratively refine their responses. We formalize the debate process mathematically, analyzing agent interactions and proving that debate amplifies correctness compared to static ensembles. To enhance efficiency, we introduce a stability detection mechanism that models judge consensus dynamics via a time-varying Beta-Binomial mixture, with adaptive stopping based on distributional similarity (Kolmogorov-Smirnov test). This mechanism models the judges' collective correct rate dynamics using a time-varying mixture of Beta-Binomial distributions and employs an adaptive stopping criterion based on distributional similarity (Kolmogorov-Smirnov statistic). Experiments across multiple benchmarks and models demonstrate that our framework improves judgment accuracy over majority voting while maintaining computational efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
