对话式非验证学习：通过元评估自我进化的大型语言模型
Conversation for Non-verifiable Learning: Self-Evolving LLMs through Meta-Evaluation
AI总结：
本文提出CoNL框架，通过多智能体自我对弈实现生成、评估与元评估的统一，利用批评质量提升解决方案来改进模型，无需外部评委或真实标签。
摘要：
训练大型语言模型（LLMs）进行不可验证任务（如创意写作、对话和伦理推理）仍然具有挑战性，因为缺乏真实标签。虽然LLM-as-Judge方法提供了替代人类反馈的可扩展方案，但其面临根本限制：性能受限于评估者的自身质量。如果评估者无法识别优质解决方案，就无法提供有用的训练信号，评估偏见（如偏好冗长而非质量）仍未解决。这促使元评估：评估和改进评估者自身的能力。我们引入CoNL，一种通过多智能体自我对弈统一生成、评估和元评估的框架。我们的关键见解：批评质量可通过是否帮助他人改进解决方案来衡量。在CoNL中，共享相同策略的多个智能体进行结构化对话，提出、批评和修订解决方案。能够促进解决方案改进的批评获得诊断奖励，创建明确的监督用于元评估，并通过自我对弈实现生成和判断能力的联合优化，无需外部评委或真实标签。在各种基准测试中，CoNL在保持稳定训练的同时，相对于自我奖励基线实现了持续改进。
英文摘要：
Training large language models (LLMs) for non-verifiable tasks, such as creative writing, dialogue, and ethical reasoning, remains challenging due to the absence of ground-truth labels. While LLM-as-Judge approaches offer a scalable alternative to human feedback, they face a fundamental limitation: performance is constrained by the evaluator's own quality. If the judge cannot recognize good solutions, it cannot provide useful training signals, and evaluation biases (e.g., favoring verbosity over quality) remain unaddressed. This motivates meta-evaluation: the ability to evaluate and improve the evaluator itself. We introduce CoNL, a framework that unifies generation, evaluation, and meta-evaluation through multi-agent self-play. Our key insight: critique quality can be measured by whether it helps others improve their solutions. In CoNL, multiple agents sharing the same policy engage in structured conversations to propose, critique, and revise solutions. Critiques that enable solution improvements earn a diagnostic reward, creating explicit supervision for meta-evaluation and enabling joint optimization of generation and judging capabilities through self-play, without external judges or ground truth. Experiments on various benchmarks show that CoNL achieves consistent improvements over self-rewarding baselines while maintaining stable training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
