GrowLoop: 由人类种子驱动的自进化对话评估
GrowLoop: Self-Evolving Conversation Evaluation Seeded by Human
AI总结：
针对开放域对话中类人性评估的隐性知识、标准分歧和动态演化三大挑战，提出GrowLoop自进化评估系统，通过最小人工种子标注和启发式学习迭代提取评估标准，并利用标准-案例协同进化机制持续适应模型进步和场景变化。
摘要：
随着大语言模型的快速发展，评估开放域对话中的类人性变得越来越重要。然而，类人性是一种隐性知识，人类可以直观感知，但其背后的标准难以明确表述。人类判断差异很大，在某些情况下高度一致，在其他情况下则存在合理分歧。同时，人类判断背后的标准仍然是隐性的，没有明确的基础来构建案例。此外，什么算作类人并非一成不变，而是随着模型能力和人类期望而演变。尽管在评估方法上取得了进展，如专家编写的基准、奖励模型和自进化基准，但没有一种方法能同时解决这三个挑战。因此，我们提出了GrowLoop，一个自进化的对话评估系统，能够随着模型进步和场景变化而持续适应。以最小的人工种子标注作为初始动力，LLM代理通过启发式学习迭代提取和细化评估标准。在标注者意见一致的地方要求人机一致，而在意见分歧的地方只要求合理性。此外，标准-案例协同进化机制实现了持续进化，当评估目标发生变化时，通过新的种子进行扩展。应用于开放域对话中的类人性评估，生成的标准不仅在与人判断的一致性上显著优于现有方法，而且还发现了标注者忽略的问题。由此产生的基准能够有效区分不同能力层级的模型，并揭示其不足之处，同时能够泛化到新场景并随着模型进步而适应。我们的工作将基准测试范式从手动更新或难度扩展转变为全面、持续的自我进化。
英文摘要：
With the rapid advancement of large language models, evaluating human-likeness in open-ended conversation has become increasingly important. However, human-likeness is a form of tacit knowledge that humans perceive intuitively, yet the underlying criteria resist explicit formulation. Human judgments vary widely, with strong agreement on some cases and legitimate disagreement on others. Meanwhile, the criteria behind human judgments remain implicit, leaving no clear basis for constructing cases. Further, what counts as human-likeness is not static, but evolving with model capability and human expectations. Despite progress in evaluation methods such as expert-authored benchmarks, Reward Models, and self-evolving benchmarks, none addresses all three challenges simultaneously. Therefore, we propose GrowLoop, a self-evolving conversation evaluation system that continuously adapts as models advance and scenarios shift. Starting from minimal human seed annotations, LLM agents iteratively extract and refine evaluation rubrics through Heuristic Learning. Human-AI agreement is required where annotators converge, while only plausibility is expected where they diverge. Moreover, the Rubric-Case co-evolution mechanism enables continuous evolution. When the evaluation target shifts, new human seeds expand the system's coverage accordingly. When applied to human-likeness evaluation in open-ended conversation, the AI judge guided by these rubrics not only substantially outperforms existing methods in alignment with human judgments, but also uncovers issues that annotators overlook. The resulting benchmark effectively discriminates models across capability tiers and reveals where they fall short, while generalizing to new scenarios and adapting as models advance. Our work shifts the benchmarking paradigm from manual updates or difficulty scaling to comprehensive, continuous self-evolution.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
