大型语言模型在真值维护与推理任务中的自主评估
Autonomous Evaluation of LLMs for Truth Maintenance and Reasoning Tasks
AI总结：
本文提出AutoEval基准，通过自动生成不同难度任务、真值及随机数据集，实现无需人工标注的LLM客观评估，其性能可指示LLM在翻译真值维护和逻辑推理等任务上的表现，为LLM评估提供高效自主方案。
摘要：
本文提出AutoEval，这是一个新颖的基准，用于在具有明确正确性概念的正式任务（如翻译中的真值维护和逻辑推理）中扩展大型语言模型（LLM）评估。AutoEval是首个基准范式，具备无需人工标注即可扩展LLM客观评估所需的若干关键优势：（a）通过自动生成不同难度级别的任务，能够评估日益复杂的LLM；（b）自动生成真值，消除对昂贵且耗时的人工标注的依赖；（c）使用自动生成的随机数据集，降低后续LLM对许多当代基准中静态数据集的过拟合能力。实证分析表明，LLM在AutoEval上的性能高度指示其在其他多种聚焦翻译和推理任务的基准上的性能，使其成为在难以获取和/或更新人工 curated 数据集的场景中极具价值的自主评估范式。
英文摘要：
This paper presents AutoEval, a novel benchmark for scaling Large Language Model (LLM) assessment in formal tasks with clear notions of correctness, such as truth maintenance in translation and logical reasoning. AutoEval is the first benchmarking paradigm that offers several key advantages necessary for scaling objective evaluation of LLMs without human labeling: (a) ability to evaluate LLMs of increasing sophistication by auto-generating tasks at different levels of difficulty; (b) auto-generation of ground truth that eliminates dependence on expensive and time-consuming human annotation; (c) the use of automatically generated, randomized datasets that mitigate the ability of successive LLMs to overfit to static datasets used in many contemporary benchmarks. Empirical analysis shows that an LLM's performance on AutoEval is highly indicative of its performance on a diverse array of other benchmarks focusing on translation and reasoning tasks, making it a valuable autonomous evaluation paradigm in settings where hand-curated datasets can be hard to obtain and/or update.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
