EQUATOR：一个用于评估大语言模型在开放式问题中推理能力的确定性框架
EQUATOR: A Deterministic Framework for Evaluating LLM Reasoning with Open-Ended Questions. # v1.0.0-beta
AI总结：
本文提出 EQUATOR 框架，通过向量数据库将开放式问题与人工评估答案配对，并结合本地小模型 LLaMA 3.2B 实现确定性自动评分，以克服多项选择评估的流畅性偏差，提升对 LLM 事实准确性和复杂推理能力的可扩展评估。
摘要：
尽管大语言模型（LLMs）表现出显著的连贯性，但现有评估方法往往受到流畅性偏差的影响，并严重依赖多项选择题形式，这使得有效评估事实准确性和复杂推理变得困难。因此，LLMs 经常生成事实不准确的回答，尤其是在复杂推理任务中，这凸显了两个突出挑战：（1）现有方法在有效评估推理和事实准确性方面存在不足，以及（2）对人工评估者进行细致判断的依赖，正如 Williams 和 Huckle（2024）[1] 所指出的，尽管自动化评分有所进步，人工评分仍然不可或缺。为了解决开放式推理任务中的评估空白，我们引入了 EQUATOR Evaluator（开放式推理中问答彻底性评估）。该框架将确定性评分与对事实准确性和稳健推理评估的关注相结合。EQUATOR 使用向量数据库将开放式问题与人工评估的答案配对，从而实现更精确和可扩展的评估。在实践中，与 Williams 和 Huckle（2004）[1] 的方法相比，EQUATOR 显著减少了对人工评估者评分的依赖，并提高了可扩展性。我们的结果表明，该框架显著优于传统的多项选择评估，同时保持了高准确性标准。此外，我们引入了一个利用较小的本地托管 LLMs 的自动化评估流程。我们使用了在 Ollama 二进制文件上运行的 LLaMA 3.2B 来简化我们的评估。这项工作建立了评估 LLM 性能的新范式，强调事实准确性和推理能力，并为未来的研究提供了稳健的方法论基础。
英文摘要：
Despite the remarkable coherence of Large Language Models (LLMs), existing evaluation methods often suffer from fluency bias and rely heavily on multiple-choice formats, making it difficult to assess factual accuracy and complex reasoning effectively. LLMs thus frequently generate factually inaccurate responses, especially in complex reasoning tasks, highlighting two prominent challenges: (1) the inadequacy of existing methods to evaluate reasoning and factual accuracy effectively, and (2) the reliance on human evaluators for nuanced judgment, as illustrated by Williams and Huckle (2024)[1], who found manual grading indispensable despite automated grading advancements. To address evaluation gaps in open-ended reasoning tasks, we introduce the EQUATOR Evaluator (Evaluation of Question Answering Thoroughness in Open-ended Reasoning). This framework combines deterministic scoring with a focus on factual accuracy and robust reasoning assessment. Using a vector database, EQUATOR pairs open-ended questions with human-evaluated answers, enabling more precise and scalable evaluations. In practice, EQUATOR significantly reduces reliance on human evaluators for scoring and improves scalability compared to Williams and Huckle's (2004)[1] methods. Our results demonstrate that this framework significantly outperforms traditional multiple-choice evaluations while maintaining high accuracy standards. Additionally, we introduce an automated evaluation process leveraging smaller, locally hosted LLMs. We used LLaMA 3.2B, running on the Ollama binaries to streamline our assessments. This work establishes a new paradigm for evaluating LLM performance, emphasizing factual accuracy and reasoning ability, and provides a robust methodological foundation for future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
