基于分布假设的无裁判大语言模型开放式生成基准
A Judge-free LLM Open-ended Generation Benchmark Based on the Distributional Hypothesis
AI总结：
针对LLMs开放式文本生成评估缺乏明确真值、人工或LLM裁判成本高的问题，提出基于n元语法统计和规则的无裁判基准，含50组问题参考集及三个新指标，与GPT-4o评估高度相关且资源需求低，为可扩展评估提供有效替代方案。
摘要：
评估大语言模型（LLMs）的开放式文本生成具有挑战性，因为缺乏明确的真值且人工或基于LLM的评估成本高昂。我们提出一种新颖的基准，无需依赖人工判断或LLM作为裁判的方法，而是使用n元语法统计和规则来评估LLMs。通过50组问题与参考答案集，我们引入基于n元语法和规则的三个新指标：流畅性、真实性和有用性。我们的基准与基于GPT-4o的评估高度相关，同时所需计算资源显著减少，证明其作为评估LLMs开放式生成能力的可扩展替代方案的有效性。
英文摘要：
Evaluating the open-ended text generation of large language models (LLMs) is challenging because of the lack of a clear ground truth and the high cost of human or LLM-based assessments. We propose a novel benchmark that evaluates LLMs using n-gram statistics and rules, without relying on human judgement or LLM-as-a-judge approaches. Using 50 question and reference answer sets, we introduce three new metrics based on n-grams and rules: Fluency, Truthfulness, and Helpfulness. Our benchmark strongly correlates with GPT-4o-based evaluations while requiring significantly fewer computational resources, demonstrating its effectiveness as a scalable alternative for assessing LLMs' open-ended generation capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
