清单对于生成式任务的自动评估真的有用吗？
Are Checklists Really Useful for Automatic Evaluation of Generative Tasks?
AI总结：
本文研究自动生成清单在生成式任务评估中的作用，发现选择性使用清单在成对比较中更有效，并指出人类评估标准存在不一致性，需更清晰地定义客观评估标准。
摘要：
使用大语言模型对生成式任务进行自动评估面临标准模糊的挑战。尽管自动生成清单是一种具有潜在前景的方法，但其有效性仍未得到充分探索。我们研究了清单应当对所有问题使用还是选择性使用，采用六种方法生成清单，在八种模型规模上评估其有效性，并识别与人类评估相关的清单条目。通过在成对比较和直接评分任务上的实验，我们发现选择性使用清单往往能在成对设置中提升评估性能，而在直接评分中其收益则不太一致。我们的分析还表明，即使与人类评分相关性较低的清单条目也常常反映了人类撰写的标准，这表明人类评估可能存在不一致性。这些发现凸显了需要更清晰地定义客观评估标准，以同时指导人类评估和自动评估。
英文摘要：
Automatic evaluation of generative tasks using large language models faces challenges due to ambiguous criteria. Although automatic checklist generation is a potentially promising approach, its usefulness remains underexplored. We investigate whether checklists should be used for all questions or selectively, generate them using six methods, evaluate their effectiveness across eight model sizes, and identify checklist items that correlate with human evaluations. Through experiments on pairwise comparison and direct scoring tasks, we find that selective checklist use tends to improve evaluation performance in pairwise settings, while its benefits are less consistent in direct scoring. Our analysis also shows that even checklist items with low correlation to human scores often reflect human-written criteria, indicating potential inconsistencies in human evaluation. These findings highlight the need to more clearly define objective evaluation criteria to guide both human and automatic evaluations. \footnote{Our code is available at~https://github.com/momo0817/checklist-effectiveness-study
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
