arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.04818cs.CL
收藏

ACORN:面向方面的常识推理解释评估

ACORN: Aspect-wise Commonsense Reasoning Explanation Evaluation

RIKEN(理化学研究所);Tohoku University(东北大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学)

Ana Brassard, Benjamin Heinzerling, Keito Kudo, Keisuke Sakaguchi, Kentaro Inui

AI总结:

本文提出ACORN数据集,包含3,500条自由文本解释及面向方面的质量评分,用于评估LLMs在解释评分任务中的表现,发现LLMs不能完全替代人类评分者,但可作为人类评分者稀缺时的补充。

摘要:

评估自由文本解释的质量是一项多层面、主观且劳动密集型的任务。大型语言模型(LLMs)因其在一致性、可扩展性和成本效益方面的潜力,提供了一种有吸引力的替代方案。在这项工作中,我们提出了ACORN,一个包含3,500条自由文本解释及面向方面质量评分的新数据集,并利用该数据集评估LLMs如何对解释进行评分。我们观察到,较大的模型输出的标签能够维持或提高标注者间一致性,这表明它们处于人类评分者之间预期方差的范围之内。然而,它们与多数投票人类评分之间的相关性在不同质量方面上存在差异,这表明它们并不能完全替代人类评分者。反过来,在某些情况下,将LLMs作为较小规模人类评分者群体的补充,能够改善与原始多数标签的相关性。但是,这种效果仅限于人类评分者稀缺的情形,并且在所有情况下,增加一名人类评分者会产生更为显著的效果。总体而言,我们建议不要将LLMs作为人类评分者的完全替代品,但鼓励在最终包含有针对性的人类参与的配置中使用它们。数据可在此获取:https://github.com/a-brassard/ACORN

英文摘要:

Evaluating the quality of free-text explanations is a multifaceted, subjective, and labor-intensive task. Large language models (LLMs) present an appealing alternative due to their potential for consistency, scalability, and cost-efficiency. In this work, we present ACORN, a new dataset of 3,500 free-text explanations and aspect-wise quality ratings, and use it to evaluate how LLMs rate explanations. We observed that larger models outputted labels that maintained or increased the inter-annotator agreement, suggesting that they are within the expected variance between human raters. However, their correlation with majority-voted human ratings varied across different quality aspects, indicating that they are not a complete replacement. In turn, using LLMs as a supplement to a smaller group of human raters in some cases improved the correlation with the original majority labels. However, the effect was limited to cases where human raters were scarce, and an additional human rater had a more pronounced effect in all cases. Overall, we recommend against using LLMs as a complete replacement for human raters but encourage using them in configurations that end with targeted human involvement. Data available here: https://github.com/a-brassard/ACORN

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑