arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.17600cs.CL
收藏

“以小见大”:LLM能否通过少量解释近似人类在自然语言推理(NLI)上的判断分布?

"Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?

University of Cambridge(剑桥大学);LMU Munich(慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心)

Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko, Anna Korhonen, Barbara Plank

AI总结:

本研究针对自然语言推理中人类判断分布标注成本高、难扩展的问题,提出用少量专家标签和解释辅助大语言模型近似该分布,验证了该方法的有效性,并指出评估模型判断分布需结合实例级距离与全局级形状指标。

摘要:

人类标签变异(HLV)是一种有价值的信息来源,产生于多名人类标注者出于合理原因给出不同标签的情况。在自然语言推理(NLI)领域,早期捕捉HLV的方法要么从大量众包工作者处收集标注以表征人类判断分布(HJD),要么请专业语言学家为其选定的标签提供详细解释。前一种方法能提供更密集的HJD信息,但获取成本高昂、耗费资源多;后一种方法能提供更丰富的文本信息,但难以扩展到大量人类评判者的场景。此外,大语言模型(LLM)正越来越多地被用作评估器(即“LLM评判者”),但效果参差不齐,且很少有研究聚焦于HJD的相关问题。本研究提出利用LLM,基于少量专家标签和解释来近似HJD。我们的实验表明,无论是否提供显式标签,少量解释都能显著提升LLM近似HJD的能力,从而为扩展HJD的标注规模提供了一种解决方案。然而,用LLM生成的模型判断分布(MJD)对具备软标签感知能力的小型模型进行微调时,结果存在部分不一致:尽管在距离指标上相近,但最终得到的微调模型以及可视化后的分布存在显著差异。我们的研究表明,将实例级的距离度量与全局级的形状指标及可视化相结合,对于更有效地对照人类判断分布评估MJD具有重要意义。

英文摘要:

Human label variation (HLV) is a valuable source of information that arises when multiple human annotators provide different labels for valid reasons. In Natural Language Inference (NLI) earlier approaches to capturing HLV involve either collecting annotations from many crowd workers to represent human judgment distribution (HJD) or use expert linguists to provide detailed explanations for their chosen labels. While the former method provides denser HJD information, obtaining it is resource-intensive. In contrast, the latter offers richer textual information but it is challenging to scale up to many human judges. Besides, large language models (LLMs) are increasingly used as evaluators ("LLM judges") but with mixed results, and few works aim to study HJDs. This study proposes to exploit LLMs to approximate HJDs using a small number of expert labels and explanations. Our experiments show that a few explanations significantly improve LLMs' ability to approximate HJDs with and without explicit labels, thereby providing a solution to scale up annotations for HJD. However, fine-tuning smaller soft-label aware models with the LLM-generated model judgment distributions (MJDs) presents partially inconsistent results: while similar in distance, their resulting fine-tuned models and visualized distributions differ substantially. We show the importance of complementing instance-level distance measures with a global-level shape metric and visualization to more effectively evaluate MJDs against human judgment distributions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑