指令调优语言模型群体的智慧：探索模型标注差异
Wisdom of Instruction-Tuned Language Model Crowds. Exploring Model Label Variation
AI总结：
本研究探索指令调优大语言模型的标注差异，通过在多语言主观任务上测试聚合标注效果，发现模型聚合标注优于单个模型但不及简单监督模型，少样本学习效果受示例质量限制，并讨论了LLM与人类标注的多方面权衡。
摘要：
大语言模型（LLM）展现出卓越的文本分类能力，在零样本学习（ZSL）和少样本学习（FSL）场景中表现突出。但由于训练数据集不同，不同模型在各类任务上的性能差异极大。近期研究强调了数据标注中考虑人类标注差异的重要性，但这种人类标注差异如何适用于LLM仍有待探索。鉴于模型可能存在专长分化，我们提出问题：LLM的聚合标注是否优于单个模型（如同人类标注者的情况）？我们评估了4个最新的指令调优LLM作为标注者，在4种语言的5项主观任务上的表现，采用ZSL和FSL设置，并使用人类标注中的标注聚合方法。结果显示，聚合标注的表现确实显著优于任何单个模型，这得益于不同模型在不同任务或语言上的专长。令人意外的是，FSL的表现并未超过ZSL，因为其效果取决于所选示例的质量，而目前似乎没有良好的信息论策略来选择这些示例。研究还发现，没有任何LLM方法能与简单的监督模型相匹敌。我们还讨论了LLM标注与人类标注在准确性、成本以及道德伦理考量方面的权衡。
英文摘要：
Large Language Models (LLMs) exhibit remarkable text classification capabilities, excelling in zero- and few-shot learning (ZSL and FSL) scenarios. However, since they are trained on different datasets, performance varies widely across tasks between those models. Recent studies emphasize the importance of considering human label variation in data annotation. However, how this human label variation also applies to LLMs remains unexplored. Given this likely model specialization, we ask: Do aggregate LLM labels improve over individual models (as for human annotators)? We evaluate four recent instruction-tuned LLMs as annotators on five subjective tasks across four languages. We use ZSL and FSL setups and label aggregation from human annotation. Aggregations are indeed substantially better than any individual model, benefiting from specialization in diverse tasks or languages. Surprisingly, FSL does not surpass ZSL, as it depends on the quality of the selected examples. However, there seems to be no good information-theoretical strategy to select those. We find that no LLM method rivals even simple supervised models. We also discuss the tradeoffs in accuracy, cost, and moral/ethical considerations between LLM and human annotation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
