探索大语言模型在德顿语相关性判断中的应用
Exploring Large Language Models for Relevance Judgments in Tetun
AI总结：
本文研究利用大语言模型在低资源语言德顿语中自动执行查询-文档相关性判断，并与人工标注比较，发现其一致性水平与高资源语言研究结果接近。
摘要：
Cranfield范式一直是构建测试集的基础方法，相关性判断通常由人工评估员完成。然而，大语言模型（LLMs）的出现为自动化这些任务带来了新的可能性。本文探讨了使用LLMs自动化相关性评估的可行性，特别是在低资源语言的背景下。在我们的研究中，通过提供一系列德顿语的查询-文档对作为输入文本，采用LLMs来自动化相关性判断任务。模型的任务是为每一对分配相关性分数，然后将这些分数与人工标注者的分数进行比较，以评估标注者间的一致性水平。我们的调查揭示的结果与高资源语言研究中报告的结果高度一致。
英文摘要：
The Cranfield paradigm has served as a foundational approach for developing test collections, with relevance judgments typically conducted by human assessors. However, the emergence of large language models (LLMs) has introduced new possibilities for automating these tasks. This paper explores the feasibility of using LLMs to automate relevance assessments, particularly within the context of low-resource languages. In our study, LLMs are employed to automate relevance judgment tasks, by providing a series of query-document pairs in Tetun as the input text. The models are tasked with assigning relevance scores to each pair, where these scores are then compared to those from human annotators to evaluate the inter-annotator agreement levels. Our investigation reveals results that align closely with those reported in studies of high-resource languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
