探索大型语言模型在引文上下文分析中的适用性
Exploring the applicability of Large Language Models to citation context analysis
AI总结：
本研究比较ChatGPT等LLM与人工标注在引文上下文分析中的表现，发现其一致性较好但预测性能不足，宜作为参考或补充标注者。
摘要：
传统引文分析假设论文中的所有引文都是等价的，而引文上下文分析与此不同，它会考虑每条引文的上下文信息。然而，引文上下文分析需要通过标注创建大量数据，这阻碍了该方法的广泛应用。本研究通过比较大型语言模型（LLM）与人工标注结果，探索了LLM——尤其是ChatGPT——在引文上下文分析中的适用性。结果表明，LLM标注在一致性方面与人工标注相当甚至更好，但在预测性能方面较差。因此，让LLM在引文上下文分析中立即取代人工标注者是不合适的。不过，当需要将多名人工标注者获得的标注结果收敛为一个结果时，LLM获得的标注结果可以作为参考信息；或者，当难以准备足够的人工标注者时，LLM可以作为标注者之一使用。本研究提供了对未来引文上下文分析发展具有重要意义的基础发现。
英文摘要：
Unlike traditional citation analysis -- which assumes that all citations in a paper are equivalent -- citation context analysis considers the contextual information of individual citations. However, citation context analysis requires creating large amounts of data through annotation, which hinders the widespread use of this methodology. This study explored the applicability of Large Language Models (LLMs) -- particularly ChatGPT -- to citation context analysis by comparing LLMs and human annotation results. The results show that the LLMs annotation is as good as or better than the human annotation in terms of consistency but poor in terms of predictive performance. Thus, having LLMs immediately replace human annotators in citation context analysis is inappropriate. However, the annotation results obtained by LLMs can be used as reference information when narrowing the annotation results obtained by multiple human annotators to one, or LLMs can be used as one of the annotators when it is difficult to prepare sufficient human annotators. This study provides basic findings important for the future development of citation context analyses.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
