arXiv 2407.064222024-07-08cs.AIcs.CL

探索ChatGPT复现社会计算任务人工标注的能力（扩展版）

Exploring the Capability of ChatGPT to Reproduce Human Labels for Social Computing Tasks (Extended Version)

The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Queen Mary University of London（伦敦玛丽女王大学）；University of Helsinki（赫尔辛基大学）

Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, Gareth Tyson

备注：: Extended version of accepted short paper to ASONAM 2024. arXiv admin note: text overlap with arXiv:2304.10145

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

该研究探索ChatGPT在社会计算任务中的数据标注能力，通过7个社会议题数据集验证其平均F1值达72%，并提出GPT-Rater工具预测其标注适配性，可降低社会计算研究的标注成本与门槛。

摘要：

挖掘ChatGPT等大语言模型（LLM）的潜力，有助于通过包容、合乎伦理且可持续的方式应对社会挑战。本文研究了ChatGPT为社会计算任务标注数据的能力上限，旨在降低开展网络研究的复杂度与成本。为评估ChatGPT的潜力，我们使用ChatGPT重新标注了7个数据集，涵盖COVID-19虚假信息、社交机器人欺骗、网络欺凌、标题党新闻、俄乌战争等紧迫社会议题相关主题。研究结果表明，ChatGPT在处理这些数据标注任务上展现出应用前景，但也存在一些挑战。在7个数据集上，ChatGPT的平均标注F1值达到72.00%，其中在标题党新闻标注任务中表现最优，数据标注正确率达89.66%。但我们也观察到，不同标注标签下的模型性能存在显著差异。本研究揭示了ChatGPT标注性能的可预测规律，据此提出了GPT-Rater工具，用于预测ChatGPT能否为给定标注任务正确标注数据，研究人员可借助该工具判断ChatGPT是否适配自身的标注需求。实验验证了GPT-Rater可有效预测ChatGPT的性能，其在标题党新闻标题数据集上表现最佳，平均F1值达到95.00%。我们认为本研究开辟了新的分析路径，能够降低开展社会计算研究的门槛。

英文摘要：

Harnessing the potential of large language models (LLMs) like ChatGPT can help address social challenges through inclusive, ethical, and sustainable means. In this paper, we investigate the extent to which ChatGPT can annotate data for social computing tasks, aiming to reduce the complexity and cost of undertaking web research. To evaluate ChatGPT's potential, we re-annotate seven datasets using ChatGPT, covering topics related to pressing social issues like COVID-19 misinformation, social bot deception, cyberbully, clickbait news, and the Russo-Ukrainian War. Our findings demonstrate that ChatGPT exhibits promise in handling these data annotation tasks, albeit with some challenges. Across the seven datasets, ChatGPT achieves an average annotation F1-score of 72.00%. Its performance excels in clickbait news annotation, correctly labeling 89.66% of the data. However, we also observe significant variations in performance across individual labels. Our study reveals predictable patterns in ChatGPT's annotation performance. Thus, we propose GPT-Rater, a tool to predict if ChatGPT can correctly label data for a given annotation task. Researchers can use this to identify where ChatGPT might be suitable for their annotation requirements. We show that GPT-Rater effectively predicts ChatGPT's performance. It performs best on a clickbait headlines dataset by achieving an average F1-score of 95.00%. We believe that this research opens new avenues for analysis and can reduce barriers to engaging in social computing research.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。