『解释有道理』：一项关于LLM在新闻分类中的表现及其在人机协作标注中影响的实证研究
"The explanation makes sense": An Empirical Study on LLM Performance in News Classification and its Influence on Judgment in Human-AI Collaborative Annotation
AI总结：
本文研究了LLM在新闻分类中的表现及其对人机协作标注的影响，发现详细解释能显著提高用户信心并影响决策。
摘要：
媒体偏见的扩散是重大关切，因为政治话语塑造了人们的信念和观点。为应对这一挑战，需要改进新闻解释的计算方法。尽管大语言模型（LLMs）可以扩展分类任务，但对其可信度仍存疑虑。为了推进人机协作，我们研究了使用LLMs按政治意识形态分类美国新闻的可行性，并考察其对用户决策的影响。我们首先将GPT模型与提示工程与最先进的监督机器学习在34000个公共数据集上进行了比较。然后我们收集了17000篇新闻文章，并测试了GPT-4的预测结果，包括简要和详细的解释。在一项被试间研究（N=124）中，我们评估了LLM生成的解释如何影响人类注释、判断和信心。结果显示，AI辅助显著提高了信心（p<.001），详细解释更具说服力，更可能改变决策。我们通过主题分析提出了AI解释的建议，并提供了我们的数据集供进一步研究。
英文摘要：
The spread of media bias is a significant concern as political discourse shapes beliefs and opinions. Addressing this challenge computationally requires improved methods for interpreting news. While large language models (LLMs) can scale classification tasks, concerns remain about their trustworthiness. To advance human-AI collaboration, we investigate the feasibility of using LLMs to classify U.S. news by political ideology and examine their effect on user decision-making. We first compared GPT models with prompt engineering to state-of-the-art supervised machine learning on a 34k public dataset. We then collected 17k news articles and tested GPT-4 predictions with brief and detailed explanations. In a between-subjects study (N=124), we evaluated how LLM-generated explanations influence human annotation, judgment, and confidence. Results show that AI assistance significantly increases confidence ($p<.001$), with detailed explanations more persuasive and more likely to alter decisions. We highlight recommendations for AI explanations through thematic analysis and provide our dataset for further research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
