面向立场分类的开源大语言模型提示与微调
Prompting and Fine-Tuning Open-Sourced Large Language Models for Stance Classification
AI总结：
本研究探索用开源大语言模型进行立场分类以减少人工标注，测试10个开源模型与7种提示方案并进行微调，发现其性能与领域内监督模型相当但不稳定，微调未必提升效果，呼吁将立场检测设为LLM优化基准。
摘要：
立场分类是预测作者对感兴趣主题所持观点的任务，长期以来一直是社会科学到机器学习等多个领域的研究焦点。当前的立场检测方法主要依赖人工标注句子，随后训练有监督机器学习模型。但这种人工标注过程需要耗费大量标注人力，因而阻碍了其在不同语境下的泛化潜力。本研究探索将大语言模型（Large Language Models, LLMs）用作立场检测方法，以减少甚至消除对人工标注的需求。我们研究了10个开源模型和7种提示方案，发现LLMs与领域内有监督模型性能相当，但性能表现不一定稳定。我们还对LLMs进行了微调，却发现微调过程不一定能带来性能提升。总体而言，我们发现LLMs并非总能优于规模更小的有监督机器学习模型，因此呼吁将立场检测设为LLM也需针对其优化的基准。本研究使用的代码可在https://github.com/ijcruic/LLM-Stance-Labeling 获取。
英文摘要：
Stance classification, the task of predicting the viewpoint of an author on a subject of interest, has long been a focal point of research in domains ranging from social science to machine learning. Current stance detection methods rely predominantly on manual annotation of sentences, followed by training a supervised machine learning model. However, this manual annotation process requires laborious annotation effort, and thus hampers its potential to generalize across different contexts. In this work, we investigate the use of Large Language Models (LLMs) as a stance detection methodology that can reduce or even eliminate the need for manual annotations. We investigate 10 open-source models and 7 prompting schemes, finding that LLMs are competitive with in-domain supervised models but are not necessarily consistent in their performance. We also fine-tuned the LLMs, but discovered that fine-tuning process does not necessarily lead to better performance. In general, we discover that LLMs do not routinely outperform their smaller supervised machine learning models, and thus call for stance detection to be a benchmark for which LLMs also optimize for. The code used in this study is available at \url{https://github.com/ijcruic/LLM-Stance-Labeling}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
