探究政治偏见对大型语言模型在立场分类任务中性能的影响
Examining the Influence of Political Bias on Large Language Model Performance in Stance Classification
AI总结：
本研究探究了大型语言模型在立场分类任务中的政治偏见，利用三个数据集、七个模型和四种提示方案，发现其性能在政治导向任务中存在显著差异，且目标歧义会降低分类准确率。
摘要：
大型语言模型（LLM）在执行基于自然语言查询的任务方面展现出卓越能力。然而，这些基于精选数据集训练的模型，固有地体现了从种族到国家和性别等多方面的偏见。这些偏见是否会影响LLM在某些任务上的性能仍不确定。在本研究中，我们调查了立场分类任务中LLM的政治偏见，具体考察这些模型是否倾向于更准确地分类具有政治色彩的立场。利用三个数据集、七个LLM和四种不同的提示方案，我们分析了LLM在政治导向陈述和目标上的性能。我们的发现表明，在各种政治导向的立场分类任务中，LLM的性能存在统计学上的显著差异。此外，我们观察到这种差异主要在数据集层面显现，模型和提示方案在不同的立场分类数据集上表现出统计学上相似的性能。最后，我们观察到，当陈述所指向的目标存在较大歧义时，LLM的立场分类准确率较差。代码和数据集：http://doi.org/10.5281/zenodo.12938478
英文摘要：
Large Language Models (LLMs) have demonstrated remarkable capabilities in executing tasks based on natural language queries. However, these models, trained on curated datasets, inherently embody biases ranging from racial to national and gender biases. It remains uncertain whether these biases impact the performance of LLMs for certain tasks. In this study, we investigate the political biases of LLMs within the stance classification task, specifically examining whether these models exhibit a tendency to more accurately classify politically-charged stances. Utilizing three datasets, seven LLMs, and four distinct prompting schemes, we analyze the performance of LLMs on politically oriented statements and targets. Our findings reveal a statistically significant difference in the performance of LLMs across various politically oriented stance classification tasks. Furthermore, we observe that this difference primarily manifests at the dataset level, with models and prompting schemes showing statistically similar performances across different stance classification datasets. Lastly, we observe that when there is greater ambiguity in the target the statement is directed towards, LLMs have poorer stance classification accuracy. Code & Dataset: http://doi.org/10.5281/zenodo.12938478
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
