信任ChatGPT：提示词的微小调整如何导致情感分类的重大差异
Trusting CHATGPT: how minor tweaks in the prompts lead to major differences in sentiment classification
AI总结：
本研究测试GPT-4o mini对提示词微小调整的敏感性，发现其情感分类结果受提示词变化影响显著，挑战了模型稳健性，强调信任需考虑技术性能与社会制度关系。
摘要：
当今社会科学的一个基本问题是：我们能在多大程度上信任ChatGPT这类高度复杂的预测模型？本研究测试了以下假设：提示词结构的细微变化不会使大型语言模型GPT-4o mini生成的情感极性分析分类结果产生显著差异。研究使用包含10万条关于四位拉美总统的西班牙语评论数据集，让模型在10次实验中对评论进行积极、消极或中性分类，每次实验都对提示词进行轻微调整。实验方法包括探索性和验证性分析，以识别分类间的显著差异。结果显示，即使是提示词的微小修改（如词汇、句法或模态变化，甚至缺乏结构）也会影响分类。在某些情况下，模型会产生不一致的响应，例如混合类别、提供未经请求的解释或使用西班牙语以外的语言。使用卡方检验的统计分析证实，大多数提示词之间的比较存在显著差异，只有在语言结构高度相似的一种情况下例外。这些发现挑战了大型语言模型在分类任务中的稳健性和可信度，凸显了它们对指令变化的脆弱性。此外，明显的是，提示词缺乏结构化语法会增加幻觉的频率。讨论强调，对大型语言模型的信任不仅基于技术性能，还基于支撑其使用的社会和制度关系。
英文摘要：
One fundamental question for the social sciences today is: how much can we trust highly complex predictive models like ChatGPT? This study tests the hypothesis that subtle changes in the structure of prompts do not produce significant variations in the classification results of sentiment polarity analysis generated by the Large Language Model GPT-4o mini. Using a dataset of 100.000 comments in Spanish on four Latin American presidents, the model classified the comments as positive, negative, or neutral on 10 occasions, varying the prompts slightly each time. The experimental methodology included exploratory and confirmatory analyses to identify significant discrepancies among classifications. The results reveal that even minor modifications to prompts such as lexical, syntactic, or modal changes, or even their lack of structure impact the classifications. In certain cases, the model produced inconsistent responses, such as mixing categories, providing unsolicited explanations, or using languages other than Spanish. Statistical analysis using Chi-square tests confirmed significant differences in most comparisons between prompts, except in one case where linguistic structures were highly similar. These findings challenge the robustness and trust of Large Language Models for classification tasks, highlighting their vulnerability to variations in instructions. Moreover, it was evident that the lack of structured grammar in prompts increases the frequency of hallucinations. The discussion underscores that trust in Large Language Models is based not only on technical performance but also on the social and institutional relationships underpinning their use.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
