arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.11980cs.AIcs.CY
收藏

提示词设计对计算社会科学任务至关重要,但其影响方式难以预测

Prompt Design Matters for Computational Social Science Tasks but in Unpredictable Ways

University of Michigan School of Information(密歇根大学信息学院)

Shubham Atreja, Joshua Ashkinaze, Lingyao Li, Julia Mendelsohn, Libby Hemphill

AI总结:

本研究通过大规模多提示词实验发现,LLM在计算社会科学任务中的合规性与准确性高度依赖提示词设计,且最佳设置因任务而异,为从业者提供了实用指南与警示。

摘要:

为计算社会科学任务手动标注数据可能成本高昂、耗时且令人情绪疲惫。尽管近期研究表明LLM可以在零样本设置下执行此类标注任务,但关于提示词设计如何影响LLM的合规性与准确性,人们知之甚少。我们开展了一项大规模多提示词实验,以测试模型选择(ChatGPT、PaLM2和Falcon7b)与提示词设计特征(是否包含定义、输出类型、解释和提示词长度)如何影响LLM生成的标注在四项CSS任务(毒性、情感、谣言立场和新闻框架)上的合规性与准确性。结果表明,LLM的合规性与准确性高度依赖于提示词。例如,要求输出数值分数而非标签会降低所有LLM的合规性与准确性。总体最佳的提示词设置因任务而异,提示词的微小改动可能导致生成标签分布的巨大变化。通过证明提示词设计显著影响LLM生成标注的质量与分布,本研究既是对研究者和从业者的警告,也是实用指南。

英文摘要:

Manually annotating data for computational social science tasks can be costly, time-consuming, and emotionally draining. While recent work suggests that LLMs can perform such annotation tasks in zero-shot settings, little is known about how prompt design impacts LLMs' compliance and accuracy. We conduct a large-scale multi-prompt experiment to test how model selection (ChatGPT, PaLM2, and Falcon7b) and prompt design features (definition inclusion, output type, explanation, and prompt length) impact the compliance and accuracy of LLM-generated annotations on four CSS tasks (toxicity, sentiment, rumor stance, and news frames). Our results show that LLM compliance and accuracy are highly prompt-dependent. For instance, prompting for numerical scores instead of labels reduces all LLMs' compliance and accuracy. The overall best prompting setup is task-dependent, and minor prompt changes can cause large changes in the distribution of generated labels. By showing that prompt design significantly impacts the quality and distribution of LLM-generated annotations, this work serves as both a warning and practical guide for researchers and practitioners.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑