对AI生成响应如何影响软件工程调查的探究
An Investigation on How AI-Generated Responses Affect SoftwareEngineering Surveys
AI总结：
本研究探讨AI生成响应对软件工程调查的影响，通过分析发现49份回答存在合成作者身份的模式，强调需结合自动化与解释性验证以提升调查可信度。
摘要：
调查研究是软件工程中一种基本的实证方法，能够系统地收集关于专业实践、看法和经历的数据。然而，近年来大型语言模型（LLMs）的进步给调查的完整性带来了新的风险，因为参与者可以使用生成工具来伪造或篡改他们的回答。本研究探讨了LLMs如何在软件工程调查中被滥用，并调查此类行为对数据真实性、有效性和研究完整性的方法论影响。我们通过Prolific平台在2025年进行了两次调查部署，并分析了参与者回答的内容以识别不规则或伪造的回答。一组被怀疑是AI生成的回答通过定性模式检查、叙述特征分析和自动化检测工具Scribbr AI Detector进行了检查。分析发现49份调查回答中存在重复的结构模式，表明合成作者身份，包括重复的顺序、统一的措辞和表面的个性化。这些虚假叙述模仿了连贯的推理，同时隐藏了伪造的内容，从而削弱了构念效度、内部效度和外部效度。我们的研究将数据真实性确定为软件工程调查中新兴的有效性维度。我们强调，可靠的证据现在需要结合自动化和解释性验证程序、透明的报告和社区标准来检测和防止AI生成的回答，从而保护软件工程中调查的可信度。
英文摘要：
Survey research is a fundamental empirical method in software engineering, enabling the systematic collection of data on professional practices, perceptions, and experiences. However, recent advances in large language models (LLMs) have introduced new risks to survey integrity, as participants can use generative tools to fabricate or manipulate their responses. This study explores how LLMs are being misused in software engineering surveys and investigates the methodological implications of such behavior for data authenticity, validity, and research integrity. We collected data from two survey deployments conducted in 2025 through the Prolific platform and analyzed the content of participants' answers to identify irregular or falsified responses. A subset of responses suspected of being AI generated was examined through qualitative pattern inspection, narrative characterization, and automated detection using the Scribbr AI Detector. The analysis revealed recurring structural patterns in 49 survey responses indicating synthetic authorship, including repetitive sequencing, uniform phrasing, and superficial personalization. These false narratives mimicked coherent reasoning while concealing fabricated content, undermining construct, internal, and external validity. Our study identifies data authenticity as an emerging dimension of validity in software engineering surveys. We emphasize that reliable evidence now requires combining automated and interpretive verification procedures, transparent reporting, and community standards to detect and prevent AI generated responses, thereby protecting the credibility of surveys in software engineering.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
