欺诈性人工智能生成的回复对软件工程调查的影响
The Influence of Fraudulent AI-Generated Responses on Software Engineering Surveys
AI总结：
研究软件工程调查中欺诈性或人工智能辅助回复对结果有效性的影响，通过对四个数据集二次分析，用人工识别、自动检测等方法，发现人工智能辅助参与因分析类型不同影响有别，强调多种验证程序结合的重要性。
AI中文摘要：
背景：大语言模型引发了对软件工程调查中欺诈或人工智能辅助参与的新担忧。目的：本研究调查可疑或潜在人工智能辅助的回复如何影响软件工程调查结果的有效性。方法：我们对四个软件工程调查数据集进行二次分析，采用人工识别可疑回复、自动检测人工智能生成文本、描述性统计分析和主题分析，并比较原始数据集和人工清理后数据集的结果。结果：过滤可疑回复后定量结果总体稳定，但一些人口统计和分析变量有适度变化，影响特定参与者群体和背景特征的解释。定性结果受背景框架、代码突出性和支持解释的证据性质变化影响更大。结论：人工智能辅助参与可能因分析类型不同而对软件工程调查结果产生不同影响，研究结果强化了多种验证程序结合的重要性，尤其在依赖开放式回复的研究中。
英文摘要：
Background: Large Language Models (LLMs) introduce new concerns regarding fraudulent or AI assisted participation in software engineering surveys. Aims: This study investigates how suspicious or potentially AI assisted responses may affect the validity of software engineering survey findings. Method: We conducted a secondary analysis of four software engineering survey datasets using manual identification of suspicious responses, automated AI generated text detection, descriptive statistical analysis, and thematic analysis. We compared findings obtained from the original and manually cleaned datasets. Results: Quantitative findings generally remained stable after filtering suspicious responses, although some demographic and analytical variables showed moderate variation, affecting the interpretation of specific participant groups and contextual characteristics. In contrast, qualitative findings were more strongly influenced by changes in contextual framing, code prominence, and the nature of the evidence supporting interpretation, shaping how participants' experiences and study contexts were interpreted and characterized. Conclusions: AI assisted participation may influence software engineering survey findings differently depending on the type of analysis being conducted. The findings reinforce the importance of combining multiple validation procedures, particularly in studies relying on open ended responses.
