arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 2307.09765cs.SE
收藏

我们准备好迎接生成式人工智能用于软件问答了吗?

Are We Ready to Embrace Generative AI for Software Q&A?

Bowen Xu, Thanh-Dat Nguyen, Thanh Le-Cong, Thong Hoang, Jiakun Liu, Kisub Kim, Chen Gong, Changan Niu, Chenyu Wang, Bach Le, David Lo

AI总结:

本文针对 Stack Overflow 因生成式 AI 流量下降并禁用 ChatGPT 的现象,通过自动与人工对比评估人类和 ChatGPT 生成的软件问答答案,发现人类答案在多个方面持续优于 ChatGPT 答案,总分高出 10%。

摘要:

Stack Overflow 作为全球最大的软件问答(SQA)网站,正因生成式人工智能技术的出现而面临显著的流量下降。ChatGPT 在发布仅 6 天后就被 Stack Overflow 禁止使用。Stack Overflow 官方给出的主要原因是 ChatGPT 生成的答案质量较低。为了验证这一点,我们对人类编写的答案和 ChatGPT 生成的答案进行了对比评估。我们的方法同时采用了自动比较和人工研究。结果表明,人类编写的答案与 ChatGPT 生成的答案在语义上相似,但人类编写的答案在多个方面持续优于 ChatGPT 生成的答案,特别是在总分上高出 10%。我们在 https://anonymous.4open.science/r/GAI4SQA-FD5C 发布了数据、分析脚本和详细结果。

英文摘要:

Stack Overflow, the world's largest software Q&A (SQA) website, is facing a significant traffic drop due to the emergence of generative AI techniques. ChatGPT is banned by Stack Overflow after only 6 days from its release. The main reason provided by the official Stack Overflow is that the answers generated by ChatGPT are of low quality. To verify this, we conduct a comparative evaluation of human-written and ChatGPT-generated answers. Our methodology employs both automatic comparison and a manual study. Our results suggest that human-written and ChatGPT-generated answers are semantically similar, however, human-written answers outperform ChatGPT-generated ones consistently across multiple aspects, specifically by 10% on the overall score. We release the data, analysis scripts, and detailed results at https://anonymous.4open.science/r/GAI4SQA-FD5C.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑