arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.12134cs.SE
收藏

大型语言模型是否提供与其生成代码相似的链接?一项针对Gemini和Bing CoPilot的研究

Do LLMs Provide Links to Code Similar to what they Generate? A Study with Gemini and Bing CoPilot

Daniele Bifolco, Pietro Cassieri, Giuseppe Scanniello, Massimiliano Di Penta, Fiorella Zampetti

AI总结:

本研究实证评估了Bing CoPilot和Gemini生成代码与提供链接的相关性,发现两者均存在来源债务,其中Bing CoPilot的66%链接相关,Gemini仅28%。

摘要:

大型语言模型(LLMs)目前被用于各种软件开发任务,包括生成代码片段以解决特定问题。与从网络重用代码不同,LLMs在提供生成代码的来源信息方面存在局限,这可能带来重要的可信度和法律后果。尽管基于LLM的助手可能提供与生成代码“相关”的外部链接,但我们不知道这些链接的相关性如何。本文介绍了一项实证研究的结果,评估了Bing CoPilot和Google Gemini生成的、涵盖六种编程语言的243个和194个代码片段,在多大程度上可能源自这两个基于LLM的助手提供的链接。该研究利用自动化代码相似性评估,并辅以细致的人工分析。研究结果表明,基于LLM的助手提供了相关和不相关链接的混合,这些链接具有不同的性质。具体而言,尽管Bing CoPilot的66%链接和Google Gemini的28%链接是相关的,但基于LLM的助手仍然存在严重的“来源债务”问题。

英文摘要:

Large Language Models (LLMs) are currently used for various software development tasks, including generating code snippets to solve specific problems. Unlike reuse from the Web, LLMs are limited in providing provenance information about the generated code, which may have important trustworthiness and legal consequences. While LLM-based assistants may provide external links that are "related" to the generated code, we do not know how relevant such links are. This paper presents the findings of an empirical study assessing the extent to which 243 and 194 code snippets, across six programming languages, generated by Bing CoPilot and Google Gemini, likely originate from the links provided by these two LLM-based assistants. The study leverages automated code similarity assessments with thorough manual analysis. The study's findings indicate that the LLM-based assistants provide a mix of relevant and irrelevant links having a different nature. Specifically, although 66% of the links from Bing CoPilot and 28% from Google Gemini are relevant, LLMs-based assistants still suffer from serious "provenance debt".

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑