arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.26091cs.SE
收藏

网页增强型大语言模型代码生成中的搜索诱导问题:检测与修复误导页面

Search-Induced Issues in Web-Augmented LLM Code Generation: Detecting and Repairing Error-Inducing Pages

Peking University(北京大学);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);Singapore Management University(新加坡管理大学);Fuzhou University(福州大学)

Guoqing Wang, Zeyu Sun, Xiaofei Xie, Yizhou Chen, Yanchao Tan, Yifan Zhao, Dan Hao

AI总结:

本文研究了网页增强型LLM代码生成中因误导页面导致的搜索诱导问题,提出Sherlock框架用于检测、调试和修复问题页面,提升系统可靠性。

摘要:

网页增强型大语言模型(LLM)在自动代码生成中展现出巨大潜力。然而,整合实时网页搜索使模型面临不可靠或恶意内容的挑战,导致搜索诱导问题(SII),即外部页面误导LLM生成错误代码。本文对三个商业搜索API和六个先进LLM的SII普遍性及影响进行了全面实证研究。分析表明,所有评估的网页增强型LLM均易受SII影响,根本原因源于搜索错误诱导页面(EIPs)中规范不一致或代码实现错误。为解决此问题,我们提出了Sherlock框架,该框架使LLM服务提供商能够大规模主动保护网页增强生成系统。Sherlock作为连续流水线,首先检测潜在SII实例,然后调试以识别责任EIPs并确定根本原因,最后通过标注不一致内容或用受信任来源评估的解决方案替换错误代码片段进行修复。实验显示,Sherlock在识别EIPs时的F1分数高达95%,在修复受影响生成方面达到71%至100%的修复率,计算开销适度。我们的发现和框架为提升实际软件工程场景中网页增强型LLM代码生成系统的可靠性提供了实用指导。

英文摘要:

Web-augmented large language models (LLMs) offer promising capabilities for automatic code generation. However, integrating live web search exposes models to unreliable or malicious content, leading to Search-Induced Issues (SII), a novel failure mode in which external pages mislead LLMs into producing incorrect code. This paper presents a comprehensive empirical study of the prevalence and impact of SII across three commercial search APIs and six advanced LLMs. Our analysis reveals that all evaluated web-augmented LLMs are vulnerable to SII, with root causes arising from either misaligned specifications or flawed code implementations in the searched Error-Inducing Pages (EIPs). To address this challenge, we propose Sherlock, an automated framework that enables LLM service providers to proactively safeguard web-augmented generation systems at scale. Sherlock operates as a continuous pipeline that first detects potential SII instances, then debugs them to identify the responsible EIPs and pinpoint their root causes, and finally repairs them by either annotating misaligned content or replacing erroneous code snippets with evaluated solutions from trusted sources. Experiments show that Sherlock identifies EIPs with an F1 score of up to 95% and repairs 71% to 100% of affected generations across the evaluated models, with modest computational overhead. Our findings and framework provide practical guidance for improving the reliability of web-augmented LLM-based code generation systems in real-world software engineering scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑