SCCGBench:面向可执行性的真实世界Web服务调用代码生成评估基准
SCCGBench: Executability-Oriented Evaluation for Real-World Web Service-Call Code Generation
AI总结:
该研究提出SCCGBench基准,评估代码LLM生成Web服务调用代码的可执行性,发现服务/API落地是瓶颈,目标文档下开源模型Mock成功率达88.73%--90.52%。
摘要:
大语言模型(LLM)生成的服务调用代码在服务计算中愈发重要,开发者依赖LLM来组合服务并调用第三方服务。然而,现有方法尚未在实际代码环境中评估服务的调用与执行。为支持基于执行的评估,我们提出服务调用代码生成基准(SCCGBench),该基准包含来自真实GitHub仓库的3135个可追踪且经去污染过滤的RapidAPI调用函数。SCCGBench采用意图生成以减少循环泄漏,并引入Mock、Robust、ParamF1以及主机、端点、方法和参数级别的诊断指标。Mock指同时匹配目标主机、端点和HTTP方法的虚拟执行命中,Robust则进一步要求状态检查与异常处理。我们在仅意图、检索增强和目标文档Oracle设置下评估了6个代码LLM和1个非神经检索基线。结果显示,模型常生成语法正确或表面相似的HTTP请求代码,但在仅意图设置下Mock成功率低,评估中最强模型仅达到3.76%的Mock。在目标文档Oracle设置(C3和C5)下,RQ2中评估的开源模型Mock达到88.73%--90.52%,DeepSeek-V4-Pro则为76.47%--81.86%,而稀疏与密集检索远低于该Oracle水平。研究发现表明,服务/API落地是主要瓶颈:Mock由主机、端点和方法对齐主导,ParamF1则单独揭示参数对齐错误。SCCGBench为Web服务调用、API推荐、服务接口检索、端点落地及可靠服务调用自动化提供了基于执行的基准。
英文摘要:
LLM-generated service-call code is increasingly important for service computing, where developers rely on large language models (LLMs) to compose services and invoke third-party services. However, current approaches have not yet evaluated the invocation and execution of services in the actual code environment. To support execution-grounded evaluation, we present the Service-Call Code Generation Benchmark (SCCGBench), a benchmark of 3,135 traceable and decontamination-filtered RapidAPI call functions from real GitHub repositories. SCCGBench uses redacted-intent generation to reduce circular leakage and introduces Mock, Robust, ParamF1, and host-, endpoint-, method-, and parameter-level diagnostics. Mock denotes a virtual-execution hit that simultaneously matches the gold host, endpoint, and HTTP method, while Robust further requires status checking and exception handling. We evaluate six code LLMs and one non-neural retrieval baseline under intent-only, retrieval-augmented, and gold-documentation Oracle settings. Results show that models often generate syntactically correct or superficially similar HTTP-request code but achieve low Mock success in the intent-only setting; the strongest evaluated model obtains only 3.76% Mock. Under the gold-documentation Oracle settings (C3 and C5), Mock reaches 88.73%--90.52% for the open-source models evaluated in RQ2 and 76.47%--81.86% for DeepSeek-V4-Pro, whereas sparse and dense retrieval remain far below this oracle level. The findings indicate that service/API grounding is the main bottleneck: Mock is dominated by host, endpoint, and method alignment, while ParamF1 separately reveals parameter-alignment errors. SCCGBench provides an execution-grounded benchmark for Web service invocation, API recommendation, service-interface retrieval, endpoint grounding, and reliable service-call automation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。