arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~

MediaWiki Code2Code搜索:用于开源软件实体语义发现的神经检索系统

MediaWiki Code2Code Search: Neural Retrieval for the Semantic Discovery of Open-Source Software Entities

Francesco Tosoni

arXiv 2607.26766首次发表:更新:

发表机构

Sant’Anna School of Advanced Studies(圣安娜高等研究大学)

机构由 AI 辅助整理,请以论文原文为准。

AI 中文总结

针对大规模代码搜索的词汇鸿沟与检索权衡问题,提出MediaWiki Code2Code神经检索系统,其在基准测试中性能优于BM25,可高效发现开源软件实体。

AI 中文摘要

大规模生态系统中的代码搜索常受用户查询与实现细节间的词汇鸿沟,以及传统信息检索(IR)低延迟与深度学习(DL)高精度间的权衡问题阻碍。我们提出MediaWiki Code2Code搜索,这是一种用于代码到代码语义发现的神经检索系统。该系统索引了2500多个MediaWiki仓库中的129万个结构实体(函数、类型和模板),支持基于计算意图而非表面标记的检索。我们采用拆分构建架构,将GPU密集型离线索引与仅CPU的服务层解耦;其FAISS IVF-PQ索引占用168.6 MB,相比flat float32基准减少了96.6%,且在商用硬件上实现了1.85秒的中位数查询延迟,满足Wikimedia Toolforge的6 GiB RAM约束。我们在包含27个查询的基准测试中评估,结果显示其性能优于BM25基准,P@10达到0.87,而BM25为0.64(严格匹配的P@10为0.52对比0.34),在词汇方法失效的名称混淆任务中提升最为显著。该系统通过Apache 2.0许可提供,可访问指定URL,并提供开放的RESTful API。

英文摘要

Code search in large-scale ecosystems is often hindered by the lexical gap between user queries and implementation details, alongside the trade-off between the low latency of traditional Information Retrieval (IR) and the precision of Deep Learning (DL). We present MediaWiki Code2Code Search, a neural retrieval system for semantic code-to-code discovery. By indexing 1.29 million structural entities (functions, types, and templates) across 2,500+ MediaWiki repositories, our system enables retrieval based on computational intent rather than surface tokens. We employ a split-build architecture, decoupling GPU-intensive offline indexing from a CPU-only serving layer; our FAISS IVF-PQ index occupies 168.6 MB: a 96.6\% reduction compared to a flat float32 baseline, and achieves a median query latency of 1.85 seconds on commodity hardware, satisfying the 6 GiB RAM constraint of Wikimedia Toolforge. Our evaluation across a 27-query benchmark demonstrates superior performance over the BM25 baseline, achieving a P@10 of 0.87 compared to 0.64 (0.52 versus 0.34 for strict matching). Gains are most pronounced in name-obfuscated tasks where lexical methods fail. The system is available at https://code2codesearch.toolforge.org under the Apache 2.0 licence and provides an open RESTful API.

Comments21 pages, 5 tables, 3 figures

论文原文

arXiv 摘要页 · PDF 原文 · HTML 原文

↑