KGym：在 Linux 内核崩溃解决任务上对大型语言模型进行基准测试的平台与数据集
KGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolution
AI总结：
提出用于评估大模型解决Linux内核崩溃能力的平台kGym与基准kBench，实验显示当前最优模型解决率极低，揭示了系统软件工程领域的挑战与研究机遇。
摘要：
大型语言模型（LLMs）在日益真实的软件工程（SE）任务上持续取得进步。在现实世界的软件栈中，大量的软件工程工作被投入到开发诸如 Linux 内核等基础系统软件上。与应用层软件不同，像 Linux 这样的系统代码库是多语言的（包含底层 C/Assembly/Bash/Rust）；规模庞大（超过 2000 万行代码）；具有关键性（影响全球数十亿台设备），且高度并发（涉及复杂的多线程）。为了评估机器学习模型在开发此类大规模系统级软件时是否具有实用价值，我们引入了 kGym（一个平台）和 kBench（一个数据集）。kGym 平台为 Linux 内核上的大规模实验提供了软件工程环境，包括在多台虚拟机中并行编译和运行内核、检测操作和崩溃、检查日志，以及对代码库进行查询和修补。我们利用 kGym 来促进在 kBench 上的评估，kBench 是一个源自现实世界 Linux 内核漏洞的崩溃解决基准。kBench 中的一个漏洞示例包含崩溃堆栈跟踪、漏洞重现文件、开发者编写的修复补丁以及其他相关数据。为了解当前的性能表现，我们通过提示 LLMs 解决 Linux 内核崩溃来进行基线实验。我们的初步评估显示，表现最好的 LLM 在无辅助和有辅助（即向模型披露包含漏洞的文件）设置下的解决率分别达到了 0.72% 和 5.38%。这些结果凸显了在软件工程任务中进一步提升模型性能的研究需求。在 kBench 上提升性能需要模型掌握新的学习技能，包括理解崩溃原因并修复故障、编写内存安全且具备硬件感知的代码，以及理解并发性。因此，这项工作开辟了机器学习与系统软件交叉领域的多条研究途径。
英文摘要：
Large Language Models (LLMs) are consistently improving at increasingly realistic software engineering (SE) tasks. In real-world software stacks, significant SE effort is spent developing foundational system software like the Linux kernel. Unlike application-level software, a systems codebase like Linux is multilingual (low-level C/Assembly/Bash/Rust); gigantic (>20 million lines); critical (impacting billions of devices worldwide), and highly concurrent (involving complex multi-threading). To evaluate if ML models are useful while developing such large-scale systems-level software, we introduce kGym (a platform) and kBench (a dataset). The kGym platform provides a SE environment for large-scale experiments on the Linux kernel, including compiling and running kernels in parallel across several virtual machines, detecting operations and crashes, inspecting logs, and querying and patching the code base. We use kGym to facilitate evaluation on kBench, a crash resolution benchmark drawn from real-world Linux kernel bugs. An example bug in kBench contains crashing stack traces, a bug-reproducer file, a developer-written fix, and other associated data. To understand current performance, we conduct baseline experiments by prompting LLMs to resolve Linux kernel crashes. Our initial evaluations reveal that the best performing LLM achieves 0.72% and 5.38% in the unassisted and assisted (i.e., buggy files disclosed to the model) settings, respectively. These results highlight the need for further research to enhance model performance in SE tasks. Improving performance on kBench requires models to master new learning skills, including understanding the cause of crashes and repairing faults, writing memory-safe and hardware-aware code, and understanding concurrency. As a result, this work opens up multiple avenues of research at the intersection of machine learning and systems software.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
