$\Phi$-Bench：大型语言模型能否工程化驱动自身的基础设施？

中文摘要 AI 辅助

大型语言模型（LLMs）在推理和代码生成方面展现了卓越的能力，这引发了它们可能协助开发和优化驱动自身的基础设施的前景。然而，现有基准主要聚焦于孤立的核函数、预定义算子或预先指定的优化目标，因此未能评估LLMs在开放式、长周期LLM基础设施工程中的能力。为填补这一空白，我们提出了$\Phi$-Bench，一个用于系统评估LLMs在工程化LLM基础设施栈方面能力的基准。$\Phi$-Bench源自前沿研究中探讨的优化问题，并基于真实世界的代码仓库，广泛覆盖了LLM基础设施栈，并涵盖了复杂度各异的任务，从局部的核级函数补全到长周期的实现及端到端系统优化。对前沿LLMs的大量实验揭示了它们在工程化复杂LLM基础设施方面的当前能力与局限，为未来AI基础设施自主优化道路上仍存在的挑战提供了见解。

英文摘要

Large language models (LLMs) have demonstrated remarkable capabilities in reasoning and code generation, raising the prospect that they could assist in developing and optimizing the very infrastructure that powers them. However, existing benchmarks mainly focus on isolated kernels, predefined operators, or pre-specified optimization targets, and therefore fail to evaluate the ability of LLMs to perform open-ended, long-horizon LLM infrastructure engineering. To address this gap, we present $Φ$-Bench, a benchmark for systematically evaluating LLMs on engineering the LLM infrastructure stack. Derived from optimization problems studied in frontier research and grounded in real-world code repositories, $Φ$-Bench provides broad coverage of the LLM infrastructure stack and spans tasks of varying complexity, ranging from localized kernel-level function completion to long-horizon implementation and end-to-end system optimization. Extensive experiments on frontier LLMs reveal their current capabilities and limitations in engineering complex LLM infrastructure, offering insights into the challenges that remain on the path toward autonomous optimization of future AI infrastructure.

发表机构

• University of Science and Technology of China（中国科学技术大学）
• StepFun（阶跃星辰）
• Peking University（北京大学）
• The Hong Kong University of Science and Technology（香港科技大学）
• Yale University（耶鲁大学）
• University of Pennsylvania（宾夕法尼亚大学）

机构由 AI 辅助整理，请以论文原文为准。