OckBench：衡量LLM推理效率
OckBench: Measuring the Efficiency of LLM Reasoning
AI总结：
提出OckBench基准，联合评估推理和编码任务中的准确性与token效率，揭示当前模型token利用率低下问题。
摘要：
大型语言模型（LLM）如GPT-5和Gemini 3已推动自动推理和代码生成的前沿。然而，当前基准强调准确性和输出质量，忽略了关键维度：token使用的效率。在实际应用中，token效率变化很大。解决相同问题且准确率相近的模型，其token长度差异可达 extbf{5.0$ imes$}，导致模型推理能力存在巨大差距。这种差异暴露了显著的冗余，凸显了对标准化基准来量化token效率差距的迫切需求。因此，我们引入OckBench，这是首个联合衡量推理和编码任务中准确性与token效率的基准。我们的评估表明，当前模型的token效率在很大程度上未得到优化，显著增加了服务成本和延迟。这些发现为社区优化潜在推理能力（即token效率）提供了具体路线图。最终，我们主张评估范式转变：token不应被无谓地倍增。我们的基准可在https://ockbench.github.io/获取。
英文摘要：
Large language models (LLMs) such as GPT-5 and Gemini 3 have pushed the frontier of automated reasoning and code generation. Yet current benchmarks emphasize accuracy and output quality, neglecting a critical dimension: efficiency of token usage. The token efficiency is highly variable in practical. Models solving the same problem with similar accuracy can exhibit up to a \textbf{5.0$\times$} difference in token length, leading to massive gap of model reasoning ability. Such variance exposes significant redundancy, highlighting the critical need for a standardized benchmark to quantify the gap of token efficiency. Thus, we introduce OckBench, the first benchmark that jointly measures accuracy and token efficiency across reasoning and coding tasks. Our evaluation reveals that token efficiency remains largely unoptimized across current models, significantly inflating serving costs and latency. These findings provide a concrete roadmap for the community to optimize the latent reasoning ability, token efficiency. Ultimately, we argue for an evaluation paradigm shift: tokens must not be multiplied beyond necessity. Our benchmarks are available at https://ockbench.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
