注意差距：前沿大语言模型能否通过标准化办公能力考试？
Mind the Gap: Can Frontier LLMs Pass a Standardized Office Proficiency Exam?
AI总结：
基于中国计算机等级考试（NCRE）的200个综合操作任务，评估7个前沿LLM在Word、Excel和PowerPoint自动化中的表现，发现单轮模型最高得分率36.6%，带执行反馈的智能体系统达68.8%，仍低于95.5%的社区参考分，表明可靠细粒度办公自动化仍是重大挑战。
摘要：
大语言模型（LLM）代理在计算机自动化领域的部署正在加速，但其在复杂、专业级生产力软件中的导航能力在很大程度上尚未得到测试。我们认为办公自动化是基准测试文档自动化能力的理想环境，因为它需要长期规划和推理、精确的参数配置以及多应用集成。为了量化这种能力，我们引入了一项基于中国国家计算机等级考试（NCRE）的评估，包含200个涵盖Word、Excel和PowerPoint的综合实践操作任务。每个任务根据7118个机器可评分标准按100分制评分，得分率（SR）表示这些任务中获得的平均评分百分比。我们对7个前沿LLM进行了基准测试，并观察到明显的局限性：单轮模型最高得分为36.6%。一个具有执行反馈、迭代修复和更广泛办公自动化访问权限的更强智能体系统达到了68.8%，但仍低于用作评分合理性检查的95.5%社区参考分。最终，我们的实验表明，尽管代码生成最近取得了进展，但对于当前的代码生成LLM和智能体系统来说，实现可靠的细粒度办公文档自动化仍然是一个重大挑战。
英文摘要：
The deployment of Large Language Model (LLM) agents for computer automation is accelerating, yet their ability to navigate complex, professional-grade productivity software is largely untested. We argue that Office automation is an ideal environment for benchmarking document-automation capability, as it requires long-horizon planning and reasoning, precise parameter configuration, and multi-application integration. To quantify this capability, we introduce an evaluation based on China's National Computer Rank Examination (NCRE), featuring 200 comprehensive practical-operation tasks across Word, Excel, and PowerPoint. Each task is scored on a 100-point rubric scale using 7,118 machine-gradable criteria, and Score Rate (SR) denotes the mean percentage of rubric points earned across these tasks. We benchmark 7 frontier LLMs and observe stark limitations: single-turn models score a maximum of 36.6%. A stronger agentic system with execution feedback, iterative repair, and broader Office automation access reaches 68.8%, but remains below the 95.5% community-reference score used as a scoring sanity check. Ultimately, our experiments demonstrate that despite recent advancements in code generation, achieving reliable fine-grained Office document automation remains a significant challenge for current code-generating LLM and agent systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
