发表机构
School of Environment, Tsinghua University; Fudan University; Nanjing University; Capital Normal University; Tencent Technology (Shenzhen) Company Limited; China University of Petroleum (Beijing); Tanwei College, Tsinghua University（清华大学环境学院; 复旦大学; 南京大学; 首都师范大学; 腾讯科技（深圳）有限公司; 中国石油大学（北京）; 清华大学探微书院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
WuYuEval是面向固体废物管理的多级基准测试，含基础与专家模块，评估33个LLM的SWM能力，发现其在专业任务表现差，为开发专用基础模型提供依据。
AI 中文摘要
大语言模型（LLM）越来越多地被用作技术助手，但它们在固体废物管理（SWM）方面的能力仍难以评估，因为现有基准测试侧重通用知识，而非工程、环境和政策约束下的专业决策。我们推出WuYuEval，一个用于评估LLM在SWM领域能力的多级基准测试，涵盖基础知识、领域推理和专家决策三个层面。经过质量审核，WuYuEval包含基础模块（Foundation Module），设有4590道封闭式选择题，涵盖6种任务类型和8个领域类别；以及专家模块（Expert Module），设有247道基于场景的开放式问题，涉及多目标优化、约束权衡和系统设计。对于专家任务，我们结合锚定校准的“LLM作为评判者”评分法与基于Elo的成对比较法。在33个LLM中，性能差异显著：领先模型在基础模块的准确率达94.64%，但平均准确率仍从简单问题的84.14%降至困难问题的42.50%，且性能较低的领域集中在计算、实验设计、城市规划及开放式专家任务。面向推理的思维模式在审核后提升了多数匹配模型对的表现，但提升效果取决于基线能力，并非均为正向。这些结果表明，仅当可见的推理过程锚定到单位、假设和工程约束时才会有效；否则，可能偏离明确的答案边界。因此，WuYuEval既提供了评估资源，也为开发具备专业推理链和显式约束控制的SWM专用基础模型提供了实证基础。
英文摘要
Large language models (LLMs) are increasingly used as technical assistants, but their competence in solid waste management (SWM) remains difficult to assess because existing benchmarks emphasize general knowledge rather than professional decisions under engineering, environmental, and policy constraints. We introduce WuYuEval, a multi-level benchmark for evaluating LLMs in SWM across foundational knowledge, domain reasoning, and expert decision-making. After quality auditing, WuYuEval contains a Foundation Module with 4,590 closed-ended multiple-choice questions across six task types and eight domain categories, together with an Expert Module with 247 scenario-based open-ended questions involving multi-objective optimization, constraint trade-offs, and system design. For expert tasks, we combine anchor-calibrated LLM-as-a-Judge scoring with Elo-based pairwise comparison. Across 33 LLMs, performance varied widely. The leading model reached 94.64\% accuracy on the Foundation Module, but average accuracy still fell from 84.14\% on easy questions to 42.50\% on hard questions, with lower performance concentrated in calculation, experimental design, urban planning, and open-ended expert tasks. Reasoning-oriented Thinking modes improve most matched model pairs after auditing, but the gains depend on baseline capability and are not uniformly positive. These results suggest that visible deliberation helps only when it remains anchored to units, assumptions, and engineering constraints; otherwise, it may drift from decisive answer boundaries. WuYuEval therefore provides both an evaluation resource and an empirical basis for developing SWM-oriented foundation models with professional reasoning chains and explicit constraint control.
