UQ：评估语言模型在未解决问题上的表现
UQ: Assessing Language Models on Unsolved Questions
AI总结：
针对当前AI基准存在的困难与现实张力问题，提出UQ测试平台，通过异步评估未解决问题、结合验证器和社区验证，推动模型在真实挑战中的能力提升，顶级模型仅15%通过率。
摘要：
基准塑造了人工智能研究的进展。一个有用的基准应该既困难又现实：问题应该挑战前沿模型，同时反映现实世界的使用场景。然而，当前范式面临困难与现实之间的张力：考试式基准往往人为设置难度，现实价值有限；而基于真实用户交互的基准则常偏向简单、高频的问题。在这项工作中，我们探索了一种截然不同的范式：在未解决问题上评估模型。我们不使用一次性评分的静态基准，而是通过验证器辅助筛选和社区验证，异步地随时间评估模型。我们引入UQ，一个包含500个具有挑战性、多样化问题的测试平台，这些问题来自Stack Exchange，涵盖从计算机科学理论、数学到科幻和历史的主题，考察推理、事实性和浏览等能力。UQ在构建上既困难又现实：未解决问题通常很难，且自然出现在人类寻求答案的过程中，因此解决它们具有直接的现实价值。我们的贡献有三方面：（1）UQ数据集及其收集流程，结合基于规则的过滤器、LLM判断器和人工审核，确保问题质量（如定义明确且困难）；（2）UQ验证器，利用生成器-验证器差距提供评估信号，并预先筛选候选解决方案供人工审核；（3）UQ平台，一个开放平台，专家可集体验证问题和解决方案。顶级模型仅在15%的问题上通过UQ验证，初步人工验证已在通过的解决方案中识别出正确答案。UQ为在现实世界、开放式挑战上评估前沿模型开辟了道路，其成功推动了人类知识的前沿。我们在https://uq.stanford.edu发布UQ。
英文摘要：
Benchmarks shape progress in AI research. A useful benchmark should be both difficult and realistic: questions should challenge frontier models while also reflecting real-world usage. Yet, current paradigms face a difficulty-realism tension: exam-style benchmarks are often made artificially difficult with limited real-world value, while benchmarks based on real user interaction often skew toward easy, high-frequency problems. In this work, we explore a radically different paradigm: assessing models on unsolved questions. Rather than a static benchmark scored once, we curate unsolved questions and evaluate models asynchronously over time with validator-assisted screening and community verification. We introduce UQ, a testbed of 500 challenging, diverse questions sourced from Stack Exchange, spanning topics from CS theory and math to sci-fi and history, probing capabilities including reasoning, factuality, and browsing. UQ is difficult and realistic by construction: unsolved questions are often hard and naturally arise when humans seek answers, thus solving them yields direct real-world value. Our contributions are threefold: (1) UQ-Dataset and its collection pipeline combining rule-based filters, LLM judges, and human review to ensure question quality (e.g., well-defined and difficult); (2) UQ-Validators, compound validation strategies that leverage the generator-validator gap to provide evaluation signals and pre-screen candidate solutions for human review; and (3) UQ-Platform, an open platform where experts collectively verify questions and solutions. The top model passes UQ-validation on only 15% of questions, and preliminary human verification has already identified correct answers among those that passed. UQ charts a path for evaluating frontier models on real-world, open-ended challenges, where success pushes the frontier of human knowledge. We release UQ at https://uq.stanford.edu.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
