Blackbox Model Provenance via Palimpsestic Membership Inference
机构 * Stanford University（斯坦福大学）
作者
Percy Liang
Natural Language Processing
Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
MLE-Smith: Scaling MLE Tasks with Automated Multi-Agent Pipeline
机构 * Georgia Institute of Technology（佐治亚理工学院） ； Stanford University（斯坦福大学）
WorldGym: World Model as An Environment for Policy Evaluation
机构 * Stanford University（斯坦福大学） ； NYU（纽约大学） ； Google DeepMind（谷歌DeepMind）
Comments https://world-model-eval.github.io
Pre-training under infinite compute
机构 * Stanford University（斯坦福大学）
Audits Under Resource, Data, and Access Constraints: Scaling Laws For Less Discriminatory Alternatives
机构 * Stanford University（斯坦福大学）
Comments 34 pages, 13 figures
Fantastic Pretraining Optimizers and Where to Find Them
机构 * Stanford University（斯坦福大学）
Comments 108 pages, 8 figures, reproducible runs available at https://wandb.ai/marin-community/optimizer-scaling
Reinforcement Learning for Machine Learning Engineering Agents
机构 * Stanford University（斯坦福大学）
AHELM: A Holistic Evaluation of Audio-Language Models
机构 * Stanford University（斯坦福大学） ； University of California, Santa Cruz（加州大学圣克ruz分校） ； Hitachi America, Ltd.（日立美国有限公司）
UQ: Assessing Language Models on Unsolved Questions
Comments FN, KZL, and NM are project co-leads and contributed equally. Project website: https://uq.stanford.edu
Establishing Best Practices for Building Rigorous Agentic Benchmarks
机构 * UIUC（伊利诺伊大学） ； Stanford University（斯坦福大学） ； University of California, Berkeley（加州大学伯克利分校） ； Yale University（耶鲁大学） ； Princeton University（普林斯顿大学） ； MIT（麻省理工学院） ； Transluce ； ML Commons ； Amazon（亚马逊） ； UK AI Safety Institute（英国人工智能安全研究所） ； University of Oxford（牛津大学）
Comments 39 pages, 15 tables, 6 figures
Advancing Science- and Evidence-based AI Policy
Comments This is the author's version of the work. It is posted here by permission of the AAAS for personal use, not for redistribution. The definitive version was published in Science on July 31, 2025
Language model developers should report train-test overlap
机构 * Stanford University（斯坦福大学）
Comments ICML 2025 Spotlight; 23 pages
Auditing Prompt Caching in Language Model APIs
机构 * stanford（斯坦福大学）
Comments Accepted at ICML 2025
The California Report on Frontier AI Policy
Comments Authored by the Joint California Policy Working Group on AI Frontier Models
MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks
机构 * Stanford University School of Medicine（斯坦福大学医学院） ； Stanford Health Care（斯坦福健康系统） ； Center for Research on Foundation Models (CRFM) & Department of Computer Science（基础模型研究中心（CRFM）及计算机科学系） ； Microsoft Corporation（微软公司） ； Stanford Institute for Human-Centered AI（斯坦福大学人本人工智能研究所）
MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering
机构 * Georgia Institute of Technology（佐治亚理工学院） ； Stanford University（斯坦福大学）
Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
Comments Accepted to Robotics: Science and Systems (RSS) 2025. Project website: https://openvla-oft.github.io/
AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons
Comments 51 pages, 8 figures and an appendix
Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models
Comments ICLR 2025 Oral
International Scientific Report on the Safety of Advanced AI (Interim Report)
Comments Available under the open government license at https://www.gov.uk/government/publications/international-scientific-report-on-the-safety-of-advanced-ai
Model Equality Testing: Which Model Is This API Serving?
Comments ICLR 2025 Camera-Ready
Language Models May Verbatim Complete Text They Were Not Explicitly Trained On
Comments Main text: 9 pages, 7 figures, 1 table. Appendix: 29 pages, 20 tables, 15 figures
In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI
Reliable and Efficient Amortized Model-based Evaluation
Independence Tests for Language Models
BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
Comments COLM 2024
The 2024 Foundation Model Transparency Index
Comments Published in TMLR 2025. Project page: https://crfm.stanford.edu/fmti
s1: Simple test-time scaling
Comments 46 pages (9 main), 10 figures, 15 tables
