Meituan(美团);Pennsylvania State University(宾夕法尼亚州立大学);Allen Institute for AI (AI2)(人工智能艾伦研究所);University of Washington(华盛顿大学);University at Buffalo(布法罗纽约州立大学)
Huaisheng Zhu, Hangfan Zhang, Teng Xiao, Siyuan Xu, Shijie Zhou, Zhimeng Guo, Zhengyu Chen, Jingang Wang
Large language model (LLM) agents are increasingly used for long-horizon research and engineering, yet evaluating these capabilities remains expensive, time-consuming, and heavily dependent on human expertise. This motivates a natural question: can LLM agents themselves construct efficient benchmarks that preserve the conclusions of costly reference evaluations? We study this capability by formulating automatic benchmark development as a meta-evaluation problem, where an agent is given a task and an expensive reference benchmark and requested to construct a lower-cost proxy benchmark that faithfully distinguishes among candidate models and solutions. Based on this formulation, we introduce EvalBuilder, a framework for evaluating such benchmark-generation capabilities along three key dimensions: fidelity, efficiency, and robustness. Specifically, we assess whether generated benchmarks preserve model rankings, generalize to held-out models and freshly collected responses, and provide well-calibrated and discriminative scores while substantially reducing evaluation cost. We further provide an iterative environment in which agents can refine their benchmarks under a fixed time budget. Our framework provides a systematic testbed and benchmark for understanding when current LLM agents can develop useful evaluators and how benchmark construction itself can be increasingly automated.
Do LLMs Build World Models From Text? A Multilingual Diagnostic of Spatial Reasoning
University of New South Wales(新南威尔士大学);Essential Energy;University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);Zhejiang University(浙江大学);Fudan University(复旦大学);Tsinghua University(清华大学)
BuildEvo: Designing Building Energy Consumption Forecasting Heuristics via LLM-driven Evolution
Department of the Built Environment, National University of Singapore(新加坡国立大学建筑环境系);Department of Industrial and Systems Engineering, Korea Advanced Institute of Science and Technology(韩国科学技术院工业与系统工程系)
Building Accurate Translation-Tailored LLMs with Language Aware Instruction Tuning
China University of Petroleum (East China)(中国石油大学(华东));The University of Sydney(悉尼大学);JD Explore Academy(京东探索研究院);Nanyang Technological University(南洋理工大学)
Towards Publicly Accountable Frontier LLMs: Building an External Scrutiny Ecosystem under the ASPIRE Framework
Centre for the Governance of AI(人工智能治理中心);Center for a New American Security(新美国安全中心);Institute for AI Policy and Strategy(人工智能政策与战略研究所);Blavatnik School of Government, University of Oxford(牛津大学布拉瓦尼克政府学院);OpenMined;Humane Intelligence(人文智能);Harvard Berkman Klein Center(哈佛大学伯克曼·克莱因中心)
NICE: A Theory-Grounded Diagnostic Benchmark for Social Intelligence of LLMs
Department of Psychology and Behavioral Sciences, Zhejiang University(浙江大学心理学与行为科学系);College of Artificial Intelligence, Zhejiang University(浙江大学人工智能学院);Human Machine Interaction Lab, Huawei Technologies Co., Ltd.(华为技术有限公司人机交互实验室);Zhejiang Key Laboratory of Neurocognitive Development and Mental Health(浙江省神经认知发展与心理健康重点实验室)
University of Liverpool(利物浦大学);Key Laboratory of System Software (Chinese Academy of Sciences)(中国科学院系统软件重点实验室);State Key Laboratory of Computer Science(计算机科学国家重点实验室);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);University of Warwick(华威大学);Institute of Digital Technologies, Loughborough University London(拉夫堡大学伦敦数字技术研究所)
Translating Regulatory Clauses into Executable Codes for Building Design Checking via Large Language Model Driven Function Matching and Composing
Tsinghua University(清华大学);Wuliangye Yibin Co., Ltd.(五粮液宜宾有限公司);Yantai University(烟台大学);Key Laboratory of Digital Construction and Digital Twin, Ministry of Housing and Urban-Rural Development(住房和城乡建设部数字建造与数字孪生重点实验室)
Configurable Foundation Models: Building LLMs from a Modular Perspective
Tsinghua University(清华大学);University of California San Diego(加利福尼亚大学圣迭戈分校);Carnegie Mellon University(卡内基梅隆大学);ModelBest Inc.(模百科技);Princeton University(普林斯顿大学);National University of Singapore(新加坡国立大学);Stanford University(斯坦福大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Renmin University of China(中国人民大学)
Distorted or Fabricated? A Survey on Hallucination in Video LLMs
Department of Electrical and Computer Engineering, Northeastern University(东北大学电气与计算机工程系);Khoury College of Computer Science, Northeastern University(东北大学科赫里计算机科学学院)