Large language models (LLMs) are increasingly deployed and democratized on edge devices. To improve the efficiency of on-device deployment, small language models (SLMs) are often adopted due to their efficient decoding latency and reduced energy consumption. However, these SLMs often generate inaccurate responses when handling complex queries. One promising solution is uncertainty-based SLM routing, offloading high-stakes queries to stronger LLMs when resulting in low-confidence responses on SLM. This follows the principle of "If you lack confidence, seek stronger support" to enhance reliability. Relying on more powerful LLMs is yet effective but increases invocation costs. Therefore, striking a routing balance between efficiency and efficacy remains a critical challenge. Additionally, efficiently generalizing the routing strategy to new datasets remains under-explored. In this paper, we conduct a comprehensive investigation into benchmarking and generalization of uncertainty-driven routing strategies from SLMs to LLMs over 1500+ settings. Our findings highlight: First, uncertainty-correctness alignment in different uncertainty quantification (UQ) methods significantly impacts routing performance. Second, uncertainty distributions depend more on both the specific SLM and the chosen UQ method, rather than downstream data. Building on the insight, we propose a calibration data construction instruction pipeline and open-source a constructed hold-out set to enhance routing generalization on new downstream scenarios. The experimental results indicate calibration data effectively bootstraps routing performance without any new data.
The Seeker's Dilemma: Realistic Formulation and Benchmarking for Hardware Trojan Detection
Klipsch School of Electrical and Computer Engineering, New Mexico State University（新墨西哥州立大学克利普什电气与计算机工程学院）；Department of Computer Systems Technology, North Carolina A&T State University（北卡罗来纳农工州立大学计算机系统技术系）；Department of Electrical and Computer Engineering, Miami University（迈阿密大学电气与计算机工程学院）
Seek Before You Move: Evidence Seeking for Progress Grounding in Vision-Language Navigation
Tsinghua University（清华大学）；Pengcheng Laboratory（鹏城实验室）；South China University of Technology（华南理工大学）；International Digital Economy Academy（国际数字经济学院）；Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Confidently making progress on multilingual modeling requires challenging, trustworthy evaluations. We present TyDi QA---a question answering dataset covering 11 typologically diverse languages with 204K question-answer pairs. The languages of TyDi QA are diverse with regard to their typology---the set of linguistic features each language expresses---such that we expect models performing well on this set to generalize across a large number of the world's languages. We present a quantitative analysis of the data quality and example-level qualitative linguistic analyses of observed language phen…
The rapid proliferation of machine learning models across domains and deployment settings has given rise to various communities (e.g. industry practitioners) which seek to benchmark models across tasks and objectives of personal value. Unfortunately, these users cannot use standard benchmark results to perform such value-driven comparisons as traditional benchmarks evaluate models on a single objective (e.g. average accuracy) and fail to facilitate a standardized training framework that controls for confounding variables (e.g. computational budget), making fair comparisons difficult. To addres…
De novo design seeks to generate molecules with required property profiles by virtual design-make-test cycles. With the emergence of deep learning and neural generative models in many application areas, models for molecular design based on neural networks appeared recently and show promising results. However, the new models have not been profiled on consistent tasks, and comparative studies to well-established algorithms have only seldom been performed. To standardize the assessment of both classical and neural models for de novo molecular design, we propose an evaluation framework, GuacaMol, …
Nanjing University（南京大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）
Entity alignment seeks to find entities in different knowledge graphs (KGs) that refer to the same real-world object. Recent advancement in KG embedding impels the advent of embedding-based entity alignment, which encodes entities in a continuous embedding space and measures entity similarities based on the learned embeddings. In this paper, we conduct a comprehensive experimental study of this emerging field. We survey 23 recent embedding-based entity alignment approaches and categorize them based on their techniques and characteristics. We also propose a new KG sampling algorithm, with which…
Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph
MBZUAI（穆罕默德·本·扎耶德人工智能大学）；ETH Zürich（苏黎世联邦理工学院）；Skoltech（斯科尔科沃科学技术学院）；AIRI（人工智能研究所）；HSE University（高等经济学院）；FRC CSC RAS（俄罗斯科学院计算机科学控制问题科学中心）；Nebius（Nebius公司）；The University of Melbourne（墨尔本大学）
Bayesian deep learning seeks to equip deep neural networks with the ability to precisely quantify their predictive uncertainty, and has promised to make deep learning more reliable for safety-critical real-world applications. Yet, existing Bayesian deep learning methods fall short of this promise; new methods continue to be evaluated on unrealistic test beds that do not reflect the complexities of downstream real-world tasks that would benefit most from reliable uncertainty quantification. We propose the RETINA Benchmark, a set of real-world tasks that accurately reflect such complexities and …
InfoGatherer: Principled Information Seeking via Evidence Retrieval and Strategic Questioning
University of British Columbia（不列颠哥伦比亚大学）；Vector Institute（向量研究所）；CIFAR AI Chair（CIFAR人工智能主席）；University of Washington（华盛顿大学）；Dalhousie University（达尔豪斯大学）
Quantum annealing (QA) is a hardware-based heuristic optimization and sampling method applicable to discrete undirected graphical models. While similar to simulated annealing, QA relies on quantum, rather than thermal, effects to explore complex search spaces. For many classes of problems, QA is known to offer computational advantages over simulated annealing. Here we report on the ability of recent QA hardware to accelerate training of fully visible Boltzmann machines. We characterize the sampling distribution of QA hardware, and show that in many cases, the quantum distributions differ signi…
CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering
Department of Computer Science, University of Southern California（南加州大学计算机科学系）；Department of Electrical and Computer Engineering, University of Southern California（南加州大学电气与计算机工程系）；Suzanne Dworak-Peck School of Social Work, University of Southern California（南加州大学苏兹安·德沃拉克-佩克社会工作学院）；Department of Psychiatry and the Behavioral Sciences, University of Southern California（南加州大学精神病学与行为科学系）；Annenberg School for Communication, University of Southern California（南加州大学安纳伯格通信学院）
OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safety
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；School of New Media and Communication, Tianjin University（天津大学新媒体与传播学院）；School of Computer and Artificial Intelligence, Zhengzhou University（郑州大学计算机与人工智能学院）
MMPostTrainBench: Benchmarking Autonomous Research for Multimodal Post-Training
University of Science and Technology of China（中国科学技术大学）；Alibaba Token Hub, Alibaba Group（阿里巴巴集团阿里云通义千问团队）；University of the Chinese Academy of Sciences（中国科学院大学）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）
