Predicting the outcomes of future events is a challenging problem for which a variety of solution methods have been explored and attempted. We present an empirical comparison of a variety of online and offline adaptive algorithms for aggregating experts' predictions of the outcomes of five years of US National Football League games (1319 games) using expert probability elicitations obtained from an Internet contest called ProbabilitySports. We find that it is difficult to improve over simple averaging of the predictions in terms of prediction accuracy, but that there is room for improvement in quadratic loss. Somewhat surprisingly, a Bayesian estimation algorithm which estimates the variance of each expert's prediction exhibits the most consistent superior performance over simple averaging among our collection of algorithms.
You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Zhongguancun Academy(中关村学院);University of Science and Technology of China(中国科学技术大学)
Mixtures of Experts Unlock Parameter Scaling for Deep RL
Mila - Québec AI Institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);Google DeepMind(谷歌DeepMind);University of Oxford(牛津大学);McGill University(麦吉尔大学)
Sample-Efficient Expert Query Control in Active Imitation Learning via Conformal Prediction
Department of Computer Science, University of Southern California(计算机科学系,南加州大学);Department of Information Technology and Electrical Engineering, ETH Zurich(信息科技与电气工程系,苏黎世联邦理工学院)
Estimating Item Difficulty with Large Language Models as Experts
Department of Methodology and Statistics, Tilburg University(蒂尔堡大学方法学与统计学系);Smart Business Technologies(智能商务技术公司);Department of Psychological Methods, University of Amsterdam(阿姆斯特丹大学心理方法系);Prowise Learn, Amsterdam(Prowise Learn公司,阿姆斯特丹);Department of Information and Computing Sciences, Utrecht University(乌得勒支大学信息与计算科学系)
Sparse Mixture-of-Experts are Domain Generalizable Learners
Nanyang Technological University(南洋理工大学);The Hong Kong University of Science and Technology(香港科技大学);Mila-Quebec AI Institute(米拉-魁北克人工智能研究所);Nvidia Research(英伟达研究院)
An AI system to help scientists write expert-level empirical software
Google DeepMind(谷歌DeepMind);Google Research(谷歌研究);Google Platforms and Devices(谷歌平台与设备);Massachusetts Institute of Technology(麻省理工学院);School of Engineering and Applied Sciences, Harvard University(哈佛大学工程与应用科学学院)
本文提出Empirical Research Assistance (ERA)系统,利用大型语言模型和树搜索技术,自动创建高质量的科学软件,以加速计算实验的开发,从而提高科研效率。
Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study
TSAI Lab, University of New Brunswick, Fredericton, NB, Canada(TSAI实验室,新不伦瑞克大学,弗雷德里克顿,NB,加拿大);Analytics Everywhere Lab, University of New Brunswick, Fredericton, NB, Canada(全方位分析实验室,新不伦瑞克大学,弗雷德里克顿,NB,加拿大)
通过实测OLMoE-1B-7B与稠密模型在Apple M2 Pro和Jetson Orin Nano上的性能,发现MoE的活跃参数优势在笔记本电脑上部分实现,在边缘设备上完全丧失,推理成本与总参数而非活跃参数相关。