Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores
The Alan Turing Institute(艾伦·图灵研究所);The Centre for Environment Fisheries and Aquaculture Science(环境渔业与水产科学研究中心);School of Computer Science, University of Leeds(利兹大学计算机科学学院)
Large language models (LLMs) are stochastic, and not all models give deterministic answers, even when setting temperature to zero with a fixed random seed. However, few benchmark studies attempt to quantify uncertainty, partly due to the time and cost of repeated experiments. We use benchmarks designed for testing LLMs' capacity to reason about cardinal directions to explore the impact of experimental repeats on mean score and prediction interval. We suggest a simple method for cost-effectively quantifying the uncertainty of a benchmark score and make recommendations concerning reproducible LLM evaluation.
Cognitive Linguistic Identity Fusion Score (CLIFS): A Scalable Cognition-Informed Approach to Quantifying Identity Fusion from Text
Center for Complex Networks and Systems Research, Luddy School of Informatics, Computing, and Engineering, Indiana University Bloomington(复杂网络与系统研究中心,信息学、计算与工程学院,印第安纳大学布卢明顿分校);Cognitive Science Program, Indiana University Bloomington(认知科学项目,印第安纳大学布卢明顿分校);School of Data Science, University of Virginia(数据科学学院,弗吉尼亚大学);CulturePulse, Inc.(CulturePulse公司)
RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty
Lehigh University(莱维大学);University of Notre Dame(诺特大学);Zhejiang Wanli University(浙江万里大学);Squirrel Ai Learning;City University of Hong Kong(香港城市大学);Duke University(杜克大学)
Quantifying and Alleviating the Language Prior Problem in Visual Question Answering
Shandong University(山东大学);Shandong Computer Science Center (National Supercomputer Center in Jinan)(山东计算机科学中心(国家超级计算济南中心));Qilu University of Technology (Shandong Academy of Sciences)(齐鲁工业大学(山东省科学院));National University of Singapore(新加坡国立大学)
Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages
New Jersey Institute of Technology(新泽西理工学院);University of Georgia(佐治亚大学);University of Minnesota at Twin Cities(明尼苏达大学双城分校);Wake Forest University(维克森林大学);Cisco Research(思科研究院)
LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Model
Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative, Institute of Digital Twin, EIT(宁波空间智能与数字衍生关键实验室,数字孪生研究院,EIT);Logic Intelligence Technology(逻辑智能技术);BUPT(北京邮电大学);Xiamen University(厦门大学)