The TextClass Benchmark project is an ongoing, continuous benchmarking process that aims to provide a comprehensive, fair, and dynamic evaluation of LLMs and transformers for text classification tasks. This evaluation spans various domains and languages in social sciences disciplines engaged in NLP and text-as-data approach. The leaderboards present performance metrics and relative ranking using a tailored Elo rating system. With each leaderboard cycle, novel models are added, fixed test sets can be replaced for unseen, equivalent data to test generalisation power, ratings are updated, and a Meta-Elo leaderboard combines and weights domain-specific leaderboards. This article presents the rationale and motivation behind the project, explains the Elo rating system in detail, and estimates Meta-Elo across different classification tasks in social science disciplines. We also present a snapshot of the first cycle of classification tasks on incivility data in Chinese, English, German and Russian. This ongoing benchmarking process includes not only additional languages such as Arabic, Hindi, and Spanish but also a classification of policy agenda topics, misinformation, among others.
Drawing Conclusions from Draws: Rethinking Preference Semantics in Arena-Style LLM Evaluation
Centre for Artificial Intelligence, University College London（人工智能中心，伦敦大学学院）；University of Waterloo（滑铁卢大学）；University of Copenhagen（哥本哈根大学）；Research and Development Center for Large Language Models, National Institute of Informatics（大语言模型研究与发展中心，日本信息处理学会）
Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generation
ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz（林茨ELLIS单位和LIT AI实验室，机器学习研究所，林茨约瑟夫·克里格大学）；NXAI GmbH（NXAI公司）
Shanghai Jiao Tong University（上海交通大学）；PengCheng Laboratory（鹏城实验室）；Harvard Medical School（哈佛医学院）；Shanghai AI Laboratory（上海人工智能实验室）；China University of Mining and Technology（中国矿业大学）
本文提出基于对抗性棋类游戏的基准框架，通过 Qi Town 平台和循环赛评估 LLM 综合能力，发现其在高压力下比人类更乐观且适应性强，但技能发挥存在不稳定性。
Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition
Zhejiang University（浙江大学）；ZJU-Hangzhou Global Scientific and Technological Innovation Center（浙江大学杭州国际科创中心）；Shanghai Electric Group Co., Ltd. Central Academe（上海电气集团股份有限公司中央研究院）；City University of Hong Kong（香港城市大学）；Shanghai Institute for Advanced Study of Zhejiang University（浙江大学上海高等研究院）；Zhoushan-Zhejiang University Ocean Research Center（舟山-浙江大学海洋研究中心）
BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution
Monash University（墨尔本大学）；CSIRO’s Data61（CSIRO的数据61）；Purdue University（普渡大学）；Independent（独立）；HKUST (Guangzhou)（香港科技大学（广州））；UCSD（加州大学圣地亚哥分校）；UVA（弗吉尼亚大学）；CNRS, France（法国国家科学研究中心）；IBM；Cisco（思科）；Comenius University in Bratislava（布拉提斯拉瓦康门纽斯大学）；University of Notre Dame（Notre Dame大学）；Uber；Tano Labs（Tano实验室）；NUS（国立大学新加坡）；Institute of Automation, CAS（中国科学院自动化研究所）；Tencent AI Lab（腾讯AI实验室）；University of Washington（华盛顿大学）；Nevsky Collective（Nevsky集体）；ETH Zurich（苏黎世联邦理工学院）；Detomo Inc（Detomo公司）；University of Oxford（牛津大学）；UIUC（伊利诺伊大学香槟分校）；Google（谷歌）；NVIDIA；Singapore Management University（新加坡管理学院）；ServiceNow Research（ServiceNow研究）；Hugging Face
