Large language models (LLMs) have exhibited exciting progress in multiple scenarios, while the huge computational demands hinder their deployments in lots of real-world applications. As an effective means to reduce memory footprint and inference cost, quantization also faces challenges in performance degradation at low bit-widths. Understanding the impact of quantization on LLM capabilities, especially the generalization ability, is crucial. However, the community's main focus remains on the algorithms and models of quantization, with insufficient attention given to whether the quantized models can retain the strong generalization abilities of LLMs. In this work, we fill this gap by providing a comprehensive benchmark suite for this research topic, including an evaluation system, detailed analyses, and a general toolbox. Specifically, based on the dominant pipeline in LLM quantization, we primarily explore the impact of calibration data distribution on the generalization of quantized LLMs and conduct the benchmark using more than 40 datasets within two main scenarios. Based on this benchmark, we conduct extensive experiments with two well-known LLMs (English and Chinese) and four quantization algorithms to investigate this topic in-depth, yielding several counter-intuitive and valuable findings, e.g., models quantized using a calibration set with the same distribution as the test data are not necessarily optimal. Besides, to facilitate future research, we also release a modular-designed toolbox, which decouples the overall pipeline into several separate components, e.g., base LLM module, dataset module, quantizer module, etc. and allows subsequent researchers to easily assemble their methods through a simple configuration. Our benchmark suite is publicly available at https://github.com/TsingmaoAI/MI-optimize
Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency
SINTEF（辛泰夫（挪威研究机构））；Singapore Management University（新加坡管理大学）；Oslo Metropolitan University（奥斯陆城市大学）；Kristiania University of Applied Sciences（克里斯蒂安尼亚应用科技大学）
本研究在Raspberry Pi 4上评估了28个量化LLM在能效、推理性能和准确率上的权衡，通过硬件级能耗分析与多数据集基准测试，为资源受限环境下的可持续LLM部署提供了可操作的见解。
A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Models
Google Research（谷歌研究院）；Google DeepMind（谷歌DeepMind）；University of Alberta（阿尔伯塔大学）；Massachusetts Institute of Technology（麻省理工学院）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Harvard T.H. Chan School of Public Health（哈佛陈曾熙公共卫生学院）
Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMs
Fraunhofer IAIS（弗劳恩霍夫智能分析与信息处理研究所）；University of Bonn（波恩大学）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔机器学习与人工智能研究所）；University of Manitoba（曼尼托巴大学）；University of Central Florida（中佛罗里达大学）；Virginia Tech（弗吉尼亚理工大学）
RPA-Check: A Multi-Stage Automated Framework for Evaluating Dynamic LLM-based Role-Playing Agents
Department of Political Sciences, Communication and International Relations, University of Macerata（马切拉塔大学政治学、传播与国际关系系）；Department of Law, University of Macerata（马切拉塔大学法律系）
A Comparative Study of Student Perspectives on Technical Writing Feedback Quality: Evaluating LLMs, SLMs, and Humans in Computer Science Topics
McMaster University（麦斯特大学）；Department of Computer Science, University of Toronto（多伦多大学计算机科学系）；Research Institute for the Study of University Pedagogy, University of Toronto Mississauga（多伦多大学密西根分校大学教学研究学院）；Department of Mathematical and Computational Sciences, University of Toronto Mississauga（多伦多大学密西根分校数学与计算科学系）
A Typology for Exploring the Mitigation of Shortcut Behavior
Technical University of Darmstadt（达姆施塔特工业大学）；Hessian Center for Artificial Intelligence (hessian.AI)（黑森人工智能中心）；LAION；German Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Centre for Cognitive Science（认知科学中心）
Forecasting day-ahead electricity prices: A review of state-of-the-art algorithms, best practices and an open-access benchmark
Delft University of Technology（代尔夫特理工大学）；Energyville（能源城（能源研究机构））；Flemish Institute for Technological Research (VITO)（弗拉芒技术研究院）；Wrocław University of Science and Technology（弗罗茨瓦夫理工大学）
The language of time: a language model perspective on time-series foundation models
College of Computer Science and Artificial Intelligence, Fudan University（计算机科学与人工智能学院，复旦大学）；Shanghai Key Laboratory of Data Science（上海数据科学重点实验室）；ZCTech；School of Mathematics and Statistics, Beijing Institute of Technology（数学与统计学学院，北京理工大学）
A Multi-Perspective Benchmark and Moderation Model for Evaluating Safety and Adversarial Robustness
Department of Computer and Information Science, Fordham University（福德汉大学计算机与信息科学系）；College of Science and Engineering, Hamad Bin Khalifa University（哈马德·本·卡西姆大学科学与工程学院）；Department of Computer Science and Engineering, Qatar University（卡塔尔大学计算机科学与工程系）
