The rapid development of large language model (LLM) evaluation methodologies and datasets has led to a profound challenge: integrating state-of-the-art evaluation techniques cost-effectively while ensuring reliability, reproducibility, and efficiency. Currently, there is a notable absence of a unified and adaptable framework that seamlessly integrates various evaluation approaches. Moreover, the reliability of evaluation findings is often questionable due to potential data contamination, with the evaluation efficiency commonly overlooked when facing the substantial costs associated with LLM inference. In response to these challenges, we introduce FreeEval, a modular and scalable framework crafted to enable trustworthy and efficient automatic evaluations of LLMs. Firstly, FreeEval's unified abstractions simplify the integration and improve the transparency of diverse evaluation methodologies, encompassing dynamic evaluation that demand sophisticated LLM interactions. Secondly, the framework integrates meta-evaluation techniques like human evaluation and data contamination detection, which, along with dynamic evaluation modules in the platform, enhance the fairness of the evaluation outcomes. Lastly, FreeEval is designed with a high-performance infrastructure, including distributed computation and caching strategies, enabling extensive evaluations across multi-node, multi-GPU clusters for open-source and proprietary LLMs.
From scratch to silver: Creating trustworthy training data for patent-SDG classification using Large Language Models
Department of Industrial Engineering, Polytechnic University of Turin（工业工程系，都灵理工学院）；Venice School of Management, Ca Foscari University of Venice（威尼斯管理学院，威尼斯福斯卡里大学）
From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations
Department of Computer Science, Yale University, CT, USA（耶鲁大学计算机科学系）；Center for Healthcare Organization and Implementation Research, VA Bedford Health Care（VA贝福德医疗中心健康组织与实施研究中心）；Miner School of Computer and Information Sciences, UMass Lowell, MA, USA（UMass洛厄尔矿尔计算机与信息科学学院）；Manning College of Information and Computer Sciences, UMass Amherst, MA, USA（UMass阿默斯特马宁信息与计算机科学学院）
From Soliloquy to Agora: Memory-Enhanced LLM Agents with Decentralized Debate for Optimization Modeling
Antai College of Economics and Management（上海交通大学安泰经济管理学院）；University of Chicago Booth School of Business（芝加哥大学布斯商学院）；The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（香港中文大学（深圳））；School of Economics and Management（清华大学经济管理学院）
