大型语言模型（LLM）的使用正在迅速扩展，开源版本也不断出现，为用户提供了更安全、更具适应性的选择。这些模型使用户无需向第三方提供数据即可保护数据隐私，并且可以针对特定任务进行定制。在本研究中，我们以GPT-4o的输出为基线，比较了多种语言模型在可持续发展目标（Sustainable Development Goal，SDG）映射任务上的性能。所选用于比较的开源模型包括Mixtral、LLaMA 2、LLaMA 3、Gemma和Qwen2。此外，研究还纳入了GPT-4o的更专用版本GPT-4o-mini，以扩展比较范围。鉴于SDG映射任务具有多标签性质，我们采用F1分数、精确率和召回率等指标，并进行微平均，以评估模型性能的不同方面。这些指标由混淆矩阵得出，以确保全面评估。我们通过绘制不同阈值下基于F1分数、精确率和召回率的曲线，对每个模型的性能给出清晰的观察和分析。实验结果表明，LLaMA 2和Gemma仍有显著的改进空间。其他四个模型在性能上没有表现出特别大的差异。全部七个模型的输出均可在Zenodo获取：https://doi.org/10.5281/zenodo.12789375。
英文摘要：
The use of large language models (LLMs) is expanding rapidly, and open-source versions are becoming available, offering users safer and more adaptable options. These models enable users to protect data privacy by eliminating the need to provide data to third parties and can be customized for specific tasks. In this study, we compare the performance of various language models on the Sustainable Development Goal (SDG) mapping task, using the output of GPT-4o as the baseline. The selected open-source models for comparison include Mixtral, LLaMA 2, LLaMA 3, Gemma, and Qwen2. Additionally, GPT-4o-mini, a more specialized version of GPT-4o, was included to extend the comparison. Given the multi-label nature of the SDG mapping task, we employed metrics such as F1 score, precision, and recall with micro-averaging to evaluate different aspects of the models' performance. These metrics are derived from the confusion matrix to ensure a comprehensive evaluation. We provide a clear observation and analysis of each model's performance by plotting curves based on F1 score, precision, and recall at different thresholds. According to the results of this experiment, LLaMA 2 and Gemma still have significant room for improvement. The other four models do not exhibit particularly large differences in performance. The outputs from all seven models are available on Zenodo: https://doi.org/10.5281/zenodo.12789375.
InMind: Evaluating LLMs in Capturing and Applying Individual Human Reasoning Styles
Nankai University（南开大学）；Shanghai Innovation Institute（上海创新研究院）；Shanghai AI Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；Johns Hopkins University（约翰霍普金斯大学）；Wuhan University（武汉大学）；University of Science and Technology of China（中国科学技术大学）
Evaluating LLM-Based Translation of a Low-Resource Technical Language: The Medical and Philosophical Greek of Galen
Department of Classical and Mediterranean Studies, Vanderbilt University（维斯尼尔大学古典与地中海研究系）；Center for Hellenic Studies, Harvard University（哈佛大学希腊研究中心）；Department of Philosophy, Vanderbilt University（维斯尼尔大学哲学系）；Machine Intelligence and Normative Theory (MINT) Lab, Australian National University（澳大利亚国立大学机器智能与规范理论实验室）；Department of the Classics, Harvard University（哈佛大学古典学系）
Assessing Socio-Cultural Alignment and Technical Safety of Sovereign LLMs
Graduate School of Data Science, Dept. of Data Science, Seoul National University（数据科学研究生院、数据科学系、首尔国立大学）；College of Engineering, Dept. of Computer Science and Engineering, Seoul National University（工程学院、计算机科学与工程系、首尔国立大学）
Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluation
Responsible Tech Research IBM Research Yorktown Heights New York USA（负责技术研究 IBM 研究 Yorktown Heights 新 York 美国）；IBM Research（IBM 研究）；The Pennsylvania State University（宾夕法尼亚州立大学）；University of Notre Dame（诺特达美大学）
OPOR-Bench: Evaluating Large Language Models on Online Public Opinion Report Generation
Communication University of China（中国传媒大学）；Harbin Institute of Technology（哈尔滨工业大学）；China Academy of Railway Sciences Corporation Limited（中国铁道科学研究院集团有限公司）；School of Engineering, Santa Clara University（圣克拉拉大学工程学院）
Telco-DPR: A Hybrid Dataset for Evaluating Retrieval Models of 3GPP Technical Specifications
Instituto de Telecomunicações（电信研究所）；CELFINET - A Cyient Company（CELFINET - 赛文特公司）；Instituto Superior de Engenharia Lisboa（里斯本高等工程学院）；Instituto Superior Técnico（里斯本高等技术学院）
CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments
School of Mathematics and Sciences, Fudan University（复旦大学数学科学学院）；Shanghai Center for Mathematical Sciences, Fudan University（复旦大学上海数学中心）；Alibaba Group（阿里巴巴集团）；Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University（复旦大学类脑智能科学与技术研究院）；Center for Applied Mathematics & Shanghai Key Laboratory of Contemporary Applied Mathematics, Fudan University（复旦大学应用数学中心暨上海市现代应用数学重点实验室）
Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification
FDM-Bench: A Comprehensive Benchmark for Evaluating Large Language Models in Additive Manufacturing Tasks
Department of Mechanical Science and Engineering, University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校机械科学与工程系）；Coordinated Science Laboratory, University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校协调科学实验室）；Department of Mechanical and Aerospace Engineering, Rutgers University（罗格斯大学机械与航空航天工程系）；Department of Mechanical Engineering, University of Michigan（密歇根大学机械工程系）
