我们为Large Language Models (LLMs，大型语言模型)（如Llama-2和Mistral）引入了一个新颖的评估框架，重点是将Precision和Recall指标从图像生成引入到文本生成中。该方法无需对齐语料库，即可对生成文本的质量和多样性进行细致评估。通过对state-of-the-art语言模型进行全面评估，本研究揭示了它们在开放式生成任务上的性能新见解，这些见解是传统基准无法充分捕捉的。研究结果突显了生成样本的质量与多样性之间的权衡，尤其是当模型在指令数据集上微调或结合人类反馈时。本工作扩展了基于分布的NLP评估工具包，为当前LLMs在生成多样化和高质量文本方面的实际能力与挑战提供了见解。我们发布了代码和数据。
英文摘要：
We introduce a novel evaluation framework for Large Language Models (LLMs) such as \textsc{Llama-2} and \textsc{Mistral}, focusing on importing Precision and Recall metrics from image generation to text generation. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora. By conducting a comprehensive evaluation of state-of-the-art language models, the study reveals new insights into their performance on open-ended generation tasks, which are not adequately captured by traditional benchmarks. The findings highlight a trade-off between the quality and diversity of generated samples, particularly when models are fine-tuned on instruction dataset or with human feedback. This work extends the toolkit for distribution-based NLP evaluation, offering insights into the practical capabilities and challenges that current LLMs face in generating diverse and high-quality text. We release our code and data.
LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video
Beijing University of Posts and Telecommunications（北京邮电大学）；Zhongguancun Academy（中关村学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；The Chinese University of Hong Kong（香港中文大学）；Xi’an Jiaotong University（西安交通大学）
Exploring the Impact of Corpus Diversity on Financial Pretrained Language Models
Department of Applied Artificial Intelligence, Hanyang University（应用人工智能系，翰阳大学）；Division of Computer Science, Hanyang University（计算机科学系，翰阳大学）；Ramply Inc.（Ramply公司）
How Diversely Can Language Models Solve Problems? Exploring the Algorithmic Diversity of Model-Generated Code
School of Computer Science and Engineering, Kyungpook National University（庆北国立大学计算机科学与工程学院）；POSTECH（浦项科技大学）；Department of Artificial Intelligence, Yonsei University（延世大学人工智能系）
Department of Computer Science, Cornell University（康奈尔大学计算机科学系）；Statistics and Data Science Center, Massachusetts Institute of Technology（麻省理工学院统计与数据科学中心）；Department of Statistics, Cornell University（康奈尔大学统计学系）
Tracking, exploring and analyzing recent developments in German-language online press in the face of the coronavirus crisis: cOWIDplus Analysis and cOWIDplus Viewer
Leibniz Institute for the German Language（莱布尼茨德语语言研究所）
Exploring Sampling Techniques for Generating Melodies with a Transformer Language Model
Institute of Computational Perception, Johannes Kepler University Linz（约翰开普勒林茨大学计算感知研究所）；Sony Computer Science Laboratories (CSL)（索尼计算机科学实验室）；LIT AI Lab, Linz Institute of Technology（林茨理工学院LIT人工智能实验室）
