Large Language Models (LLMs) have ushered in a new era of text annotation, as their ease-of-use, high accuracy, and relatively low costs have meant that their use has exploded in recent months. However, the rapid growth of the field has meant that LLM-based annotation has become something of an academic Wild West: the lack of established practices and standards has led to concerns about the quality and validity of research. Researchers have warned that the ostensible simplicity of LLMs can be misleading, as they are prone to bias, misunderstandings, and unreliable results. Recognizing the transformative potential of LLMs, this paper proposes a comprehensive set of standards and best practices for their reliable, reproducible, and ethical use. These guidelines span critical areas such as model selection, prompt engineering, structured prompting, prompt stability analysis, rigorous model validation, and the consideration of ethical and legal implications. The paper emphasizes the need for a structured, directed, and formalized approach to using LLMs, aiming to ensure the integrity and robustness of text annotation practices, and advocates for a nuanced and critical engagement with LLMs in social scientific research.
"You are an expert annotator": Automatic Best-Worst-Scaling Annotations for Emotion Intensity Modeling
Merck Data & AI Organization, Merck Group（默克集团默克数据与人工智能部门）；Institut für Maschinelle Sprachverarbeitung, University of Stuttgart（斯图加特大学机器语言处理研究所）；Fundamentals of Natural Language Processing, University of Bamberg（班贝格大学自然语言处理基础研究所）；Merck IT Centre, Merck Group（默克集团默克IT中心）
Charting the European LLM Benchmarking Landscape: A New Taxonomy and a Set of Best Practices
Jožef Stefan Institute（乔塞夫·斯塔芬研究所）；University of Ljubljana（卢布尔雅那大学）；Faculty of Arts, University of Ljubljana（艺术学院，卢布尔雅那大学）；Faculty of Computer and Information Science, University of Ljubljana（计算机与信息科学学院，卢布尔雅那大学）；Institute of Contemporary History（现代史研究所）
Spoken Language Dialogue Systems and Components: Best practice in development and evaluation (DISC 24823) - Periodic Progress Report 1: Basic Details of the Action
