This document provides the annotation guidelines for MaiBaam, a Bavarian corpus manually annotated with part-of-speech (POS) tags, syntactic dependencies, and German lemmas. MaiBaam belongs to the Universal Dependencies (UD) project, and our annotations elaborate on the general and German UD version 2 guidelines. In this document, we detail how to preprocess and tokenize Bavarian data, provide an overview of the POS tags and dependencies we use, explain annotation decisions that would also apply to closely related languages like German, and lastly we introduce and motivate decisions that are specific to Bavarian grammar.
MaiBaam: A Multi-Dialectal Bavarian Universal Dependency Treebank
Center for Information and Language Processing, LMU Munich（慕尼黑大学信息与语言处理中心）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；IT University of Copenhagen（哥本哈根信息技术大学）
This document describes the annotation guidelines used to construct the Turku Paraphrase Corpus. These guidelines were developed together with the corpus annotation, revising and extending the guidelines regularly during the annotation work. Our paraphrase annotation scheme uses the base scale 1-4, where labels 1 and 2 are used for negative candidates (not paraphrases), while labels 3 and 4 are paraphrases at least in the given context if not everywhere. In addition to base labeling, the scheme is enriched with additional subcategories (flags) for categorizing different types of paraphrases in…
National Electronics and Computer Technology Center（国家电子与计算机技术中心）
This report presents the annotation guideline for LST20, a large-scale corpus with multiple layers of linguistic annotation for Thai language processing. Our guideline consists of five layers of linguistic annotation: word segmentation, POS tagging, named entities, clause boundaries, and sentence boundaries. The dataset complies to the CoNLL-2003-style format for ease of use. LST20 Corpus offers five layers of linguistic annotation as aforementioned. At a large scale, it consists of 3,164,864 words, 288,020 named entities, 248,962 clauses, and 74,180 sentences, while it is annotated with 16 di…
Università degli Studi di Cagliari（卡利亚里大学）；Università degli Studi di Torino（都灵大学）；Dublin City University（都柏林城市大学）；University of Stuttgart（斯图加特大学）；Universitat Politècnica de València（瓦伦西亚理工大学）；University of Mannheim（曼海姆大学）；Leibniz-Institut für Deutsche Sprache Mannheim（曼海姆莱布尼茨德语研究所）；INRIA Paris（法国国家信息与自动化研究所巴黎中心）；Georgetown University（乔治城大学）
This article presents a discussion on the main linguistic phenomena which cause difficulties in the analysis of user-generated texts found on the web and in social media, and proposes a set of annotation guidelines for their treatment within the Universal Dependencies (UD) framework of syntactic analysis. Given on the one hand the increasing number of treebanks featuring user-generated content, and its somewhat inconsistent treatment in these resources on the other, the aim of this article is twofold: (1) to provide a condensed, though comprehensive, overview of such treebanks -- based on avai…
Allen Institute for AI（艾伦人工智能研究所）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Yale University（耶鲁大学）
While human evaluation remains best practice for accurately judging the faithfulness of automatically-generated summaries, few solutions exist to address the increased difficulty and workload when evaluating long-form summaries. Through a survey of 162 papers on long-form summarization, we first shed light on current human evaluation practices surrounding long-form summaries. We find that 73% of these papers do not perform any human evaluation on model-generated summaries, while other works face new difficulties that manifest when dealing with long documents (e.g., low inter-annotator agreemen…
Accenture Labs（埃森哲实验室）；Indian Institute of Science（印度科学学院）；Arizona State University（亚利桑那州立大学）；Duke University（杜克大学）
Understanding causal narratives communicated in clinical notes can help make strides towards personalized healthcare. Extracted causal information from clinical notes can be combined with structured EHR data such as patients' demographics, diagnoses, and medications. This will enhance healthcare providers' ability to identify aspects of a patient's story communicated in the clinical notes and help make more informed decisions. In this work, we propose annotation guidelines, develop an annotated corpus and provide baseline scores to identify types and direction of causal relations between a pai…
The Abstract Meaning Representation (AMR) formalism, designed originally for English, has been adapted to a number of languages. We build on previous work proposing the annotation of AMR in Spanish, which resulted in the release of 50 Spanish AMR annotations for the fictional text "The Little Prince." In this work, we present the first sizable, general annotation project for Spanish Abstract Meaning Representation. Our approach to annotation makes use of Spanish rolesets from the AnCora-Net lexicon and extends English AMR with semantic features specific to Spanish. In addition to our guideline…
University of St.Gallen（圣加仑大学）；University of Passau（帕绍大学）；University of Kassel（卡塞尔大学）
In this paper, we present a novel annotation approach to capture claims and premises of arguments and their relations in student-written persuasive peer reviews on business models in German language. We propose an annotation scheme based on annotation guidelines that allows to model claims and premises as well as support and attack relations for capturing the structure of argumentative discourse in student-written peer reviews. We conduct an annotation study with three annotators on 50 persuasive essays to evaluate our annotation scheme. The obtained inter-rater agreement of $α=0.57$ for argum…
In argumentative discourse, persuasion is often achieved by refuting or attacking others arguments. Attacking is not always straightforward and often comprise complex rhetorical moves such that arguers might agree with a logic of an argument while attacking another logic. Moreover, arguer might neither deny nor agree with any logics of an argument, instead ignore them and attack the main stance of the argument by providing new logics and presupposing that the new logics have more value or importance than the logics present in the attacked argument. However, no existing studies in the computati…
During the past decade, neural network models have made tremendous progress on in-domain semantic role labeling (SRL). However, performance drops dramatically under the out-of-domain setting. In order to facilitate research on cross-domain SRL, this paper presents MuCPAD, a multi-domain Chinese predicate-argument dataset, which consists of 30,897 sentences and 92,051 predicates from six different domains. MuCPAD exhibits three important features. 1) Based on a frame-free annotation methodology, we avoid writing complex frames for new predicates. 2) We explicitly annotate omitted core arguments…
National University of Singapore（新加坡国立大学）；Koc University（科奇大学）；Rijksuniversiteit Groningen（格罗宁根大学）；Radboud University（拉德堡德大学）；National Institute of Japanese Literature（日本文学研究所）；Birmingham City University（伯明翰城市大学）；Instituto Politécnico Nacional（国立理工学院）；Mersin University（梅尔辛大学）；University of East Anglia（东英吉利大学）；ETH Zürich（苏黎世联邦理工学院）
Despite the importance of understanding causality, corpora addressing causal relations are limited. There is a discrepancy between existing annotation guidelines of event causality and conventional causality corpora that focus more on linguistics. Many guidelines restrict themselves to include only explicit relations or clause-based arguments. Therefore, we propose an annotation schema for event causality that addresses these concerns. We annotated 3,559 event sentences from protest event news with labels on whether it contains causal relations or not. Our corpus is known as the Causal News Co…
Grammarly；University of Mannheim（曼海姆大学）；Dataminr, Inc.（Dataminr公司）
Computational models of argument quality (AQ) have focused primarily on assessing the overall quality or just one specific characteristic of an argument, such as its convincingness or its clarity. However, previous work has claimed that assessment based on theoretical dimensions of argumentation could benefit writers, but developing such models has been limited by the lack of annotated data. In this work, we describe GAQCorpus, the first large, domain-diverse annotated corpus of theory-based AQ. We discuss how we designed the annotation task to reliably collect a large number of judgments with…
Hochschule für Technik und Wirtschaft Berlin（柏林工程与经济应用技术大学）；Alice Salomon Hochschule Berlin（柏林爱丽丝·萨洛蒙应用科学大学）
Over the course of the COVID-19 pandemic, existing conspiracy theories were refreshed and new ones were created, often interwoven with antisemitic narratives, stereotypes and codes. The sheer volume of antisemitic and conspiracy theory content on the Internet makes data-driven algorithmic approaches essential for anti-discrimination organizations and researchers alike. However, the manifestation and dissemination of these two interrelated phenomena is still quite under-researched in scholarly empirical research of large text corpora. Algorithmic approaches for the detection and classification …
RAAMove: A Corpus for Analyzing Moves in Research Article Abstracts
Faculty of Information Technology, Beijing University of Technology（北京工业大学信息技术学院）；School of Computer Science, Beijing Institute of Technology（北京理工大学计算机学院）
IT University of Copenhagen（哥本哈根信息技术大学）；Center for Information and Language Processing (CIS), LMU Munich（慕尼黑大学信息与语言处理中心（CIS））
Skill Extraction (SE) is an important and widely-studied task useful to gain insights into labor market dynamics. However, there is a lacuna of datasets and annotation guidelines; available datasets are few and contain crowd-sourced labels on the span-level or labels from a predefined skill inventory. To address this gap, we introduce SKILLSPAN, a novel SE dataset consisting of 14.5K sentences and over 12.5K annotated spans. We release its respective guidelines created over three different sources annotated for hard and soft skills by domain experts. We introduce a BERT baseline (Devlin et al.…
Hebrew University of Jerusalem（耶路撒冷希伯来大学）；Georgetown University（乔治城大学）
This is the annotation manual for Universal Conceptual Cognitive Annotation (UCCA; Abend and Rappoport, 2013), specifically the Foundational Layer. UCCA is a graph-based semantic annotation scheme based on typological linguistic principles. It has been applied to several languages; for ease of exposition these guidelines give examples mainly in English. New annotators may wish to start with the tutorial on the UCCA framework (Abend et al., 2020). Further resources are available at the project homepage: https://universalconceptualcognitiveannotation.github.io
Refining and Reusing Annotation Guidelines for LLM Annotation
The Graduate University for Advanced Studies, SOKENDAI（Graduate University for Advanced Studies, SOKENDAI）；National Institute of Informatics (NII)（National Institute of Informatics）；BioData Science Initiative (BSI), National Institute of Genetics (NIG)（BioData Science Initiative (BSI), National Institute of Genetics）
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Brown University（布朗大学）；Google（谷歌）
Large-scale, high-quality corpora are critical for advancing research in coreference resolution. However, existing datasets vary in their definition of coreferences and have been collected via complex and lengthy guidelines that are curated for linguistic experts. These concerns have sparked a growing interest among researchers to curate a unified set of guidelines suitable for annotators with various backgrounds. In this work, we develop a crowdsourcing-friendly coreference annotation methodology, ezCoref, consisting of an annotation tool and an interactive tutorial. We use ezCoref to re-anno…
Instituto Politécnico Nacional (IPN)（国立理工学院）；Centro de Investigación en Computación (CIC)（计算机研究中心）
Hope is characterized as openness of spirit toward the future, a desire, expectation, and wish for something to happen or to be true that remarkably affects human's state of mind, emotions, behaviors, and decisions. Hope is usually associated with concepts of desired expectations and possibility/probability concerning the future. Despite its importance, hope has rarely been studied as a social media analysis task. This paper presents a hope speech dataset that classifies each tweet first into "Hope" and "Not Hope", then into three fine-grained hope categories: "Generalized Hope", "Realistic Ho…
Coconstructions in spoken data: UD annotation guidelines and first results
University of Bologna（博洛尼亚大学）；Université Paris Nanterre, Modyco（巴黎楠泰尔大学，莫迪科）；University of Ljubljana（卢布尔雅那大学）；Jozef Stefan Institute（约瑟夫·斯特凡研究所）；Université de Lorraine, CNRS, Inria, LORIA（洛林大学，法国国家科学研究中心，法国国家信息与自动化研究所，洛林计算机科学实验室）；Masaryk University（马萨里克大学）
Guidelines for Fine-grained Sentence-level Arabic Readability Annotation
Computational Approaches to Modeling Language Lab, New York University Abu Dhabi（纽约大学阿布扎克分校计算语言学建模实验室）；Zai Arabic Language Research Centre, Zayed University（扎伊阿拉伯语言研究中心，扎耶德大学）；Abu Dhabi Arabic Language Centre（阿布扎克阿拉伯语言中心）
TIB Leibniz Information Centre for Science and Technology（莱布尼茨科学与技术信息中心）
Purpose: The aim of this work is to normalize the NLPCONTRIBUTIONS scheme (henceforward, NLPCONTRIBUTIONGRAPH) to structure, directly from article sentences, the contributions information in Natural Language Processing (NLP) scholarly articles via a two-stage annotation methodology: 1) pilot stage - to define the scheme (described in prior work); and 2) adjudication stage - to normalize the graphing model (the focus of this paper). Design/methodology/approach: We re-annotate, a second time, the contributions-pertinent information across 50 prior-annotated NLP scholarly articles in terms of a d…
DementiaBank-Emotion: A Multi-Rater Emotion Annotation Corpus for Alzheimer's Disease Speech (Version 1.0)
Sue & Bill Gross School of Nursing, University of California, Irvine (UCI)（加州大学厄尔布雷特分校护理学院）；Donald Bren School of Information and Computer Sciences, UCI, USA（加州大学厄尔布雷特分校信息与计算机科学学院）；Smart Forward, Rancho Palos Verdes, USA（Smart Forward公司）；Dept. of Psychiatry and Human Behavior, UCI, USA（加州大学厄尔布雷特分校精神病学与人类行为系）；Amore Senior Living, Laguna Niguel, USA（Amore Senior Living公司）；School of Medicine, University of California, Irvine（加州大学厄尔布雷特分校医学院）；Purdue University, West Lafayette, USA（普渡大学）；The George Washington University, Washington, D.C., USA（乔治华盛顿大学）
CMDAG: A Chinese Metaphor Dataset with Annotated Grounds as CoT for Boosting Metaphor Generation
University of California, San Diego（加利福尼亚大学圣迭戈分校）；Massachusetts Institute of Technology（麻省理工学院）；HKUST（香港科技大学）；University of Manchester（曼彻斯特大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Waterloo（滑铁卢大学）；Multimodal Art Projection Research Community（多模态艺术投影研究社区）
University of St.Gallen（圣加仑大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Passau（帕绍大学）；University of Kassel（卡塞尔大学）
We present an annotation approach to capturing emotional and cognitive empathy in student-written peer reviews on business models in German. We propose an annotation scheme that allows us to model emotional and cognitive empathy scores based on three types of review components. Also, we conducted an annotation study with three annotators based on 92 student essays to evaluate our annotation scheme. The obtained inter-rater agreement of α=0.79 for the components and the multi-π=0.41 for the empathy scores indicate that the proposed annotation scheme successfully guides annotators to a substanti…
This document provides extensive guidelines and examples for Rhetorical Structure Theory (RST) annotation in Mandarin Chinese. The guideline is divided into three sections. We first introduce preprocessing steps to prepare data for RST annotation. Secondly, we discuss syntactic criteria to segment texts into Elementary Discourse Units (EDUs). Lastly, we provide examples to define and distinguish discourse relations in different genres. We hope that this reference manual can facilitate RST annotations in Chinese and accelerate the development of the RST framework across languages.
Repurposing Annotation Guidelines to Instruct LLM Annotators: A Case Study
The Graduate University for Advanced Studies, SOKENDAI（综合研究大学院大学）；National Institute of Informatics（国立信息学研究所）；National Library of Medicine（美国国家医学图书馆）；Joint Support-Center for Data Science Research（数据科学联合支持中心）；Japanese-French Laboratory of Informatics, CNRS（日法信息学联合实验室（法国国家科学研究中心））；Nantes University（南特大学）
iPOE: Interpretable Prompt Optimization via Explanations
Fundamentals of Natural Language Processing, University of Bamberg, Germany（自然语言处理基础，巴姆堡大学，德国）；Leibniz-Institut für Psychologie (ZPID), Trier, Germany（莱比锡心理学研究所（ZPID），特里尔，德国）
University of Montpellier（蒙彼利埃大学）；CIRAD（法国农业国际合作研究发展中心）；INRAE（法国国家农业、食品与环境研究院）；AgroParisTech（巴黎高科农业学院）；CNRS（法国国家科学研究中心）；Sciensano（比利时国家公共卫生研究院）；Public Service of Wallonia（瓦隆公共服务局）
This paper describes a method for annotation of epidemiological information in animal disease-related news articles. The annotation guidelines are generic and aim to embrace all animal or zoonotic infectious diseases, regardless of the pathogen involved or its way of transmission (e.g. vector-borne, airborne, by contact). The framework relies on the successive annotation of all the sentences from a news article. The annotator evaluates the sentences in a specific epidemiological context, corresponding to the publication of the news article.
