This article reports on a survey carried out across the Natural Language Processing (NLP) community. The survey aimed to capture the opinions of the research community on issues surrounding shared tasks, with respect to both participation and organisation. Amongst the 175 responses received, both positive and negative observations were made. We carried out and report on an extensive analysis of these responses, which leads us to propose a Shared Task Organisation Checklist that could support future participants and organisers. The proposed Checklist is flexible enough to accommodate the wide diversity of shared tasks in our field and its goal is not to be prescriptive, but rather to serve as a tool that encourages shared task organisers to foreground ethical behaviour, beginning with the common issues that the 175 respondents deemed important. Its usage would not only serve as an instrument to reflect on important aspects of shared tasks, but would also promote increased transparency around them.
Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Task
University of Maryland(马里兰大学);University of Washington(华盛顿大学);Microsoft Research New York(微软纽约研究院)
本文介绍了第一次研讨会上建立语言概括的自然语言处理系统,以及相关的Build It Break It,语言版共享任务的总结。这次研讨会的目的是将NLP和语言学的研究人员聚集在一起,共同完成一项任务,旨在测试NLP系统在其训练数据分布之外的通用性。我们描述了共享任务的动机、设置和参与,讨论了一些突出的结果,并讨论了经验教训。
AraFinNLP 2024: The First Arabic Financial NLP Shared Task
Birzeit University(比尔宰特大学);Lancaster University(兰卡斯特大学);King Saud University(沙特国王大学);Mohammed VI Polytechnic University(穆罕默德六世理工大学);Carnegie Mellon University(卡内基梅隆大学)
LeWiDi-2025 at NLPerspectives: Third Edition of the Learning with Disagreements Shared Task
Fondazione Bruno Kessler(布鲁诺·科塞拉基金会);LMU Munich & MCML(慕尼黑大学及MCML);Università Milano Bicocca(米兰Bicocca大学);Università di Torino(都灵大学);University of Gothenburg(哥德堡大学);Queen Mary University of London(伦敦大学玛丽女王学院);Utrecht University(乌得勒支大学)
Findings of the NLP4IF-2021 Shared Tasks on Fighting the COVID-19 Infodemic and Censorship Detection
Qatar Computing Research Institute, HBKU(卡塔尔计算研究所(哈马德·本·哈利法大学));University of Padova(帕多瓦大学);Hamad bin Khalifa University(哈马德·本·哈利法大学);Montclair State University(蒙特克莱尔州立大学);Sofia University “St. Kliment Ohridski”(索非亚大学“圣克莱门特·奥里季斯基”)
Overview of the Problem List Summarization (ProbSum) 2023 Shared Task on Summarizing Patients' Active Diagnoses and Problems from Electronic Health Record Progress Notes
University of Wisconsin(威斯康星大学);Loyola University Chicago(芝加哥洛约拉大学);Boston Children’s Hospital(波士顿儿童医院);Harvard Medical School(哈佛医学院)
Language Resources in Spanish for Automatic Text Simplification across Domains
Universidad Autónoma de Madrid(马德里自治大学);Consejo Superior de Investigaciones Científicas(西班牙高等科学研究理事会);Universidad Nacional de Educación a Distancia(国立远程教育大学)
NLP-CIC @ DIACR-Ita: POS and Neighbor Based Distributional Models for Lexical Semantic Change in Diachronic Italian Corpora
Instituto Politécnico Nacional(国家理工学院);Instituto Caro y Cuervo(卡罗-奎尔沃研究所)
我们在EVALITA 2020的DIACR-Ita共享任务中展示了我们关于意大利语无监督词汇语义变化的系统和发现。该任务是确定目标词是否随着时间的推移而演变其含义,仅依赖于来自两个特定时间数据集的原始文本。我们提出了两个模型,代表目标词的各个时期预测变化的话,使用阈值和投票计划。我们的第一个模型仅仅依赖于词性的使用和距离测量的集合。第二个模型使用词嵌入表示法提取相邻词在空间上的相对距离,并提出“绝对差平均值”来估计词汇语义变化。我们的模型取得了令人满意的结果,在DIACR-Ita竞赛中排名第三。此外,我们使用第二个模型的k_neighbor参数进行实验,以比较使用“绝对差的平均值”与Hamilton et al.(2016)中使用的余弦距离的影响。
SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)
Fondazione Bruno Kessler(布鲁诺·凯塞勒基金会);Heriot-Watt University(赫瑞瓦特大学);Queen Mary University of London(伦敦大学玛丽皇后学院);Università di Torino(都灵大学);Università Bocconi(博科尼大学);University of Munich(慕尼黑大学)
Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art
Technical University of Darmstadt(达姆施塔特工业大学);Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA)(康拉德·楚泽学习与智能系统卓越学院(ELIZA));University of Cambridge(剑桥大学)
DAMO-NLP at SemEval-2022 Task 11: A Knowledge-based System for Multilingual Named Entity Recognition
Zhejiang University(浙江大学);Singapore University of Technology and Design(新加坡科技设计大学);DAMO Academy, Alibaba Group(阿里巴巴集团达摩院);ShanghaiTech University(上海科技大学)
Progress Note Understanding -- Assessment and Plan Reasoning: Overview of the 2022 N2C2 Track 3 Shared Task
University of Wisconsin Madison(威斯康星大学麦迪逊分校);Loyola University Chicago(芝加哥洛约拉大学);Boston Children’s Hospital(波士顿儿童医院);Harvard University(哈佛大学);George Mason University(乔治梅森大学)
A Parameter-Efficient Transfer Learning Approach through Multitask Prompt Distillation and Decomposition for Clinical NLP
Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida(佛罗里达大学医学院健康结果与生物医学信息学系);Cancer Informatics Shared Resource, University of Florida Health Cancer Center(佛罗里达大学健康癌症中心癌症信息学共享资源)
DAMO-NLP at SemEval-2023 Task 2: A Unified Retrieval-augmented System for Multilingual Named Entity Recognition
DAMO Academy, Alibaba Group(阿里巴巴集团达摩院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Information Science and Technology, ShanghaiTech University(上海科技大学信息科学与技术学院);Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院)
Towards More Accurate Prediction of Human Empathy and Emotion in Text and Multi-turn Conversations by Combining Advanced NLP, Transformers-based Networks, and Linguistic Methodologies
Edinburgh Clinical NLP at MEDIQA-CORR 2024: Guiding Large Language Models with Hints
School of Informatics, University of Edinburgh(爱丁堡大学信息学院);Usher Institute, University of Edinburgh(爱丁堡大学厄舍研究所);Edinburgh Futures Institute, University of Edinburgh(爱丁堡未来研究所);School of Literatures, Languages and Cultures, University of Edinburgh(爱丁堡大学文学、语言与文化学院)