Progress Note Understanding -- Assessment and Plan Reasoning: Overview of the 2022 N2C2 Track 3 Shared Task
University of Wisconsin Madison（威斯康星大学麦迪逊分校）；Loyola University Chicago（芝加哥洛约拉大学）；Boston Children’s Hospital（波士顿儿童医院）；Harvard University（哈佛大学）；George Mason University（乔治梅森大学）
Yanjun Gao, Dmitriy Dligach, Timothy Miller, Matthew M Churpek, Ozlem Uzuner, Majid Afshar
Daily progress notes are common types in the electronic health record (EHR) where healthcare providers document the patient's daily progress and treatment plans. The EHR is designed to document all the care provided to patients, but it also enables note bloat with extraneous information that distracts from the diagnoses and treatment plans. Applications of natural language processing (NLP) in the EHR is a growing field with the majority of methods in information extraction. Few tasks use NLP methods for downstream diagnostic decision support. We introduced the 2022 National NLP Clinical Challenge (N2C2) Track 3: Progress Note Understanding - Assessment and Plan Reasoning as one step towards a new suite of tasks. The Assessment and Plan Reasoning task focuses on the most critical components of progress notes, Assessment and Plan subsections where health problems and diagnoses are contained. The goal of the task was to develop and evaluate NLP systems that automatically predict causal relations between the overall status of the patient contained in the Assessment section and its relation to each component of the Plan section which contains the diagnoses and treatment plans. The goal of the task was to identify and prioritize diagnoses as the first steps in diagnostic decision support to find the most relevant information in long documents like daily progress notes. We present the results of 2022 n2c2 Track 3 and provide a description of the data, evaluation, participation and system performance.
Spotify（声田）；CLARIN ERIC；Dublin City University（都柏林城市大学）
The Podcast Track is new at the Text Retrieval Conference (TREC) in 2020. The podcast track was designed to encourage research into podcasts in the information retrieval and NLP research communities. The track consisted of two shared tasks: segment retrieval and summarization, both based on a dataset of over 100,000 podcast episodes (metadata, audio, and automatic transcripts) which was released concurrently with the track. The track generated considerable interest, attracted hundreds of new registrations to TREC and fifteen teams, mostly disjoint between search and summarization, made final s…
Johns Hopkins University Human Language Technology Center of Excellence（约翰霍普金斯大学人机语言技术卓越中心）；University of Glasgow（格拉斯哥大学）；Allen Institute for AI（人工智能研究院）
Microsoft AI & Research（微软人工智能研究院）；University College London（伦敦大学学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
This is the second year of the TREC Deep Learning Track, with the goal of studying ad hoc ranking in the large training data regime. We again have a document retrieval task and a passage retrieval task, each with hundreds of thousands of human-labeled training queries. We evaluate using single-shot TREC-style evaluation, to give us a picture of which ranking methods work best when large data is available, with much more comprehensive relevance labeling on the small number of test queries. This year we have further evidence that rankers with BERT-style pretraining outperform other rankers in th…
Microsoft AI & Research（微软人工智能与研究部）；University College London（伦敦大学学院）；NIST（美国国家标准与技术研究院（NIST））
The Deep Learning Track is a new track for TREC 2019, with the goal of studying ad hoc ranking in a large data regime. It is the first track with large human-labeled training sets, introducing two sets corresponding to two tasks, each with rigorous TREC-style blind evaluation and reusable test sets. The document retrieval task has a corpus of 3.2 million documents with 367 thousand training queries, for which we generate a reusable test set of 43 queries. The passage retrieval task has a corpus of 8.8 million passages with 503 thousand training queries, for which we generate a reusable test se…
University of Glasgow（格拉斯哥大学）；Microsoft Research（微软研究院）；Carnegie Mellon University（卡内基梅隆大学）
The Conversational Assistance Track (CAsT) is a new track for TREC 2019 to facilitate Conversational Information Seeking (CIS) research and to create a large-scale reusable test collection for conversational search systems. The document corpus is 38,426,252 passages from the TREC Complex Answer Retrieval (CAR) and Microsoft MAchine Reading COmprehension (MARCO) datasets. Eighty information seeking dialogues (30 train, 50 test) are an average of 9 to 10 questions long. Relevance assessments are provided for 30 training topics and 20 test topics. This year 21 groups submitted a total of 65 runs …
Microsoft（微软公司）；University College London（伦敦大学学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Neural Magic Inc（Neural Magic公司）；University of Waterloo（滑铁卢大学）；NIST（国家标准与技术研究院）
本文概述了 TREC 2022 深度学习赛道，利用 MS MARCO 数据集和扩充后的集合构建了更完整的段落检索测试集，并证实大规模预训练的深度神经排序模型仍优于传统方法。
Overview of the MedHopQA track at BioCreative IX: track description, participation and evaluation of systems for multi-hop medical question answering
National Library of Medicine (NLM), National Institutes of Health (NIH)（美国国家医学图书馆（NLM）、国家卫生研究院（NIH））；University of Illinois at Urbana Champaign（伊利诺伊大学厄巴纳-香槟分校）；Korea University（韩国大学）；VNU University of Engineering and Technology, Hanoi, Vietnam（越南河内工程大学）；Concordia University, Montreal, QC, CA（蒙特利尔大学）；Institute of Biomedical Chemistry (IBMC), 10 bld. 8, Pogodinskaya str., 119121 Moscow, Russia（俄罗斯生物医学化学研究所（IBMC））；LASIGE, Departamento de Informática, Faculdade de Ciências, Universidade de Lisboa, 1749-016 Lisbon, Portugal（葡萄牙里斯本大学 LASIGE 实验室）；Faculty of Engineering, Computer Engineering Department Cairo University（埃及开罗大学工程学院）；Menzies School of Health Research, Charles Darwin University, NT, Australia（澳大利亚查尔斯达尔文大学梅恩兹健康研究中心）；CaresAI, Australia（澳大利亚 CaresAI）
本文介绍了BioCreative IX MedHopQA共享任务，旨在评估大型语言模型在多跳推理中的表现，通过构建1000个挑战性问题，展示了检索增强生成策略的重要性，并提供了公开数据集和评估结果。
Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Tsinghua University（清华大学）；Yidu Cloud（医渡云（北京）技术有限公司）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Baidu（百度）；School of Computer Science & Technology, Soochow University（苏州大学计算机科学与技术学院）；Gowild（狗尾草智能科技有限公司）；University of Chinese Academy of Sciences（中国科学院大学）；Ant Financial（蚂蚁金服（杭州）网络技术有限公司）；Southeast University（东南大学）；Peking University（北京大学）
Knowledge graph models world knowledge as concepts, entities, and the relationships between them, which has been widely used in many real-world tasks. CCKS 2019 held an evaluation track with 6 tasks and attracted more than 1,600 teams. In this paper, we give an overview of the knowledge graph evaluation tract at CCKS 2019. By reviewing the task definition, successful methods, useful resources, good strategies and research challenges associated with each task in CCKS 2019, this paper can provide a helpful reference for developing knowledge graph applications and conducting future knowledge grap…
Microsoft（微软公司）；University College London（伦敦大学学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Neural Magic Inc（Neural Magic公司）；University of Waterloo（滑铁卢大学）
University of Eastern Finland（东芬兰大学）；Carnegie Mellon University（卡内基梅隆大学）；Tencent AI Lab（腾讯人工智能实验室）；NetEase Games AI Lab（网易游戏人工智能实验室）；University of Geneva（日内瓦大学）；HSE University（高等经济大学）；JetBrains Research（JetBrains研究院）
Reinforcement learning competitions advance the field by providing appropriate scope and support to develop solutions toward a specific problem. To promote the development of more broadly applicable methods, organizers need to enforce the use of general techniques, the use of sample-efficient methods, and the reproducibility of the results. While beneficial for the research community, these restrictions come at a cost -- increased difficulty. If the barrier for entry is too high, many potential participants are demoralized. With this in mind, we hosted the third edition of the MineRL ObtainDia…
The authors present an overview of a hierarchical framework for coordinating task- and motion-level operations in multirobot systems. Their framework is based on the idea of using simple temporal networks to simultaneously reason about precedence/causal constraints required for task-level coordination and simple temporal constraints required to take some kinematic constraints of robots into account. In the plan-generation phase, the framework provides a computationally scalable method for generating plans that achieve high-level tasks for groups of robots and take some of their kinematic const…
