Manually annotated data is key to developing text-mining and information-extraction algorithms. However, human annotation requires considerable time, effort and expertise. Given the rapid growth of biomedical literature, it is paramount to build tools that facilitate speed and maintain expert quality. While existing text annotation tools may provide user-friendly interfaces to domain experts, limited support is available for image display, project management, and multi-user team annotation. In response, we developed TeamTat (teamtat.org), a web-based annotation tool (local setup available), equipped to manage team annotation projects engagingly and efficiently. TeamTat is a novel tool for managing multi-user, multi-label document annotation, reflecting the entire production life cycle. Project managers can specify annotation schema for entities and relations and select annotator(s) and distribute documents anonymously to prevent bias. Document input format can be plain text, PDF or BioC, (uploaded locally or automatically retrieved from PubMed or PMC), and output format is BioC with inline annotations. TeamTat displays figures from the full text for the annotators convenience. Multiple users can work on the same document independently in their workspaces, and the team manager can track task completion. TeamTat provides corpus-quality assessment via inter-annotator agreement statistics, and a user-friendly interface convenient for annotation review and inter-annotator disagreement resolution to improve corpus quality.
"The explanation makes sense": An Empirical Study on LLM Performance in News Classification and its Influence on Judgment in Human-AI Collaborative Annotation
Measuring Cognitive Engagement in Collaborative Discourse with an Extended ICAP Framework: Comparing Human Annotation, In-Context Learning, and Reflective LLM Agents
A Collaborative Content Moderation Framework for Toxicity Detection based on Conformalized Estimates of Annotation Disagreement
TECNALIA, Basque Research and Technology Alliance (BRTA)(TECNALIA巴斯克研究与技术联盟);University of the Basque Country (UPV/EHU)(巴斯克国家大学);Faculty of Engineering, University of Deusto(德乌斯托大学工程学院)
HistoColAi: An Open-Source Web Platform for Collaborative Digital Histology Image Annotation with AI-Driven Predictive Integration
Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, Universitat Politècnica de València(瓦伦西亚理工大学以人为中心技术大学研究院);valgrAI: Valencian Graduate School and Research Network of Artificial Intelligence(瓦伦西亚人工智能研究生院及研究网络);ÉTS Montréal(蒙特利尔工程学院)
End-to-End Trainable Non-Collaborative Dialog System
University of California, Davis(加州大学戴维斯分校)
端到端的面向任务的对话模型在协作任务上取得了很好的性能,用户愿意与系统协调完成给定的任务。而在非协作环境中,例如谈判和说服,用户和系统没有共同的目标。因此,与协作任务相比,人们使用社交内容在这些非协作环境中建立融洽关系和信任,以推进他们的目标。为了处理社交内容,我们引入了一个层次化的意图注释方案,它可以推广到不同的非协作对话任务。在TransferTransfo(Wolf et al. 2019)的基础上,我们提出了一个端到端的神经网络模型,以生成不同的连贯响应。我们的模型利用意图和语义槽作为中间句子表示来指导生成过程。此外,我们设计了一个过滤器,以选择适当的反应的基础上,这些中间表示是否适合设计的任务和会话的限制。我们的非协作对话模型指导用户完成任务,同时保持他们参与。我们在新提出的ANTISCAM数据集和现有的PERSUASIONFORGOOD数据集上测试我们的方法。自动和人工评估都表明,我们的模型在这两个非协作任务中的表现优于多个基线。
Confident Teacher, Confident Student? A Novel User Study Design for Investigating the Didactic Potential of Explanations and their Impact on Uncertainty
Berliner Hochschule für Technik(柏林应用科学大学);Einstein Center Digital Future(爱因斯坦数字未来中心)