This paper has been peer-reviewed and presented at the "CHI 2024 Workshop on LLMs as Research Tools: Applications and Evaluations in HCI Data Work, May 12, 2024, Honolulu, HI, USA."
In this position paper, we discuss the potential for leveraging LLMs as interactive research tools to facilitate collaboration between human coders and AI to effectively annotate online risk data at scale. Collaborative human-AI labeling is a promising approach to annotating large-scale and complex data for various tasks. Yet, tools and methods to support effective human-AI collaboration for data annotation are under-studied. This gap is pertinent because co-labeling tasks need to support a two-way interactive discussion that can add nuance and context, particularly in the context of online risk, which is highly subjective and contextualized. Therefore, we provide some of the early benefits and challenges of using LLMs-based tools for risk annotation and suggest future directions for the HCI research community to leverage LLMs as research tools to facilitate human-AI collaboration in contextualized online data annotation. Our research interests align very well with the purposes of the LLMs as Research Tools workshop to identify ongoing applications and challenges of using LLMs to work with data in HCI research. We anticipate learning valuable insights from organizers and participants into how LLMs can help reshape the HCI community's methods for working with data.
Enhancing Text Annotation through Rationale-Driven Collaborative Few-Shot Prompting
Institute of Artificial Intelligence and Future Networks, Beijing Normal University(北京师范大学人工智能与未来网络研究院);Hong Kong Baptist University(香港浸会大学);BNU-HKBU United International College(北京师范大学-香港浸会大学联合国际学院);College of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院)
Towards Event Extraction with Massive Types: LLM-based Collaborative Annotation and Partitioning Extraction
School of Computer Science and Technology, University of Chinese Academy of Sciences(中国科学院大学计算机科学与技术学院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);State Key Laboratory of Geo-Information Engineering(地理信息工程国家重点实验室)
Dynamic Framework for Collaborative Learning: Leveraging Advanced LLM with Adaptive Feedback Mechanisms
Swinburne University of Technology, Melbourne, Australia(斯winburne技术大学,墨尔本,澳大利亚);Western Sydney University, Australia(西澳大学,澳大利亚);Department of Computer Science, LUMS, Pakistan(计算机科学系,拉瓦尔大学,巴基斯坦);UAE University, UAE(阿联酋大学,阿联酋);Zayed University, UAE(扎耶德大学,阿联酋);School of Business Western Sydney University, Australia(商学院,西澳大学,澳大利亚)
MELT: Towards Automated Multimodal Emotion Data Annotation by Leveraging LLM Embedded Knowledge
Technical University of Munich(慕尼黑技术大学);Imperial College London(伦敦帝国理工学院);CHI – Chair of Health Informatics(健康信息学系);Munich Centre for Machine Learning(慕尼黑机器学习中心);GLAM – Group on Language, Audio, & Music(语言、音频与音乐小组)
Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialogues
Polytechnic of Turin(都灵理工大学);University of Verona(维罗纳大学);ENSTA Paris(巴黎国立高等先进技术学校);Institut Polytechnique de Paris(巴黎综合理工学院);Fondazione Bruno Kessler(布鲁诺·凯塞勒基金会)
A Collaborative Content Moderation Framework for Toxicity Detection based on Conformalized Estimates of Annotation Disagreement
TECNALIA, Basque Research and Technology Alliance (BRTA)(TECNALIA巴斯克研究与技术联盟);University of the Basque Country (UPV/EHU)(巴斯克国家大学);Faculty of Engineering, University of Deusto(德乌斯托大学工程学院)
"The explanation makes sense": An Empirical Study on LLM Performance in News Classification and its Influence on Judgment in Human-AI Collaborative Annotation
Measuring Cognitive Engagement in Collaborative Discourse with an Extended ICAP Framework: Comparing Human Annotation, In-Context Learning, and Reflective LLM Agents
HistoColAi: An Open-Source Web Platform for Collaborative Digital Histology Image Annotation with AI-Driven Predictive Integration
Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, Universitat Politècnica de València(瓦伦西亚理工大学以人为中心技术大学研究院);valgrAI: Valencian Graduate School and Research Network of Artificial Intelligence(瓦伦西亚人工智能研究生院及研究网络);ÉTS Montréal(蒙特利尔工程学院)
A Qualitative Analysis of Common Practices in Annotations: A Taxonomy and Design Space
University of Utah(犹他大学);University of Oklahoma(俄克拉荷马大学);University of South Florida(南佛罗里达大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校)