The use of artificial intelligence (AI) in working environments with individuals, known as Human-AI Collaboration (HAIC), has become essential in a variety of domains, boosting decision-making, efficiency, and innovation. Despite HAIC's wide potential, evaluating its effectiveness remains challenging due to the complex interaction of components involved.
This paper provides a detailed analysis of existing HAIC evaluation approaches and develops a fresh paradigm for more effectively evaluating these systems.
Our framework includes a structured decision tree which assists to select relevant metrics based on distinct HAIC modes (AI-Centric, Human-Centric, and Symbiotic). By including both quantitative and qualitative metrics, the framework seeks to represent HAIC's dynamic and reciprocal nature, enabling the assessment of its impact and success. This framework's practicality can be examined by its application in an array of domains, including manufacturing, healthcare, finance, and education, each of which has unique challenges and requirements. Our hope is that this study will facilitate further research on the systematic evaluation of HAIC in real-world applications.
Evaluating Assistive Technologies on a Trade Fair: Methodological Overview and Lessons Learned
Frankfurt University of Applied Sciences(法兰克福应用科学大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);TU Dortmund University(多特蒙德工业大学);University of Duisburg-Essen(杜伊斯堡-埃森大学)
Designing Harvesting Tools for Olive Trees: Methodological Reflections on Exploring and Incorporating Plant Perspectives in the Early Stages of Design Process
Neurodiversity and Technostress: Towards a Multimodal Research Design for Evaluating Subjective, Physiological, and Behavioral Responses
University of Applied Science Burgenland(布尔根兰应用科学大学);Lancaster University(兰卡斯特大学);University of Applied Sciences Upper Austria(上奥地利应用科学大学);Johannes Kepler University(约翰内斯·开普勒大学)