arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.05036cs.HCcs.AI
收藏

映射人机交互的挑战:利用ChatGPT进行大规模洞察挖掘的应用与评估

Mapping the Challenges of HCI: An Application and Evaluation of ChatGPT for Mining Insights at Scale

University of Oulu, Center for Applied Computing, Faculty of Information Technology and Electrical Engineering(奥卢大学,应用计算中心,信息科技与电气工程学院);University of Jyväskylä, Software and Communications Engineering, Faculty of Information Technology(耶夫尼耶斯基大学,软件与通信工程,信息科技学院)

Jonas Oppenlaender, Joonas Hämäläinen

AI总结:

本文评估ChatGPT在从大规模人机交互文献中提取研究挑战的应用,通过两步法提取并筛选挑战,发现与伦理和可及性等领域的强关联,以及人机协作等领域的不足,验证了LLM在大规模定性分析中的实用性。

摘要:

大型语言模型(LLMs)越来越多地用于分析任务,但其在现实应用中的有效性仍缺乏充分研究,部分原因在于专有模型的不透明性。我们评估了ChatGPT(GPT-3.5和GPT-4)在从2023年ACM CHI会议论文集的879篇论文中提取研究挑战的实用任务。通过两步方法,我们首先使用GPT-3.5提取候选挑战,然后使用GPT-4筛选每篇论文中最相关的挑战。这一过程产生了4,392个研究挑战,涵盖113个主题,通过主题建模进行组织并以交互式可视化呈现。我们比较了识别出的挑战与已确立的HCI重大挑战和联合国可持续发展目标,发现伦理和可及性等领域的强关联,以及人机协作等领域的不足。一项任务特定的评估与人类评估者确认,提取的陈述在研究挑战方面具有几乎完美的一致性(κ=0.97)。两步方法在完整文献库上成本约为50美元,表明LLM为大规模定性文本分析提供了实用手段,尤其适用于研究想法的原型设计和多视角分析文献。

英文摘要:

Large language models (LLMs) are increasingly used for analytical tasks, yet their effectiveness in real-world applications remains underexamined, partly due to the opacity of proprietary models. We evaluate ChatGPT (GPT-3.5 and GPT-4) on the practical task of extracting research challenges from a large scholarly corpus in Human-Computer Interaction (HCI). Using a two-step approach, we first apply GPT-3.5 to extract candidate challenges from the 879 papers in the 2023 ACM CHI Conference proceedings, then use GPT-4 to select the most relevant challenges per paper. This process yielded 4,392 research challenges across 113 topics, which we organized through topic modeling and present in an interactive visualization. We compare the identified challenges with previously established HCI grand challenges and the United Nations Sustainable Development Goals, finding both strong alignment in areas such as ethics and accessibility, and gaps in areas such as human-AI collaboration. A task-specific evaluation with human raters confirmed near-perfect agreement that the extracted statements represent plausible research challenges (\k{appa} = 0.97). The two-step approach proved cost-effective at approximately US$50 for the full corpus, suggesting that LLMs offer a practical means for qualitative text analysis at scale, particularly for prototyping research ideas and examining corpora from multiple analytical perspectives.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑