arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.18173cs.HCcs.IR
收藏

人机交互数据工作中的LLMs:弥合信息检索与负责任研究实践之间的差距

LLMs in HCI Data Work: Bridging the Gap Between Information Retrieval and Responsible Research Practices

Neda Taghizadeh Serajeh, Iman Mohammadi, Vittorio Fuccella, Mattia De Rosa

AI总结:

本文提出结合LLMs与结构化文本分析的信息检索系统,从300篇CHI论文中提取实验数据,评估GPT-3.5与Llama-2-70b性能,并探讨LLMs在HCI研究中的风险与伦理指南。

摘要:

在文献综述过程中,从科学论文中高效且准确地提取信息对于快速发展的HCI研究具有重要意义。本文介绍并分析了一种新的信息检索系统,该系统结合了最先进的大型语言模型(LLMs)与结构化文本分析技术,以从HCI文献中提取实验数据,并强调关键要素。随后,我们分析了在研究领域使用LLMs所面临的挑战与风险。我们在整理的数据集上进行了全面分析,该数据集包含300篇CHI 2020-2022论文的指定信息,旨在评估两个大型语言模型GPT-3.5 (text-davinci-003)和Llama-2-70b结合结构化文本分析技术的性能。GPT-3.5模型达到了58%的准确率和7.00的平均绝对误差。相比之下,Llama2模型的准确率为56%,平均绝对误差为7.63。该系统还集成了回答问题的能力,以便处理流线型数据。通过评估LLMs带来的风险与机遇,我们的工作为关于在HCI数据工作中确立LLM使用的方法论有效性和伦理指南的持续对话做出了贡献。

英文摘要:

Efficient and accurate information extraction from scientific papers is significant in the rapidly developing human-computer interaction research in the literature review process. Our paper introduces and analyses a new information retrieval system using state-of-the-art Large Language Models (LLMs) in combination with structured text analysis techniques to extract experimental data from HCI literature, emphasizing key elements. Then We analyze the challenges and risks of using LLMs in the world of research. We performed a comprehensive analysis on our conducted dataset, which contained the specified information of 300 CHI 2020-2022 papers, to evaluate the performance of the two large language models, GPT-3.5 (text-davinci-003) and Llama-2-70b, paired with structured text analysis techniques. The GPT-3.5 model gains an accuracy of 58\% and a mean absolute error of 7.00. In contrast, the Llama2 model indicates an accuracy of 56\% with a mean absolute error of 7.63. The ability to answer questions was also included in the system in order to work with streamlined data. By evaluating the risks and opportunities presented by LLMs, our work contributes to the ongoing dialogue on establishing methodological validity and ethical guidelines for LLM use in HCI data work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑