arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.01962cs.HCcs.AIcs.CLcs.CYcs.ET
收藏

开放式生成模型在以人为中心的数据科学工作中的影响:以事实核查组织为案例研究

The Implications of Open Generative Models in Human-Centered Data Science Work: A Case Study with Fact-Checking Organizations

University of Washington(华盛顿大学)

Robert Wolfe, Tanushree Mitra

AI总结:

本文访谈六大洲20家事实核查组织的24名专业人员,提出生成式AI介入数据科学流程的五环节模型,并分类比较其选择开放模型与专有模型的动机及限制。

摘要:

在学术研究中使用开放式生成语言模型的呼吁,凸显了科学研究对可复现性和透明度的需求。然而,生成式AI的影响远远超出了学术界,因为企业和公共利益组织已经开始将这些模型整合到其数据科学流程中。我们将这一视角扩展到开放模型对组织的影响,特别聚焦于事实核查组织;这些组织使用AI观察和分析大量流传的错误信息,同时还必须确保其工作的可复现性和公正性。我们希望了解事实核查组织在数据科学流程的哪些环节使用开放模型;是什么促使其使用开放模型或专有模型;以及它们对开放模型或专有模型的使用如何为生成式AI社会影响研究提供启示。为回答这些问题,我们对六大洲20家事实核查组织的N=24名专业人员开展了访谈研究。基于这些访谈,我们提出了一个五部分概念模型,说明事实核查组织在何处运用生成式AI来支持或自动化其数据科学流程的组成部分,包括数据摄取、数据分析、数据检索、数据交付和数据共享。随后,我们对事实核查组织使用开放模型的动机以及阻碍其进一步采用开放模型的局限进行了分类,发现它们出于组织自主性、数据隐私与所有权、应用专用性和能力透明度而偏好开放模型。尽管如此,它们仍会使用专有模型,原因在于其在性能、可用性和安全性方面可感知的优势,以及参与新兴生成式AI生态系统所涉及的机会成本。我们的工作为数据驱动型组织中的开放模型提供了新的视角。

英文摘要:

Calls to use open generative language models in academic research have highlighted the need for reproducibility and transparency in scientific research. However, the impact of generative AI extends well beyond academia, as corporations and public interest organizations have begun integrating these models into their data science pipelines. We expand this lens to include the impact of open models on organizations, focusing specifically on fact-checking organizations, which use AI to observe and analyze large volumes of circulating misinformation, yet must also ensure the reproducibility and impartiality of their work. We wanted to understand where fact-checking organizations use open models in their data science pipelines; what motivates their use of open models or proprietary models; and how their use of open or proprietary models can inform research on the societal impact of generative AI. To answer these questions, we conducted an interview study with N=24 professionals at 20 fact-checking organizations on six continents. Based on these interviews, we offer a five-component conceptual model of where fact-checking organizations employ generative AI to support or automate parts of their data science pipeline, including Data Ingestion, Data Analysis, Data Retrieval, Data Delivery, and Data Sharing. We then provide taxonomies of fact-checking organizations' motivations for using open models and the limitations that prevent them for further adopting open models, finding that they prefer open models for Organizational Autonomy, Data Privacy and Ownership, Application Specificity, and Capability Transparency. However, they nonetheless use proprietary models due to perceived advantages in Performance, Usability, and Safety, as well as Opportunity Costs related to participation in emerging generative AI ecosystems. Our work provides novel perspective on open models in data-driven organizations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑