arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.15320cs.DLcs.AIcs.CL
收藏

使用大型语言模型丰富机器学习数据集文档

Using Large Language Models to Enrich the Documentation of Datasets for Machine Learning

Internet Interdisciplinary Institute (IN3), Universitat Oberta de Catalunya (UOC)(开放加泰罗尼亚大学互联网跨学科研究所);Luxembourg Institute of Science and Technology(卢森堡科学技术研究院);University of Luxembourg(卢森堡大学)

Joan Giner-Miguelez, Abel Gómez, Jordi Cabot

AI总结:

本研究利用大型语言模型和提示策略,从数据集文档中自动提取关键维度信息,以生成机器可读文档,提升数据集可发现性、法规合规性及模型质量,实验显示GPT3.5优于FLAN-UL2。

摘要:

近期如欧洲人工智能法案等监管举措以及机器学习社区的相关声音强调,为了可信人工智能,需要沿多个关键维度描述数据集,例如来源过程和社会关切。然而,这些信息通常以非结构化文本形式出现在随附文档中,妨碍了其自动化分析和处理。在本工作中,我们探索使用大型语言模型(LLM)和一组提示策略,从文档中自动提取这些维度,并以此丰富数据集描述。我们的方法可以帮助数据发布者和实践者创建机器可读的文档,以提高其数据集的可发现性,评估其是否符合当前人工智能法规,并改善基于这些数据集训练的机器学习模型的整体质量。在本文中,我们在两个科学期刊(Nature的Scientific Data和Elsevier的Data in Brief)上发表的12篇科学数据集论文上评估了该方法,使用了两种不同的LLM(GPT3.5和Flan-UL2)。结果表明,我们的提示提取策略具有良好准确性。具体结果因维度而异,但总体而言,GPT3.5的准确性(81.21%)略高于FLAN-UL2(69.13%),尽管它更容易产生幻觉。我们已在开源仓库中发布了实现我们方法的开源工具和复制包,包括实验代码和结果。

英文摘要:

Recent regulatory initiatives like the European AI Act and relevant voices in the Machine Learning (ML) community stress the need to describe datasets along several key dimensions for trustworthy AI, such as the provenance processes and social concerns. However, this information is typically presented as unstructured text in accompanying documentation, hampering their automated analysis and processing. In this work, we explore using large language models (LLM) and a set of prompting strategies to automatically extract these dimensions from documents and enrich the dataset description with them. Our approach could aid data publishers and practitioners in creating machine-readable documentation to improve the discoverability of their datasets, assess their compliance with current AI regulations, and improve the overall quality of ML models trained on them. In this paper, we evaluate the approach on 12 scientific dataset papers published in two scientific journals (Nature's Scientific Data and Elsevier's Data in Brief) using two different LLMs (GPT3.5 and Flan-UL2). Results show good accuracy with our prompt extraction strategies. Concrete results vary depending on the dimensions, but overall, GPT3.5 shows slightly better accuracy (81,21%) than FLAN-UL2 (69,13%) although it is more prone to hallucinations. We have released an open-source tool implementing our approach and a replication package, including the experiments' code and results, in an open-source repository.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑