在AI中理解数据集文档:对 Hugging Face 数据集卡片的大规模分析
Navigating Dataset Documentations in AI: A Large-Scale Analysis of Dataset Cards on Hugging Face
AI总结:
本文大规模分析 Hugging Face 上 7,433 份数据集卡片,揭示文档完成率、内容侧重、主题分布、可复现性及人工质量感知等问题。
摘要:
机器学习的进展与数据集的创建密切相关。尽管数据文档被广泛认为对机器学习(ML)的可靠性、可复现性和透明度至关重要,但我们对当前数据集文档实践仍缺乏系统性的实证理解。为阐明这一问题,本文将 Hugging Face——最大的机器学习模型与数据集共享及协作平台之一——作为一个具有代表性的案例研究。通过分析 Hugging Face 上全部 7,433 份数据集文档,我们的研究概述了 Hugging Face 数据集生态系统,并深入考察了数据集文档实践,得出 5 项主要发现:(1)数据集卡片的完成率表现出显著异质性,并与数据集流行度相关。(2)对数据集卡片中各个部分的细致考察显示,从业者似乎优先填写 Dataset Description(数据集描述)和 Dataset Structure(数据集结构)部分,而 Considerations for Using the Data(使用数据的注意事项)部分拥有内容的比例最低。(3)通过分析每个部分中的子部分,并利用主题建模识别关键主题,我们揭示了各部分讨论的内容,并强调了 Considerations for Using the Data 部分中涵盖技术影响、社会影响以及局限性的重要主题。(4)我们的发现还凸显出,Usage(使用)部分需要改善数据集的可访问性和可复现性。(5)此外,人工标注评估强调,全面的数据集内容在塑造个人对数据集卡片整体质量的感知方面发挥关键作用。总体而言,本研究为通过大规模数据科学分析来考察数据集文档提供了独特视角,并强调机器学习研究需要更详尽的数据集文档。
英文摘要:
Advances in machine learning are closely tied to the creation of datasets. While data documentation is widely recognized as essential to the reliability, reproducibility, and transparency of ML, we lack a systematic empirical understanding of current dataset documentation practices. To shed light on this question, here we take Hugging Face -- one of the largest platforms for sharing and collaborating on ML models and datasets -- as a prominent case study. By analyzing all 7,433 dataset documentation on Hugging Face, our investigation provides an overview of the Hugging Face dataset ecosystem and insights into dataset documentation practices, yielding 5 main findings: (1) The dataset card completion rate shows marked heterogeneity correlated with dataset popularity. (2) A granular examination of each section within the dataset card reveals that the practitioners seem to prioritize Dataset Description and Dataset Structure sections, while the Considerations for Using the Data section receives the lowest proportion of content. (3) By analyzing the subsections within each section and utilizing topic modeling to identify key topics, we uncover what is discussed in each section, and underscore significant themes encompassing both technical and social impacts, as well as limitations within the Considerations for Using the Data section. (4) Our findings also highlight the need for improved accessibility and reproducibility of datasets in the Usage sections. (5) In addition, our human annotation evaluation emphasizes the pivotal role of comprehensive dataset content in shaping individuals' perceptions of a dataset card's overall quality. Overall, our study offers a unique perspective on analyzing dataset documentation through large-scale data science analysis and underlines the need for more thorough dataset documentation in machine learning research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。