我们对 Hugging Face 了解多少?一项系统文献综述及对定性论断的定量验证
What do we know about Hugging Face? A systematic literature review and quantitative validation of qualitative claims
AI总结:
本文通过系统文献综述识别 Hugging Face 平台上关于预训练模型复用的 12 条论断,并对其中缺乏定量验证的论断进行测量,证实了 PTM 更替率高于传统软件以及文档质量与流行度强相关等结论。
摘要:
背景:协作式软件包注册表(SPR)是软件供应链的重要组成部分。大量工程工作将 SPR 中的软件包合成到应用程序中。此前的研究已经考察了传统软件的 SPR,例如 NPM(JavaScript)和 PyPI(Python)。预训练模型(PTM)注册表是一类新兴且日益重要的 SPR,因为它们支撑着深度学习供应链。目的:近期的实证研究已经从漏洞、复用过程和演化等方面考察了 PTM 注册表。然而,现有研究尚未对这些内容进行综合,以提供对当前知识的系统性理解。部分现有研究包含缺乏定量分析的定性论断。我们的研究通过提供知识综合和定量分析来填补这些空白。方法:我们首先开展了一项系统文献综述(SLR)。随后我们观察到其中一些论断是定性的。我们识别出与这些论断相关的可量化指标,并进行测量以证实这些论断。结果:通过系统文献综述,我们识别出关于 HuggingFace 平台上 PTM 复用的 12 条论断,其中 4 条缺乏定量验证。我们通过定量分析成功检验了其中 3 条论断,并将其中 1 条与传统软件进行了直接比较。我们的发现用定量测量结果印证了定性论断。我们的发现包括:(1)PTM 的更替率远高于传统软件,表明 PTM 生态系统中存在动态且快速演化的复用环境;以及(2)文档质量与 PTM 流行度之间存在强相关性。结论:我们用具体指标证实了定性研究论断,支持了此前的定性和案例研究。我们的测量结果进一步揭示了 PTM 复用的动态特征,为研究基础设施和新的度量方法提供了启发。
英文摘要:
Background: Collaborative Software Package Registries (SPRs) are an integral part of the software supply chain. Much engineering work synthesizes SPR package into applications. Prior research has examined SPRs for traditional software, such as NPM (JavaScript) and PyPI (Python). Pre-Trained Model (PTM) Registries are an emerging class of SPR of increasing importance, because they support the deep learning supply chain. Aims: Recent empirical research has examined PTM registries in ways such as vulnerabilities, reuse processes, and evolution. However, no existing research synthesizes them to provide a systematic understanding of the current knowledge. Some of the existing research includes qualitative claims lacking quantitative analysis. Our research fills these gaps by providing a knowledge synthesis and quantitative analyses. Methods: We first conduct a systematic literature review (SLR). We then observe that some of the claims are qualitative. We identify quantifiable metrics associated with those claims, and measure in order to substantiate these claims. Results: From our SLR, we identify 12 claims about PTM reuse on the HuggingFace platform, 4 of which lack quantitative validation. We successfully test 3 of these claims through a quantitative analysis, and directly compare one with traditional software. Our findings corroborate qualitative claims with quantitative measurements. Our findings are: (1) PTMs have a much higher turnover rate than traditional software, indicating a dynamic and rapidly evolving reuse environment within the PTM ecosystem; and (2) There is a strong correlation between documentation quality and PTM popularity. Conclusions: We confirm qualitative research claims with concrete metrics, supporting prior qualitative and case study research. Our measures show further dynamics of PTM reuse, inspiring research infrastructure and new measures.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。