不让任何语言数据掉队：Hugging Face生态系统中中日韩语言数据集的比较研究
No Language Data Left Behind: A Comparative Study of CJK Language Datasets in the Hugging Face Ecosystem
AI总结：
本研究比较分析HuggingFace上3300多个中日韩数据集，揭示文化、机构与社区驱动的差异，并提出改进文档、许可与跨语言共享的策略，以促进东亚LLM开发。
摘要：
自然语言处理（NLP）的最新进展凸显了高质量数据集在构建大型语言模型（LLM）中的关键作用。然而，尽管针对英语存在大量资源和分析，东亚语言——尤其是中文、日文和韩文（CJK）——的格局仍然零散且研究不足，尽管这些语言合计服务超过16亿使用者。为弥补这一空白，我们从跨语言视角考察HuggingFace生态系统，重点关注文化规范、研究环境和机构实践如何影响数据集的可获取性与质量。基于3300多个数据集，我们采用定量与定性方法，考察这些因素如何驱动中文、日文和韩文NLP社区各自独特的创建与策展模式。我们的发现凸显了中文数据集的规模化且往往由机构驱动的特性、韩文NLP中草根社区主导的发展，以及日文语料库对娱乐和亚文化的侧重。通过揭示这些模式，我们提出了增强数据集文档、许可清晰度和跨语言资源共享的实用策略——最终引导东亚地区更有效且更具文化适应性的LLM开发。最后，我们讨论了未来数据集策展与合作的最佳实践，旨在加强所有三种语言的资源建设。
英文摘要：
Recent advances in Natural Language Processing (NLP) have underscored the crucial role of high-quality datasets in building large language models (LLMs). However, while extensive resources and analyses exist for English, the landscape for East Asian languages - particularly Chinese, Japanese, and Korean (CJK) - remains fragmented and underexplored, despite these languages together serving over 1.6 billion speakers. To address this gap, we investigate the HuggingFace ecosystem from a cross-linguistic perspective, focusing on how cultural norms, research environments, and institutional practices shape dataset availability and quality. Drawing on more than 3,300 datasets, we employ quantitative and qualitative methods to examine how these factors drive distinct creation and curation patterns across Chinese, Japanese, and Korean NLP communities. Our findings highlight the large-scale and often institution-driven nature of Chinese datasets, grassroots community-led development in Korean NLP, and an entertainment- and subculture-focused emphasis on Japanese collections. By uncovering these patterns, we reveal practical strategies for enhancing dataset documentation, licensing clarity, and cross-lingual resource sharing - ultimately guiding more effective and culturally attuned LLM development in East Asia. We conclude by discussing best practices for future dataset curation and collaboration, aiming to strengthen resource development across all three languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
