TEDI：用于分析和比较数据集文档的可信与伦理数据集指标
TEDI: Trustworthy and Ethical Dataset Indicators to Analyze and Compare Dataset Documentation
AI总结：
针对多模态数据集文档透明度不足且难以比较的问题，提出TEDI指标体系，通过143个细粒度指标分析超100个含语音数据集，发现伦理指标记录稀缺且与收集方法相关，为提升数据集透明度奠定基础。
摘要：
数据集透明度是负责任人工智能的关键推动因素，但对影响人工智能应用可信与伦理方面的多模态数据集属性的洞察仍然稀缺，且难以跨数据集进行比较。为应对这一挑战，我们引入了可信与伦理数据集指标（TEDI），以促进对数据集文档进行系统性的实证分析。TEDI包含143个细粒度指标，用于表征多模态数据集及其收集过程的可信与伦理属性。这些指标旨在从数据集文档中提取可验证的信息。利用TEDI，我们手动标注并分析了超过100个包含人类语音的多模态数据集。我们进一步标注了数据来源、规模及模态细节，以深入了解影响跨数据集可信与伦理维度的因素。我们发现，只有少数数据集记录了与同意、隐私和有害内容指标相关的属性和实践。这些及其他伦理指标被提及的程度因数据收集方法而异，通过众包和直接收集方法获取的数据集的文档更可能提及这些指标。抓取以牺牲伦理指标为代价主导了规模，但并非唯一可行的收集方法。我们的方法和实证洞察有助于提高数据集在可信与伦理维度上的透明度，并为未来自动化从数据集文档中提取信息的繁琐任务铺平道路。
英文摘要：
Dataset transparency is a key enabler of responsible AI, but insights into multimodal dataset attributes that impact trustworthy and ethical aspects of AI applications remain scarce and are difficult to compare across datasets. To address this challenge, we introduce Trustworthy and Ethical Dataset Indicators (TEDI) that facilitate the systematic, empirical analysis of dataset documentation. TEDI encompasses 143 fine-grained indicators that characterize trustworthy and ethical attributes of multimodal datasets and their collection processes. The indicators are framed to extract verifiable information from dataset documentation. Using TEDI, we manually annotated and analyzed over 100 multimodal datasets that include human voices. We further annotated data sourcing, size, and modality details to gain insights into the factors that shape trustworthy and ethical dimensions across datasets. We find that only a select few datasets have documented attributes and practices pertaining to consent, privacy, and harmful content indicators. The extent to which these and other ethical indicators are addressed varies based on the data collection method, with documentation of datasets collected via crowdsourced and direct collection approaches being more likely to mention them. Scraping dominates scale at the cost of ethical indicators, but is not the only viable collection method. Our approach and empirical insights contribute to increasing dataset transparency along trustworthy and ethical dimensions and pave the way for automating the tedious task of extracting information from dataset documentation in future.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
