精挑细选：收获丰富、可扩展且可迁移的多模态数据用于指令微调
Cream of the Crop: Harvesting Rich, Scalable and Transferable Multi-Modal Data for Instruction Fine-Tuning
AI总结：
本文提出 mmSSR 数据整理框架，通过 14 项视觉-语言能力评分与交互风格识别，从 260 万条多模态指令数据中高效选择 30%，达到完整性能的 99.1%。
摘要：
预训练大语言模型（LLM）在微调（SFT）阶段仅需极少监督的假设（Zhou 等人，2024）已得到近期数据整理与选择研究进展的证实。然而，由于易受实验设置和验证协议影响，其稳定性和泛化性受到损害，未能超越随机采样（Diddee & Ippolito，2024；Xia 等人，2024b）。建立在 LLM 之上的多模态 LLM（MLLM），加之庞大的 token 数量和数据源异质性的增强，进一步放大了数据选择的重要性与复杂性。为以稳健且高效的方式收获多模态指令数据，我们将质量指标的粒度重新定义为分解出的 14 项视觉-语言相关能力，并引入多模态丰富评分器来评估每个候选数据的能力。为提升多样性，结合对齐阶段的内在目标，我们将交互风格作为多样性指标，并使用多模态丰富风格器识别数据指令模式。通过这种方式，我们的多模态丰富评分器与风格器（mmSSR）确保高分信息以多样化形式传递给用户。mmSSR 不依赖基于嵌入的聚类或贪心采样，可在不同预算约束下高效扩展至数百万条数据，支持针对通用或特定能力获取的定制，并有助于在无需训练的情况下泛化到新领域进行数据整理。在 10 余种实验设置中，经 14 个多模态基准验证，我们证明其相较随机采样、基线策略和最先进选择方法均取得持续提升，仅使用 260 万条数据中的 30% 即达到完整性能的 99.1%。
英文摘要：
The hypothesis that pretrained large language models (LLMs) necessitate only minimal supervision during the fine-tuning (SFT) stage (Zhou et al., 2024) has been substantiated by recent advancements in data curation and selection research. However, their stability and generalizability are compromised due to the vulnerability to experimental setups and validation protocols, falling short of surpassing random sampling (Diddee & Ippolito, 2024; Xia et al., 2024b). Built upon LLMs, multi-modal LLMs (MLLMs), combined with the sheer token volume and heightened heterogeneity of data sources, amplify both the significance and complexity of data selection. To harvest multi-modal instructional data in a robust and efficient manner, we re-define the granularity of the quality metric by decomposing it into 14 vision-language-related capabilities, and introduce multi-modal rich scorers to evaluate the capabilities of each data candidate. To promote diversity, in light of the inherent objective of the alignment stage, we take interaction style as diversity indicator and use a multi-modal rich styler to identify data instruction patterns. In doing so, our multi-modal rich scorers and styler (mmSSR) guarantee that high-scoring information is conveyed to users in diversified forms. Free from embedding-based clustering or greedy sampling, mmSSR efficiently scales to millions of data with varying budget constraints, supports customization for general or specific capability acquisition, and facilitates training-free generalization to new domains for curation. Across 10+ experimental settings, validated by 14 multi-modal benchmarks, we demonstrate consistent improvements over random sampling, baseline strategies and state-of-the-art selection methods, achieving 99.1% of full performance with only 30% of the 2.6M data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
