arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.15317cs.AI
收藏

VERITAS:利用视觉先验与专家融合提升多模态数据质量

VERITAS: Leveraging Vision Priors and Expert Fusion to Improve Multimodal Data

Shanghai Jiao Tong University(上海交通大学);Fudan University(复旦大学)

Tingqiao Xu, Ziru Zeng, Jiayu Chen

AI总结:

VERITAS通过整合视觉先验与多专家LMMs的统计融合,提升多模态SFT数据质量,训练轻量级批评模型并细化答案,在六个基准上优于原始数据。

摘要:

监督微调(SFT)数据的质量对于大型多模态模型(LMMs)的性能至关重要,然而当前的数据增强方法常常因视觉感知不足而出现事实性错误和幻觉。为应对这一挑战,我们提出了VERITAS,一个系统性地整合视觉先验、多个最先进的LMMs以及统计方法来提升SFT数据质量的流程。VERITAS利用视觉识别模型(RAM++)和OCR系统(PP-OCRv4)提取结构化的视觉先验,并将其与图像、问题和答案相结合。三个LMMs(GPT-4o、Gemini-2.5-Pro、豆包-1.5-pro)对原始答案进行评估,提供批评性理由和评分,这些评分通过统计融合形成一个高置信度的共识分数,作为真实标签。利用这一共识,我们通过群体相对策略优化(GRPO)训练一个轻量级批评模型,高效地增强推理能力。随后,每个LMM根据批评意见对原始答案进行细化,生成新的候选答案;我们选择得分最高的一个作为最终细化答案。在六个多模态基准上的实验表明,使用VERITAS处理的数据微调的模型始终优于使用原始数据的模型,尤其是在文本丰富和细粒度推理任务中。我们的批评模型展现出与最先进的LMMs相当的能力,同时效率显著更高。我们发布我们的流程、数据集和模型检查点,以推动多模态数据优化的研究。

英文摘要:

The quality of supervised fine-tuning (SFT) data is crucial for the performance of large multimodal models (LMMs), yet current data enhancement methods often suffer from factual errors and hallucinations due to inadequate visual perception. To address this challenge, we propose VERITAS, a pipeline that systematically integrates vision priors and multiple state-of-the-art LMMs with statistical methods to enhance SFT data quality. VERITAS leverages visual recognition models (RAM++) and OCR systems (PP-OCRv4) to extract structured vision priors, which are combined with images, questions, and answers. Three LMMs (GPT-4o, Gemini-2.5-Pro, Doubao-1.5-pro) evaluate the original answers, providing critique rationales and scores that are statistically fused into a high-confidence consensus score serving as ground truth. Using this consensus, we train a lightweight critic model via Group Relative Policy Optimization (GRPO), enhancing reasoning capabilities efficiently. Each LMM then refines the original answers based on the critiques, generating new candidate answers; we select the highest-scoring one as the final refined answer. Experiments across six multimodal benchmarks demonstrate that models fine-tuned with data processed by VERITAS consistently outperform those using raw data, particularly in text-rich and fine-grained reasoning tasks. Our critic model exhibits enhanced capability comparable to state-of-the-art LMMs while being significantly more efficient. We release our pipeline, datasets, and model checkpoints to advance research in multimodal data optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑