LaV-CoT：具有多方面奖励优化的语言感知视觉CoT
LaV-CoT: Language-Aware Visual CoT with Multi-Aspect Reward Optimization for Real-World Multilingual VQA
AI总结：
LaV-CoT通过多阶段推理流程和多方面奖励优化，提升多语言视觉问答的准确性和泛化能力，实现在公开数据集上的9.5%准确率提升。
摘要：
随着大视觉语言模型（VLMs）的发展，其在多语言视觉问答（mVQA）中的能力显著提升。链式思维（CoT）推理已被证明能增强可解释性和复杂推理能力。然而，大多数现有方法主要依赖文本CoT，对多语言多模态推理支持有限，限制了其在现实应用中的部署。为解决这一差距，我们引入LaV-CoT，首个具有多方面奖励优化的语言感知视觉CoT框架。LaV-CoT包含一个可解释的多阶段推理流程，包括带边界框的文本摘要、语言识别、空间对象级描述和逐步逻辑推理。随后，我们设计了一种自动化数据整理方法，通过迭代生成、修正和优化生成多语言CoT注释，实现可扩展的高质量训练数据。为了提高推理和泛化能力，LaV-CoT采用结合监督微调（SFT）与语言感知组相对策略优化（GRPO）的两阶段训练范式，由可验证的多方面奖励指导，包括语言一致性、结构准确性及语义对齐。在MMM B、多语言MMBench和MTVQA等公开数据集上的广泛评估显示，LaV-CoT在与之相似规模的开源基线上实现了高达约9.5%的准确率提升，甚至超越了2倍规模的模型约2.6%。此外，LaV-CoT在高级专有模型如GPT-4o-0513和Gemini-2.5-flash上表现更优。我们进一步进行了在线A/B测试以验证我们的方法在现实数据上的有效性，突显了其在工业部署中的有效性。我们的代码可在此链接获取：https://github.com/HJNVR/LaV-CoT
英文摘要：
As large vision language models (VLMs) advance, their capabilities in multilingual visual question answering (mVQA) have significantly improved. Chain-of-thought (CoT) reasoning has been proven to enhance interpretability and complex reasoning. However, most existing approaches rely primarily on textual CoT and provide limited support for multilingual multimodal reasoning, constraining their deployment in real-world applications. To address this gap, we introduce LaV-CoT, the first Language-aware Visual CoT framework with Multi-Aspect Reward Optimization. LaV-CoT incorporates an interpretable multi-stage reasoning pipeline consisting of Text Summary with Bounding Box (BBox), Language Identification, Spatial Object-level Captioning, and Step-by-step Logical Reasoning. Following this reasoning pipeline, we design an automated data curation method that generates multilingual CoT annotations through iterative generation, correction, and refinement, enabling scalable and high-quality training data. To improve reasoning and generalization, LaV-CoT adopts a two-stage training paradigm combining Supervised Fine-Tuning (SFT) with Language-aware Group Relative Policy Optimization (GRPO), guided by verifiable multi-aspect rewards including language consistency, structural accuracy, and semantic alignment. Extensive evaluations on public datasets including MMMB, Multilingual MMBench, and MTVQA show that LaV-CoT achieves up to ~9.5% accuracy improvements over open-source baselines of similar size and even surpasses models with 2$\times$ larger scales by ~2.6%. Moreover, LaV-CoT outperforms advanced proprietary models such as GPT-4o-0513 and Gemini-2.5-flash. We further conducted an online A/B test to validate our method on real-world data, highlighting its effectiveness for industrial deployment. Our code is available at this link: https://github.com/HJNVR/LaV-CoT
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
