超越英语：揭示视觉-语言-动作模型中的多语言差距
Beyond English: Uncovering the Multilingual Gap in Vision-Language-Action Models
AI总结：
本研究首次系统探究VLA模型的多语言指令跟随能力，发现英语训练模型在其他语言上性能显著下降，并提出多语言主成分对齐方法缩小差距。
摘要：
视觉-语言-动作模型最近展示了从大规模多模态数据学习通用机器人策略的能力。然而，大多数现有的VLA系统主要使用英语指令进行训练和评估，使得它们理解和执行其他语言指令的能力在很大程度上未被探索。虽然底层的大语言模型通常具备多语言能力，但这些多语言能力在训练过程中是否能迁移到VLA尚不清楚。在这项工作中，我们首次对VLA模型中的多语言指令跟随进行了系统研究。我们首先通过扩展现有基准测试并翻译其指令来构建多语言指令。利用这些指令，我们在模拟环境中评估了几个代表性的VLA模型在一系列任务上的表现。我们的实验揭示了一个显著的多语言差距：主要用英语指令训练的模型在评估其他语言时表现出显著的性能下降，即使底层语言骨干是多语言的。我们提供了若干发现和分析来理解多语言差距。跨语言迁移行为分析表明，性能下降与指令理解和动作执行都相关。表示分析表明，多语言指令引起的表示偏移可能导致了多语言差距。受这些发现的启发，我们进一步探索了提高VLA多语言性能的策略。我们提出了一种简单而有效的多语言微调方法——多语言主成分对齐，该方法利用主成分分析获取主成分子空间并对齐投影后的多语言表示，有效缩小了多语言性能差距。
英文摘要：
Vision-Language-Action models have recently demonstrated promising capabilities in learning generalist robot policies from large-scale multimodal data. However, most existing VLA systems are trained and evaluated primarily with English instructions, leaving their ability to understand and execute instructions in other languages largely unexplored. While the underlying large language models often possess multilingual capabilities, it remains unclear whether these multilingual capabilities transfer to VLAs during training. In this work, we present the first systematic study of multilingual instruction following in VLA models. We first construct multilingual instructions by extending existing benchmarks with translations of their instructions. Using these instructions, we evaluate several representative VLA models across a range of tasks in simulation settings. Our experiments reveal a significant multilingual gap: models trained primarily on English instructions exhibit substantial performance degradation when evaluated on other languages, even when the underlying language backbone is multilingual. We provide several findings and analyses to understand the multilingual gap. Cross-lingual transfer behavior analysis shows that performance drops correlate with both instruction understanding and action execution. Representation analyses suggest that multilingual instruction-caused representation shifts may contribute to the multilingual gap. Motivated by these findings, we further explore strategies to improve multilingual performance in VLAs. We propose a simple yet effective multilingual fine-tuning approach, Multilingual Principal Component Alignment, which leverages Principal Component Analysis to get the principal component subspace and align projected multilingual representations, effectively reducing the multilingual performance gap.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
