arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.28301cs.LG
收藏

LIBERO-Para:一种用于视觉语言动作模型中同义词鲁棒性诊断的基准和指标

LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models

Soongsil University(崇实大学);Chung-Ang University(中央大学)

Chanyoung Kim, Minwoo Kim, Minseok Kang, Hyunwoo Kim, Dahuin Jung

AI总结:

本文提出LIBERO-Para基准,通过独立变化动作表达和物体参考,分析语言泛化能力。研究发现,不同规模的VLA模型在同义词替换下性能下降22-52个百分点,主要由物体层面的词汇变化导致。提出PRIDE指标量化同义词难度,揭示模型对任务识别的依赖。

摘要:

视觉语言动作(VLA)模型通过利用预训练的视觉语言骨干网络,在机器人操作中实现了强大的性能。然而,在下游机器人场景中,它们通常仅用有限的数据进行微调,导致过度拟合特定的指令格式,从而忽视了对改写指令的鲁棒性。为研究这一差距,我们引入了LIBERO-Para,一个受控的基准,通过独立变化动作表达和物体参考,对语言泛化能力进行细粒度分析。在七个VLA配置(0.6B-7.5B)中,我们观察到在改写情况下性能下降了22-52个百分点。这种下降主要由物体层面的词汇变化驱动:即使简单的同义词替换也会导致大幅下降,表明模型依赖于表层匹配而非语义基础。此外,80-96%的失败源于规划层面的轨迹偏离,而非执行错误,表明改写会破坏任务识别。二元成功率将所有改写视为同等,模糊了模型在不同难度水平上是否表现一致或依赖于较简单的案例。为此,我们提出了PRIDE指标,该指标利用语义和语法因素量化改写难度。我们的基准和相应代码可在:https://github.com/cau-hai-lab/LIBERO-Para获取。

英文摘要:

Vision-Language-Action (VLA) models achieve strong performance in robotic manipulation by leveraging pre-trained vision-language backbones. However, in downstream robotic settings, they are typically fine-tuned with limited data, leading to overfitting to specific instruction formulations and leaving robustness to paraphrased instructions underexplored. To study this gap, we introduce LIBERO-Para, a controlled benchmark that independently varies action expressions and object references for fine-grained analysis of linguistic generalization. Across seven VLA configurations (0.6B-7.5B), we observe consistent performance degradation of 22-52 pp under paraphrasing. This degradation is primarily driven by object-level lexical variation: even simple synonym substitutions cause large drops, indicating reliance on surface-level matching rather than semantic grounding. Moreover, 80-96% of failures arise from planning-level trajectory divergence rather than execution errors, showing that paraphrasing disrupts task identification. Binary success rate treats all paraphrases equally, obscuring whether models perform consistently across difficulty levels or rely on easier cases. To address this, we propose PRIDE, a metric that quantifies paraphrase difficulty using semantic and syntactic factors. Our benchmark and corresponding code are available at: https://github.com/cau-hai-lab/LIBERO-Para

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑