行动前改写：表征并缓解视觉-语言-动作模型中的语言敏感性
Rephrase Before You Act: Characterizing and Mitigating Language Sensitivity in Vision-Language-Action Models
浏览论文内容
中文总结 AI 辅助
本研究揭示视觉-语言-动作模型对指令措辞高度敏感，通过大型语言模型提炼改写规则，在不修改策略的情况下显著提升分布外任务成功率，并零样本泛化至未见任务。
中文摘要 AI 辅助
视觉-语言-动作模型（VLAs）对指令措辞极为敏感，并未继承其构建所基于的视觉-语言模型的语言鲁棒性。一个词的改动即可使成功率发生数十个百分点的变化：$\pi_{0.5}$ 在 LIBERO 任务中对“switch on the stove”的指令100%成功，而对“switch on the hot plate”的指令成功率仅为2%；经过改写增强微调的 $\pi_0$ 检查点仍显示出高达61个百分点的波动。我们通过经统计检验的单次编辑波动和 oracle 短语搜索来表征这种敏感性，结果表明仅凭措辞变化就几乎弥合了分布内与分布外任务之间21个百分点的差距。随后，我们在不修改策略的情况下降低了这种敏感性。由于这种敏感性是系统性的，可以将其表达为显式规则：我们对少量训练任务的多种措辞进行评分，让大型语言模型将证据提炼成十到二十条改写规则，并在部署时根据这些规则对每条传入指令进行一次改写。这些规则使冻结的 $\pi_0$ 在十二个保留任务上，针对对抗性、VLM 生成和人类生成的措辞，相对性能提升了16%至27%，且增益集中在分布外任务上。该流程在 $\pi_{0.5}$ 和 LIBERO 上复现，将微调内成功率从93.6%提升至97.8%。该方法无需重新训练，也无需逐步验证，并可零样本应用于未见任务和指令。项目网站：此 https URL
英文摘要
Vision-language-action models (VLAs) are strikingly sensitive to instruction phrasing and do not inherit the language robustness of the vision-language models they are built on. A one-word edit can move success by tens of points: $π_{0.5}$ turns on a LIBERO stove 100% of the time for "switch on the stove" and 2% for "switch on the hot plate", and a $π_0$ checkpoint finetuned with rephrase augmentation still shows swings of up to 61 points. We characterize this sensitivity with statistically tested single-edit swings and an oracle phrase search, which shows that phrasing alone nearly closes the 21-point gap between in-distribution and out-of-distribution tasks. We then reduce it without modifying the policy. Because the sensitivity is systematic, it can be expressed as explicit rules: we score many phrasings of a few training tasks, have a large language model distill the evidence into ten to twenty rephrasing rules, and at deployment rewrite each incoming instruction once under these rules. The rules improve the frozen $π_0$ by 16 to 27% relative on twelve held-out tasks across adversarial, VLM-generated, and human-generated phrasings, with gains concentrated on out-of-distribution tasks. The pipeline replicates on $π_{0.5}$ and LIBERO, lifting in-finetune success from 93.6% to 97.8%. The method requires no retraining and no per-step verification, and applies zero-shot to unseen tasks and instructions. Project website: https://sttawm.github.io/rephrase-before-you-act
发表机构
- University of California, Los Angeles（加州大学洛杉矶分校）
机构由 AI 辅助整理，请以论文原文为准。
