脑条件动作策略用于神经运动解码

中文摘要 AI 辅助

运动脑机接口（BCI）旨在解码运动意图，使瘫痪患者能够控制外部设备。神经运动解码通常学习从神经活动到运动学数据的任务特定映射，但仍受限于稀缺的配对神经-动作数据。我们提出BrainVLA，一个通过语言介导的对齐，利用预训练的视觉-语言-动作（VLA）模型实现神经运动解码的框架。BrainVLA通过利用VLA策略减轻对稀缺配对神经-动作数据的依赖。我们首先构建VLA兼容数据集，包括配对的神经活动、动作信号、语言指令和渲染的视觉观察。然后，我们通过LoRA微调将OpenVLA-OFT策略适配到目标动作空间。为了建立有效的接口，使神经活动能够将运动意图传达给适配的VLA策略并指导动作生成，我们通过神经-语言对齐训练一个神经编码器，使用语言表示作为语义目标，从神经活动中捕获潜在的运动意图。由此产生的神经表示作为内源性意图信号，指导VLA策略生成可执行的动作，而视觉观察提供关于任务状态演变的补充信息。BrainVLA在两个具有不同动作维度的神经运动数据集上使用因果展开解码进行评估。它在跨会话解码$R^2$和任务成功率方面优于评估的基线，同时展示了高训练数据效率。这些结果为神经运动解码通过脑条件VLA策略利用大规模机器人先验建立了一条路径。

英文摘要

Motor brain-computer interfaces (BCIs) aim to decode motor intention, enabling people with paralysis to control external devices. Neural motor decoding typically learns task-specific mappings from neural activity to kinematics, yet remains constrained by scarce paired neural-action data. We propose BrainVLA, a framework that enables neural motor decoding by drawing on a pretrained vision-language-action (VLA) model through language-mediated alignment. BrainVLA mitigates reliance on scarce paired neural-action data by leveraging VLA policies. We first construct VLA-compatible datasets including paired neural activity, action signals, language instructions, and rendered visual observations. Then, we adapt the OpenVLA-OFT policy to the target action spaces through LoRA fine-tuning. To establish an effective interface through which neural activity can convey motor intention to adapted VLA policies and guide action generation, we train a neural encoder via neural-language alignment, using language representations as semantic targets to capture latent motor intent from neural activity. The resulting neural representations serve as an endogenous intention signal to guide VLA policies to generate executable actions, while visual observations provide complementary information about the evolving task state. BrainVLA is evaluated on two neural motor datasets with different action dimensionalities using causal rollout decoding. It outperforms the evaluated baselines in cross-session decoding $R^2$ and task success rate, while demonstrating high training data efficiency. These results establish a route for neural motor decoding to draw on large-scale robotic priors through brain-conditioned VLA policies.

发表机构

• The Chinese University of Hong Kong（香港中文大学）
• University of Hong Kong（香港大学）

机构由 AI 辅助整理，请以论文原文为准。