SignVLA: 通过注意力LSTM和视觉-语言-动作模型实现实时手语引导的机器人操作
SignVLA: Real-Time Sign Language-Guided Robotic Manipulation via Attention LSTM and Vision-Language-Action Models
AI总结：
提出SignVLA框架，通过注意力LSTM网络将手语手势实时转换为语义指令，驱动VLA模型执行机器人操作任务，为听障用户提供无障碍交互。
摘要：
视觉-语言-动作（VLA）模型使机器人能够根据基于视觉观察的自然语言指令执行操作任务。然而，现有的VLA接口主要依赖语音或文本输入，限制了聋哑、听力障碍和言语障碍用户的可访问性。我们提出SignVLA，一个实时手语引导的VLA框架，用于无障碍人机交互。该系统引入了一个模块化的手语到文本接口，将视觉手语手势转换为与下游VLA策略兼容的语义指令。给定视频流，SignVLA提取手部地标特征，并采用注意力增强的长短期记忆（LSTM）网络捕捉时间手势动态，用于字母和命令级别的手语识别。一个时间稳定模块进一步提高了实时交互中的预测一致性。生成的指令序列随后传递给下游VLA策略，用于手语条件化的机器人操作。实验结果表明，稳定的实时手语识别和由手语输入驱动的操作任务成功执行。我们的发现表明，轻量级时间手语识别可以作为多模态具身智能的有效且实用的无障碍层。
英文摘要：
Vision-Language-Action (VLA) models enable robots to execute manipulation tasks from natural-language instructions grounded in visual observations. However, existing VLA interfaces primarily rely on speech or text input, limiting accessibility for deaf, hard-of-hearing, and speech-impaired users. We present SignVLA, a real-time sign-language-guided VLA framework for accessible human-robot interaction. The system introduces a modular sign-to-text interface that converts visual sign gestures into semantic instructions compatible with downstream VLA policies. Given video streams, SignVLA extracts hand landmark features and employs an attention-enhanced Long Short-Term Memory (LSTM) network to capture temporal gesture dynamics for alphabet- and command-level sign recognition. A temporal stabilization module further improves prediction consistency in real-time interaction settings.The generated instruction sequence is then passed to a downstream VLA policy for sign-conditioned robotic manipulation. Experimental results demonstrate stable real-time sign recognition and successful execution of manipulation tasks driven by sign-language inputs. Our findings suggest that lightweight temporal sign recognition can serve as an effective and practical accessibility layer for multimodal embodied intelligence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
