SignVLA:一种无 gloss 的视觉-语言-动作框架,用于实时 sign language 引导的机器人操作
SignVLA: A Gloss-Free Vision-Language-Action Framework for Real-Time Sign Language-Guided Robotic Manipulation
AI总结:
SignVLA 提出了一种无 gloss 的视觉-语言-动作框架,用于实时手语引导的机器人操作,通过几何归一化、时间平滑和词汇精炼提升多模态交互的稳定性和可扩展性。
摘要:
我们提出了一种视觉-语言-动作(VLA)框架,这是首个基于手语驱动的直观和包容性人机交互框架。与传统方法依赖 gloss 注释作为中间监督不同,所提出的系统采用无 gloss 的范式,直接将视觉手语手势映射到语义指令。这种设计减少了标注成本,并避免了由 gloss 表示引入的信息损失,使多模态交互更加自然和可扩展。在本工作中,我们专注于实时字母级手指拼写界面,为机器人控制提供稳健且低延迟的通信通道。与大规模连续手语识别相比,字母级交互在可靠性、可解释性和部署可行性方面具有改进,特别是在安全关键的躯体环境中。所提出的流程通过几何归一化、时间平滑和词汇精炼将连续手势流转换为连贯的语言指令,确保稳定一致的交互。此外,该框架设计用于支持未来 transformer 基础的无 gloss 手语模型的集成,实现可扩展的词级和句级语义理解。实验结果展示了所提出系统在多样交互场景中将手语指令转化为精确机器人动作的有效性。这些结果突显了该框架在推动可访问、可扩展和多模态躯体智能方面的潜力。
英文摘要:
We present, to our knowledge, the first sign language-driven Vision-Language-Action (VLA) framework for intuitive and inclusive human-robot interaction. Unlike conventional approaches that rely on gloss annotations as intermediate supervision, the proposed system adopts a gloss-free paradigm and directly maps visual sign gestures to semantic instructions. This design reduces annotation cost and avoids the information loss introduced by gloss representations, enabling more natural and scalable multimodal interaction. In this work, we focus on a real-time alphabet-level finger-spelling interface that provides a robust and low-latency communication channel for robotic control. Compared with large-scale continuous sign language recognition, alphabet-level interaction offers improved reliability, interpretability, and deployment feasibility in safety-critical embodied environments. The proposed pipeline transforms continuous gesture streams into coherent language commands through geometric normalization, temporal smoothing, and lexical refinement, ensuring stable and consistent interaction. Furthermore, the framework is designed to support future integration of transformer-based gloss-free sign language models, enabling scalable word-level and sentence-level semantic understanding. Experimental results demonstrate the effectiveness of the proposed system in grounding sign-derived instructions into precise robotic actions under diverse interaction scenarios. These results highlight the potential of the framework to advance accessible, scalable, and multimodal embodied intelligence.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。