Cross-modality Force and Language Embeddings for Natural Human-Robot Communication
Electrical Engineering and Computer Science Massachusetts Institute of Technology(电气工程与计算机科学麻省理工学院);Department of Physical Medicine, Harvard Medical School Rehabilitation, Spaulding Rehabilitation Hospital(物理医学系哈佛医学院康复科,斯波尔丁康复医院);Mechanical Engineering Massachusetts Institute of Technology(机械工程麻省理工学院)
Ravi Tejwani, Karl Velazquez, John Payne, Paolo Bonato, Harry Asada
A method for cross-modality embedding of force profile and words is presented for synergistic coordination of verbal and haptic communication. When two people carry a large, heavy object together, they coordinate through verbal communication about the intended movements and physical forces applied to the object. This natural integration of verbal and physical cues enables effective coordination. Similarly, human-robot interaction could achieve this level of coordination by integrating verbal and haptic communication modalities. This paper presents a framework for embedding words and force profiles in a unified manner, so that the two communication modalities can be integrated and coordinated in a way that is effective and synergistic. Here, it will be shown that, although language and physical force profiles are deemed completely different, the two can be embedded in a unified latent space and proximity between the two can be quantified. In this latent space, a force profile and words can a) supplement each other, b) integrate the individual effects, and c) substitute in an exchangeable manner. First, the need for cross-modality embedding is addressed, and the basic architecture and key building block technologies are presented. Methods for data collection and implementation challenges will be addressed, followed by experimental results and discussions.
LCMF: Lightweight Cross-Modality Mambaformer for Embodied Robotics VQA
School of Software Northwestern Polytechnical University Xi'an, China(软件学院 西安理工大学中国);Laboratoire L2Tl University Sorbonne Paris Nord Paris, France(L2Tl实验室 索邦巴黎北大学巴黎法国)
Exploring Human-Robot Collaboration: Analysis of Interaction Modalities in Challenging Tasks
Dalle Molle Institute for Artificial Intelligence (IDSIA), USI-SUPSI(达勒莫勒人工智能研究所(IDSIA));Department of Surgery, Medicine, Dentistry and Morphological Sciences with interest in Transplant, Oncology and Regenerative Medicine, University of Modena and Reggio Emilia(外科、医学、牙科和形态学科学系(关注移植、肿瘤学和再生医学),摩德纳和雷吉奥艾米利亚大学);Department of Sciences and Methods for Engineering (DISMI), University of Modena and Reggio Emilia(工程科学与方法系(DISMI),摩德纳和雷吉奥艾米利亚大学)
LiDAR-BIND-T: Improved and Temporally Consistent Sensor Modality Translation and Fusion for Robotic Applications
IDLab - Faculty of Applied Engineering, University of Antwerp - imec(IDLab - 应用工程学院,安特卫普大学 - imec);Cosys-Lab - Faculty of Applied Engineering, University of Antwerp(Cosys-Lab - 应用工程学院,安特卫普大学);Flanders Make Strategic Research Centre(弗拉芒制造战略研究中心)
本文提出 LiDAR-BIND-T,通过时间嵌入相似度、运动对齐损失和窗口化时间融合增强异构传感器到 LiDAR 的转换,并以新指标验证其提升 SLAM 稳定性与精度。
RoboPEPP: Vision-Based Robot Pose and Joint Angle Estimation through Embedding Predictive Pre-Training
New York University Tandon School of Engineering(纽约大学Tandon工程学院);New York University Courant Institute of Mathematical Sciences(纽约大学Courant数学科学研究所);Meta-FAIR
Modality-Decoupled Federated Learning for Privacy-Preserving Embodied Intelligence in 6G
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Shanghai Jiao Tong University(上海交通大学);Eastern Institute of Technology(东方理工大学);ITMO University(圣彼得堡国立信息技术机械与光学大学);University of Hong Kong(香港大学);Singapore University of Technology and Design(新加坡科技设计大学);Mohamed Bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
GATER: Learning Grasp-Action-Target Embeddings and Relations for Task-Specific Grasping
Shanghai Jiao Tong University(上海交通大学)
智能服务机器人需要在动态环境中执行各种任务的能力。尽管机器人抓取技术取得了重大进展,但在非结构化的现实生活环境中,机器人在执行不同任务时确定抓取位置仍然是一个挑战。为了克服这一挑战,创建一个合适的知识表示框架是关键。与以往的工作不同,本文将任务定义为一个三元组,包括抓取工具、期望动作和目标对象。我们提出的算法GATER(Grasp-Action-Target Embeddings and Relations)在嵌入空间中建模抓取工具-动作-目标对象之间的关系。为了验证我们的方法,创建了一个新的数据集,用于特定任务的抓取。GATER在新的数据集上进行训练,获得了94.6%的任务特定抓取推理成功率。最后,在一个实际的服务机器人平台上对GATER算法的有效性进行了测试。GATER算法在人类行为预测和人机交互方面有很大的潜力。
Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);University of Chinese Academy of Sciences(中国科学院大学);The Chinese University of Hong Kong(香港中文大学)
Robo-MUTUAL: Robotic Multimodal Task Specification via Unimodal Learning
Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));College of Engineering, Peking University(北京大学工学院);Sensetime Research(商汤科技研究院);University of Toronto(多伦多大学);University College London(伦敦大学学院);Shanghai AI Lab(上海人工智能实验室)
M3ET: Efficient Vision-Language Learning for Robotics based on Multimodal Mamba-Enhanced Transformer
School of Software Northwestern Polytechnical University Xi'an, China(软件学院 西安理工大学 西安);Laboratoire L2Tl University Sorbonne Paris Nord Paris, France(L2Tl实验室 索邦巴黎北大学 巴黎)
TACT: Humanoid Whole-body Contact Manipulation through Deep Imitation Learning with Tactile Modality
CNRS-AIST JRL (Joint Robotics Laboratory), IRL and National Institute of Advanced Industrial Science and Technology (AIST)(CNRS-AIST联合机器人实验室、IRL和国家先进工业科学与技术研究院);Tokyo University of Science(东京科学大学);University of Tsukuba(茨口大学)
Multi-Domain Motion Embedding: Expressive Real-Time Mimicry for Legged Robots
Robotic Systems Lab, ETH Zurich, Zurich, Switzerland(苏黎世联邦理工学院机器人系统实验室);Computational Robotics Lab, ETH Zurich, Zurich, Switzerland(苏黎世联邦理工学院计算机器人实验室);ETH Zürich(苏黎世联邦理工学院)
Robust Bimanual Vision-Language-Action Models via Embarrassingly Simple Modality Masking
Shanghai Innovation Institute(上海创新研究院);Wuhan University(武汉大学);Zhejiang University(浙江大学);Shanghai Jiao Tong University(上海交通大学);University of Science and Technology of China(中国科学技术大学);City University of Hong Kong(香港城市大学);Southeast University(东南大学)