State recognition of objects and environment in robots has been conducted in various ways. In most cases, this is executed by processing point clouds, learning images with annotations, and using specialized sensors. In contrast, in this study, we propose a state recognition method that applies Visual Question Answering (VQA) in a Pre-Trained Vision-Language Model (PTVLM) trained from a large-scale dataset. By using VQA, it is possible to intuitively describe robotic state recognition in the spoken language. On the other hand, there are various possible ways to ask about the same event, and the performance of state recognition differs depending on the question. Therefore, in order to improve the performance of state recognition using VQA, we search for an appropriate combination of questions using a genetic algorithm. We show that our system can recognize not only the open/closed of a refrigerator door and the on/off of a display, but also the open/closed of a transparent door and the state of water, which have been difficult to recognize.
LCMF: Lightweight Cross-Modality Mambaformer for Embodied Robotics VQA
School of Software Northwestern Polytechnical University Xi'an, China(软件学院 西安理工大学中国);Laboratoire L2Tl University Sorbonne Paris Nord Paris, France(L2Tl实验室 索邦巴黎北大学巴黎法国)
Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgery
The Chinese University of Hong Kong(香港中文大学);Shenzhen Research Institute, The Chinese University of Hong Kong(香港中文大学深圳研究院);University College London(伦敦大学学院);National University of Singapore(新加坡国立大学)
M3ET: Efficient Vision-Language Learning for Robotics based on Multimodal Mamba-Enhanced Transformer
School of Software Northwestern Polytechnical University Xi'an, China(软件学院 西安理工大学 西安);Laboratoire L2Tl University Sorbonne Paris Nord Paris, France(L2Tl实验室 索邦巴黎北大学 巴黎)
MQA: Answering the Question via Robotic Manipulation
Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Tsinghua University(清华大学)
An adaptive experience-based discrete genetic algorithm for multi-trip picking robot task scheduling in smart orchards
School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450007, China(电气与信息工程学院,郑州大学);School of Electrical Engineering and Automation, Henan Institute of Technology, Xinxiang 453003, China(电气工程与自动化学院,河南理工大学);School of Engineering, RMIT University, Melbourne, VIC, 3000, Australia(工程学院,皇家墨尔本理工大学);Longmen Laboratory, Luoyang 471000, China(洛阳长门实验室);KINDI Center for Computing Research, College of Engineering, Qatar University, Doha, Qatar(计算研究KINDI中心,工程学院,卡塔尔大学);Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Canada(电气与计算机工程系,阿尔伯塔大学);Systems Research Institute of the Polish Academy of Sciences, Warsaw, Poland(波兰科学院系统研究所)
Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery
The Chinese University of Hong Kong(香港中文大学);Universiti Malaya(马来亚大学);Centre for Artificial Intelligence and Robotics, HKISI-CAS(香港智能系统研究所人工智能与机器人中心);University College London(伦敦大学学院)
CAT-ViL: Co-Attention Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London(伦敦大学学院惠康/工程与物理科学研究委员会介入与外科科学中心(WEISS));The Chinese University of Hong Kong (CUHK)(香港中文大学(CUHK));Shun Hing Institute of Advanced Engineering, CUHK(香港中文大学信兴高等工程研究所)
Surgical-VQLA: Transformer with Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery
National University of Singapore(新加坡国立大学);The Chinese University of Hong Kong (CUHK)(香港中文大学);University College London(伦敦大学学院);Shun Hing Institute of Advanced Engineering(信兴高等工程研究所)