When embodied AI is expanding from traditional object detection and recognition to more advanced tasks of robot manipulation and actuation planning, visual spatial reasoning from the video inputs is necessary to perceive the spatial relationships of objects and guide device actions. However, existing visual language models (VLMs) have very weak capabilities in spatial reasoning due to the lack of knowledge about 3D spatial information, especially when the reasoning task involve complex spatial relations across multiple video frames. In this paper, we present a new inference-time computing technique for on-device embodied AI, namely \emph{MosaicThinker}, which enhances the on-device small VLM's spatial reasoning capabilities on difficult cross-frame reasoning tasks. Our basic idea is to integrate fragmented spatial information from multiple frames into a unified space representation of global semantic map, and further guide the VLM's spatial reasoning over the semantic map via a visual prompt. Experiment results show that our technique can greatly enhance the accuracy of cross-frame spatial reasoning on resource-constrained embodied AI devices, over reasoning tasks with diverse types and complexities.
Embodied Tree of Thoughts: Deliberate Manipulation Planning with Embodied World Model
University of Chinese Academy of Sciences (UCAS)(中国科学院大学);Tsinghua University(清华大学);JD Explore Academy(京东探索学院);ShanghaiTech University(上海科技大学);Nanjing University(南京大学)
E-TTS: A New Embodied Test-Time Scaling Framework for Robotic Manipulation
New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所模式识别国家重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所基础模型研究中心);FiveAges(五时代)
Position: Intelligent Science Laboratory Requires the Integration of Cognitive and Embodied AI
University of Science and Technology of China(中国科学技术大学);Nanjing University(南京大学);University of New South Wales(新南威尔士大学);Shanghai Jiao Tong University(上海交通大学);Peking University(北京大学);Harbin Institute of Technology(哈尔滨工业大学);Tongji University(同济大学);The University of Sydney, Oxford(悉尼大学,牛津);Renmin University of China(中国人民大学);Swiss Federal Institute of Technology Lausanne(苏黎世联邦理工学院);Shanghai Institute of Ceramics, Chinese Academy of Sciences(中国科学院上海硅酸盐研究所);Shanghai AI Laboratory(上海人工智能实验室);Shanghai AI Laboratory & The Chinese University of Hong Kong(上海人工智能实验室 & 香港中文大学)
ERQA-Plus: A Diagnostic Benchmark for Reasoning in Embodied AI
Centre for Frontier AI Research, Agency for Science, Technology and Research(新加坡科技研究局前沿人工智能研究中心);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院)
Guava: Distilling Frontier VLMs into a Compact Agent through a Robotic Manipulation Harness
University of Maryland College Park(马里兰大学帕克分校);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Waterloo(滑铁卢大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of Pennsylvania(宾夕法尼亚大学);Amazon FAR(亚马逊 FAR)
CoEnv: Code-Driven Exploration in Real-to-Sim Digital Twins for Multi-Arm Manipulation
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Harbin Institute of Technology(哈尔滨工业大学);University of Science and Technology of China(中国科学技术大学);CUHK-Shenzhen(香港中文大学(深圳));Fudan University(复旦大学);Dalian University of Technology(大连理工大学);Carnegie Mellon University(卡内基梅隆大学);University of Oxford(牛津大学)
Look, Zoom, Understand: The Robotic Eyeball for Embodied Perception
School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Dalian University of Technology(大连理工大学)
EvolveNav: Empowering LLM-Based Vision-Language Navigation via Self-Improving Embodied Reasoning
Shanghai Jiao Tong University(上海交通大学);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Peng Cheng Laboratory(鹏城实验室);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);Mohamed Bin Zayed University of Artificial Intelligence(Mohamed Bin Zayed人工智能大学);Yinwang Intelligent Technology Co.(亿维智能技术有限公司)
InfBaGel: Human-Object-Scene Interaction Generation with Dynamic Perception and Iterative Refinement
Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Sichuan University(四川大学);Shenzhen University(深圳大学);The University of Hong Kong(香港大学)
Compromising Embodied Agents with Contextual Backdoor Attacks
Beihang University(北京航空航天大学);National University of Singapore(新加坡国立大学);Zhongguancun Laboratory(中关村实验室);Nanyang Technological University(南洋理工大学);University of Science and Technology of China(中国科学技术大学);A*STAR(新加坡科技研究局)
EmbodiedVSR: Dynamic Scene Graph-Guided Chain-of-Thought Reasoning for Visual Spatial Tasks
Beijing Innovation Center of Humanoid Robotics(北京人形机器人创新中心);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Hong Kong University of Science and Technology(香港科技大学);University of Technology Sydney(悉尼科技大学)
Value-Guided Iterative Refinement and the DIQ-H Benchmark for Evaluating VLM Robustness
The School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, China(香港中文大学(深圳)科学与工程学院);The School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China(香港中文大学(深圳)人工智能学院);The Shenzhen Institute of Artificial Intelligence and Robotics for Society, Shenzhen, China(深圳人工智能与机器人社会研究院)
Recurrent Reasoning with Vision-Language Models for Estimating Long-Horizon Embodied Task Progress
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学北京校区人工智能学院);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大型模型与智能治理研究重点实验室);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(教育部下一代智能搜索与推荐工程研究中心);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院)
CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoning
School of Computer Science and Technology, East China Normal University(东华大学计算机科学与技术学院);King Abdullah University of Science and Technology(科廷大学);Fudan University(复旦大学)
Visual Environment-Interactive Planning for Embodied Complex-Question Answering
School of Artificial Intelligence, Xidian University(西安电子科技大学人工智能学院);Guangzhou Institute of Technology, Xidian University(西安电子科技大学广州研究院);Pazhou Laboratory(琶洲实验室);Peng Cheng Laboratory(鹏城实验室)
PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments
Beijing Jiaotong University(北京交通大学);Tencent Robotics X & Futian Laboratory, Shenzhen(腾讯机器人X及福田实验室,深圳);Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(基础模型研究中心,中国科学院自动化研究所);ObjectEye Inc(ObjectEye公司)