Achieving human-like reasoning in deep learning models for complex tasks in unknown environments remains a critical challenge in embodied intelligence. While advanced vision-language models (VLMs) excel in static scene understanding, their limitations in spatio-temporal reasoning and adaptation to dynamic, open-set tasks like task-oriented navigation and embodied question answering (EQA) persist due to inadequate modeling of fine-grained spatio-temporal cues and physical world comprehension. To address this, we propose VEME, a novel cross-modal alignment method that enhances generalization in unseen scenes by learning an ego-centric, experience-centered world model. Our framework integrates three key components: (1) a cross-modal alignment framework bridging objects, spatial representations, and visual semantics with spatio-temporal cues to enhance VLM in-context learning; (2) a dynamic, implicit cognitive map activated by world embedding to enable task-relevant geometric-semantic memory recall; and (3) an instruction-based navigation and reasoning framework leveraging embodied priors for long-term planning and efficient exploration. By embedding geometry-aware spatio-temporal episodic experiences, our method significantly improves reasoning and planning in dynamic environments. Experimental results on VSI-Bench and VLN-CE demonstrate 1%-3% accuracy and exploration efficiency improvement compared to traditional approaches.
A Progressive Training Strategy for Embodied Vision-Language Models to Mitigate Spatio-Temporal Hallucinations
Zhejiang University（浙江大学）；The Hong Kong University of Science and Technology（香港科技大学）；Qingdao University（青岛大学）；The University of Hong Kong（香港大学）；Shanghai Jiao Tong University（上海交通大学）；The Chinese University of Hong Kong（香港中文大学）；University of YYY（YYY大学）；Institute of WWW（WWW研究所）
SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoning
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Esslingen University of Applied Sciences（埃斯林根应用科学大学）；Dr. Ing. h.c. F. Porsche AG（德意志联邦汽车工业联合会）；University of Michigan（密歇根大学）；Voxel51 Inc.（Voxel51公司）
CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoning
School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；King Abdullah University of Science and Technology（科廷大学）；Fudan University（复旦大学）
From Perception to Planning: Evolving Ego-Centric Task-Oriented Spatiotemporal Reasoning via Curriculum Learning
Zhejiang University（浙江大学）；Tianjin University（天津大学）；Qingdao University（青岛大学）；Shanghai Jiao Tong University（上海交通大学）；National University of Singapore（新加坡国立大学）
R4: Retrieval-Augmented Reasoning for Vision-Language Models in 4D Spatio-Temporal Space
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Esslingen University of Applied Sciences（埃斯林根应用科学大学）；Dr. Ing. h.c. F. Porsche AG（德意志联邦汽车工业协会）；University of Michigan（密歇根大学）；Voxel51 Inc.（Voxel51公司）
Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level
University of Central Florida（中佛罗里达大学）；University of Western Australia（西澳大学）；Amazon Web Services（亚马逊网络服务）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）
Beyond Pixels: A Vector-to-Graph Framework for Reliable Schematic Auditing
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（人工智能与数字经济广东实验室）；Guangdong Power Grid Co., Ltd.（广东电网公司）；Shanghai University（上海大学）；Carleton University（卡尔顿大学）
ERMV: Editing 4D Robotic Multi-view images to enhance embodied agents
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University and Key Laboratory of System Control and Information Processing, Ministry of Education of China（自动化与智能感知学院，上海交通大学；系统控制与信息处理重点实验室，中华人民共和国教育部）
Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis
Northeastern University, Boston, MA, USA；The Chinese University of Hong Kong, Hong Kong, China；Peking University, Beijing, China；Westlake University, Hangzhou, China；Harvard University, Cambridge, MA, USA；Purdue University, West Lafayette, IN, USA；University of Oxford, Oxford, United Kingdom
Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（信息多媒体国家重点实验室，计算机学院，北京大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Institute for Brain and Intelligence, Fudan University（脑与智能研究院，复旦大学）；University of Science and Technology Beijing（北京科技大学）；Beijing Innovation Center of Humanoid Robotics（北京人形机器人创新中心）
DynTrace: Tracking Dynamic Object Evidence for 4D Spatio-Temporal Reasoning in MLLMs
Xiamen University（厦门大学）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；China University of Mining and Technology（中国矿业大学）；Shenzhen Technology University（深圳技术大学）
Carnegie Mellon University（卡内基梅隆大学）；University of British Columbia（不列颠哥伦比亚大学）；SenseTime Research（商汤科技研究院）；University of California, Merced（加州大学默塞德分校）
Existing denoising methods typically restore clear results by aggregating pixels from the noisy input. Instead of relying on hand-crafted aggregation schemes, we propose to explicitly learn this process with deep neural networks. We present a spatial pixel aggregation network and learn the pixel sampling and averaging strategies for image denoising. The proposed model naturally adapts to image structures and can effectively improve the denoised results. Furthermore, we develop a spatio-temporal pixel aggregation network for video denoising to efficiently sample pixels across the spatio-tempora…
University of Bremen（不来梅大学）；The DesignSpace Group（设计空间小组）；MANIT（迈普尔国家理工学院）
We present a general theory and corresponding declarative model for the embodied grounding and natural language based analytical summarisation of dynamic visuo-spatial imagery. The declarative model ---ecompassing spatio-linguistic abstractions, image schemas, and a spatio-temporal feature based language generator--- is modularly implemented within Constraint Logic Programming (CLP). The implemented model is such that primitives of the theory, e.g., pertaining to space and motion, image schemata, are available as first-class objects with `deep semantics' suited for inference and query. We demo…
Harbin Institute of Technology（哈尔滨工业大学）；Tencent AI Lab（腾讯人工智能实验室）
The key success factor of the video deblurring methods is to compensate for the blurry pixels of the mid-frame with the sharp pixels of the adjacent video frames. Therefore, mainstream methods align the adjacent frames based on the estimated optical flows and fuse the alignment frames for restoration. However, these methods sometimes generate unsatisfactory results because they rarely consider the blur levels of pixels, which may introduce blurry pixels from video frames. Actually, not all the pixels in the video frames are sharp and beneficial for deblurring. To address this problem, we propo…
Universidade Federal de Minas Gerais（米纳斯吉拉斯联邦大学）；University of Campinas（坎皮纳斯大学）；Universidade Estadual Paulista（圣保罗州立大学）；University of Stirling（斯特灵大学）
Plant phenology studies rely on long-term monitoring of life cycles of plants. High-resolution unmanned aerial vehicles (UAVs) and near-surface technologies have been used for plant monitoring, demanding the creation of methods capable of locating and identifying plant species through time and space. However, this is a challenging task given the high volume of data, the constant data missing from temporal dataset, the heterogeneity of temporal profiles, the variety of plant visual patterns, and the unclear definition of individuals' boundaries in plant communities. In this letter, we propose a…
Unsupervised Domain Adaptation (UDA) of semantic segmentation transfers labeled source knowledge to an unlabeled target domain by relying on accessing both the source and target data. However, the access to source data is often restricted or infeasible in real-world scenarios. Under the source data restrictive circumstances, UDA is less practical. To address this, recent works have explored solutions under the Source-Free Domain Adaptation (SFDA) setup, which aims to adapt a source-trained model to the target domain without accessing source data. Still, existing SFDA approaches use only image-…
DMTrack: Spatio-Temporal Multimodal Tracking via Dual-Adapter
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（中国科学院大学杭州高等研究院）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Department of Computer Science and Engineering, University of North Texas（北卡罗来纳州立大学计算机科学与工程系）；OPPO AI Center（OPPO人工智能中心）
Tsinghua University（清华大学）；Beijing Innovation Center for Future Chip（北京未来芯片创新中心）
Super-resolution (SR) is a useful technology to generate a high-resolution (HR) visual output from the low-resolution (LR) visual inputs overcoming the physical limitations of the cameras. However, SR has not been applied to enhance the resolution of spatiotemporal event-stream images captured by the frame-free dynamic vision sensors (DVSs). SR of event-stream image is fundamentally different from existing frame-based schemes since basically each pixel value of DVS images is an event sequence. In this work, a two-stage scheme is proposed to solve the SR problem of the spatiotemporal event-stre…
We introduce a novel video-rate hyperspectral imager with high spatial, and temporal resolutions. Our key hypothesis is that spectral profiles of pixels in a super-pixel of an oversegmented image tend to be very similar. Hence, a scene-adaptive spatial sampling of an hyperspectral scene, guided by its super-pixel segmented image, is capable of obtaining high-quality reconstructions. To achieve this, we acquire an RGB image of the scene, compute its super-pixels, from which we generate a spatial mask of locations where we measure high-resolution spectrum. The hyperspectral image is subsequently…
Embody4D: A Generalist Data Engine for Embodied 4D World Modeling
Zhejiang University（浙江大学）；Beijing Zhongguancun Academy（北京中关村学院）；University of Science and Technology of China（中国科学技术大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Shanghai Jiao Tong University（上海交通大学）；Beihang University（北京航空航天大学）
Canon Information Systems Research Australia（佳能澳大利亚信息系统研究公司）
We present the SAMMI lightweight object detection method which has a high level of accuracy and robustness, and which is able to operate in an environment with a large number of cameras. Background modeling is based on DCT coefficients provided by cameras. Foreground detection uses similarity in temporal characteristics of adjacent blocks of pixels, which is a computationally inexpensive way to make use of object coherence. Scene model updating uses the approximated median method for improved performance. Evaluation at pixel level and application level shows that SAMMI object detection perform…
This paper addresses the problem of video object segmentation, where the initial object mask is given in the first frame of an input video. We propose a novel spatio-temporal Markov Random Field (MRF) model defined over pixels to handle this problem. Unlike conventional MRF models, the spatial dependencies among pixels in our model are encoded by a Convolutional Neural Network (CNN). Specifically, for a given object, the probability of a labeling to a set of spatially neighboring pixels can be predicted by a CNN trained for this specific object. As a result, higher-order, richer dependencies a…
In this paper, we address the problem of quantifying reliability of computational saliency for videos, which can be used to improve saliency-based video processing and enable more reliable performance and risk assessment of such processing. Our approach is twofold. First, we explore spatial correlations in both saliency map and eye-fixation map. Then, we learn spatiotemporal correlations that define a reliable saliency map. We first study spatiotemporal eye-fixation data from a public dataset and investigate a common feature in human visual attention, which dictates correlation in saliency bet…
A Comprehensive Survey on World Models for Embodied AI
College of Computer Science and the Academy for Advanced Interdisciplinary Studies, Nankai University（南开大学计算机科学学院与前沿交叉学科研究院）；School of Computer Science and Engineering, Tianjin University of Technology（天津理工大学计算机科学与工程学院）；School of Information and Communication Engineering, University of Electronic Science and Technology of China（电子科技大学信息与通信工程学院）；Institute for Infocomm Research (I2R), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局资讯通信研究院）；Information Systems Technology and Design (ISTD) Pillar, Singapore University of Technology and Design (SUTD)（新加坡科技设计大学信息系统科技与设计系）
Learning semantical dynamics and spatiotemporal collaboration for human pose estimation in video
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）
