Multi-modal Large Language Models (LLM) have advanced conversational abilities but struggle with providing live, interactive step-by-step guidance, a key capability for future AI assistants. Effective guidance requires not only delivering instructions but also detecting their successful execution, as well as identifying and alerting users to mistakes, all of which has to happen in real-time. This requires models that are not turn-based, but that can react asynchronously to a video stream, as well as video data showing users performing tasks including mistakes and their corrections. To this end, we introduce Qualcomm Interactive Cooking, a new benchmark and dataset built upon CaptainCook4D, which contains user mistakes during task execution. Our dataset and benchmark features densely annotated, timed instructions and feedback messages, specifically including mistake alerts precisely timestamped to their visual occurrence in the video. We evaluate state-of-the-art multi-modal LLMs on the Qualcomm Interactive Cooking benchmark and introduce LiveMamba, a streaming multi-modal LLM designed for interactive instructional guidance. This work provides the first dedicated benchmark and a strong baseline for developing and evaluating on live, situated coaching.
The University of Tokyo（东京大学）；National Institute of Informatics（国立信息学研究所）；Microsoft Research Asia（微软亚洲研究院）
We study the challenging problem of recovering detailed motion from a single motion-blurred image. Existing solutions to this problem estimate a single image sequence without considering the motion ambiguity for each region. Therefore, the results tend to converge to the mean of the multi-modal possibilities. In this paper, we explicitly account for such motion ambiguity, allowing us to generate multiple plausible solutions all in sharp detail. The key idea is to introduce a motion guidance representation, which is a compact quantization of 2D optical flow with only four discrete motion direct…
LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content
Faculty of Engineering Tel-Aviv University（特拉维夫大学工程学院）；IBM Research（IBM研究院）；Department of Statistics, University of Michigan（密歇根大学统计学系）；JKU Linz, Austria（林茨大学，奥地利）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；MIT-IBM（麻省理工学院-IBM）
University of Science and Technology of China（中国科学技术大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Rochester（罗切斯特大学）
We study the problem of localizing audio-visual events that are both audible and visible in a video. Existing works focus on encoding and aligning audio and visual features at the segment level while neglecting informative correlation between segments of the two modalities and between multi-scale event proposals. We propose a novel MultiModulation Network (M2N) to learn the above correlation and leverage it as semantic guidance to modulate the related auditory, visual, and fused features. In particular, during feature encoding, we propose cross-modal normalization and intra-modal normalization…
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；Multimedia Laboratory, The Chinese University of Hong Kong（香港中文大学多媒体实验室）；Adobe NextCam；Adobe Research；Shanghai AI Laboratory（上海人工智能实验室）；CPII under InnoHK（InnoHK下的CPII）；Shenzhen University of Advanced Technology（深圳先进技术大学）
A Multimodal Transformer for Live Streaming Highlight Prediction
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；KuaiShou Inc.（快手公司）；CAIR, HK Institute of Science and Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院人工智能与机器人中心）
Zhejiang University of Science & Technology（浙江科技大学）
RGB thermal scene parsing has recently attracted increasing research interest in the field of computer vision. However, most existing methods fail to perform good boundary extraction for prediction maps and cannot fully use high level features. In addition, these methods simply fuse the features from RGB and thermal modalities but are unable to obtain comprehensive fused features. To address these problems, we propose an edge-aware guidance fusion network (EGFNet) for RGB thermal scene parsing. First, we introduce a prior edge map generated using the RGB and thermal images to capture detailed …
Multi-Modal Face Anti-Spoofing via Cross-Modal Feature Transitions
College of Computer and Cyber Security, Fujian Normal University（计算机与网络安全学院，福建师范大学）；Department of Computer Science, National Tsing Hua University（计算机科学系，国立清华大学）
Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Kitware（Kitware公司）；Nanjing Institute of Software Technology（南京软件技术研究院）
Image inpainting is an ill-posed problem to recover missing or damaged image content based on incomplete images with masks. Previous works usually predict the auxiliary structures (e.g., edges, segmentation and contours) to help fill visually realistic patches in a multi-stage fashion. However, imprecise auxiliary priors may yield biased inpainted results. Besides, it is time-consuming for some methods to be implemented by multiple stages of complex neural networks. To solve this issue, we develop an end-to-end multi-modality guided transformer network, including one inpainting branch and two …
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Lenovo Research（联想研究院）；Lenovo Ltd.（联想集团有限公司）
Liver cancer is one of the most common cancers worldwide. Due to inconspicuous texture changes of liver tumor, contrast-enhanced computed tomography (CT) imaging is effective for the diagnosis of liver cancer. In this paper, we focus on improving automated liver tumor segmentation by integrating multi-modal CT images. To this end, we propose a novel mutual learning (ML) strategy for effective and robust multi-modal liver tumor segmentation. Different from existing multi-modal methods that fuse information from different modalities by a single model, with ML, an ensemble of modality-specific mo…
Multi-modal Liver Segmentation and Fibrosis Staging Using Real-world MRI Images
Multiscale X-ray Imaging (MXI) Lab, Department of Mechanical Engineering, University College London（多尺度X射线成像实验室，机械工程系，伦敦大学学院）；Centre for Clinical Brain Sciences, University of Edinburgh（临床脑科学中心，爱丁堡大学）；MRC Weatherall Institute of Molecular Medicine, University of Oxford（MRC韦伯尔分子医学研究所，牛津大学）
Shenzhen Research Institute of Big Data, the Chinese University of Hong Kong（香港中文大学深圳大数据研究院）；Huazhong University of Science and Technology（华中科技大学）；Shenzhen Third People’s Hospital（深圳市第三人民医院）
With the development of radiomics, noninvasive diagnosis like ultrasound (US) imaging plays a very important role in automatic liver fibrosis diagnosis (ALFD). Due to the noisy data, expensive annotations of US images, the application of Artificial Intelligence (AI) assisting approaches encounters a bottleneck. Besides, the use of mono-modal US data limits the further improve of the classification results. In this work, we innovatively propose a multi-modal fusion network with active learning (MMFN-AL) for ALFD to exploit the information of multiple modalities, eliminate the noisy data and red…
METDrive: Multi-modal End-to-end Autonomous Driving with Temporal Guidance
Intelligent Space Robotics Laboratory, Center for Digital Engineering, Skolkovo Institute of Science and Technology（斯克尔科沃科学与技术研究所智能空间机器人实验室，数字工程中心）；Institute of Automation, Qilu University of Technology (Shandong Academy of Sciences)（齐鲁科技大学自动化研究所，山东科学院）
University of Rochester（罗切斯特大学）；University of Chinese Academy of Sciences（中国科学院大学）；University of North Texas（北得克萨斯大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）
Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models
Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）；Monash University（莫纳什大学）；Temple University（天普大学）；Peng Cheng Laboratory（鹏城实验室）
Image outpainting technology generates visually plausible content regardless of authenticity, making it unreliable to be applied in practice. Thus, we propose a reliable image outpainting task, introducing the sparse depth from LiDARs to extrapolate authentic RGB scenes. The large field view of LiDARs allows it to serve for data enhancement and further multimodal tasks. Concretely, we propose a Depth-Guided Outpainting Network to model different feature representations of two modalities and learn the structure-aware cross-modal fusion. And two components are designed: 1) The Multimodal Learnin…
Nankai University（南开大学）；Tianjin University of Finance and Economics（天津财经大学）
The multi-modal salient object detection model based on RGB-D information has better robustness in the real world. However, it remains nontrivial to better adaptively balance effective multi-modal information in the feature fusion phase. In this letter, we propose a novel gated recoding network (GRNet) to evaluate the information validity of the two modes, and balance their influence. Our framework is divided into three phases: perception phase, recoding mixing phase and feature integration phase. First, A perception encoder is adopted to extract multi-level single-modal features, which lays t…
Diffusion generative models have recently greatly improved the power of text-conditioned image generation. Existing image generation models mainly include text conditional diffusion model and cross-modal guided diffusion model, which are good at small scene image generation and complex scene image generation respectively. In this work, we propose a simple yet effective approach, namely UPainting, to unify simple and complex scene image generation, as shown in Figure 1. Based on architecture improvements and diverse guidance schedules, UPainting effectively integrates cross-modal guidance from …
Helmholtz Zentrum München（慕尼黑亥姆霍兹中心）；Technical University of Munich（慕尼黑工业大学）；University of Freiburg（弗莱堡大学）；University of Bonn（波恩大学）
Given the high incidence and effective treatment options for liver diseases, they are of great socioeconomic importance. One of the most common methods for analyzing CT and MRI images for diagnosis and follow-up treatment is liver segmentation. Recent advances in deep learning have demonstrated encouraging results for automatic liver segmentation. Despite this, their success depends primarily on the availability of an annotated database, which is often not available because of privacy concerns. Federated Learning has been recently proposed as a solution to alleviate these challenges by trainin…
Imperial College London（伦敦帝国学院）；Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希·亚历山大大学）
This paper presents the use of Multi-Agent Reinforcement Learning (MARL) to perform navigation in 3D anatomical volumes from medical imaging. We utilize Neural Style Transfer to create synthetic Computed Tomography (CT) agent gym environments and assess the generalization capabilities of our agents to clinical CT volumes. Our framework does not require any labelled clinical data and integrates easily with several image translation techniques, enabling cross modality applications. Further, we solely condition our agents on 2D slices, breaking grounds for 3D guidance in much more difficult imagi…
Johns Hopkins University（约翰斯·霍普金斯大学）；Ludwig-Maximilians-Universität München（慕尼黑大学）；Johns Hopkins Hospital（约翰斯·霍普金斯医院）；Technische Universität München（慕尼黑工业大学）
Fluoroscopic X-ray guidance is a cornerstone for percutaneous orthopaedic surgical procedures. However, two-dimensional observations of the three-dimensional anatomy suffer from the effects of projective simplification. Consequently, many X-ray images from various orientations need to be acquired for the surgeon to accurately assess the spatial relations between the patient's anatomy and the surgical tools. In this paper, we present an on-the-fly surgical support system that provides guidance using augmented reality and can be used in quasi-unprepared operating rooms. The proposed system build…
Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）；Taobao (China) Software Co.,Ltd.（淘宝（中国）软件有限公司）；Tsinghua University（清华大学）
Depth map records distance between the viewpoint and objects in the scene, which plays a critical role in many real-world applications. However, depth map captured by consumer-grade RGB-D cameras suffers from low spatial resolution. Guided depth map super-resolution (DSR) is a popular approach to address this problem, which attempts to restore a high-resolution (HR) depth map from the input low-resolution (LR) depth and its coupled HR RGB image that serves as the guidance. The most challenging problems for guided DSR are how to correctly select consistent structures and propagate them, and pro…
MM-TRELLIS: Point-Cloud Guided Multi-Modal 3D Vehicle Generation in Autonomous Driving
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院教育部人工智能重点实验室）；Academy of Military Science（军事科学院）；Bosch innovation software development (Wuxi) Co., Ltd.（博世创新软件开发（无锡）有限公司）；College of Computer Science and Technology, National University of Defense Technology（国防科技大学计算机学院）；Shopee Pte. Ltd.（Shopee私人有限公司）
