我们提出了一种基于潜在流匹配的新型视频预测生成模型，该方法是扩散模型的高效替代方案。与现有研究不同，我们在图像生成过程的每个积分步中仅以少量随机选取的过去帧为条件，从而避免了训练和推理期间对过去信息建模的高昂成本。此外，为实现高分辨率视频生成并加快训练速度，我们在预训练VQGAN（矢量量化生成对抗网络）的潜在空间中进行运算。最后，我们提出用前一帧带噪帧来近似流常微分方程（ODE）的初始条件，这一设计可减少积分步数量，进而提升推理时的采样速度。我们将该模型命名为Random frame conditioned flow Integration for VidEo pRediction，简称RIVER。实验表明，在常用视频预测基准测试中，RIVER的性能优于或持平于现有方法，同时所需计算资源减少了一个数量级。
英文摘要：
We introduce a novel generative model for video prediction based on latent flow matching, an efficient alternative to diffusion-based models. In contrast to prior work, we keep the high costs of modeling the past during training and inference at bay by conditioning only on a small random set of past frames at each integration step of the image generation process. Moreover, to enable the generation of high-resolution videos and to speed up the training, we work in the latent space of a pretrained VQGAN. Finally, we propose to approximate the initial condition of the flow ODE with the previous noisy frame. This allows to reduce the number of integration steps and hence, speed up the sampling at inference time. We call our model Random frame conditioned flow Integration for VidEo pRediction, or, in short, RIVER. We show that RIVER achieves superior or on par performance compared to prior work on common video prediction benchmarks, while requiring an order of magnitude fewer computational resources.
Can We Build Scene Graphs, Not Classify Them? FlowSG: Progressive Image-Conditioned Scene Graph Generation with Flow Matching
The Laboratory of Intelligent Collaborative Computing of UESTC（UESTC智能协同计算实验室）；Tianfu Jiangxi Laboratory（天斧江西实验室）；Monash University（墨尔本大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东人工智能与数字经济实验室（深圳））
SubFlow: Sub-mode Conditioned Flow Matching for Diverse One-Step Generation
Sydney AI Centre, The University of Sydney（悉尼人工智能中心，悉尼大学）；School of Artificial Intelligence, Xidian University（西安电子科技大学人工智能学院）；Australian Artificial Intelligence Institute, University of Technology Sydney（澳大利亚人工智能研究所，技术大学悉尼）；Department of Information Systems, City University of Hong Kong（香港城市大学信息系统学院）
StimFlow: Synthesizing Brain Activity from Images via Stimulus-conditioned Flow Matching
Chonnam National University（全南大学）；AISEED, Inc.（AISEED公司）；Ho Chi Minh University of Science（胡志明市科学大学）；MIREA – Russian Technological University（俄罗斯国立研究型技术大学（莫斯科国立精密机械与计算机科学研究所））
SCMA: Structure-Conditioned and Metal-Aware Flow Matching for CT Metal Artifact Reduction
School of Mathematical Sciences, Capital Normal University（首都师范大学数学科学学院）；College of Mathematics Science, Inner Mongolia Normal University（内蒙古师范大学数学科学学院）；National Center for Applied Mathematics Beijing, Capital Normal University（首都师范大学北京应用数学中心）；Academy for Multidisciplinary Studies, Capital Normal University（首都师范大学交叉科学研究院）
MTFlow: Time-Conditioned Flow Matching for Microtubule Segmentation in Noisy Microscopy Images
CNRS, Univ. Rennes, Institute of Genetics and Development of Rennes (IGDR), Rennes, France（法国里昂大学CNRS研究所、里昂大学基因与发育研究所）；TIAD Laboratory, Sciences and Technology Faculty, Sultan Moulay Slimane Univ.（摩洛哥苏丹穆莱·西拉曼大学科技学院TIAD实验室）
Shortest-Path Flow Matching with Mixture-Conditioned Bases for OOD Generalization to Unseen Conditions
Wellcome Sanger Institute（Wellcome桑格研究所）；Institute for Molecular Medicine Finland (FIMM)（芬兰分子医学研究所）；University of Helsinki（赫尔辛基大学）；School of Computing and Information Systems（计算与信息系统学院）；University of Cambridge（剑桥大学）；Cambridge Centre for AI in Medicine（剑桥人工智能医学中心）；MediaTek Research（MediaTek研究）；Cambridge Stem Cell Institute（剑桥干细胞研究所）
CurvSegFlow: Time-Conditioned Flow Matching for Robust Segmentation of Curvilinear Structures in Noisy Biomedical Images
CNRS, Univ. Rennes, Institute of Genetics and Development of Rennes (IGDR)（法国国家科学研究中心、雷恩大学、雷恩遗传与发育研究所）；TIAD Laboratory, Sciences and Technology Faculty, Sultan Moulay Slimane Univ（苏丹穆莱·斯利曼大学科学与技术学院TIAD实验室）；Institute of Biotechnology, Czech Academy of Sciences（捷克科学院生物技术研究所）
BridgeMatch: Conditional Transport Bridges in Matching Matrix Space for 3D Deformable Registration
Nantong University（南通大学）；Nanyang Technological University（南洋理工大学）；Nanjing University（南京大学）；Nanjing University of Science and Technology（南京理工大学）；Southeast University（东南大学）
Rutgers University（罗格斯大学）；NEC Labs America（NEC美国实验室）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Conditional Generative Adversarial Networks (cGANs) extend the standard unconditional GAN framework to learning joint data-label distributions from samples, and have been established as powerful generative models capable of generating high-fidelity imagery. A challenge of training such a model lies in properly infusing class information into its generator and discriminator. For the discriminator, class conditioning can be achieved by either (1) directly incorporating labels as input or (2) involving labels in an auxiliary classification loss. In this paper, we show that the former directly ali…
Unified Trajectory Matching Policy Optimization: Diverse T2I Generation and VLA Generalization
Huazhong University of Science and Technology（华中科技大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）
University of Oxford（牛津大学）；VGG（视觉几何组）；Dept. of Eng. Science（工程科学系）
We propose a new approach to determine correspondences between image pairs in the wild under large changes in illumination, viewpoint, context, and material. While other approaches find correspondences between pairs of images by treating the images independently, we instead condition on both images to implicitly take account of the differences between them. To achieve this, we introduce (i) a spatial attention mechanism (a co-attention module, CoAM) for conditioning the learned features on both images, and (ii) a distinctiveness score used to choose the best matches at test time. CoAM can be a…
MRI and CT are most widely used medical imaging modalities. It is often necessary to acquire multi-modality images for diagnosis and treatment such as radiotherapy planning. However, multi-modality imaging is not only costly but also introduces misalignment between MRI and CT images. To address this challenge, computational conversion is a viable approach between MRI and CT images, especially from MRI to CT images. In this paper, we propose to use an emerging deep learning framework called diffusion and score-matching models in this context. Specifically, we adapt denoising diffusion probabili…
Tel Aviv University（特拉维夫大学）；Technion（以色列理工学院）；NetApp
We present a method for matching a text sentence from a given corpus to a given video clip and vice versa. Traditionally video and text matching is done by learning a shared embedding space and the encoding of one modality is independent of the other. In this work, we encode the dataset data in a way that takes into account the query's relevant information. The power of the method is demonstrated to arise from pooling the interaction data between words and frames. Since the encoding of the video clip depends on the sentence compared to it, the representation needs to be recomputed for each pot…
Tsinghua-Berkeley Shenzhen Institute（清华-伯克利深圳学院）；Tsinghua University（清华大学）；Tencent AI Lab（腾讯人工智能实验室）
Temporal sentence grounding in videos aims to detect and localize one target video segment, which semantically corresponds to a given sentence. Existing methods mainly tackle this task via matching and aligning semantics between a sentence and candidate video segments, while neglect the fact that the sentence information plays an important role in temporally correlating and composing the described contents in videos. In this paper, we propose a novel semantic conditioned dynamic modulation (SCDM) mechanism, which relies on the sentence semantics to modulate the temporal convolution operations …
Action- and Language-Conditioned Video Assessment for Embodied Control
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院电气工程学院）
I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning
Beijing Jiaotong University（北京交通大学）；HUJING Digital Media & Entertainment Group（汇晶数字媒体与娱乐集团）；MAIS Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所MAIS）
