Differentiable image sampling in the form of backward warping has seen broad adoption in tasks like depth estimation and optical flow prediction. In contrast, how to perform forward warping has seen less attention, partly due to additional challenges such as resolving the conflict of mapping multiple pixels to the same target location in a differentiable way. We propose softmax splatting to address this paradigm shift and show its effectiveness on the application of frame interpolation. Specifically, given two input frames, we forward-warp the frames and their feature pyramid representations based on an optical flow estimate using softmax splatting. In doing so, the softmax splatting seamlessly handles cases where multiple source pixels map to the same target location. We then use a synthesis network to predict the interpolation result from the warped representations. Our softmax splatting allows us to not only interpolate frames at an arbitrary time but also to fine tune the feature pyramid and the optical flow. We show that our synthesis approach, empowered by softmax splatting, achieves new state-of-the-art results for video frame interpolation.
GS2E: Gaussian Splatting is an Effective Data Generator for Event Stream Generation
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Holcombe Department of Electrical and Computer Engineering, Clemson University（克莱姆森大学霍尔科姆电子与计算机工程系）
Frame interpolation is an essential video processing technique that adjusts the temporal resolution of an image sequence. While deep learning has brought great improvements to the area of video frame interpolation, techniques that make use of neural networks can typically not easily be deployed in practical applications like a video editor since they are either computationally too demanding or fail at high resolutions. In contrast, we propose a deep learning approach that solely relies on splatting to synthesize interpolated frames. This splatting-based synthesis for video frame interpolation …
NG-GS: NeRF-Guided 3D Gaussian Splatting Segmentation
Key Laboratory of Big Data and Artificial Intelligence in Transportation, Ministry of Education（交通大数据与人工智能联合实验室，教育部）；School of Computer and Information Technology, Beijing Jiaotong University（北京交通大学计算机与信息学院）；Department of Artificial Intelligence, Westlake University（西湖大学人工智能学院）
Enhancing Novel View Synthesis from extremely sparse views with SfM-free 3D Gaussian Splatting Framework
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University（电子与电气工程系，香港理工大学）；School of Intelligent Systems Engineering, Sun Yat-sen University（智能系统工程学院，中山大学）
SuperGaussians: Enhancing Gaussian Splatting Using Primitives with Spatially Varying Colors
The University of Hong Kong（香港大学）；Shenzhen Graduate School of Peking University（北京大学深圳研究生院）；Nanyang Technological University（南洋理工大学）；Hong Kong University of Science and Technology（香港科技大学）；Macau University of Science and Technology（澳门科技大学）；Shandong University（山东大学）；Texas A&M University（德克萨斯农工大学）
MG-SLAM: Structure Gaussian Splatting SLAM with Manhattan World Hypothesis
Department of Information Science and Technology and Department of Complexity Science and Engineering, The University of Tokyo（信息科学与技术系和复杂科学与工程系，东京大学）；Institute of Medical Robotics and Department of Automation, Shanghai Jiao Tong University（医疗机器人研究所和自动化系，上海交通大学）；Department of Mechanical Engineering, Columbia University（机械工程系，哥伦比亚大学）；School of Electrical and Electronic Engineering, Nanyang Technological University（电子与电气工程学院，南洋理工大学）；Department of Computer Science, Dalian University of Technology（计算机科学系，大连理工大学）
MrGS: Multi-modal Radiance Fields with 3D Gaussian Splatting for RGB-Thermal Novel View Synthesis
Minnesota Robotics Institute (MnRI), University of Minnesota, Twin Cities（明尼苏达大学罗学院（MnRI）、明尼苏达大学双城分校）；Department of Information Convergence Engineering (Artificial Intelligence Major), Pusan National University（信息融合工程系（人工智能专业），釜山国立大学）；Department of Energy Engineering, Korea Institute of Energy Technology (KENTECH)（能源工程系，韩国能源技术研究所（KENTECH））；School of Computer Science and Engineering, Pusan National University（计算机科学与工程学院，釜山国立大学）
Boston University（波士顿大学）；Adobe Research（奥多比研究院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Motion-based video frame interpolation commonly relies on optical flow to warp pixels from the inputs to the desired interpolation instant. Yet due to the inherent challenges of motion estimation (e.g. occlusions and discontinuities), most state-of-the-art interpolation approaches require subsequent refinement of the warped result to generate satisfying outputs, which drastically decreases the efficiency for multi-frame interpolation. In this work, we propose a fully differentiable Many-to-Many (M2M) splatting framework to interpolate frames efficiently. Specifically, given a frame pair, we es…
FLOWING: Implicit Neural Flows for Structure-Preserving Morphing
EPFL（苏黎世联邦理工学院）；University of Buenos Aires（布宜诺斯艾利斯大学）；University of São Paulo（圣保罗大学）；IMPA（巴西科学院）；Universidade Federal Rural de Pernambuco（皮钦查联邦农业大学）；ISR-UC（UC ISR）；INCM；University of Georgia（佐治亚大学）
Megvii Technology（旷视科技）；Tsinghua University（清华大学）；University of Electronic Science and Technology of China（电子科技大学）
Obtaining the ground truth labels from a video is challenging since the manual annotation of pixel-wise flow labels is prohibitively expensive and laborious. Besides, existing approaches try to adapt the trained model on synthetic datasets to authentic videos, which inevitably suffers from domain discrepancy and hinders the performance for real-world applications. To solve these problems, we propose RealFlow, an Expectation-Maximization based framework that can create large-scale optical flow datasets directly from any unlabeled realistic videos. Specifically, we first estimate optical flow be…
