The Reference-based Super-resolution (RefSR) super-resolves a low-resolution (LR) image given an external high-resolution (HR) reference image, where the reference image and LR image share similar viewpoint but with significant resolution gap x8. Existing RefSR methods work in a cascaded way such as patch matching followed by synthesis pipeline with two independently defined objective functions, leading to the inter-patch misalignment, grid effect and inefficient optimization. To resolve these issues, we present CrossNet, an end-to-end and fully-convolutional deep neural network using cross-scale warping. Our network contains image encoders, cross-scale warping layers, and fusion decoder: the encoder serves to extract multi-scale features from both the LR and the reference images; the cross-scale warping layers spatially aligns the reference feature map with the LR feature map; the decoder finally aggregates feature maps from both domains to synthesize the HR output. Using cross-scale warping, our network is able to perform spatial alignment at pixel-level in an end-to-end fashion, which improves the existing schemes both in precision (around 2dB-4dB) and efficiency (more than 100 times faster).
ETH Zurich(苏黎世联邦理工学院);Meta Inc.(Meta公司);KU Leuven(鲁汶大学)
视频恢复(例如,视频超分辨率)旨在从低质量帧恢复高质量帧。与单个图像复原不同,视频复原通常需要利用来自多个相邻但通常未对准的视频帧的时间信息。现有的深度方法通常通过利用滑动窗口策略或递归架构来解决这个问题,这要么受到逐帧恢复的限制,要么缺乏长期建模能力。在本文中,我们提出了一个视频恢复Transformer(VRT)与并行帧预测和长期的时间依赖建模能力。更具体地说,VRT由多个尺度组成,每个尺度由两种类型的模块组成:时间相互自我注意(TMSA)和并行扭曲。TMSA将视频分割成小片段,在小片段上应用相互注意进行联合运动估计、特征对齐和特征融合,而自注意用于特征提取。为了启用跨剪辑交互,视频序列每隔一层移动一次。此外,并行变形被用来进一步融合信息从相邻帧的并行特征变形。在视频超分辨率、视频去模糊、视频去噪、视频帧内插和时空视频超分辨率等5个任务上的实验结果表明,VRT在14个基准数据集上的性能明显优于现有方法($\textbf{up to 2.16dB}$).
H2-Stereo: High-Speed, High-Resolution Stereoscopic Video System
Shanghai Jiao Tong University(上海交通大学);Cooperative Medianet Innovation Center(未来媒体网络协同创新中心);University of California at Riverside(加州大学河滨分校);Nanjing University(南京大学)
Learning Flow-based Feature Warping for Face Frontalization with Illumination Inconsistent Supervision
Harbin Institute of Technology(哈尔滨工业大学);University of Burgundy Franche-Comté(勃艮第-弗朗什-孔泰大学);University of the Basque Country(巴斯克大学);Anyvision;Peng Cheng Lab(鹏城实验室)
Eliminating Warping Shakes for Unsupervised Online Video Stitching
Institute of Information Science, Beijing Jiaotong University(北京交通大学信息科学研究所);Nanyang Technological University(南洋理工大学);Communication University of Zhejiang(浙江传媒学院);University of Electronic Science and Technology of China(电子科技大学)
Liquid Warping GAN with Attention: A Unified Framework for Human Image Synthesis
ShanghaiTech University(上海科技大学);Chinese Academy of Sciences(中国科学院);Shanghai Institute of Microsystem and Information Technology(上海微系统与信息技术研究所);University of Chinese Academy of Sciences(中国科学院大学)
我们在一个统一的框架内处理人体图像合成,包括人体运动模仿,外观转移和新颖的视图合成。这意味着模型一旦经过训练,就可以用来处理所有这些任务。现有的特定任务的方法主要使用2D关键点来估计人体结构。然而,他们只表示的位置信息,没有能力来表征个性化的形状的人和模型的肢体旋转。在本文中,我们建议使用一个三维人体网格恢复模块来解开姿势和形状。它不仅可以模拟关节的位置和旋转,而且可以表征个性化的身体形状。为了保留源信息,如纹理,风格,颜色和人脸身份,我们提出了一个注意液体Warping GAN与注意液体Warping块(AttLWB),传播图像和特征空间的源信息的合成参考。具体地说,源特征提取的去噪卷积自动编码器的特征源身份很好。此外,我们提出的方法可以支持更灵活的多个来源的扭曲。为了进一步提高不可见源图像的泛化能力,应用了一次/Few-Shot对抗学习。具体来说,它首先在一个广泛的训练集中训练一个模型。然后,它以自我监督的方式通过一个/Few-Shot不可见图像对模型进行微调,以生成高分辨率(512 x 512和1024 x 1024)结果。此外,我们建立了一个新的数据集,即iPER数据集,用于评估人体运动模仿,外观转移和新的视图合成。大量的实验证明了我们的方法在保持人脸身份,形状一致性和衣服细节方面的有效性。所有代码和数据集都可以在https://impersonator.org/work/…
AttenIR: Aligning Dense Visual Attention with Mesh Warping for Unsupervised Image Retargeting
School of Media Engineering, Communication University of Zhejiang(浙江传媒学院媒体工程学院);Samsung Research America(三星美国研究院);School of Computer Science and Engineering, The University of New South Wales(新南威尔士大学计算机科学与工程学院)
Navigating Large-Pose Challenge for High-Fidelity Face Reenactment with Video Diffusion Model
National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu, 610065, China(合成视觉基础科学国家重点实验室,四川大学,成都,610065,中国);School of Cyber Science and Engineering, Sichuan University, Chengdu, 610065, China(网络科学与工程学院,四川大学,成都,610065,中国);College of Computer Science, Sichuan University, Chengdu, 610065, China(计算机科学学院,四川大学,成都,610065,中国)
MonoPCC: Photometric-invariant Cycle Constraint for Monocular Depth Estimation of Endoscopic Images
Huazhong University of Science and Technology(华中科技大学);Wuhan United Imaging Healthcare Surgical Technology Co., Ltd.(武汉联影外科技术有限公司);Tongji Hospital(同济医院);Tongji Medical College(同济医学院);Britton Chance Center for Biomedical Photonics(布里顿·钱斯生物光子学中心);Wuhan National Laboratory for Optoelectronics(武汉光电国家实验室)
Object-IR: Leveraging Object Consistency and Mesh Deformation for Self-Supervised Image Retargeting
Key Laboratory of Grain Information Processing and Control(粮食信息处理与控制重点实验室);College of Information Science and Engineering(信息科学与工程学院);Institute for Complexity Science(复杂科学研究所)