arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.02672cs.CV
收藏

Latent-INR:一种具有判别语义的视频隐式表示的灵活框架

Latent-INR: A Flexible Framework for Implicit Representations of Videos with Discriminative Semantics

University of Maryland, College Park(马里兰大学帕克分校)

Shishira R Maiya, Anubhav Gupta, Matthew Gwilliam, Max Ehrlich, Abhinav Shrivastava

AI总结:

本文提出 Latent-INR 框架,通过逐帧潜变量字典与视频特定超网络解耦视频 INR 的空间与时间方面,使学习到的潜变量可与 CLIP、VideoLlama 等大模型对齐,从而在保持压缩效率的同时赋予视频隐式表示判别语义,支持检索、对话和插值等任务。

摘要:

隐式神经网络(INRs)已成为编码各种形式数据(包括图像、视频、音频和场景)的强大表示方法。在视频方面,许多用于视频的 INR 已被提出用于压缩任务,近期方法在编码时间、存储和重建质量方面取得了显著改进。然而,这些编码表示缺乏语义含义,因此无法用于任何需要此类属性的下游任务,例如检索。这可能成为视频 INR 相对于传统编解码器被采用的障碍,因为除了压缩之外,它们没有提供任何显著优势。为了缓解这一问题,我们提出了一个灵活的框架,将视频 INR 的空间和时间方面解耦。我们通过一个逐帧潜变量字典来实现这一点,该字典与一组视频特定的超网络联合学习,使得给定一个潜变量,这些超网络可以预测 INR 权重以重建给定帧。该框架不仅保持了压缩效率,而且学习到的潜变量可以与大型视觉模型的特征对齐,从而赋予它们判别属性。我们将这些潜变量与 CLIP 对齐,并在压缩和视频检索任务中展示了良好的性能。通过与 VideoLlama 对齐,我们能够以学习到的潜变量作为视觉输入进行开放式对话。此外,学习到的潜变量可作为底层权重的代理,使我们能够执行视频插值等任务。这些语义属性和应用,与执行压缩、插值和超分辨率属性的能力同时存在,在该工作领域中尚属首次。

英文摘要:

Implicit Neural Networks (INRs) have emerged as powerful representations to encode all forms of data, including images, videos, audios, and scenes. With video, many INRs for video have been proposed for the compression task, and recent methods feature significant improvements with respect to encoding time, storage, and reconstruction quality. However, these encoded representations lack semantic meaning, so they cannot be used for any downstream tasks that require such properties, such as retrieval. This can act as a barrier for adoption of video INRs over traditional codecs as they do not offer any significant edge apart from compression. To alleviate this, we propose a flexible framework that decouples the spatial and temporal aspects of the video INR. We accomplish this with a dictionary of per-frame latents that are learned jointly with a set of video specific hypernetworks, such that given a latent, these hypernetworks can predict the INR weights to reconstruct the given frame. This framework not only retains the compression efficiency, but the learned latents can be aligned with features from large vision models, which grants them discriminative properties. We align these latents with CLIP and show good performance for both compression and video retrieval tasks. By aligning with VideoLlama, we are able to perform open-ended chat with our learned latents as the visual inputs. Additionally, the learned latents serve as a proxy for the underlying weights, allowing us perform tasks like video interpolation. These semantic properties and applications, existing simultaneously with ability to perform compression, interpolation, and superresolution properties, are a first in this field of work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑