This project implements the paper "Video Textures" by Szeliski. The aim is to create a "Moving Picture" or as we popularly call it, a GIF; which is "somewhere between a photograph and a video". The idea is to input a video which has some repeated motion (the texture), such as a flag waving, rain, or a candle flame. The output is a new video that infinitely extends the original video in a seamless way. In practice, the output isn't really infinte, but is instead looped using a video player and is sufficiently long as to appear to never repeat.
Our goal from this implementation was to: improve distance metric by switching from a crude sum of squared distance to most sophisticated wavelet-based distance; add intensity normalization, cross-fading and morphing to the suggested basic algorithm. We also experiment on the trade-off between variety and smoothness.
We investigate how well CLIP understands texture in natural images described by natural language. To this end, we analyze CLIP's ability to: (1) perform zero-shot learning on various texture and material classification datasets; (2) represent compositional properties of texture such as red dots or yellow stripes on the Describable Texture in Detail(DTDD) dataset; and (3) aid fine-grained categorization of birds in photographs described by color and texture of their body parts.
On the dynamic evolution of CLIP texture-shape bias and its relationship to human alignment and model robustness
Image Processing Lab, Universitat de Valencia, Spain（瓦伦西亚大学图像处理实验室）；Centro de Biomateriales e Ingenieria Tisular, Universitat Politecnica de Valencia, Spain（瓦伦西亚理工大学生物材料与组织工程中心）；Computer Vision Center, Spain（西班牙计算机视觉中心）；Universitat Autònoma de Barcelona, Spain（巴塞罗那自治大学）
We present a technique for zero-shot generation of a 3D model using only a target text prompt. Without any 3D supervision our method deforms the control shape of a limit subdivided surface along with its texture map and normal map to obtain a 3D asset that corresponds to the input text prompt and can be easily deployed into games or modeling applications. We rely only on a pre-trained CLIP model that compares the input text prompt with differentiably rendered images of our 3D model. While previous works have focused on stylization or required training of generative models we perform optimizati…
Seeing 3D Through 2D Lenses: 3D Few-Shot Class-Incremental Learning via Cross-Modal Geometric Rectification
South China University of Technology（南方科技大学）；State Key Laboratory of Subtropical Building Science（亚热带建筑科学国家重点实验室）；Guangdong Provincial Key Lab of Computational Intelligence and Cyberspace Information（广东省计算智能与网络信息重点实验室）；Ministry of Education Key Laboratory of Big Data and Intelligent Robot（教育部大数据与智能机器人重点实验室）；Singapore Management University（新加坡国立大学）
Tel Aviv University（特拉维夫大学）；Cornell Tech, Cornell University（康奈尔科技学院，康奈尔大学）
As image generation techniques mature, there is a growing interest in explainable representations that are easy to understand and intuitive to manipulate. In this work, we turn to co-occurrence statistics, which have long been used for texture analysis, to learn a controllable texture synthesis model. We propose a fully convolutional generative adversarial network, conditioned locally on co-occurrence statistics, to generate arbitrarily large images while having local, interpretable control over the texture appearance. To encourage fidelity to the input condition, we introduce a novel differen…
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Adobe Research（奥多比研究院）
Human speech is often accompanied by body gestures including arm and hand gestures. We present a method that reenacts a high-quality video with gestures matching a target speech audio. The key idea of our method is to split and re-assemble clips from a reference video through a novel video motion graph encoding valid transitions between clips. To seamlessly connect different clips in the reenactment, we propose a pose-aware video blending network which synthesizes video frames around the stitched frames between two clips. Moreover, we developed an audio-based gesture searching algorithm to fin…
Tel Aviv University（特拉维夫大学）；University of Copenhagen（哥本哈根大学）
We make the distinction between (i) style transfer, in which a source image is manipulated to match the textures and colors of a target image, and (ii) essence transfer, in which one edits the source image to include high-level semantic attributes from the target. Crucially, the semantic attributes that constitute the essence of an image may differ from image to image. Our blending operator combines the powerful StyleGAN generator and the semantic encoder of CLIP in a novel way that is simultaneously additive in both latent spaces, resulting in a mechanism that guarantees both identity preserv…
TSCLIP: Robust CLIP Fine-Tuning for Worldwide Cross-Regional Traffic Sign Recognition
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Wuhan Polytechnic University（武汉轻工大学）；The Hong Kong University of Science and Technology（香港科技大学）
Content-based multimedia information retrieval is an interesting research area since it allows retrieval based on inherent characteristic of multimedia objects. For example retrieval based on visual characteristics such as colour, shapes or textures of objects in images or retrieval based on spatial relationships among objects in the media (images or video clips). This paper reviews some work done in image and video retrieval and then proposes an integrated model that can handle images and video clips uniformly. Using this model retrieval on images or video clips can be done based on the same …
Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation
University of Toronto（多伦多大学）；China Agricultural University（中国农业大学）；Concordia University（Concordia大学）；McMaster University（麦马斯特大学）；Beijing Jiaotong University（北京交通大学）
SegTalker: Segmentation-based Talking Face Generation with Mask-guided Local Editing
South China University of Technology（华南理工大学）；Zhejiang University（浙江大学）；Huawei Cloud Computing Technologies Co., Ltd（华为云计算技术有限公司）；Peking University（北京大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））
Isotropic3D: Image-to-3D Generation Based on a Single CLIP Embedding
Academy for Engineering and Technology, Fudan University（复旦大学工程技术研究院）；Beijing National Research Center for Information Science and Technology (BNRist)（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）
