Conventional image quality metrics (IQMs), such as PSNR and SSIM, are designed for perceptually uniform gamma-encoded pixel values and cannot be directly applied to perceptually non-uniform linear high-dynamic-range (HDR) colors. Similarly, most of the available datasets consist of standard-dynamic-range (SDR) images collected in standard and possibly uncontrolled viewing conditions. Popular pre-trained neural networks are likewise intended for SDR inputs, restricting their direct application to HDR content. On the other hand, training HDR models from scratch is challenging due to limited available HDR data. In this work, we explore more effective approaches for training deep learning-based models for image quality assessment (IQA) on HDR data. We leverage networks pre-trained on SDR data (source domain) and re-target these models to HDR (target domain) with additional fine-tuning and domain adaptation. We validate our methods on the available HDR IQA datasets, demonstrating that models trained with our combined recipe outperform previous baselines, converge much quicker, and reliably generalize to HDR inputs.
Cranberry-Lemon University（蔓越莓柠檬大学）；University of the Witwatersrand（金山大学）
Anomaly detection methods require high-quality features. In recent years, the anomaly detection community has attempted to obtain better features using advances in deep self-supervised feature learning. Surprisingly, a very promising direction, using pretrained deep features, has been mostly overlooked. In this paper, we first empirically establish the perhaps expected, but unreported result, that combining pretrained features with simple anomaly detection and segmentation methods convincingly outperforms, much more complex, state-of-the-art methods. In order to obtain further performance gain…
Image quality assessment (IQA) is an important element of a broad spectrum of applications ranging from automatic video streaming to display technology. Furthermore, the measurement of image quality requires a balanced investigation of image content and features. Our proposed approach extracts visual features by attaching global average pooling (GAP) layers to multiple Inception modules of on an ImageNet database pretrained convolutional neural network (CNN). In contrast to previous methods, we do not take patches from the input image. Instead, the input image is treated as a whole and is run …
Recent advances in text-to-image synthesis have led to large pretrained transformers with excellent capabilities to generate visualizations from a given text. However, these models are ill-suited for specialized tasks like story visualization, which requires an agent to produce a sequence of images given a corresponding sequence of captions, forming a narrative. Moreover, we find that the story visualization task fails to accommodate generalization to unseen plots and characters in new narratives. Hence, we first propose the task of story continuation, where the generated visual story is condi…
Center for Artificial Intelligence in Medicine and Imaging（医学与成像人工智能中心）；Stanford University（斯坦福大学）
Multi-modal foundation models are typically trained on millions of pairs of natural images and text captions, frequently obtained through web-crawling approaches. Although such models depict excellent generative capabilities, they do not typically generalize well to specific domains such as medical images that have fundamentally shifted distributions compared to natural images. Building generative models for medical images that faithfully depict clinical context may help alleviate the paucity of healthcare datasets. Thus, in this study, we seek to research and expand the representational capab…
Amazon Web Services（亚马逊云科技）；USC Information Sciences Institute（南加州大学信息科学研究所）；Shanghai Jiao Tong University（上海交通大学）
Layout-to-image generation refers to the task of synthesizing photo-realistic images based on semantic layouts. In this paper, we propose LayoutDiffuse that adapts a foundational diffusion model pretrained on large-scale image or text-image datasets for layout-to-image generation. By adopting a novel neural adaptor based on layout attention and task-aware prompts, our method trains efficiently, generates images with both high perceptual quality and layout alignment, and needs less data. Experiments on three datasets show that our method significantly outperforms other 10 generative models base…
University of Michigan（密歇根大学）；Adobe Research（奥多比研究院）
Large-scale pretrained image-text models have shown incredible zero-shot performance in a handful of tasks, including video ones such as action recognition and text-to-video retrieval. However, these models have not been adapted to video, mainly because they do not account for the time dimension but also because video frames are different from the typical images (e.g., containing motion blur, and less sharpness). In this paper, we present a fine-tuning strategy to refine these large-scale pretrained image-text models for zero-shot video understanding tasks. We show that by carefully adapting t…
Xi’an Jiaotong University（西安交通大学）；University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）
Frozen pretrained models have become a viable alternative to the pretraining-then-finetuning paradigm for transfer learning. However, with frozen models there are relatively few parameters available for adapting to downstream tasks, which is problematic in computer vision where tasks vary significantly in input/output format and the type of information that is of value. In this paper, we present a study of frozen pretrained models when applied to diverse and representative computer vision tasks, including object detection, semantic segmentation and video action recognition. From this empirical…
Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models
School of Informatics, Xiamen University（厦门大学信息学院）；Center for Smart Health, School of Nursing, The Hong Kong Polytechnic University（香港理工大学护理学院智能健康中心）；Department of Biomedical Informatics, Harvard Medical School（哈佛医学院生物医学信息学系）；Department of Radiology, Zhongshan Hospital (Xiamen), Fudan University（复旦大学中山医院放射科）；National Institute for Data Science in Health and Medicine（医学与数据科学国家研究院）；Department of Computer Science, School of Informatics, Xiamen University（厦门大学信息学院计算机科学系）
GenMask: Adapting DiT for Segmentation via Direct Mask Generation
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University（上海交通大学协作中位网创新中心）；School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）；Institute of Artificial Intelligence for Medicine, Shanghai Jiao Tong University School of Medicine（上海交通大学医学院人工智能医学研究院）；Alibaba Group（阿里巴巴集团）
Harbin Institute of Technology（哈尔滨工业大学）；National University of Singapore（新加坡国立大学）；Peng Cheng Laboratory（鹏城实验室）
For long-tailed classification, most works often pretrain a big model on a large-scale dataset, and then fine-tune the whole model for adapting to long-tailed data. Though promising, fine-tuning the whole pretrained model tends to suffer from high cost in computation and deployment of different models for different tasks, as well as weakened generalization ability for overfitting to certain features of long-tailed data. To alleviate these issues, we propose an effective Long-tailed Prompt Tuning method for long-tailed classification. LPT introduces several trainable prompts into a frozen pretr…
LARGO: Low-Rank Regulated Gradient Projection for Robust Parameter Efficient Fine-Tuning
School of Artificial Intelligence, Beihang University（北航人工智能学院）；Hangzhou International Innovation Institute, Beihang University（北航杭州国际创新研究院）；Nanyang Technological University（南洋理工大学）；University of Leicester（莱斯特大学）
Tem-adapter: Adapting Image-Text Pretraining for Video Question Answer
Carnegie Mellon University（卡内基梅隆大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Eindhoven University of Technology（埃因霍温理工大学）；University of Oxford（牛津大学）；Tsinghua University（清华大学）；Toronto Metropolitan University（多伦多都会大学）
Adapting 2D Multi-Modal Large Language Model for 3D CT Image Analysis
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（香港科技大学电子及计算机工程学系）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程学系）；Institute of Medical Intelligence and XR, The Chinese University of Hong Kong（香港中文大学医学智能与扩展现实研究所）；Center for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院人工智能与机器人创新中心）
Recent breakthroughs in text-to-image synthesis have been driven by diffusion models trained on billions of image-text pairs. Adapting this approach to 3D synthesis would require large-scale datasets of labeled 3D data and efficient architectures for denoising 3D data, neither of which currently exist. In this work, we circumvent these limitations by using a pretrained 2D text-to-image diffusion model to perform text-to-3D synthesis. We introduce a loss based on probability density distillation that enables the use of a 2D diffusion model as a prior for optimization of a parametric image gener…
