Recently, researchers have proposed various deep learning methods to accurately detect infrared targets with the characteristics of indistinct shape and texture. Due to the limited variety of infrared datasets, training deep learning models with good generalization poses a challenge. To augment the infrared dataset, researchers employ data augmentation techniques, which often involve generating new images by combining images from different datasets. However, these methods are lacking in two respects. In terms of realism, the images generated by mixup-based methods lack realism and are difficult to effectively simulate complex real-world scenarios. In terms of diversity, compared with real-world scenes, borrowing knowledge from another dataset inherently has a limited diversity. Currently, the diffusion model stands out as an innovative generative approach. Large-scale trained diffusion models have a strong generative prior that enables real-world modeling of images to generate diverse and realistic images. In this paper, we propose Diff-Mosaic, a data augmentation method based on the diffusion model. This model effectively alleviates the challenge of diversity and realism of data augmentation methods via diffusion prior. Specifically, our method consists of two stages. Firstly, we introduce an enhancement network called Pixel-Prior, which generates highly coordinated and realistic Mosaic images by harmonizing pixels. In the second stage, we propose an image enhancement strategy named Diff-Prior. This strategy utilizes diffusion priors to model images in the real-world scene, further enhancing the diversity and realism of the images. Extensive experiments have demonstrated that our approach significantly improves the performance of the detection network. The code is available at https://github.com/YupeiLin2388/Diff-Mosaic
MosaicMem: Hybrid Spatial Memory for Controllable Video World Models
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；The University of Osaka（大阪大学）；Georgia Institute of Technology（佐治亚理工学院）；Mujin Inc.（Mujin公司）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Taiyuan University of Technology（太原本科技大学）
This article proposes a novel approach for augmenting generative adversarial network (GAN) with a self-supervised task in order to improve its ability for encoding video representations that are useful in downstream tasks such as human activity recognition. In the proposed method, input video frames are randomly transformed by different spatial transformations, such as rotation, translation and shearing or temporal transformations such as shuffling temporal order of frames. Then discriminator is encouraged to predict the applied transformation by introducing an auxiliary loss. Subsequently, re…
Augmenting medical image classifiers with synthetic data from latent diffusion models
Harvard Medical School（哈佛医学院）；University of Oxford（牛津大学）；Northwestern University（西北大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Memorial Sloan Kettering Cancer Center（纪念斯隆-凯特琳癌症中心）；Stanford School of Medicine（斯坦福医学院）
A panoramic image mosaic is an attractive visualization for viewing many overlapping photos, but its images must be both captured and processed correctly to produce an acceptable composite. We propose Swipe Mosaics, an interactive visualization that places the individual video frames on a 2D planar map that represents the layout of the physical scene. Compared to traditional panoramic mosaics, our capture is easier because the user can both translate the camera center and film moving subjects. Processing and display degrade gracefully if the footage lacks distinct, overlapping, non-repeating t…
The problem of overlapping occlusion in target recognition has been a difficult research problem, and the situation of mutual occlusion of ship targets in narrow waters still exists. In this paper, an improved mosaic data enhancement method is proposed, which optimizes the reading method of the data set, strengthens the learning ability of the detection algorithm for local features, improves the recognition accuracy of overlapping targets while keeping the test speed unchanged, reduces the decay rate of recognition ability under different resolutions, and strengthens the robustness of the algo…
The Ohio State University（俄亥俄州立大学）；Google Research（谷歌研究院）；University of Southern California（南加州大学）
Many objects do not appear frequently enough in complex scenes (e.g., certain handbags in living rooms) for training an accurate object detector, but are often found frequently by themselves (e.g., in product images). Yet, these object-centric images are not effectively leveraged for improving object detection in scene-centric images. In this paper, we propose Mosaic of Object-centric images as Scene-centric images (MosaicOS), a simple and novel framework that is surprisingly effective at tackling the challenges of long-tailed object detection. Keys to our approach are three-fold: (i) pseudo s…
This paper presents a novel framework for generating texture mosaics with convolutional neural networks. Our method is called GANosaic and performs optimization in the latent noise space of a generative texture model, which allows the transformation of a content image into a mosaic exhibiting the visual properties of the underlying texture manifold. To represent that manifold, we use a state-of-the-art generative adversarial method for texture synthesis, which can learn expressive texture representations from data and produce mosaic images with very high resolution. This fully convolutional mo…
This paper introduces a novel method to super-resolve multi-spectral images captured by modern real-time single-shot mosaic image sensors, also known as multi-spectral cameras. Our contribution is two-fold. Firstly, we super-resolve multi-spectral images from mosaic images rather than image cubes, which helps to take into account the spatial offset of each wavelength. Secondly, we introduce an external multi-scale feature aggregation network (Multi-FAN) which concatenates the feature maps with different levels of semantic information throughout a super-resolution (SR) network. A cascade of con…
Method for a mosaic image representation (MIR) is proposed for a selective treatment of image fragments of different transition frequency. MIR method is based on piecewise-constant image approximation on a non-uniform orthogonal grid constructed by the following recurrent multigrid algorithm. A sequence of nested uniform grids is built, such that each cell of a current grid is subdivided into four smaller cells for the next grid designing. In each grid the cells are selected, where the color intensity function can be approximated by its average value with a given precision (thereafter 'good' c…
CLIM: Contrastive Language-Image Mosaic for Region Representation
Nanyang Technological University（南洋理工大学）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；SenseTime Research（商汤科技研究院）；Shanghai AI Laboratory（上海人工智能实验室）
SMILENet: Unleashing Extra-Large Capacity Image Steganography via a Synergistic Mosaic InvertibLE Hiding Network
College of Computer Science and Technology, National University of Defense Technology（国防科技大学计算机科学与技术学院）；School of Electronic Information Engineering, Beihang University（北京航空航天大学电子信息工程学院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；Department of Electrical and Electronic Engineering, Imperial College London（帝国理工学院电气与电子工程系）
CSIRO-Data61（澳大利亚联邦科学与工业研究组织-Data61）；Australian National University（澳大利亚国立大学）
Advances in the design of multi-spectral cameras have led to great interests in a wide range of applications, from astronomy to autonomous driving. However, such cameras inherently suffer from a trade-off between the spatial and spectral resolution. In this paper, we propose to address this limitation by introducing a novel method to carry out super-resolution on raw mosaic images, multi-spectral or RGB Bayer, captured by modern real-time single-shot mosaic sensors. To this end, we design a deep super-resolution architecture that benefits from a sequential feature pyramid along the depth of th…
Automated Mosaic Tesserae Segmentation via Deep Learning Techniques
School of Electrical and Computer Engineering, Technical University of Crete, Chania, Crete, Greece（电气与计算机工程学院，希腊克里特技术大学，克里特，希腊）；School of Production Engineering and Management, Technical University of Crete, Chania, Crete, Greece（生产工程与管理学院，希腊克里特技术大学，克里特，希腊）
A Recipe for CAC: Mosaic-based Generalized Loss for Improved Class-Agnostic Counting
National Yang Ming Chiao Tung University（国立阳明交通大学）；University of Michigan（密歇根大学）；Research Center of Information Technology Innovation, Academia Sinica（中央研究院信息技术创新研究中心）
We introduce MosAIc, an interactive web app that allows users to find pairs of semantically related artworks that span different cultures, media, and millennia. To create this application, we introduce Conditional Image Retrieval (CIR) which combines visual similarity search with user supplied filters or "conditions". This technique allows one to find pairs of similar images that span distinct subsets of the image corpus. We provide a generic way to adapt existing image retrieval data-structures to this new domain and provide theoretical bounds on our approach's efficiency. To quantify the per…
Constraint Model for the Satellite Image Mosaic Selection Problem
University of Luxembourg（卢森堡大学）；Interdisciplinary Centre for Security, Reliability and Trust (SnT)（安全、可靠性与信任跨学科中心（SnT））；Poznan University of Technology（波兹南理工大学）
In this paper, we propose to augment image-based place recognition with structural cues. Specifically, these structural cues are obtained using structure-from-motion, such that no additional sensors are needed for place recognition. This is achieved by augmenting the 2D convolutional neural network (CNN) typically used for image-based place recognition with a 3D CNN that takes as input a voxel grid derived from the structure-from-motion point cloud. We evaluate different methods for fusing the 2D and 3D features and obtain best performance with global average pooling and simple concatenation. …
ETH Zürich（苏黎世联邦理工学院）；Institute of Molecular Systems Biology（分子系统生物学研究所）
Recent advances in computer vision and robotics enabled automated large-scale biological image analysis. Various machine learning approaches have been successfully applied to phenotypic profiling. However, it remains unclear how they compare in terms of biological feature extraction. In this study, we propose a simple CNN architecture and implement 4 different representation learning approaches. We train 16 deep learning setups on the 770k cancer cell images dataset under identical conditions, using different augmenting and cropping strategies. We compare the learned representations by evaluat…
University of Waterloo（滑铁卢大学）；Vector Institute（向量研究所）；Ontario Tech University（安大略理工大学）
Deep learning methods such as convolutional neural networks (CNNs) are difficult to directly utilize to analyze whole slide images (WSIs) due to the large image dimensions. We overcome this limitation by proposing a novel two-stage approach. First, we extract a set of representative patches (called mosaic) from a WSI. Each patch of a mosaic is encoded to a feature vector using a deep network. The feature extractor model is fine-tuned using hierarchical target labels of WSIs, i.e., anatomic site and primary diagnosis. In the second stage, a set of encoded patch-level features from a WSI is used…
Amazon Web Services（亚马逊云服务）；National University of Singapore（新加坡国立大学）
Emotion recognition and understanding is a vital component in human-machine interaction. Dimensional models of affect such as those using valence and arousal have advantages over traditional categorical ones due to the complexity of emotional states in humans. However, dimensional emotion annotations are difficult and expensive to collect, therefore they are not as prevalent in the affective computing community. To address these issues, we propose a method to generate synthetic images from existing categorical emotion datasets using face morphing as well as dimensional labels in the circumplex…
The University of Sydney（悉尼大学）；Stevens Institute of Technology（斯蒂文斯理工学院）
Data mixing augmentation has proved effective in training deep models. Recent methods mix labels mainly based on the mixture proportion of image pixels. As the main discriminative information of a fine-grained image usually resides in subtle regions, methods along this line are prone to heavy label noise in fine-grained recognition. We propose in this paper a novel scheme, termed as Semantically Proportional Mixing (SnapMix), which exploits class activation map (CAM) to lessen the label noise in augmenting fine-grained data. SnapMix generates the target label for a mixed image by estimating it…
SenseBrain；SenseTime Research（商汤科技研究院）；Shanghai AI Laboratory（上海人工智能实验室）；Nanyang Technological University（南洋理工大学）
Developing and integrating advanced image sensors with novel algorithms in camera systems are prevalent with the increasing demand for computational photography and imaging on mobile platforms. However, the lack of high-quality data for research and the rare opportunity for in-depth exchange of views from industry and academia constrain the development of mobile intelligent photography and imaging (MIPI). To bridge the gap, we introduce the first MIPI challenge, including five tracks focusing on novel image sensors and imaging algorithms. In this paper, Quad Joint Remosaic and Denoise, one of …
Weizmann Institute of Science（魏茨曼科学研究所）；Facebook AI Research（Facebook AI研究院）
Implicit neural representation is a recent approach to learn shape collections as zero level-sets of neural networks, where each shape is represented by a latent code. So far, the focus has been shape reconstruction, while shape generalization was mostly left to generic encoder-decoder or auto-decoder regularization. In this paper we advocate deformation-aware regularization for implicit neural representations, aiming at producing plausible deformations as latent code changes. The challenge is that implicit representations do not capture correspondences between different shapes, which makes it…
