Image Aesthetics Assessment using Multi Channel Convolutional Neural Networks
Dhirubhai Ambani Institute of Information and Communication Technology（迪鲁巴希·阿姆巴尼信息与通信技术研究所）；C R Rao Advanced Institute of Mathematics, Statistics and Computer Science（C R Rao高级数学、统计与计算机科学研究所）
Image Aesthetics Assessment is one of the emerging domains in research. The domain deals with classification of images into categories depending on the basis of how pleasant they are for the users to watch. In this article, the focus is on categorizing the images in high quality and low quality image. Deep convolutional neural networks are used to classify the images. Instead of using just the raw image as input, different crops and saliency maps of the images are also used, as input to the proposed multi channel CNN architecture. The experiments reported on widely used AVA database show improvement in the aesthetic assessment performance over existing approaches.
Aesthetic assessment is subjective, and the distribution of the aesthetic levels is imbalanced. In order to realize the auto-assessment of photo aesthetics, we focus on retraining the CNN-based aesthetic assessment model by dropping out the unavailable samples in the middle levels from the training data set repetitively to overcome the effect of imbalanced aesthetic data on classification. Further, the method of extracting aesthetics highlight region of the photo image by using the two repetitively trained models is presented. Therefore, the correlation of the extracted region with the aesthet…
Recent research has widely explored the problem of aesthetics assessment of images with generic content. However, few approaches have been specifically designed to predict the aesthetic quality of images containing human faces, which make up a massive portion of photos in the web. This paper introduces a method for aesthetic quality assessment of images with faces. We exploit three different Convolutional Neural Networks to encode information regarding perceptual quality, global image aesthetics, and facial attributes; then, a model is trained to combine these features to explicitly predict th…
Beijing Electronic Science and Technology Institute（北京电子科技学院）；CETC Big Data Research Institute Co.,Ltd.（中电科大数据研究院有限公司）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；SenseTime Research（商汤科技研究院）；Beihang University（北京航空航天大学）；Peng Cheng Laboratory（鹏城实验室）
Image aesthetic quality assessment has been a relatively hot topic during the last decade. Most recently, comments type assessment (aesthetic captions) has been proposed to describe the general aesthetic impression of an image using text. In this paper, we propose Aesthetic Attributes Assessment of Images, which means the aesthetic attributes captioning. This is a new formula of image aesthetic assessment, which predicts aesthetic attributes captions together with the aesthetic score of each attribute. We introduce a new dataset named \emph{DPC-Captions} which contains comments of up to 5 aest…
Image aesthetic quality assessment has got much attention in recent years, but not many works have been done on a specific genre of photos: Group photograph. In this work, we designed a set of high-level features based on the experience and principles of group photography: Opened-eye, Gaze, Smile, Occluded faces, Face Orientation, Facial blur, Character center. Then we combined them and 83 generic aesthetic features to build two aesthetic assessment models. We also constructed a large dataset of group photographs - GPD- annotated with the aesthetic score. The experimental result shows that our…
University of Science and Technology of China（中国科学技术大学）；Beijing Electronic Science and Technology Institute（北京电子科技学院）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；University of California, Davis（加州大学戴维斯分校）
Image aesthetic quality assessment is popular during the last decade. Besides numerical assessment, nature language assessment (aesthetic captioning) has been proposed to describe the generally aesthetic impression of an image. In this paper, we propose aesthetic attribute assessment, which is the aesthetic attributes captioning, i.e., to assess the aesthetic attributes such as composition, lighting usage and color arrangement. It is a non-trivial task to label the comments of aesthetic attributes, which limit the scale of the corresponding datasets. We construct a novel dataset, named DPC-Cap…
Beijing Lab of Intelligent Information Technology（北京市智能信息技术实验室）；School of Computer Science, Beijing Institute of Technology（北京理工大学计算机学院）
We model the photo cropping problem as a cascade of attention box regression and aesthetic quality classification, based on deep learning. A neural network is designed that has two branches for predicting attention bounding box and analyzing aesthetics, respectively. The predicted attention box is treated as an initial crop window where a set of cropping candidates are generated around it, without missing important information. Then, aesthetics assessment is employed to select the final crop as the one with the best aesthetic quality. With our network, cropping candidates share features within…
Beijing Electronic Science and Technology Institute（北京电子科技学院）；University of California, Davis（加州大学戴维斯分校）；National Institute of Technology Patna（巴特那国家理工学院）
With the continuous development of social software and multimedia technology, images have become a kind of important carrier for spreading information and socializing. How to evaluate an image comprehensively has become the focus of recent researches. The traditional image aesthetic assessment methods often adopt single numerical overall assessment scores, which has certain subjectivity and can no longer meet the higher aesthetic requirements. In this paper, we construct an new image attribute dataset called aesthetic mixed dataset with attributes(AMD-A) and design external attribute features …
Nanyang Technological University（南洋理工大学）；DSO National Laboratories（DSO国家实验室）；Alibaba Group（阿里巴巴集团）；Alibaba-NTU Singapore Joint Research Institute（阿里巴巴-南洋理工大学新加坡联合研究院）
This survey aims at reviewing recent computer vision techniques used in the assessment of image aesthetic quality. Image aesthetic assessment aims at computationally distinguishing high-quality photos from low-quality ones based on photographic rules, typically in the form of binary classification or quality scoring. A variety of approaches has been proposed in the literature trying to solve this challenging problem. In this survey, we present a systematic listing of the reviewed approaches based on visual feature types (hand-crafted features and deep features) and evaluation criteria (dataset…
Beijing Electronic Science and Technology Institute（北京电子科技学院）；University of Science and Technology of China（中国科学技术大学）；University of California, Davis（加州大学戴维斯分校）
Aesthetic assessment of images can be categorized into two main forms: numerical assessment and language assessment. Aesthetics caption of photographs is the only task of aesthetic language assessment that has been addressed. In this paper, we propose a new task of aesthetic language assessment: aesthetic visual question and answering (AVQA) of images. If we give a question of images aesthetics, model can predict the answer. We use images from \textit{www.flickr.com}. The objective QA pairs are generated by the proposed aesthetic attributes analysis algorithms. Moreover, we introduce subjectiv…
University of Engineering and Technology, Peshawar（白沙瓦工程技术大学）；COMSATS University Islamabad, Wah Campus, Pakistan（伊斯兰堡COMSATS大学瓦赫校区）；Saudi Electronic University, Saudi Arabia（沙特电子大学）；Imaging and Computer vision group, Data61-CSIRO, Australia（澳大利亚CSIRO数据61中心成像与计算机视觉组）；Norwegian University of Life Sciences, Norway（挪威生命科学大学）
Automatic image aesthetics assessment is a computer vision problem dealing with categorizing images into different aesthetic levels. The categorization is usually done by analyzing an input image and computing some measure of the degree to which the image adheres to the fundamental principles of photography such as balance, rhythm, harmony, contrast, unity, look, feel, tone and texture. Due to its diverse applications in many areas, automatic image aesthetic assessment has gained significant research attention in recent years. This article presents a review of the contemporary automatic image …
Deep convolutional neural networks have recently achieved great success on image aesthetics assessment task. In this paper, we propose an efficient method which takes the global, local and scene-aware information of images into consideration and exploits the composite features extracted from corresponding pretrained deep learning models to classify the derived features with support vector machine. Contrary to popular methods that require fine-tuning or training a new model from scratch, our training-free method directly takes the deep features generated by off-the-shelf models for image classi…
SUNY at Buffalo（布法罗纽约州立大学）；Tianjin University（天津大学）
Deep convolutional neural networks (CNN) have recently been shown to generate promising results for aesthetics assessment. However, the performance of these deep CNN methods is often compromised by the constraint that the neural network only takes the fixed-size input. To accommodate this requirement, input images need to be transformed via cropping, warping, or padding, which often alter image composition, reduce image resolution, or cause image distortion. Thus the aesthetics of the original images is impaired because of potential loss of fine grained details and holistic image layout. Howev…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Youtu Lab, Tencent（腾讯优图实验室）；Snap Inc.（Snap公司）；North China Electric Power University（华北电力大学）；Xiamen University（厦门大学）；Kuaishou Technology（快手科技）
Visual aesthetic assessment has been an active research field for decades. Although latest methods have achieved promising performance on benchmark datasets, they typically rely on a large number of manual annotations including both aesthetic labels and related image attributes. In this paper, we revisit the problem of image aesthetic assessment from the self-supervised feature learning perspective. Our motivation is that a suitable feature representation for image aesthetic assessment should be able to distinguish different expert-designed image manipulations, which have close relationships w…
Automatic image aesthetics assessment is important for a wide variety of applications such as on-line photo suggestion, photo album management and image retrieval. Previous methods have focused on mapping the holistic image content to a high or low aesthetics rating. However, the composition information of an image characterizes the harmony of its visual elements according to the principles of art, and provides richer information for learning aesthetics. In this work, we propose to model the image composition information as the mutual dependency of its local regions, and design a novel archite…
Image aesthetics assessment (IAA) is a challenging task due to its highly subjective nature. Most of the current studies rely on large-scale datasets (e.g., AVA and AADB) to learn a general model for all kinds of photography images. However, little light has been shed on measuring the aesthetic quality of artistic images, and the existing datasets only contain relatively few artworks. Such a defect is a great obstacle to the aesthetic assessment of artistic images. To fill the gap in the field of artistic image aesthetics assessment (AIAA), we first introduce a large-scale AIAA dataset: Boldbr…
This paper presents our proposed methods to ICDAR 2021 Robust Reading Challenge - Integrated Circuit Text Spotting and Aesthetic Assessment (ICDAR RRC-ICTEXT 2021). For the text spotting task, we detect the characters on integrated circuit and classify them based on yolov5 detection model. We balance the lowercase and non-lowercase by using SynthText, generated data and data sampler. We adopt semi-supervised algorithm and distillation to furtherly improve the model's accuracy. For the aesthetic assessment task, we add a classification branch of 3 classes to differentiate the aesthetic classes …
Beihang University（北京航空航天大学）；Beijing Electronic Science and Technology Institute（北京电子科技学院）；Peng Cheng Laboratory（鹏城实验室）
Despite the growing number of unmanned aerial vehicles (UAVs) and aerial videos, there is a paucity of studies focusing on the aesthetics of aerial videos that can provide valuable information for improving the aesthetic quality of aerial photography. In this article, we present a method of deep multimodality learning for UAV video aesthetic quality assessment. More specifically, a multistream framework is designed to exploit aesthetic attributes from multiple modalities, including spatial appearance, drone camera motion, and scene structure. A novel specially designed motion stream network is…
Evaluating aesthetic value of digital photographs is a challenging task, mainly due to numerous factors that need to be taken into account and subjective manner of this process. In this paper, we propose to approach this problem using deep convolutional neural networks. Using a dataset of over 1.7 million photos collected from Flickr, we train and evaluate a deep learning model whose goal is to classify input images by analysing their aesthetic value. The result of this work is a publicly available Web-based application that can be used in several real-life applications, e.g. to improve the wo…
Paintings and Drawings Aesthetics Assessment with Rich Attributes for Various Artistic Categories
Beijing Electronic Science and Technology Institute（北京电子科技学院）；Central Academy of Fine Arts（中央美术学院）；Beijing University of Technology（北京工业大学）；University of Science and Technology of China（中国科学技术大学）；Beijing Institute of Fashion Technology（北京服装学院）
An Order-Complexity Aesthetic Assessment Model for Aesthetic-aware Music Recommendation
Beijing Electronic Science and Technology Institute（北京电子科技学院）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；Tianjin Conservatory of Music（天津音乐学院）；Qingdao Academy of Intelligent Industries（青岛智能产业技术研究院）；Tongji University（同济大学）；Sun Yat-sen University（中山大学）
Trinity College Dublin（都柏林圣三一学院）；Accenture Labs（埃森哲实验室）
Aspect ratio and spatial layout are two of the principal factors determining the aesthetic value of a photograph. But, incorporating these into the traditional convolution-based frameworks for the task of image aesthetics assessment is problematic. The aspect ratio of the photographs gets distorted while they are resized/cropped to a fixed dimension to facilitate training batch sampling. On the other hand, the convolutional filters process information locally and are limited in their ability to model the global spatial layout of a photograph. In this work, we present a two-stage framework base…
Friedrich Schiller University（弗里德里希·席勒大学）；University of Jena School of Medicine（耶拿大学医学院）；Jena University Hospital（耶拿大学医院）
Computational aesthetics is an emerging field of research which has attracted different research groups in the last few years. In this field, one of the main approaches to evaluate the aesthetic quality of paintings and photographs is a feature-based approach. Among the different features proposed to reach this goal, color plays an import role. In this paper, we introduce a novel dataset that consists of paintings of Western provenance from 36 well-known painters from the 15th to the 20th century. As a first step and to assess this dataset, using a classifier, we investigate the correlation be…
OPPO Research Institute（OPPO研究院）；Xidian University（西安电子科技大学）
Personalized image aesthetics assessment (PIAA) is challenging due to its highly subjective nature. People's aesthetic tastes depend on diversified factors, including image characteristics and subject characters. The existing PIAA databases are limited in terms of annotation diversity, especially the subject aspect, which can no longer meet the increasing demands of PIAA research. To solve the dilemma, we conduct so far, the most comprehensive subjective study of personalized image aesthetics and introduce a new Personalized image Aesthetics database with Rich Attributes (PARA), which consists…
To leverage deep learning for image aesthetics assessment, one critical but unsolved issue is how to seamlessly incorporate the information of image aspect ratios to learn more robust models. In this paper, an adaptive fractional dilated convolution (AFDC), which is aspect-ratio-embedded, composition-preserving and parameter-free, is developed to tackle this issue natively in convolutional kernel level. Specifically, the fractional dilated kernel is adaptively constructed according to the image aspect ratios, where the interpolation of nearest two integers dilated kernels is used to cope with …
The aesthetic quality of an image is defined as the measure or appreciation of the beauty of an image. Aesthetics is inherently a subjective property but there are certain factors that influence it such as, the semantic content of the image, the attributes describing the artistic aspect, the photographic setup used for the shot, etc. In this paper we propose a method for the automatic prediction of the aesthetics of an image that is based on the analysis of the semantic content, the artistic style and the composition of the image. The proposed network includes: a pre-trained network for semant…
Multi-modal Learnable Queries for Image Aesthetics Assessment
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Alibaba-NTU Singapore Joint Research Institute, Nanyang Technological University（阿里巴巴-南洋理工大学新加坡联合研究院）；Alibaba Group（阿里巴巴集团）
ArtiMuse: Fine-Grained Image Aesthetics Assessment with Joint Scoring and Expert-Level Understanding
USTC（中国科学技术大学）；Shanghai AI Lab（上海人工智能实验室）；China Academy of Art（中国美术学院）；Peking University（北京大学）；SJTU（上海交通大学）；Sun Yat-sen University（中山大学）；CUHK（香港中文大学）；Hong Kong PolyU（香港理工大学）
Isfahan University of Technology（伊斯法罕理工大学）；McMaster University（麦克马斯特大学）
Image retargeting aims at altering an image size while preserving important content and minimizing noticeable distortions. However, previous image retargeting methods create outputs that suffer from artifacts and distortions. Besides, most previous works attempt to retarget the background and foreground of the input image simultaneously. Simultaneous resizing of the foreground and background causes changes in the aspect ratios of the objects. The change in the aspect ratio is specifically not desirable for human objects. We propose a retargeting method that overcomes these problems. The propos…
UniQA: Unified Vision-Language Pre-training for Image Quality and Aesthetic Assessment
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；Media Analytics and Computing Lab, Department of Artificial Intelligence, School of Informatics, Xiamen University（媒体分析与计算实验室，人工智能系，厦门大学）；School of Computer Science and Technology, Ocean University of China（计算机科学与技术学院，中国海洋大学）；Institute of Image Communication and Information Processing, Shanghai Jiao Tong University（图像通信与信息处理研究所，上海交通大学）
Zhengzhou University（郑州大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Personalized image aesthetic assessment (PIAA) has recently become a hot topic due to its usefulness in a wide variety of applications such as photography, film and television, e-commerce, fashion design and so on. This task is more seriously affected by subjective factors and samples provided by users. In order to acquire precise personalized aesthetic distribution by small amount of samples, we propose a novel user-guided personalized image aesthetic assessment framework. This framework leverages user interactions to retouch and rank images for aesthetic assessment based on deep reinforcemen…
Learning to Look before Learning to Like: Incorporating Human Visual Cognition into Aesthetic Quality Assessment
Faculty of Data Science, City University of Macau（澳门城市大学数据科学学院）；Zhongshan Ophthalmic Center, Sun Yat-sen University（中山眼科中心，中山大学）；School of computer science, Torrens University Australia（计算机科学学院，托伦斯大学澳大利亚分校）
University of Nantes（南特大学）；Tencent（腾讯）；Ocean University of China（中国海洋大学）
With the proliferation of various gaming technology, services, game styles, and platforms, multi-dimensional aesthetic assessment of the gaming contents is becoming more and more important for the gaming industry. Depending on the diverse needs of diversified game players, game designers, graphical developers, etc. in particular conditions, multi-modal aesthetic assessment is required to consider different aesthetic dimensions/perspectives. Since there are different underlying relationships between different aesthetic dimensions, e.g., between the `Colorfulness' and `Color Harmony', it could b…
VADB: A Large-Scale Video Aesthetic Database with Professional and Multi-Dimensional Annotations
Nanjing University（南京大学）；Huazhong University of Science and Technology（华中科技大学）；Beijing Film Academy（北京电影学院）；University of Science and Technology of China（中国科学技术大学）；Beijing Electronic Science and Technology Institute（北京电子科技学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）
University of Malaya（马来亚大学）；ViTrox Technologies Sdn. Bhd.（ViTrox科技有限公司）；The University of Adelaide（阿德莱德大学）；The Chinese University of Hong Kong（香港中文大学）；South China University of Technology（华南理工大学）；Baidu（百度）；Webank（微众银行）
With hundreds of thousands of electronic chip components are being manufactured every day, chip manufacturers have seen an increasing demand in seeking a more efficient and effective way of inspecting the quality of printed texts on chip components. The major problem that deters this area of research is the lacking of realistic text on chips datasets to act as a strong foundation. Hence, a text on chips dataset, ICText is used as the main target for the proposed Robust Reading Challenge on Integrated Circuit Text Spotting and Aesthetic Assessment (RRC-ICText) 2021 to encourage the research on …
University College London（伦敦大学学院）；Royal Free & University College Medical School（皇家自由与大学学院医学院）；Royal Free London Foundation Trust（皇家自由伦敦基金会信托）
Breast cancer is the most prevalent cancer type in women, and while its survival rate is generally high the aesthetic outcome is an increasingly important factor when evaluating different treatment alternatives. 3D scanning and reconstruction techniques offer a flexible tool for building detailed and accurate 3D breast models that can be used both pre-operatively for surgical planning and post-operatively for aesthetic evaluation. This paper aims at comparing the accuracy of low-cost 3D scanning technologies with the significantly more expensive state-of-the-art 3D commercial scanners in the c…
One Model, Two Minds: Task-Conditioned Reasoning for Unified Image Quality and Aesthetic Assessment
University of Electronic Science and Technology of China（电子科技大学）；Faculty of Information Technology, Monash University（莫纳什大学信息科技学院）；The Laboratory of Intelligent Collaborative Computing of UESTC（UESTC智能协同计算实验室）
Beijing Electronic Science and Technology Institute（北京电子科技学院）
The task of aesthetic quality assessment is complicated due to its subjectivity. In recent years, the target representation of image aesthetic quality has changed from a one-dimensional binary classification label or numerical score to a multi-dimensional score distribution. According to current methods, the ground truth score distributions are straightforwardly regressed. However, the subjectivity of aesthetics is not taken into account, that is to say, the psychological processes of human beings are not taken into consideration, which limits the performance of the task. In this paper, we pro…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Human beings often assess the aesthetic quality of an image coupled with the identification of the image's semantic content. This paper addresses the correlation issue between automatic aesthetic quality assessment and semantic recognition. We cast the assessment problem as the main task among a multi-task deep model, and argue that semantic recognition task offers the key to address this problem. Based on convolutional neural networks, we employ a single and simple multi-task framework to efficiently utilize the supervision of aesthetic and semantic labels. A correlation item between these tw…
In this work, we point out that the major dilemma of image aesthetics assessment (IAA) comes from the abstract nature of aesthetic labels. That is, a vast variety of distinct contents can correspond to the same aesthetic label. On the one hand, during inference, the IAA model is required to relate various distinct contents to the same aesthetic label. On the other hand, when training, it would be hard for the IAA model to learn to distinguish different contents merely with the supervision from aesthetic labels, since aesthetic labels are not directly related to any specific content. To deal wi…
