Aesthetic Visual Question Answering of Photographs
Beijing Electronic Science and Technology Institute（北京电子科技学院）；University of Science and Technology of China（中国科学技术大学）；University of California, Davis（加州大学戴维斯分校）
Xin Jin, Wu Zhou, Xinghui Zhou, Shuai Cui, Le Zhang, Jianwen Lv, Shu Zhao
Aesthetic assessment of images can be categorized into two main forms: numerical assessment and language assessment. Aesthetics caption of photographs is the only task of aesthetic language assessment that has been addressed. In this paper, we propose a new task of aesthetic language assessment: aesthetic visual question and answering (AVQA) of images. If we give a question of images aesthetics, model can predict the answer. We use images from \textit{www.flickr.com}. The objective QA pairs are generated by the proposed aesthetic attributes analysis algorithms. Moreover, we introduce subjective QA pairs that are converted from aesthetic numerical labels and sentiment analysis from large-scale pre-train models. We build the first aesthetic visual question answering dataset, AesVQA, that contains 72,168 high-quality images and 324,756 pairs of aesthetic questions. Two methods for adjusting the data distribution have been proposed and proved to improve the accuracy of existing models. This is the first work that both addresses the task of aesthetic VQA and introduces subjectiveness into VQA tasks. The experimental results reveal that our methods outperform other VQA models on this new task.
V-SENSE School of Computer Science Statistics Trinity College Dublin（V-SENSE计算机科学与统计学院都柏林大学三一学院）
Aesthetic image captioning (AIC) refers to the multi-modal task of generating critical textual feedbacks for photographs. While in natural image captioning (NIC), deep models are trained in an end-to-end manner using large curated datasets such as MS-COCO, no such large-scale, clean dataset exists for AIC. Towards this goal, we propose an automatic cleaning strategy to create a benchmarking AIC dataset, by exploiting the images and noisy comments easily available from photography websites. We propose a probabilistic caption-filtering method for cleaning the noisy web-data, and compile a large-…
AesTest: Measuring Aesthetic Intelligence from Perception to Production
University of International Business and Economics（国际商务经济大学）；University of Science and Technology of China（中国科学技术大学）；Huawei Technologies Co., Ltd（华为技术有限公司）；Beijing Electronic Science and Technology Institute（北京电子科技研究所）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）
Image aesthetic quality assessment has got much attention in recent years, but not many works have been done on a specific genre of photos: Group photograph. In this work, we designed a set of high-level features based on the experience and principles of group photography: Opened-eye, Gaze, Smile, Occluded faces, Face Orientation, Facial blur, Character center. Then we combined them and 83 generic aesthetic features to build two aesthetic assessment models. We also constructed a large dataset of group photographs - GPD- annotated with the aesthetic score. The experimental result shows that our…
Evaluating aesthetic value of digital photographs is a challenging task, mainly due to numerous factors that need to be taken into account and subjective manner of this process. In this paper, we propose to approach this problem using deep convolutional neural networks. Using a dataset of over 1.7 million photos collected from Flickr, we train and evaluate a deep learning model whose goal is to classify input images by analysing their aesthetic value. The result of this work is a publicly available Web-based application that can be used in several real-life applications, e.g. to improve the wo…
PhotoFlow: Agentic 3D Virtual Photography Missions
Shanghai Jiao Tong University（上海交通大学）；Northeastern University（东北大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Cornell University（康奈尔大学）；Shanghai AI Laboratory（上海人工智能实验室）；Sichuan University（四川大学）
Beijing Electronic Science and Technology Institute（北京电子科技学院）
With the vigorous development of mobile photography technology, major mobile phone manufacturers are scrambling to improve the shooting ability of equipments and the photo beautification algorithm of software. However, the improvement of intelligent equipments and algorithms cannot replace human subjective photography technology. In this paper, we propose the aesthetic language guidance of image (ALG). We divide ALG into ALG-T and ALG-I according to whether the guiding rules are based on photography templates or guidance images. Whether it is ALG-T or ALG-I, we guide photography from three att…
University of California, Davis（加利福尼亚大学戴维斯分校）；National Taiwan University（台湾大学）；National Tsing Hua University（清华大学（台湾））
Photo composition is an important factor affecting the aesthetics in photography. However, it is a highly challenging task to model the aesthetic properties of good compositions due to the lack of globally applicable rules to the wide variety of photographic styles. Inspired by the thinking process of photo taking, we formulate the photo composition problem as a view finding process which successively examines pairs of views and determines their aesthetic preferences. We further exploit the rich professional photographs on the web to mine unlimited high-quality ranking samples and demonstrate …
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing University of Posts and Telecommunications（北京邮电大学）
Aesthetic quality assessment (AQA) is a challenging task due to complex aesthetic factors. Currently, it is common to conduct AQA using deep neural networks that require fixed-size inputs. Existing methods mainly transform images by resizing, cropping, and padding or employ adaptive pooling to alternately capture the aesthetic features from fixed-size inputs. However, these transformations potentially damage aesthetic features. To address this issue, we propose a simple but effective method to accomplish full-resolution image AQA by combining image padding with region of image (RoM) pooling. P…
Trinity College Dublin（都柏林圣三一学院）；Accenture Labs（埃森哲实验室）
Aspect ratio and spatial layout are two of the principal factors determining the aesthetic value of a photograph. But, incorporating these into the traditional convolution-based frameworks for the task of image aesthetics assessment is problematic. The aspect ratio of the photographs gets distorted while they are resized/cropped to a fixed dimension to facilitate training batch sampling. On the other hand, the convolutional filters process information locally and are limited in their ability to model the global spatial layout of a photograph. In this work, we present a two-stage framework base…
UniPercept: Towards Unified Perceptual-Level Image Understanding across Aesthetics, Quality, Structure, and Texture
University of Science and Technology of China（中国科学技术大学）；Shanghai AI Laboratory（上海人工智能实验室）；Peking University（北京大学）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；Nanjing University（南京大学）；Sun Yat-sen University（中山大学）；Tele-AI Website（Tele-AI网站）
Machine-learning excels in many areas with well-defined goals. However, a clear goal is usually not available in art forms, such as photography. The success of a photograph is measured by its aesthetic value, a very subjective concept. This adds to the challenge for a machine learning approach. We introduce Creatism, a deep-learning system for artistic content creation. In our system, we break down aesthetics into multiple aspects, each can be learned individually from a shared dataset of professional examples. Each aspect corresponds to an image operation that can be optimized efficiently. A …
International Institute of Information Technology（国际信息技术学院）；University of Hyderabad（海得拉巴大学）
Automatic photo aesthetic assessment is a challenging artificial intelligence task. Existing computational approaches have focused on modeling a single aesthetic score or a class (good or bad), however these do not provide any details on why the photograph is good or bad, or which attributes contribute to the quality of the photograph. To obtain both accuracy and human interpretation of the score, we advocate learning the aesthetic attributes along with the prediction of the overall score. For this purpose, we propose a novel multitask deep convolution neural network, which jointly learns eigh…
Beihang University（北京航空航天大学）；Beijing Electronic Science and Technology Institute（北京电子科技学院）；Peng Cheng Laboratory（鹏城实验室）
Despite the growing number of unmanned aerial vehicles (UAVs) and aerial videos, there is a paucity of studies focusing on the aesthetics of aerial videos that can provide valuable information for improving the aesthetic quality of aerial photography. In this article, we present a method of deep multimodality learning for UAV video aesthetic quality assessment. More specifically, a multistream framework is designed to exploit aesthetic attributes from multiple modalities, including spatial appearance, drone camera motion, and scene structure. A novel specially designed motion stream network is…
The capacity of automatically modeling photographic composition is valuable for many real-world machine vision applications such as digital photography, image retrieval, image understanding, and image aesthetics assessment. The triangle technique is among those indispensable composition methods on which professional photographers often rely. This paper proposes a system that can identify prominent triangle arrangements in two major categories of photographs: natural or urban scenes, and portraits. For the natural or urban scene pictures, the focus is on the effect of linear perspective. For po…
Researchers try to model the aesthetic quality of photographs into low and high- level features, drawing inspiration from art theory, psychology and marketing. We attempt to describe every feature extraction measure employed in the above process. The contribution of this literature review is the taxonomy of each feature by its implementation complexity, considering real-world applications and integration in mobile apps and digital cameras. Also, we discuss the machine learning results along with some unexplored research areas as future work.
Predicting Scores of Various Aesthetic Attribute Sets by Learning from Overall Score Labels
University of Science and Technology of China（中国科学技术大学）；Beijing Electronic Science and Technology Institute（北京电子科技学院）；University of Edinburgh（爱丁堡大学）；Sensetime Research（商汤科技研究院）
Paintings and Drawings Aesthetics Assessment with Rich Attributes for Various Artistic Categories
Beijing Electronic Science and Technology Institute（北京电子科技学院）；Central Academy of Fine Arts（中央美术学院）；Beijing University of Technology（北京工业大学）；University of Science and Technology of China（中国科学技术大学）；Beijing Institute of Fashion Technology（北京服装学院）
My goal in this paper is twofold: to study how well deep models can understand the images generated by DALL-E 2 and Midjourney, and to quantitatively evaluate these generative models. Two sets of generated images are collected for object recognition and visual question answering (VQA) tasks. On object recognition, the best model, out of 10 state-of-the-art object recognition models, achieves about 60\% and 80\% top-1 and top-5 accuracy, respectively. These numbers are much lower than the best accuracy on the ImageNet dataset (91\% and 99\%). On VQA, the OFA model scores 77.3\% on answering 241…
Beautifeye；IADT（爱尔兰艺术设计与技术学院）；Xerox Research Centre Europe（施乐欧洲研究中心）
Aesthetic image analysis is the study and assessment of the aesthetic properties of images. Current computational approaches to aesthetic image analysis either provide accurate or interpretable results. To obtain both accuracy and interpretability by humans, we advocate the use of learned and nameable visual attributes as mid-level features. For this purpose, we propose to discover and learn the visual appearance of attributes automatically, using a recently introduced database, called AVA, which contains more than 250,000 images together with their aesthetic scores and textual comments given …
University of Texas at Austin（得克萨斯大学奥斯汀分校）；University of Science and Technology of China（中国科学技术大学）；University of Colorado Boulder（科罗拉多大学博尔德分校）；Carnegie Mellon University（卡内基梅隆大学）；University of Rochester（罗切斯特大学）
The study of algorithms to automatically answer visual questions currently is motivated by visual question answering (VQA) datasets constructed in artificial VQA settings. We propose VizWiz, the first goal-oriented VQA dataset arising from a natural VQA setting. VizWiz consists of over 31,000 visual questions originating from blind people who each took a picture using a mobile phone and recorded a spoken question about it, together with 10 crowdsourced answers per visual question. VizWiz differs from the many existing VQA datasets because (1) images are captured by blind photographers and so a…
University of Engineering and Technology, Peshawar（白沙瓦工程技术大学）；COMSATS University Islamabad, Wah Campus, Pakistan（伊斯兰堡COMSATS大学瓦赫校区）；Saudi Electronic University, Saudi Arabia（沙特电子大学）；Imaging and Computer vision group, Data61-CSIRO, Australia（澳大利亚CSIRO数据61中心成像与计算机视觉组）；Norwegian University of Life Sciences, Norway（挪威生命科学大学）
Automatic image aesthetics assessment is a computer vision problem dealing with categorizing images into different aesthetic levels. The categorization is usually done by analyzing an input image and computing some measure of the degree to which the image adheres to the fundamental principles of photography such as balance, rhythm, harmony, contrast, unity, look, feel, tone and texture. Due to its diverse applications in many areas, automatic image aesthetic assessment has gained significant research attention in recent years. This article presents a review of the contemporary automatic image …
This survey aims at reviewing recent computer vision techniques used in the assessment of image aesthetic quality. Image aesthetic assessment aims at computationally distinguishing high-quality photos from low-quality ones based on photographic rules, typically in the form of binary classification or quality scoring. A variety of approaches has been proposed in the literature trying to solve this challenging problem. In this survey, we present a systematic listing of the reviewed approaches based on visual feature types (hand-crafted features and deep features) and evaluation criteria (dataset…
Friedrich Schiller University（弗里德里希·席勒大学）；University of Jena School of Medicine（耶拿大学医学院）；Jena University Hospital（耶拿大学医院）
Computational aesthetics is an emerging field of research which has attracted different research groups in the last few years. In this field, one of the main approaches to evaluate the aesthetic quality of paintings and photographs is a feature-based approach. Among the different features proposed to reach this goal, color plays an import role. In this paper, we introduce a novel dataset that consists of paintings of Western provenance from 36 well-known painters from the 15th to the 20th century. As a first step and to assess this dataset, using a classifier, we investigate the correlation be…
The aesthetic quality of an image is defined as the measure or appreciation of the beauty of an image. Aesthetics is inherently a subjective property but there are certain factors that influence it such as, the semantic content of the image, the attributes describing the artistic aspect, the photographic setup used for the shot, etc. In this paper we propose a method for the automatic prediction of the aesthetics of an image that is based on the analysis of the semantic content, the artistic style and the composition of the image. The proposed network includes: a pre-trained network for semant…
Media Technology Center (MTC)（媒体技术中心（MTC））；ETH Zurich（苏黎世联邦理工学院）；DISCo；University of Milano-Bicocca（米兰比可卡大学）
Computational inference of aesthetics is an ill-defined task due to its subjective nature. Many datasets have been proposed to tackle the problem by providing pairs of images and aesthetic scores based on human ratings. However, humans are better at expressing their opinion, taste, and emotions by means of language rather than summarizing them in a single number. In fact, photo critiques provide much richer information as they reveal how and why users rate the aesthetics of visual stimuli. In this regard, we propose the Reddit Photo Critique Dataset (RPCD), which contains tuples of image and p…
Rating how aesthetically pleasing an image appears is a highly complex matter and depends on a large number of different visual factors. Previous work has tackled the aesthetic rating problem by ranking on a 1-dimensional rating scale, e.g., incorporating handcrafted attributes. In this paper, we propose a rather general approach to automatically map aesthetic pleasingness with all its complexity into an "aesthetic space" to allow for a highly fine-grained resolution. In detail, making use of deep learning, our method directly learns an encoding of a given image into this high-dimensional feat…
Indian Institute of Information Technology Sri City（斯里城印度信息技术学院）
This paper proposes a technique to assess the aesthetic quality of photographs. The goal of the study is to predict whether a given photograph is captured by professional photographers, or by common people, based on a measurement of artistic quality of the photograph. We propose a Multi-Layer-Perceptron based system to analyze some low, mid and high level image features and find their effectiveness to measure artistic quality of the image and produce a measurement of the artistic quality of the image on a scale of 10. We validate the proposed system on a large dataset, containing images downlo…
University of Science and Technology of China（中国科学技术大学）；Beijing Electronic Science and Technology Institute（北京电子科技学院）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；University of California, Davis（加州大学戴维斯分校）
Image aesthetic quality assessment is popular during the last decade. Besides numerical assessment, nature language assessment (aesthetic captioning) has been proposed to describe the generally aesthetic impression of an image. In this paper, we propose aesthetic attribute assessment, which is the aesthetic attributes captioning, i.e., to assess the aesthetic attributes such as composition, lighting usage and color arrangement. It is a non-trivial task to label the comments of aesthetic attributes, which limit the scale of the corresponding datasets. We construct a novel dataset, named DPC-Cap…
Media Integration and Communication Center (MICC)（媒体集成与通信中心（MICC））；University of Florence（佛罗伦萨大学）
Technology and the fruition of cultural heritage are becoming increasingly more entwined, especially with the advent of smart audio guides, virtual and augmented reality, and interactive installations. Machine learning and computer vision are important components of this ongoing integration, enabling new interaction modalities between user and museum. Nonetheless, the most frequent way of interacting with paintings and statues still remains taking pictures. Yet images alone can only convey the aesthetics of the artwork, lacking is information which is often required to fully understand and app…
Image aesthetics assessment (IAA) is a challenging task due to its highly subjective nature. Most of the current studies rely on large-scale datasets (e.g., AVA and AADB) to learn a general model for all kinds of photography images. However, little light has been shed on measuring the aesthetic quality of artistic images, and the existing datasets only contain relatively few artworks. Such a defect is a great obstacle to the aesthetic assessment of artistic images. To fill the gap in the field of artistic image aesthetics assessment (AIAA), we first introduce a large-scale AIAA dataset: Boldbr…
School of Information Science and Engineering, Xiamen University（厦门大学信息科学与技术学院）；Institute of Information Science, Academia Sinica（中央研究院信息科学研究所）
Despite recent progress, computational visual aesthetic is still challenging. Image cropping, which refers to the removal of unwanted scene areas, is an important step to improve the aesthetic quality of an image. However, it is challenging to evaluate whether cropping leads to aesthetically pleasing results because the assessment is typically subjective. In this paper, we propose a novel cascaded cropping regression (CCR) method to perform image cropping by learning the knowledge from professional photographers. The proposed CCR method improves the convergence speed of the cascaded method, wh…
We address the problem of Visual Question Answering (VQA), which requires joint image and language understanding to answer a question about a given photograph. Recent approaches have applied deep image captioning methods based on convolutional-recurrent networks to this problem, but have failed to model spatial inference. To remedy this, we propose a model we call the Spatial Memory Network and apply it to the VQA task. Memory networks are recurrent neural networks with an explicit attention mechanism that selects certain parts of the information stored in memory. Our Spatial Memory Network st…
DEAR: Dataset for Evaluating the Aesthetics of Rendering
Color Reproduction and Synthesis Institute（颜色再现与合成研究所）；Moscow Independent Research Institute of Artificial Intelligence（莫斯科独立人工智能研究 institute）；Computational color photography Laboratory（计算色彩摄影实验室）；University of Milano-Bicocca（米兰-比科卡大学）
Zhengzhou University（郑州大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Personalized image aesthetic assessment (PIAA) has recently become a hot topic due to its usefulness in a wide variety of applications such as photography, film and television, e-commerce, fashion design and so on. This task is more seriously affected by subjective factors and samples provided by users. In order to acquire precise personalized aesthetic distribution by small amount of samples, we propose a novel user-guided personalized image aesthetic assessment framework. This framework leverages user interactions to retouch and rank images for aesthetic assessment based on deep reinforcemen…
