Modern vision models are trained on very large noisy datasets. While these models acquire strong capabilities, they may not follow the user's intent to output the desired results in certain aspects, e.g., visual aesthetic, preferred style, and responsibility. In this paper, we target the realm of visual aesthetics and aim to align vision models with human aesthetic standards in a retrieval system. Advanced retrieval systems usually adopt a cascade of aesthetic models as re-rankers or filters, which are limited to low-level features like saturation and perform poorly when stylistic, cultural or knowledge contexts are involved. We find that utilizing the reasoning ability of large language models (LLMs) to rephrase the search query and extend the aesthetic expectations can make up for this shortcoming. Based on the above findings, we propose a preference-based reinforcement learning method that fine-tunes the vision models to distill the knowledge from both LLMs reasoning and the aesthetic models to better align the vision models with human aesthetics. Meanwhile, with rare benchmarks designed for evaluating retrieval systems, we leverage large multi-modality model (LMM) to evaluate the aesthetic performance with their strong abilities. As aesthetic assessment is one of the most subjective tasks, to validate the robustness of LMM, we further propose a novel dataset named HPIR to benchmark the alignment with human aesthetics. Experiments demonstrate that our method significantly enhances the aesthetic behaviors of the vision models, under several metrics. We believe the proposed algorithm can be a general practice for aligning vision models with human values.
Aligning Step-by-Step Instructional Diagrams to Video Demonstrations
The Australian National University（澳大利亚国立大学）；Mitsubishi Electric Research Labs（三菱电机研究院）；Technion Israel Institute of Technology（以色列理工学院）；The Australian Institute for Machine Learning（澳大利亚机器学习研究所）
Università degli Studi di Modena e Reggio Emilia（摩德纳和雷焦艾米利亚大学）
This paper presents a novel retrieval pipeline for video collections, which aims to retrieve the most significant parts of an edited video for a given query, and represent them with thumbnails which are at the same time semantically meaningful and aesthetically remarkable. Videos are first segmented into coherent and story-telling scenes, then a retrieval algorithm based on deep learning is proposed to retrieve the most significant scenes for a textual query. A ranking strategy based on deep features is finally used to tackle the problem of visualizing the best thumbnail. Qualitative and quant…
Faculty of Engineering, University of Porto（波尔图大学工程学院）；INESC TEC（葡萄牙国家工程与系统创新研究所）；Champalimaud Foundation（尚帕利莫基金会）；Nova Medical School（诺瓦医学院）
Treatments for breast cancer have continued to evolve and improve in recent years, resulting in a substantial increase in survival rates, with approximately 80\% of patients having a 10-year survival period. Given the serious impact that breast cancer treatments can have on a patient's body image, consequently affecting her self-confidence and sexual and intimate relationships, it is paramount to ensure that women receive the treatment that optimizes both survival and aesthetic outcomes. Currently, there is no gold standard for evaluating the aesthetic outcome of breast cancer treatment. In ad…
Achieving Text-based Person Retrieval with Any Granularity
National Key Laboratory of Multispectral Information Intelligent Processing Technology, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院多光谱信息智能处理技术国家重点实验室）；School of Software Engineering, Huazhong University of Science and Technology（华中科技大学软件工程学院）
Institute for Datability Science（数据可持续性科学研究所）；Osaka University（大阪大学）
In computer vision, visual arts are often studied from a purely aesthetics perspective, mostly by analysing the visual appearance of an artistic reproduction to infer its style, its author, or its representative features. In this work, however, we explore art from both a visual and a language perspective. Our aim is to bridge the gap between the visual appearance of an artwork and its underlying meaning, by jointly analysing its aesthetics and its semantics. We introduce the use of multi-modal techniques in the field of automatic art analysis by 1) collecting a multi-modal dataset with fine-ar…
Beautifeye；IADT（爱尔兰艺术设计与技术学院）；Xerox Research Centre Europe（施乐欧洲研究中心）
Aesthetic image analysis is the study and assessment of the aesthetic properties of images. Current computational approaches to aesthetic image analysis either provide accurate or interpretable results. To obtain both accuracy and interpretability by humans, we advocate the use of learned and nameable visual attributes as mid-level features. For this purpose, we propose to discover and learn the visual appearance of attributes automatically, using a recently introduced database, called AVA, which contains more than 250,000 images together with their aesthetic scores and textual comments given …
Multilingual Text-to-Image Person Retrieval via Bidirectional Relation Reasoning and Aligning
School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；Key Laboratory of New Generation Artificial Intelligence Technology & Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新 generation 人工智能技术及交叉应用重点实验室（东南大学），教育部，中国）
AesRM: Improving Video Aesthetics with Expert-Level Feedback
The University of Hong Kong（香港大学）；Fudan University（复旦大学）；Zhejiang University（浙江大学）；Hong Kong University of Science and Technology（香港科学与技术大学）；University of Sydney（悉尼大学）；Alibaba Group（阿里巴巴集团）
Computational visual aesthetics has recently become an active research area. Existing state-of-art methods formulate this as a binary classification task where a given image is predicted to be beautiful or not. In many applications such as image retrieval and enhancement, it is more important to rank images based on their aesthetic quality instead of binary-categorizing them. Furthermore, in such applications, it may be possible that all images belong to the same category. Hence determining the aesthetic ranking of the images is more appropriate. To this end, we formulate a novel problem of ra…
Learning to Look before Learning to Like: Incorporating Human Visual Cognition into Aesthetic Quality Assessment
Faculty of Data Science, City University of Macau（澳门城市大学数据科学学院）；Zhongshan Ophthalmic Center, Sun Yat-sen University（中山眼科中心，中山大学）；School of computer science, Torrens University Australia（计算机科学学院，托伦斯大学澳大利亚分校）
Tsinghua University（清华大学）；Nanjing University（南京大学）；National University of Singapore (Suzhou) Research Institute（新加坡国立大学（苏州）研究院）
Many aesthetic models in computer vision suffer from two shortcomings: 1) the low descriptiveness and interpretability of those hand-crafted aesthetic criteria (i.e., nonindicative of region-level aesthetics), and 2) the difficulty of engineering aesthetic features adaptively and automatically toward different image sets. To remedy these problems, we develop a deep architecture to learn aesthetically-relevant visual attributes from Flickr1, which are localized by multiple textual attributes in a weakly-supervised setting. More specifically, using a bag-ofwords (BoW) representation of the frequ…
Text-to-image person retrieval aims to identify the target person based on a given textual description query. The primary challenge is to learn the mapping of visual and textual modalities into a common latent space. Prior works have attempted to address this challenge by leveraging separately pre-trained unimodal models to extract visual and textual features. However, these approaches lack the necessary underlying alignment capabilities required to match multimodal data effectively. Besides, these works use prior information to explore explicit part alignments, which may lead to the distortio…
Aligning image and text encoders from scratch using contrastive learning requires large amounts of paired image-text data. We alleviate this need by aligning individually pre-trained language and vision representation models using a much smaller amount of paired data, augmented with a curriculum learning algorithm to learn fine-grained vision-language alignments. TOnICS (Training with Ontology-Informed Contrastive Sampling) initially samples minibatches whose image-text pairs contain a wide variety of objects to learn object-level alignment, and progressively samples minibatches where all imag…
Boston University（波士顿大学）；Kuaishou Technology（快手科技）
Most existing methods in vision-language retrieval match two modalities by either comparing their global feature vectors which misses sufficient information and lacks interpretability, detecting objects in images or videos and aligning the text with fine-grained features which relies on complicated model designs, or modeling fine-grained interaction via cross-attention upon visual and textual tokens which suffers from inferior efficiency. To address these limitations, some recent works simply aggregate the token-wise similarities to achieve fine-grained alignment, but they lack intuitive expla…
Semantic Hardness Is Not Visual Hardness: Sign-Aware Hard Negative Mining for Sign Language Retrieval
School of Computing（计算机学院）；Graduate School of Culture Technology（文化技术研究生院）；Korea Advanced Institute of Science and Technology（韩国科学技术院）；ETRI Medical Informatics Laboratory（电子通信研究院医学信息学实验室）
ISTI-CNR（意大利国家研究委员会信息科学与技术研究所）；Univ. of Modena and Reggio Emilia（摩德纳和雷焦艾米利亚大学）
Image-text matching is gaining a leading role among tasks involving the joint understanding of vision and language. In literature, this task is often used as a pre-training objective to forge architectures able to jointly deal with images and texts. Nonetheless, it has a direct downstream application: cross-modal retrieval, which consists in finding images related to a given query text or vice-versa. Solving this task is of critical importance in cross-modal search engines. Many recent methods proposed effective solutions to the image-text matching problem, mostly using recent large vision-lan…
University of Science and Technology Beijing（北京科技大学）；Sun Yat-sen University（中山大学）；Microsoft Research（微软研究院）
We propose Pixel-BERT to align image pixels with text by deep multi-modal transformers that jointly learn visual and language embedding in a unified end-to-end framework. We aim to build a more accurate and thorough connection between image pixels and language semantics directly from image and sentence pairs instead of using region-based image features as the most recent vision and language tasks. Our Pixel-BERT which aligns semantic connection in pixel and text level solves the limitation of task-specific visual representation for vision and language tasks. It also relieves the cost of boundi…
