Multimodal Large Language Models (MLLMs) excel in vision--language tasks by pre-training solely on coarse-grained concept annotations (e.g., image captions). We hypothesize that integrating fine-grained concept annotations (e.g., object labels and object regions) will further improve performance, as both data granularities complement each other in terms of breadth and depth in concept representation. We introduce a new dataset featuring Multimodal Multi-Grained Concept annotations (MMGiC) for MLLMs. In constructing MMGiC, we explore the impact of different data recipes on multimodal comprehension and generation. Our analyses reveal that multi-grained concept annotations integrate and complement each other, under our structured template and a general MLLM framework. We clearly explore and demonstrate the potential of MMGiC to help MLLMs better locate and learn concepts, aligning vision and language at multiple granularities. We further validate our hypothesis by investigating the fair comparison and effective collaboration between MMGiC and image--caption data on 12 multimodal comprehension and generation benchmarks, e.g., their appropriate combination achieve 3.95% and 2.34% absolute improvements over image--caption data alone on POPE and SEED-Bench. Code, data and models will be available at https://github.com/LooperXX/MMGiC.
Stony Brook University（石溪大学）；University of Oxford（牛津大学）
Crowd-sourcing is an increasingly popular tool for image analysis in animal ecology. Computer vision methods that can utilize crowd-sourced annotations can help scale up analysis further. In this work we study the potential to do so on the challenging task of fine-grained counting. As opposed to the standard crowd counting task, fine-grained counting also involves classifying attributes of individuals in dense crowds. We introduce a new dataset from animal ecology to enable this study that contains 1.7M crowd-sourced annotations of 8 fine-grained classes. It is the largest available dataset fo…
Existing 3D pose datasets of object categories are limited to generic object types and lack of fine-grained information. In this work, we introduce a new large-scale dataset that consists of 409 fine-grained categories and 31,881 images with accurate 3D pose annotation. Specifically, we augment three existing fine-grained object recognition datasets (StanfordCars, CompCars and FGVC-Aircraft) by finding a specific 3D model for each sub-category from ShapeNet and manually annotating each 2D image by adjusting a full set of 7 continuous perspective parameters. Since the fine-grained shapes allow …
Cornell University（康奈尔大学）；Google（谷歌）；University of Edinburgh（爱丁堡大学）；University of Copenhagen（哥本哈根大学）
We present a new benchmark dataset, Sapsucker Woods 60 (SSW60), for advancing research on audiovisual fine-grained categorization. While our community has made great strides in fine-grained visual categorization on images, the counterparts in audio and video fine-grained categorization are relatively unexplored. To encourage advancements in this space, we have carefully constructed the SSW60 dataset to enable researchers to experiment with classifying the same set of categories in three different modalities: images, audio, and video. The dataset covers 60 species of birds and is comprised of i…
We introduce a novel dataset for architectural style classification, consisting of 9,485 images of church buildings. Both images and style labels were sourced from Wikipedia. The dataset can serve as a benchmark for various research fields, as it combines numerous real-world challenges: fine-grained distinctions between classes based on subtle visual features, a comparatively small sample size, a highly imbalanced class distribution, a high variance of viewpoints, and a hierarchical organization of labels, where only some images are labeled at the most precise level. In addition, we provide 63…
ABCD-LINK: Annotation Bootstrapping for Cross-Document Fine-Grained Links
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), TU Darmstadt（图腾知识处理实验室（UKP实验室）、计算机科学系和海德堡人工智能中心（hessian.AI）、图腾达姆斯塔特大学）；Hebrew University of Jerusalem and The Allen Institute for AI (AI2)（耶路撒冷希伯来大学和人工智能研究所（AI2））
Automated Data Enrichment using Confidence-Aware Fine-Grained Debate among Open-Source LLMs for Mental Health and Online Safety
University of Southampton, UK（英国南安普顿大学）；Queen Mary University of London, UK（伦敦大学玛丽女王学院）；The Alan Turing Institute, UK（阿兰·图灵研究所）；Bar Ilan University, Israel（巴伊兰大学）
Exploring Imbalanced Annotations for Effective In-Context Learning
Department of Statistics and Data Science, Southern University of Science and Technology（南方科技大学统计与数据科学系）；School of Economics and Finance, Xi’an Jiaotong University（西安交通大学经济金融学院）；School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学学院）
Exploring LLM-based Data Annotation Strategies for Medical Dialogue Preference Alignment
School of Computer Science, Peking University（北京大学计算机学院）；Key Laboratory of High Confidence Software Technologies(PKU), MOE, China（教育部高可信软件技术重点实验室（北京大学））；Beijing Jiaotong University（北京交通大学）
WBCAtt+: Fine-Grained Pixel-Level Morphological Annotations for White Blood Cell Images
Rapid-Rich Object Search (ROSE) Lab, School of Electrical and Electronic Engineering, Nanyang Technological University（快速丰富目标搜索（ROSE）实验室，电气与电子工程学院，南洋理工大学）；Shanghai University of Finance and Economics（上海财经大学）
Can Frontier LLMs Replace Annotators in Biomedical Text Mining? Analyzing Challenges and Exploring Solutions
The University of Tokyo（东京大学）；Graduate School of Frontier Sciences（前沿科学研究生院）；Department of Computational Biology and Medical Sciences（计算生物学与医学科学系）；Database Center for Life Science（生命科学数据库中心）；Joint Support-Center for Data Science Research（数据科学研究联合支持中心）
FACap: A Large-scale Fashion Dataset for Fine-grained Composed Image Retrieval
Inria, \'Ecole normale sup\'erieure, CNRS, PSL Research University；Courant Institute of Mathematical Sciences；Center for Data Science, New York University
LPLCv2: An Expanded Dataset for Fine-Grained License Plate Legibility Classification
1 0.15mm Federal University of Paran\'a, Curitiba, Brazil；2 0.15mm Department of Technological Development；3 0.15mm Pontifical Catholic University of Paran\'a, Curitiba, Brazil
Hybrid Annotation for Propaganda Detection: Integrating LLM Pre-Annotations with Human Intelligence
Quality and Usability Lab, Technische Universität Berlin, Germany（柏林技术大学质量与可用性实验室）；German Research Center for Artificial Intelligence (DFKI), Berlin, Germany（德国人工智能研究中心（DFKI））
Unveiling Perceptual Artifacts: A Fine-Grained Benchmark for Interpretable AI-Generated Image Detection
Sun Yat-sen University（中山大学）；Peng Cheng Laboratory（鹏城实验室）；Xi’an Jiaotong University（西安交通大学）；Australian National University（澳大利亚国立大学）；Tsinghua University（清华大学）
MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search
Nanyang Technological University（南洋理工大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Wuhan University（武汉大学）；National University of Singapore（新加坡国立大学）；University of New South Wales（新南威尔士大学）；MiroMind；The Chinese University of Hong Kong（香港中文大学）
Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers
MBZUAI（穆罕默德·本·扎耶德人工智能大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Copenhagen（哥本哈根大学）；University of California, Santa Barbara（加州大学圣巴巴拉分校）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）
FineSkiing: A Fine-grained Benchmark for Skiing Action Quality Assessment
Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, College of Computer Science and Technology（教育部符号计算与知识工程重点实验室、计算机科学与技术学院）；Jilin University（吉林大学）；BNRist, THUIBCS, BLBCI, School of Software（BNRist、THUIBCS、BLBCI、软件学院）；Tsinghua University（清华大学）
AffectSpeech: A Large-Scale Emotional Speech Dataset with Fine-Grained Textual Descriptions for Speech Emotion Captioning and Synthesis
School of Biological Science and Medical Engineering, Southeast University（东南大学生物科学与医学工程学院）；Key Laboratory of Child Development and Learning Science (Southeast University), Ministry of Education（教育部儿童发展与学习科学重点实验室（东南大学））；Chair of Health Informatics (CHI), Technical University of Munich（慕尼黑工业大学健康信息学教席）；Group on Language, Audio, & Music (GLAM), Imperial College London（帝国理工学院语言、音频与音乐组）；Shenzhen Loop Area Institute（深圳河套学院）；Shenzhen Research Institute of Big Data, School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen（深圳市大数据研究院，香港中文大学（深圳）人工智能学院）
DeepFashion is a widely used clothing dataset with 50 categories and more than overall 200k images where each image is annotated with fine-grained attributes. This dataset is often used for clothes recognition and although it provides comprehensive annotations, the attributes distribution is unbalanced and repetitive specially for training fine-grained attribute recognition models. In this work, we tailored DeepFashion for fine-grained attribute recognition task by focusing on each category separately. After selecting categories with sufficient number of images for training, we remove very sca…
Advanced Digital Sciences Center（高级数字科学中心）；Nanjing University（南京大学）；Nanyang Technological University（南洋理工大学）
In this paper, we categorize fine-grained images without using any object / part annotation neither in the training nor in the testing stage, a step towards making it suitable for deployments. Fine-grained image categorization aims to classify objects with subtle distinctions. Most existing works heavily rely on object / part detectors to build the correspondence between object parts by using object or object part annotations inside training images. The need for expensive object annotations prevents the wide usage of these methods. Instead, we propose to select useful parts from multi-scale pa…
Universidad de Valladolid（巴利亚多利德大学）；University of San Francisco（旧金山大学）
Existing computer vision research in categorization struggles with fine-grained attributes recognition due to the inherently high intra-class variances and low inter-class variances. SOTA methods tackle this challenge by locating the most informative image regions and rely on them to classify the complete image. The most recent work, Vision Transformer (ViT), shows its strong performance in both traditional and fine-grained classification tasks. In this work, we propose a multi-stage ViT framework for fine-grained image classification tasks, which localizes the informative image regions withou…
Fine-grained recognition is a challenging task due to the small intra-category variances. Most of top-performing fine-grained recognition methods leverage parts of objects for better performance. Therefore, part annotations which are extremely computationally expensive are required. In this paper, we propose a novel cascaded deep CNN detection framework for fine-grained recognition which is trained to detect the whole object without considering parts. Nevertheless, most of current top-performing detection networks use the N+1 class (N object categories plus background) softmax loss, and the ba…
Industrial Institute of Science, The University of Tokyo（东京大学生产技术研究所）
Object affordance is an important concept in hand-object interaction, providing information on action possibilities based on human motor capacity and objects' physical property thus benefiting tasks such as action anticipation and robot imitation learning. However, the definition of affordance in existing datasets often: 1) mix up affordance with object functionality; 2) confuse affordance with goal-related action; and 3) ignore human motor capacity. This paper proposes an efficient annotation scheme to address these issues by combining goal-irrelevant motor actions and grasp types as affordan…
Fine-grained image classification involves identifying different subcategories of a class which possess very subtle discriminatory features. Fine-grained datasets usually provide bounding box annotations along with class labels to aid the process of classification. However, building large scale datasets with such annotations is a mammoth task. Moreover, this extensive annotation is time-consuming and often requires expertise, which is a huge bottleneck in building large datasets. On the other hand, self-supervised learning (SSL) exploits the freely available data to generate supervisory signal…
Self-supervised contrastive learning has demonstrated great potential in learning visual representations. Despite their success in various downstream tasks such as image classification and object detection, self-supervised pre-training for fine-grained scenarios is not fully explored. We point out that current contrastive methods are prone to memorizing background/foreground texture and therefore have a limitation in localizing the foreground object. Analysis suggests that learning to extract discriminative texture information and localization are equally crucial for fine-grained self-supervis…
Fine-Grained Rib Fracture Diagnosis with Hyperbolic Embeddings: A Detailed Annotation Framework and Multi-Label Classification Model
Indian Institute of Technology Jodhpur（印度理工学院贾姆pur分校）；All India Institute of Medical Sciences Delhi（全印度医学科学研究所德里）；Government Medical College Srinagar（政府医学院斯里 Nagar）
Given a food image, can a fine-grained object recognition engine tell "which restaurant which dish" the food belongs to? Such ultra-fine grained image recognition is the key for many applications like search by images, but it is very challenging because it needs to discern subtle difference between classes while dealing with the scarcity of training data. Fortunately, the ultra-fine granularity naturally brings rich relationships among object classes. This paper proposes a novel approach to exploit the rich relationships through bipartite-graph labels (BGL). We show how to model BGL in an over…
Stanford University（斯坦福大学）；University of Michigan（密歇根大学）
We present a crowdsourcing workflow to collect image annotations for visually similar synthetic categories without requiring experts. In animals, there is a direct link between taxonomy and visual similarity: e.g. a collie (type of dog) looks more similar to other collies (e.g. smooth collie) than a greyhound (another type of dog). However, in synthetic categories such as cars, objects with similar taxonomy can have very different appearance: e.g. a 2011 Ford F-150 Supercrew-HD looks the same as a 2011 Ford F-150 Supercrew-LL but very different from a 2011 Ford F-150 Supercrew-SVT. We introduc…
Defense Innovation Institute, Chinese Academy of Military Science（国防科技研究院，中国军事科学院）；Tianjin Artificial Intelligence Innovation Center（天津人工智能创新中心）；School of Mechanical Engineering, Tianjin University（天津大学机械工程学院）
