This paper proposes to go beyond the state-of-the-art deep convolutional neural network (CNN) by incorporating the information from object detection, focusing on dealing with fine-grained image classification. Unfortunately, CNN suffers from over-fiting when it is trained on existing fine-grained image classification benchmarks, which typically only consist of less than a few tens of thousands training images. Therefore, we first construct a large-scale fine-grained car recognition dataset that consists of 333 car classes with more than 150 thousand training images. With this large-scale dataset, we are able to build a strong baseline for CNN with top-1 classification accuracy of 81.6%. One major challenge in fine-grained image classification is that many classes are very similar to each other while having large within-class variation. One contributing factor to the within-class variation is cluttered image background. However, the existing CNN training takes uniform window sampling over the image, acting as blind on the location of the object of interest. In contrast, this paper proposes an \emph{object-centric sampling} (OCS) scheme that samples image windows based on the object location information. The challenge in using the location information lies in how to design powerful object detector and how to handle the imperfectness of detection results. To that end, we design a saliency-aware object detection approach specific for the setting of fine-grained image classification, and the uncertainty of detection results are naturally handled in our OCS scheme. Our framework is demonstrated to be very effective, improving top-1 accuracy to 89.3% (from 81.6%) on the large-scale fine-grained car classification dataset.
UC Berkeley（加州大学伯克利分校）；ICSI（国际计算机科学研究所）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Saarland University（萨尔大学）；Technische Universität München（慕尼黑工业大学）；Stanford University（斯坦福大学）；SPIEGEL-Verlag（明镜出版社）
Activity recognition has shown impressive progress in recent years. However, the challenges of detecting fine-grained activities and understanding how they are combined into composite activities have been largely overlooked. In this work we approach both tasks and present a dataset which provides detailed annotations to address them. The first challenge is to detect fine-grained activities, which are defined by low inter-class variability and are typically characterized by fine-grained body motions. We explore how human pose and hands can help to approach this challenge by comparing two pose-b…
Deep learning has a wide range of applications in industrial scenario, but reducing false alarm (FA) remains a major difficulty. Optimizing network architecture or network parameters is used to tackle this challenge in academic circles, while ignoring the essential characteristics of data in application scenarios, which often results in increased FA in new scenarios. In this paper, we propose a novel paradigm for fine-grained design of datasets, driven by industrial applications. We flexibly select positive and negative sample sets according to the essential features of the data and applicatio…
Attribute-Centric Vision-Language Alignment for Fine-Grained Gait Understanding
School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；The Hong Kong University of Science and Technology（香港科技大学）；Zhejiang University（浙江大学）；School of Artificial Intelligence, Shenzhen University（深圳大学人工智能学院）；Southern University of Science and Technology（南方科技大学）；Watrix Technology Limited Co. Ltd.（华瑞芯科技有限公司）
Beihang University（北京航空航天大学）；Sea AI Lab（Sea AI实验室）
Vision and language understanding techniques have achieved remarkable progress, but currently it is still difficult to well handle problems involving very fine-grained details. For example, when the robot is told to "bring me the book in the girl's left hand", most existing methods would fail if the girl holds one book respectively in her left and right hand. In this work, we introduce a new task named human-centric relation segmentation (HRS), as a fine-grained case of HOI-det. HRS aims to predict the relations between the human and surrounding entities and identify the relation-correlated hu…
VidEmo: Affective-Tree Reasoning for Emotion-Centric Video Foundation Models
Nankai University（南开大学）；Pengcheng Laboratory（鹏城实验室）；Kuaishou Technology（快手科技）；Nankai International Advanced Research Institute (SHENZHEN·FUTIAN)（南开国际先进研究院（深圳·福田））
EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generation
ByteDance Intelligent Creation（字节跳动智能创作）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of California, Merced（加州大学默塞德分校）；University of California, Los Angeles（加州大学洛杉矶分校）
Focus on What Really Matters in Low-Altitude Governance: A Management-Centric Multi-Modal Benchmark with Implicitly Coordinated Vision-Language Reasoning Framework
National University of Defense Technology（国防科技大学）；Zidong Taichu (Beijing) Technology Co., Ltd.（紫东太初（北京）科技有限公司）；Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Wuhan AI Research（武汉人工智能研究院）
FineParser: A Fine-grained Spatio-temporal Action Parser for Human-centric Action Quality Assessment
School of Intelligence Science and Technology, University of Science and Technology Beijing（北京科技大学智能科学与技术学院）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）
EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models
X-Humanoid；Fudan University（复旦大学）；University of Science and Technology of China（中国科学技术大学）；University of Manchester（曼彻斯特大学）；Monash University（墨尔本大学）；Celonis AI；University of New South Wales（新南威尔士大学）
University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）；University of Rochester（罗切斯特大学）
Learning subtle yet discriminative features (e.g., beak and eyes for a bird) plays a significant role in fine-grained image recognition. Existing attention-based approaches localize and amplify significant parts to learn fine-grained details, which often suffer from a limited number of parts and heavy computational cost. In this paper, we propose to learn such fine-grained features from hundreds of part proposals by Trilinear Attention Sampling Network (TASN) in an efficient teacher-student manner. Specifically, TASN consists of 1) a trilinear attention module, which generates attention maps b…
Representation of semantic context and local details is the essential issue for building modern semantic segmentation models. However, the interrelationship between semantic context and local details is not well explored in previous works. In this paper, we propose a Dynamic Dual Sampling Module (DDSM) to conduct dynamic affinity modeling and propagate semantic context to local details, which yields a more discriminative representation. Specifically, a dynamic sampling strategy is used to sparsely sample representative pixels and channels in the higher layer, forming adaptive compact support f…
We contribute HAA500, a manually annotated human-centric atomic action dataset for action recognition on 500 classes with over 591K labeled frames. To minimize ambiguities in action classification, HAA500 consists of highly diversified classes of fine-grained atomic actions, where only consistent actions fall under the same label, e.g., "Baseball Pitching" vs "Free Throw in Basketball". Thus HAA500 is different from existing atomic action datasets, where coarse-grained atomic actions were labeled with coarse action-verbs such as "Throw". HAA500 has been carefully curated to capture the precise…
Deep learning in general domains has constantly been extended to domain-specific tasks requiring the recognition of fine-grained characteristics. However, real-world applications for fine-grained tasks suffer from two challenges: a high reliance on expert knowledge for annotation and necessity of a versatile model for various downstream tasks in a specific domain (e.g., prediction of categories, bounding boxes, or pixel-wise annotations). Fortunately, the recent self-supervised learning (SSL) is a promising approach to pretrain a model without annotations, serving as an effective initializatio…
MICAS: Multi-grained In-Context Adaptive Sampling for 3D Point Cloud Processing
Zhejiang University-University of Illinois Urbana-Champaign Institute（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合研究院）；Zhejiang University（浙江大学）；University of Nevada, Reno（内华达大学雷诺分校）；Ningbo Innovation Center, Zhejiang University（浙江大学宁波创新中心）；School of Software Technology, Zhejiang University（浙江大学软件学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
Generative latent-variable models are emerging as promising tools in robotics and reinforcement learning. Yet, even though tasks in these domains typically involve distinct objects, most state-of-the-art generative models do not explicitly capture the compositional nature of visual scenes. Two recent exceptions, MONet and IODINE, decompose scenes into objects in an unsupervised fashion. Their underlying generative processes, however, do not account for component interactions. Hence, neither of them allows for principled sampling of novel scenes. Here we present GENESIS, the first object-centri…
EdiVal-Agent: An Object-Centric Framework for Automated, Fine-Grained Evaluation of Multi-Turn Editing
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of California, Los Angeles（加州大学洛杉矶分校）；Microsoft AI Superintelligence（微软人工智能超智实验室）；Lambda, Inc（Lambda公司）
Adept: Annotation-Denoising Auxiliary Tasks with Discrete Cosine Transform Map and Keypoint for Human-Centric Pretraining
College of Electrical Engineering, Zhejiang University（浙江大学电气工程学院）；School of Mechanical Engineering, Zhejiang University（浙江大学机械工程学院）；Ocean College, Zhejiang University（浙江大学海洋学院）；Hainan Institute, Zhejiang University（海南研究院）；Shanghai AI Laboratory（上海人工智能实验室）
