Significant effort has been recently devoted to modeling visual relations. This has mostly addressed the design of architectures, typically by adding parameters and increasing model complexity. However, visual relation learning is a long-tailed problem, due to the combinatorial nature of joint reasoning about groups of objects. Increasing model complexity is, in general, ill-suited for long-tailed problems due to their tendency to overfit. In this paper, we explore an alternative hypothesis, denoted the Devil is in the Tails. Under this hypothesis, better performance is achieved by keeping the model simple but improving its ability to cope with long-tailed distributions. To test this hypothesis, we devise a new approach for training visual relationships models, which is inspired by state-of-the-art long-tailed recognition literature. This is based on an iterative decoupled training scheme, denoted Decoupled Training for Devil in the Tails (DT2). DT2 employs a novel sampling approach, Alternating Class-Balanced Sampling (ACBS), to capture the interplay between the long-tailed entity and predicate distributions of visual relations. Results show that, with an extremely simple architecture, DT2-ACBS significantly outperforms much more complex state-of-the-art methods on scene graph generation tasks. This suggests that the development of sophisticated models must be considered in tandem with the long-tailed nature of the problem.
The Devil is in Classification: A Simple Framework for Long-tail Object Detection and Instance Segmentation
NGS, National University of Singapore(新加坡国立大学);ECE Department, National University of Singapore(新加坡国立大学电机与计算机工程系);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Salesforce Research Asia(Salesforce亚洲研究院)
The Devil is in Fine-tuning and Long-tailed Problems:A New Benchmark for Scene Text Detection
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);VCIP & TMCC & DISSec, College of Computer Science, Nankai University(南开大学计算机学院);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院)
Devil's in the Details: Aligning Visual Clues for Conditional Embedding in Person Re-Identification
Tencent Youtu Lab(腾讯优图实验室);Sun Yat-sen University(中山大学);Huazhong University of Science and Technology(华中科技大学);Southern University of Science and Technology(南方科技大学)
The Devil is in the Few Shots: Iterative Visual Knowledge Completion for Few-shot Learning
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);Ant Group(蚂蚁集团);Institute of AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院)
The Devil is in the Distributions: Explicit Modeling of Scene Content is Key in Zero-Shot Video Captioning
School of Computer Science and Technology, University of Chinese Academy of Sciences(中国科学院大学计算机科学与技术学院);Macquarie University(麦考瑞大学);Australian Institute for Machine Learning, The University of Adelaide(阿德莱德大学澳大利亚机器学习研究所)
Exploring Visual Interpretability for Contrastive Language-Image Pre-training
The Hong Kong University of Science and Technology(香港科技大学);University of Technology Sydney(悉尼科技大学)
对比图像预训练(CLIP)通过对自然语言的现成监督来学习丰富的表示。它提高了下游视觉任务的性能,包括但不限于zero-shot、长尾、分割、检索、字幕和视频。然而,CLIP的视觉可解释性研究很少,特别是对于原始特征图。为了提供其预测的视觉解释,我们提出了图像-文本相似性图(ITSM)。在此基础上,我们惊讶地发现,CLIP喜欢的背景区域比前景,并显示错误的可视化结果对人类的理解。这种现象在视觉Transformer和卷积网络中都是普遍存在的,这表明这个问题是独特的,而不是由于特定的网络。通过实验,我们发现问题出在池化部分,不适当的池化方法会导致一种称为语义转换的现象。针对这个问题,我们提出了可解释的对比图像预训练(ECLIP),它通过Masked Max Pooling来纠正可解释性。具体来说,为了避免语义转移,我们用max pooling代替了原来的注意力池,专注于自信的前景,并在训练过程中得到自由注意力的指导。在三个数据集上的实验表明,ECLIP大大提高了CLIP的可解释性,并在很大程度上超过了以前的可解释性方法。代码将在稍后发布。
GazeLT: Visual attention-guided long-tailed disease classification in chest radiographs
Department of Biomedical Informatics, Stony Brook University, NY, US(斯通比克大学生物医学信息学系);Department of Radiology, Columbia University, NY, US(哥伦比亚大学放射学系);Department of Computer Science, Stony Brook University, NY, US(斯通比克大学计算机科学系)
Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognition
Guangming Laboratory(光明实验室);Xiamen University(厦门大学);Guangdong University of Technology(广东工业大学);Hong Kong Baptist University(香港浸会大学);Shenzhen University(深圳大学)
针对长尾识别中尾部类泛化差及 SAM 计算开销高的问题,本文提出 RSAM-PT,通过随机邻域单步梯度更新与延迟再加权提升性能。
Feature Fusion from Head to Tail for Long-Tailed Visual Recognition
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室(深圳));Shenzhen University(深圳大学);Hong Kong Baptist University(香港浸会大学);Xiamen University(厦门大学)
SuperDisco: Super-Class Discovery Improves Visual Recognition for the Long-Tail
AIM Lab, University of Amsterdam(阿姆斯特丹大学AIM实验室);Inception Institute of Artificial Intelligence(Inception人工智能研究所);United Imaging Healthcare, Co., Ltd.(联合影象医疗科技有限公司)
PI-H2T: Enhancing Long-Tailed Visual Recognition with Permutation-Invariant and Head-to-Tail Feature Fusion
Guangdong Provincial Key Laboratory of Visual Media and Multidimensional Intelligence, CSSE, Shenzhen University, China(广东省视觉媒体与多维智能重点实验室,计算机科学与电子技术学院,深圳大学,中国);Fujian Key Laboratory of Sensing and Computing for Smart City, School of Informatics, Xiamen University, China(福建省智慧城市感知与计算重点实验室,信息学院,厦门大学,中国);Department of Computer Science, Hong Kong Baptist University, China(计算机科学系,香港 Baptist 大学,中国);Inspur Smart City Technology Co., Ltd., China(Inspur 智能城市技术有限公司,中国)
Category-Prompt Refined Feature Learning for Long-Tailed Multi-Label Image Classification
School of Big Data & Software Engineering, Chongqing University(重庆大学大数据与软件学院);College of Computer Science, Chongqing University(重庆大学计算机学院);Fowler College of Business, San Diego State University(圣地亚哥州立大学福勒商学院);School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院)
Sparsity Meets Similarity: Leveraging Long-Tail Distribution for Dynamic Optimized Token Representation in Multimodal Large Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institution of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Concept Drift and Long-Tailed Distribution in Fine-Grained Visual Categorization: Benchmark and Method
School of Electronic Information and Communications, Huazhong University of Science and Technology(华中科技大学电子信息与通信学院);Alibaba Group, DEEPROUTE.AI(阿里巴巴集团DEEPROUTE.AI);Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Australian National University(澳大利亚国立大学);Linkoping University(林雪平大学);University of Chinese Academy of Sciences(中国科学院大学)
Long-Tailed Visual Recognition via Self-Heterogeneous Integration with Knowledge Excavation
Xiamen University(厦门大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳));Hong Kong Baptist University(香港浸会大学)
NCL++: Nested Collaborative Learning for Long-Tailed Visual Recognition
Institute of Deep Learning, Baidu Research(百度研究院深度学习研究所);Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)(中国科学院大学人工智能学院)
Prior-Anchored Debiasing for Long-Tailed Multi-Organ Pathology Report Generation
City University of Hong Kong, Hong Kong SAR(香港城市大学);University of Nottingham, United Kingdom(诺丁汉大学);University of Massachusetts Boston, United States(马萨诸塞大学波士顿分校)
LLM-empowered Dynamic Prompt Routing for Vision-Language Models Tuning under Long-Tailed Distributions
Shandong University(山东大学);The Hong Kong University of Science and Technology(香港科技大学);AiLF Instruments(AiLF仪器);Shandong Key Laboratory of Intelligent Electronic Packaging Testing and Application(山东省智能电子封装测试与应用重点实验室)
Unleashing the Power of Vision-Language Models for Long-Tailed Multi-Label Visual Recognition
School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Key Laboratory of Computer Network and Information Integration (Southeast University)(计算机网络与信息集成重点实验室);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)(Mohamed bin Zayed人工智能大学);Carnegie Mellon University(卡内基梅隆大学)
Exploring Long Tail Visual Relationship Recognition with Large Vocabulary
King Abdullah University of Science and Technology (KAUST)(阿卜杜拉国王科技大学);IIT Roorkee(印度理工学院鲁尔基分校);Stanford University(斯坦福大学);Baidu(百度);NTU Singapore(新加坡南洋理工大学)