Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey
The University of Tokyo(东京大学);S-Lab, Nanyang Technological University(南洋理工大学S实验室);Duke University(杜克大学);Salesforce AI Research(Salesforce人工智能研究);Nanyang Technological University(南洋理工大学);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);Tokyo University of Science(东京科学大学)
Detecting out-of-distribution (OOD) samples is crucial for ensuring the safety of machine learning systems and has shaped the field of OOD detection. Meanwhile, several other problems are closely related to OOD detection, including anomaly detection (AD), novelty detection (ND), open set recognition (OSR), and outlier detection (OD). To unify these problems, a generalized OOD detection framework was proposed, taxonomically categorizing these five problems. However, Vision Language Models (VLMs) such as CLIP have significantly changed the paradigm and blurred the boundaries between these fields, again confusing researchers. In this survey, we first present a generalized OOD detection v2, encapsulating the evolution of these fields in the VLM era. Our framework reveals that, with some field inactivity and integration, the demanding challenges have become OOD detection and AD. Then, we highlight the significant shift in the definition, problem settings, and benchmarks; we thus feature a comprehensive review of the methodology for OOD detection and related tasks to clarify their relationship to OOD detection. Finally, we explore the advancements in the emerging Large Vision Language Model (LVLM) era, such as GPT-4V. We conclude with open challenges and future directions. The resource is available at https://github.com/AtsuMiyai/Awesome-OOD-VLM.
Generative Visual Communication in the Era of Vision-Language Models
Raymond and Beverly Sackler Faculty of Exact Sciences(雷蒙德和贝弗利·萨克勒精密科学学院);Blavatnik School of Computer Science(布拉瓦尼克计算机科学学院);School of Computer Science at Tel Aviv University(特拉维夫大学计算机科学学院)
Beyond Superficial Unlearning: Sharpness-Aware Robust Erasure of Hallucinations in Multimodal LLMs
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics(南京航空航天大学计算机科学与技术学院);Institute for AI, Tsinghua University(清华大学人工智能研究院);Huzhou University(湖州大学);Institute of Dataspace, Hefei Comprehensive National Science Center(合肥综合性国家科学中心数据空间研究院);University of Science and Technology of China(中国科学技术大学)
Foundational Models Defining a New Era in Vision: A Survey and Outlook
MBZ University of AI(穆罕默德·本·扎耶德人工智能大学);Australian National University(澳大利亚国立大学);Linköping University(林雪平大学);University of Central Florida(中佛罗里达大学);University of California, Merced(加州大学默塞德分校);Yonsei University(延世大学);Google Research(谷歌研究院)
LAION-C: An Out-of-Distribution Benchmark for Web-Scale Vision Models
Max Planck Institute for Intelligent Systems, Tübingen, Germany(马克斯·普朗克智能系统研究所,图宾根,德国);ELLIS Institute Tübingen(图宾根ELLIS研究所);Tübingen AI Center(图宾根人工智能中心);Google DeepMind(谷歌DeepMind)
Beyond MedQA: Towards Real-world Clinical Decision Making in the Era of LLMs
Department of Computer Science, Emory University(埃默里大学计算机科学系);Children’s Healthcare of Atlanta(亚特兰大儿童医院);Nell Hodgson Woodruff School of Nursing, Emory University(埃默里大学奈尔·霍德森·伍德鲁夫护理学院)
ZeroSlide: Is Zero-Shot Classification Adequate for Lifelong Learning in Whole-Slide Image Analysis in the Era of Pathology Vision-Language Foundation Models?
Nara Institute of Science and Technology(奈良科学技术研究所)
ERASE: Eliminating Redundant Visual Tokens via Adaptive Two-Stage Token Pruning
Department of Electrical and Computer Engineering, Sungkyunkwan University, Republic of Korea(电气与计算机工程系,成均馆大学,大韩民国);Department of Semiconductor Systems Engineering, Sungkyunkwan University, Republic of Korea(半导体系统工程系,成均馆大学,大韩民国)
VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training
University College London(伦敦大学学院);The University of Queensland(昆士兰大学);Southeast University(东南大学);University of Science and Technology of China(中国科学技术大学);Auckland University of Technology(奥克兰理工大学)
A Survey on Computer Vision based Human Analysis in the COVID-19 Era
Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);Istanbul Technical University(伊斯坦布尔理工大学);Fraunhofer Institute for Computer Graphics Research IGD(弗劳恩霍夫计算机图形学研究所);TU Darmstadt(达姆施塔特工业大学);Qatar Computing Research Institute(卡塔尔计算研究所);HBKU(哈马德·本·哈利法大学);University of Ljubljana(卢布尔雅那大学);Utrecht University(乌得勒支大学);Boğaziçi University(博阿齐奇大学);Carnegie Mellon University(卡内基梅隆大学)
More performant and scalable: Rethinking contrastive vision-language pre-training of radiology in the LLM era
School of Biomedical Engineering, Division of Life Sciences;Medicine, University of Science;Technology of China (USTC), Hefei Anhui, 230026, China Center for Medical Imaging, Robotics, Analytic Computing \& Learning (MIRACLE), Suzhou Institute for Advance Research, USTC, Suzhou Jiangsu, 215123, China The First Affiliated Hospital of USTC, Division of Life Sciences;Medicine, USTC, Hefei Anhui, 230001, China Jiangsu Provincial Key Laboratory of Multimodal Digital Twin Technology, Suzhou Jiangsu, 215123, China State Key Laboratory of Precision
本文提出利用 LLMs 自动从放射学报告中提取诊断标签以低成本构建大规模“银标准”数据集,从而促进大规模监督预训练,仅使用 3D ResNet-18 结合原始 CLIP 训练便在零样本诊断和跨模态检索任务上取得了最先进的性能。
Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers
IDEAS NCBR;Jagiellonian University, Faculty of Mathematics and Computer Science(雅盖隆大学数学与计算机科学学院);Jagiellonian University, Doctoral School of Exact and Natural Sciences(雅盖隆大学精确与自然科学博士学院);University of Warsaw(华沙大学);Warsaw University of Technology(华沙技术大学);Tooploox
Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Beyond Masking: Demystifying Token-Based Pre-Training for Vision Transformers
University of Chinese Academy of Sciences(中国科学院大学);Huawei Inc.(华为公司);Huazhong University of Science and Technology(华中科技大学);Chinese Academy of Sciences(中国科学院)