In this paper, we introduce DetailCLIP: A Detail-Oriented CLIP to address the limitations of contrastive learning-based vision-language models, particularly CLIP, in handling detail-oriented and fine-grained tasks like segmentation. While CLIP and its variants excel in the global alignment of image and text representations, they often struggle to capture the fine-grained details necessary for precise segmentation. To overcome these challenges, we propose a novel framework that employs patch-level comparison of self-distillation and pixel-level reconstruction losses, enhanced with an attention-based token removal mechanism. This approach selectively retains semantically relevant tokens, enabling the model to focus on the image's critical regions aligned with the specific functions of our model, including textual information processing, patch comparison, and image reconstruction, ensuring that the model learns high-level semantics and detailed visual features. Our experiments demonstrate that DetailCLIP surpasses existing CLIP-based and traditional self-supervised learning (SSL) models in segmentation accuracy and exhibits superior generalization across diverse datasets. DetailCLIP represents a significant advancement in vision-language modeling, offering a robust solution for tasks that demand high-level semantic understanding and detailed feature extraction. https://github.com/KishoreP1/DetailCLIP.
For video captioning, "pre-training and fine-tuning" has become a de facto paradigm, where ImageNet Pre-training (INP) is usually used to encode the video content, then a task-oriented network is fine-tuned from scratch to cope with caption generation. This paper first investigates the impact of the recently proposed CLIP (Contrastive Language-Image Pre-training) on video captioning. Through the empirical study on INP vs. CLIP, we identify the potential deficiencies of INP and explore the key factors for accurate description generation. The results show that the INP-based model is tricky to ca…
COSMIC: Clique-Oriented Semantic Multi-space Integration for Robust CLIP Test-Time Adaptation
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Shenzhen Technology University（深圳技术大学）；University of Science and Technology of China（中国科学技术大学）
School of Artificial Intelligence Beijing Univerisity of Posts\ Telecommunications Beijing, China Email；Department of Electrical；School of Artificial Intelligence Beijing Univerisity of Posts
Traditional video captioning requests a holistic description of the video, yet the detailed descriptions of the specific objects may not be available. Without associating the moving trajectories, these image-based data-driven methods cannot understand the activities from the spatio-temporal transitions in the inter-object visual features. Besides, adopting ambiguous clip-sentence pairs in training, it goes against learning the multi-modal functional mappings owing to the one-to-many nature. In this paper, we propose a novel task to understand the videos in object-level, named object-oriented v…
A Sober Look at the Robustness of CLIPs to Spurious Features
Hong Kong Baptist University（香港浸会大学）；The Hong Kong University of Science and Technology（香港科技大学）；The Chinese University of Hong Kong（香港中文大学）；University of Washington（华盛顿大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
MEVL-STP: Multi-Encoder and Vision Language Model for Arbitrarily Shaped Scene Text Spotting
Rajiv Gandhi Institute of Petroleum Technology（拉吉夫·甘地石油技术学院）；Indian Institute of Technology (ISM) Dhanbad（印度理工学院（ISM）丹巴德分校）；University of Salford（索尔福德大学）
Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasoner
Department of Pathology, West China Hospital, Sichuan University（四川大学华西医院病理科部门）；Institute of Clinical Pathology, West China Hospital, Sichuan University（四川大学华西医院临床病理科研究所）；University of Toronto（多伦多大学）；Business School, Sichuan University（四川大学商学院）；Department of Pathology, Shengjing Hospital of China Medical University（中国医科大学盛京医院病理科部门）
CNNs, Transformers, Hybrid, and Vision Language Models for Skin Cancer Detection
Department of Computer Science and Software Engineering, Concordia University, Montreal, Canada（计算机科学与软件工程系，康科迪亚大学，加拿大蒙特利尔）；Ebovir Biotechnologie Inc., Montreal, Canada（Ebovir生物技术公司，加拿大蒙特利尔）
Shouldn't language and vision features be treated equally in vision-language (VL) tasks? Many VL approaches treat the language component as an afterthought, using simple language models that are either built upon fixed word embeddings trained on text-only data or are learned from scratch. We believe that language features deserve more attention, and conduct experiments which compare different word embeddings, language models, and embedding augmentation steps on five common VL tasks: image-sentence retrieval, image captioning, visual question answering, phrase grounding, and text-to-clip retrie…
ML-CLIPSim: Multi-Layer CLIP Similarity for Machine-Oriented Image Quality
Simon Fraser University（西蒙弗雷泽大学）；University of British Columbia（不列颠哥伦比亚大学）；Eastern Institute of Technology（东部技术学院）；Xi’an Jiaotong University（西安交通大学）；Google Inc（谷歌公司）
VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval
Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；The Hong Kong Polytechnic University（香港理工大学）
GOOD: Towards Domain Generalized Orientated Object Detection
School of Computer Science, Wuhan University（武汉大学计算机学院）；Faculty of Science, University of Amsterdam（阿姆斯特丹大学理学院）；School of Medicine, Yale University（耶鲁大学医学院）；Faculty of Information Technology, Monash University（莫纳什大学信息技术学院）
Recently, there have been breakthroughs in computer vision ("CV") models that are more generalizable with the advent of models such as CLIP and ALIGN. In this paper, we analyze CLIP and highlight some of the challenges such models pose. CLIP reduces the need for task specific training data, potentially opening up many niche tasks to automation. CLIP also allows its users to flexibly specify image classification classes in natural language, which we find can shift how biases manifest. Additionally, through some preliminary probes we find that CLIP can inherit biases found in prior computer visi…
Exploring the Potential of Contrastive Language-Image Pre-training for Multi-Source Remote Sensing Data
School of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Space-based Computing System Research Center, Zhejiang Lab（之江实验室天基计算系统研究中心）
Understanding-Enhanced Model Collaboration for Long-Tailed Egocentric Mistake Detection
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；School of Computer Science and Tech., University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Institute of Information Engineering, CAS（信息工程研究所，中国科学院）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）
SonoCLIP: Mask-Guided Region-Aware Vision-Language Pretraining for Fetal Ultrasound Analysis
School of Computer Science, Wuhan University, Wuhan, China（武汉大学计算机学院，武汉，中国）；Institute of Artificial Intelligence, Wuhan University, Wuhan, China（武汉大学人工智能研究院，武汉，中国）；Department of Ultrasound, Renmin Hospital of Wuhan University, Wuhan, China（武汉大学仁民医院超声科，武汉，中国）；National Engineering Research Center for Multimedia Software, Wuhan University, Wuhan, China（武汉大学多媒体软件国家工程研究中心，武汉，中国）；Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, Wuhan, China（湖北省多媒体与网络通信工程重点实验室，武汉大学，武汉，中国）
Clinician-Friendly Foundation Models for Ophthalmic Image Diagnostics without Fine-Tuning or Technical Barriers
Centre for Innovation and Precision Eye Health（创新与精准眼科健康中心）；Yong Loo Lin School of Medicine（尤洛林医学院）；National University of Singapore（新加坡国立大学）；Department of Ophthalmology（眼科学系）；Shantou University（汕大）；Chinese University of Hong Kong（香港中文大学）；School of Computer Science and Engineering（计算机科学与工程学院）；Big Vision Medical Technology Ltd.（大视医学技术有限公司）；Singapore Eye Research Institute（新加坡眼科研究所）；Singapore National Eye Centre（新加坡国家眼科中心）；The Chinese University of Hong Kong（香港中文大学）；University of Exeter（埃克塞特大学）；University of Science and Technology Hospital（科学技术大学医院）；Shenzhen Qianhai Shekou Free Trade Zone Hospital（深圳前海蛇口自由贸易区医院）；Meizhou People’s Hospital（梅州人民医院）；Yeungnam University College of Medicine（延世大学医学院）；Aier Eye Hospital of Wuhan University（武汉大学爱尔眼科医院）；School of Biomedical Engineering（生物医学工程学院）；College of Computer Science and Technology（计算机科学与技术学院）
