This paper presents Tag2Text, a vision language pre-training (VLP) framework, which introduces image tagging into vision-language models to guide the learning of visual-linguistic features. In contrast to prior works which utilize object tags either manually labeled or automatically detected with an off-the-shelf detector with limited performance, our approach explicitly learns an image tagger using tags parsed from image-paired text and thus provides a strong semantic guidance to vision-language models. In this way, Tag2Text can utilize large-scale annotation-free image tags in accordance with image-text pairs, and provides more diverse tag categories beyond objects. As a result, Tag2Text demonstrates the ability of a foundational image tagging model, with superior zero-shot performance even comparable to fully supervised models. Moreover, by leveraging the tagging guidance, Tag2Text effectively enhances the performance of vision-language models on both generation-based and alignment-based tasks. Across a wide range of downstream benchmarks, Tag2Text achieves state-of-the-art results with similar model sizes and data scales, demonstrating the efficacy of the proposed tagging guidance. Code, demo and pre-trained models are available at https://github.com/xinyu1205/recognize-anything.
Guiding Medical Vision-Language Models with Explicit Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations
Sichuan University Pittsburgh Institute(四川大学匹兹堡学院);Beijing University of Posts and Telecommunications(北京邮电大学);Brown University(布朗大学);Case Western Reserve University(凯斯西储大学);Shanghai Jiao Tong University(上海交通大学)
Are Multimodal Large Language Models Good Annotators for Image Tagging?
RIKEN Center for Advanced Intelligence Project, Japan(日本先进人工智能研究中心);The University of Tokyo, Japan(东京大学);Southeast University, China(东南大学);China University of Mining(中国矿业大学)
Scale, Don't Fine-tune: Guiding Multimodal LLMs for Efficient Visual Place Recognition at Test-Time
Hong Kong University of Science(香港科学与技术大学);South China Normal University, Shanwei, Guangdong, China(华南师范大学,汕尾,广东,中国);Shenzhen Technology University, Shenzhen, Guangdong, China(深圳科技大学,深圳,广东,中国);University of Science(科学大学)
MiniVLM: A Smaller and Faster Vision-Language Model
Microsoft(微软)
最近的视觉语言(VL)的研究已经取得了显着的进展,学习通用表示与Transformer模型,然后微调下游VL任务的大量图像-文本对。虽然现有的研究一直专注于通过大型预训练模型实现高准确性,但构建轻量级模型在实践中具有很大的价值,但探索较少。在本文中,我们提出了一个更小,更快的VL模型,MiniVLM,它可以微调,在各种下游任务的良好性能,就像它的大型同行。MiniVLM由两个模块组成,一个视觉特征提取器和一个基于变换的视觉语言融合模块。我们设计了一个两阶段有效的特征提取器(TEE),灵感来自一个阶段的EfficientDet网络,显着减少视觉特征提取的时间成本$95\%$,相比基线模型。通过比较不同的紧凑型BERT模型,采用MiniLM结构来降低Transformer模块的计算量。此外,我们通过添加$7M$ Open Images数据来改进MiniVLM预训练,这些数据由最先进的字幕模型进行伪标记。我们还使用从强标记模型中获得的高质量图像标记进行预训练,以增强跨模态对齐。大型模型可以离线使用,而不会在微调和推理中增加任何开销。通过上述设计选择,我们的MiniVLM通过$73\%$减少了模型大小,通过$94\%$减少了推理时间成本,同时能够在多个VL任务上保持$94-97\%$的准确性。我们希望MiniVLM有助于简化最先进的VL研究在边缘应用中的使用。
Learning Visual Prompts for Guiding the Attention of Vision Transformers
School of Computation, Information and Technology(计算、信息与技术学院);Technical University of Munich(慕尼黑工业大学);AI Innovation, Data:Lab Munich(慕尼黑人工智能创新数据实验室);Volkswagen AG(大众汽车集团);Department of Engineering Science(工程科学系);University of Oxford(牛津大学)
Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift
National University of Singapore(新加坡国立大学);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Sea AI lab(海天人工智能实验室);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Nanyang Technological University(南洋理工大学)
VisualAD: Language-Free Zero-Shot Anomaly Detection via Vision Transformer
State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Anhui University, Hefei, China(光电信息采集与防护技术国家重点实验室,安徽大学,合肥,中国);School of Artificial Intelligence, Anhui University, Hefei, China(人工智能学院,安徽大学,合肥,中国);College of Intelligence Science and Technology, National University of Defense Technology, Changsha, China(智能科学与技术学院,国防科技大学,长沙,中国)
A Multi-Modal Foundation Model to Assist People with Blindness and Low Vision in Environmental Interaction
Tandon School of Engineering, New York University(纽约大学坦登工程学院);NYU Langone Health, New York University(纽约大学朗格尼健康中心);New York University Abu Dhabi(纽约大学阿布扎比分校)
Toward Robust Medical Fairness: Debiased Dual-Modal Alignment via Text-Guided Attribute-Disentangled Prompt Learning for Vision-Language Models
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology(集成空天地海大数据应用技术国家工程实验室);Northwestern Polytechnical University(西北工业大学);Huiying Medical Technology Company Ltd.(慧影医疗技术有限公司);The School of Computer Science(计算机学院);The University of Sydney(悉尼大学);Department of Computer Science and Engineering(计算机科学与工程系);The Chinese University of Hong Kong(香港中文大学)
Efficient and Comprehensive Feature Extraction in Large Vision-Language Model for Pathology Analysis
School of Software Technology, Zhejiang University(浙江大学软件技术学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);First Affiliated Hospital, College of Medicine, Zhejiang University(浙江大学医学院第一附属医院)
StructXLIP: Enhancing Vision-language Models with Multimodal Structural Cues
University of Verona(维罗纳大学);Sapienza University of Rome(罗马大学);University of Trento(特伦托大学);Fondazione Bruno Kessler(布鲁诺·凯斯勒基金会);Reykjavik University(雷克雅未克大学)
Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models
The Hong Kong Polytechnic University(香港理工大学);Hong Kong University of Science and Technology(香港科技大学);Tencent AI Lab(腾讯AI实验室);Huazhong University of Science and Technology(华中科技大学);Tsinghua University(清华大学)
Large Vision-Language Models Get Lost in Attention
School of Cyberspace Security, Beijing University of Posts(信息安全学院,北京邮电大学);State Key Laboratory of Networking and Switching Technology, Beijing University of Posts(网络与交换技术国家重点实验室,北京邮电大学);School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts(计算机科学学院(国家试点软件工程学院),北京邮电大学);Nanyang Technological University, Singapore(新加坡南洋理工大学);Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China(信息工程研究所,中国科学院北京研究院)
Vision-Language Model Purified Semi-Supervised Semantic Segmentation for Remote Sensing Images
Faculty of Data Science, City University of Macau(数据科学学院,澳门城市大学);School of Automation, Southeast University(自动化学院,东南大学);College of Oceanography and Space Informatics, China University of Petroleum (East China)(海洋与空间信息学院,中国石油大学(华东))
ScreenAgent: A Vision Language Model-driven Computer Control Agent
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Ministry of Education, China(教育部知识驱动人机智能工程研究中心)