Exploiting LMM-based knowledge for image classification tasks
Information Technologies Institute (ITI)（信息技术研究所）；Centre of Research and Technology Hellas (CERTH)（希腊研究与技术中心）
Maria Tzelepi, Vasileios Mezaris
备注：
Accepted for publication, 25th Int. Conf. on Engineering Applications of Neural Networks (EANN/EAAAI 2024), Corfu, Greece, June 2024. This is the "submitted manuscript"
In this paper we address image classification tasks leveraging knowledge encoded in Large Multimodal Models (LMMs). More specifically, we use the MiniGPT-4 model to extract semantic descriptions for the images, in a multimodal prompting fashion. In the current literature, vision language models such as CLIP, among other approaches, are utilized as feature extractors, using only the image encoder, for solving image classification tasks. In this paper, we propose to additionally use the text encoder to obtain the text embeddings corresponding to the MiniGPT-4-generated semantic descriptions. Thus, we use both the image and text embeddings for solving the image classification task. The experimental evaluation on three datasets validates the improved classification performance achieved by exploiting LMM-based knowledge.
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Shanghai AI Laboratory（上海人工智能实验室）；Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院）
CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark
Multimodal Art Projection Research Community（多模态艺术投影研究共同体）；Hong Kong University of Science and Technology（香港科技大学）；University of Waterloo（滑铁卢大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Peking University（北京大学）；The Hong Kong Polytechnic University（香港理工大学）；Waseda University（早稻田大学）；University of Manchester（曼彻斯特大学）
Knowledgeable In-Context Tuning: Exploring and Exploiting Factual Knowledge for In-Context Learning
School of Data Science and Engineering, East China Normal University（华东师范大学数据科学与工程学院）；Alibaba Group（阿里巴巴集团）；School of Statistics, East China Normal University（华东师范大学统计学院）
VSLLaVA: a pipeline of large multimodal foundation model for industrial vibration signal analysis
State Key Laboratory of Tribology, Department of Mechanical Engineering, Tsinghua University（摩擦学国家重点实验室，清华大学机械工程系）；Department of Statistics and Data Science, Yale University（耶鲁大学统计与数据科学系）
"Glue pizza and eat rocks" -- Exploiting Vulnerabilities in Retrieval-Augmented Generative Models
Arizona State University（亚利桑那州立大学）；Michigan State University（密歇根州立大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Virginia（弗吉尼亚大学）
We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?
Beijing University of Posts and Telecommunications（北京邮电大学）；Wechat（腾讯微信）；Tencent Inc.（腾讯公司）；Huazhong University of Science and Technology（华中科技大学）；Beijing Institute of Technology（北京理工大学）
TG-LMM: Enhancing Medical Image Segmentation Accuracy through Text-Guided Large Multi-Modal Model
Sun Yat-sen University（中山大学）；National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital & Shenzhen Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（中国医学科学院北京协和医学院国家癌症中心/国家肿瘤临床医学研究中心/肿瘤医院&深圳医院）；University of Macau（澳门大学）
Logic-RAG: Augmenting Large Multimodal Models with Visual-Spatial Knowledge for Road Scene Understanding
College of Information Sciences and Technology, Pennsylvania State University（宾夕法尼亚州立大学信息科学与技术学院）；Department of Mathematics and Computer Science, Drake University（德雷克大学数学与计算机科学系）
本文提出 Logic-RAG 框架，通过一阶逻辑动态知识库与逻辑推理增强 LMM 的道路场景空间理解，显著提升 GPT-4V 和 Claude 3.5 的问答准确率。
A Culturally-diverse Multilingual Multimodal Video Benchmark & Model
Mohamed bin Zayed University of Artificial Intelligence（莫德赫·本·扎耶德人工智能大学）；University of Central Florida（中央佛罗里达大学）；Islamic University of Technology（伊斯兰技术大学）；Air University（空军大学）；ETH Zurich（苏黎世联邦理工学院）；Technische Universität München（慕尼黑技术大学）；National Institute of Informatics（国家信息研究所）；Australian National University（澳大利亚国立大学）；Linköping University（利尔贝里大学）
Ludwig Maximilian University of Munich（路德维希·马克西米利安慕尼黑大学）
Training scene graph classification models requires a large amount of annotated image data. Meanwhile, scene graphs represent relational knowledge that can be modeled with symbolic data from texts or knowledge graphs. While image annotation demands extensive labor, collecting textual descriptions of natural scenes requires less effort. In this work, we investigate whether textual scene descriptions can substitute for annotated image data. To this end, we employ a scene graph classification framework that is trained not only from annotated images but also from symbolic data. In our architecture…
Idea23D: Collaborative LMM Agents Enable 3D Model Generation from Interleaved Multimodal Inputs
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；College of Computer Science, Zhejiang University（浙江大学计算机学院）；College of Computer Science and Technology, Harbin Engineering University（哈尔滨工程大学计算机科学与技术学院）
Agri-LLaVA: Knowledge-Infused Large Multimodal Assistant on Agricultural Pests and Diseases
China Three Gorges University（三峡大学）；College of Computer and Information Technology, China Three Gorges University（三峡大学计算机与信息学院）；University of Birmingham（伯明翰大学）；Southern University of Science and Technology（南方科技大学）
Beijing Institute of Technology（北京理工大学）；Tsinghua University（清华大学）；Inceptio Tech.（嬴彻科技）
Unsupervised Domain Adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. Most existing UDA approaches enable knowledge transfer via learning domain-invariant representation and sharing one classifier across two domains. However, ignoring the domain-specific information that are related to the task, and forcing a unified classifier to fit both domains will limit the feature expressiveness in each domain. In this paper, by observing that the Transformer architecture with comparable parameters can generate more transferable representations than C…
TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models
National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；University of Science and Technology of China（中国科学技术大学）；Hong Kong Polytechnic University（香港理工大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
ADS-Edit: A Multimodal Knowledge Editing Dataset for Autonomous Driving Systems
Zhejiang University,\ University - Ant \ Joint Laboratory of Knowledge Graph Hangzhou China；Zhejiang University,\ University - Ant \ Joint Laboratory of Knowledge Graph
AG-VAS: Anchor-Guided Zero-Shot Visual Anomaly Segmentation with Large Multimodal Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Casivision（中科视语）；Weiqiao-UCAS Science and Technology Park（魏桥国科科技园）
University of Trento（特伦托大学）；Huawei Technologies Ireland（华为技术爱尔兰公司）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）
Nowadays, the majority of state of the art monocular depth estimation techniques are based on supervised deep learning models. However, collecting RGB images with associated depth maps is a very time consuming procedure. Therefore, recent works have proposed deep architectures for addressing the monocular depth prediction task as a reconstruction problem, thus avoiding the need of collecting ground-truth depth. Following these works, we propose a novel self-supervised deep model for estimating depth maps. Our framework exploits two main strategies: refinement via cycle-inconsistency and distil…
In recent years, the performance of object detection has advanced significantly with the evolving deep convolutional neural networks. However, the state-of-the-art object detection methods still rely on accurate bounding box annotations that require extensive human labelling. Object detection without bounding box annotations, i.e, weakly supervised detection methods, are still lagging far behind. As weakly supervised detection only uses image level labels and does not require the ground truth of bounding box location and label of each object in an image, it is generally very difficult to disti…
AttentionDrag: Exploiting Latent Correlation Knowledge in Pre-trained Diffusion Models for Image Editing
Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University（上海数据科学关键实验室，计算机科学学院，复旦大学）；Rajax Network Technology (ele.me), Alibaba Group（RAJAX网络技术（饿了么），阿里巴巴集团）
Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learning
School of Computer Science and Engineering, Tianjin University of Technology（天津理工大学计算机科学与工程学院）；Computer Vision Center, Universitat Autonoma de Barcelona（巴塞罗那自治大学计算机视觉中心）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学与技术学院）；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院多模态人工智能系统国家重点实验室）；School of Artificial Intelligence, University of the Chinese Academy of Sciences（中国科学院大学人工智能学院）
The Chinese University of Hong Kong（香港中文大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Stanford University（斯坦福大学）
Annotation scarcity is a long-standing problem in medical image analysis area. To efficiently leverage limited annotations, abundant unlabeled data are additionally exploited in semi-supervised learning, while well-established cross-modality data are investigated in domain adaptation. In this paper, we aim to explore the feasibility of concurrently leveraging both unlabeled data and cross-modality data for annotation-efficient cardiac segmentation. To this end, we propose a cutting-edge semi-supervised domain adaptation framework, namely Dual-Teacher++. Besides directly learning from limited l…
