Often, deep network models are purely inductive during training and while performing inference on unseen data. Thus, when such models are used for predictions, it is well known that they often fail to capture the semantic information and implicit dependencies that exist among objects (or concepts) on a population level. Moreover, it is still unclear how domain or prior modal knowledge can be specified in a backpropagation friendly manner, especially in large-scale and noisy settings. In this work, we propose an end-to-end vision and language model incorporating explicit knowledge graphs. We also introduce an interactive out-of-distribution (OOD) layer using implicit network operator. The layer is used to filter noise that is brought by external knowledge base. In practice, we apply our model on several vision and language downstream tasks including visual question answering, visual reasoning, and image-text retrieval on different datasets. Our experiments show that it is possible to design models that perform similarly to state-of-art results but with significantly fewer samples and training time.
Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approach
College of Future Information Technology, Fudan University(未来信息科技学院,复旦大学);Shanghai Innovation Institute(上海创新研究院);The Chinese University of Hong Kong(香港中文大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Zhangjiang Laboratory(张江实验室)
MODE-RAG: Manifold Outlier Diagnosis and Energy-based Retrieval-Augmented Generation Evaluation
School of Computer Science & Tech, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院);School of AI and Automation, Huazhong University of Science and Technology(华中科技大学人工智能与自动化学院)
Universal Skeleton Understanding via Differentiable Rendering and MLLMs
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School, China(人工智能通用基础理论国家重点实验室,北京大学深圳研究生院,中国);Tencent(腾讯);Nanjing University of Aeronautics(南京航空航天大学)
SnapBench: Benchmarking Snap-and-Ask Multimodal Retrieval for Mobile Interactions
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Tencent Yuanbao(腾讯元宝);Tsinghua University(清华大学);ARC Lab, Tencent(腾讯ARC实验室);The University of Hong Kong(香港大学)
Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment
Shanghai AI Laboratory(上海人工智能实验室);Zhejiang University(浙江大学);University of Science and Technology of China(中国科学技术大学);Shanghai Jiao Tong University(上海交通大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究所);Nanjing University(南京大学);Shanghai Innovation Institute(上海创新研究院)
PromptDx: Differentiable Prompt Tuning for Multimodal In-Context Alzheimer's Diagnosis
Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California(斯蒂文斯神经影像与信息学研究所,凯克医学院,南加州大学);Ming Hsieh Department of Electrical and Computer Engineering, Viterbi School of Engineering, University of Southern California(明希德电气与计算机工程系,维特比工程学院,南加州大学);Alfred E. Mann Department of Biomedical Engineering, Viterbi School of Engineering, University of Southern California(阿尔弗雷德·E·曼生物医学工程系,维特比工程学院,南加州大学)
Variational Neural Belief Parameterizations for Robust Dexterous Grasping under Multimodal Uncertainty
Department of Electrical & Computer Engineering and Institute for Systems Research, University of Maryland, College Park, MD, USA(电气与计算机工程系和系统研究所,马里兰大学,College Park, MD, USA);Maryland Applied Graduate Engineering, A. James Clark School of Engineering, University of Maryland, College Park, MD, USA(马里兰应用研究生工程学院,A. James Clark工程学院,马里兰大学,College Park, MD, USA)
Vision Also You Need: Navigating Out-of-Distribution Detection with Multimodal Large Language Model
Zhejiang University(浙江大学);Tsinghua University(清华大学);University of Chinese Academy of Sciences(中国科学院大学);Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences)(教育部计算电力网络与信息安全重点实验室,山东计算机科学中心(国家超算中心济南中心),齐鲁工业大学(山东科学院));Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing, Shandong Fundamental Research Center for Computer Science(山东省计算电力互联网与服务计算重点实验室,山东省计算机科学基础研究中心);University of Electronic Science and Technology of China(电子科技大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);RWTH Aachen University(亚琛工业大学)