视觉大模型 / VLM

Revisiting Vision-Language Features Adaptation and Inconsistency for Social Media Popularity Prediction

Chih-Chung Hsu, Chia-Ming Lee, Yu-Fan Lin, Yi-Shiuan Chou, Chih-Yu Jian, Chi-Han Tsai

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)

Comments Submission of the 7th Social Media Prediction Challenge

MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment

Jihao Liu, Xin Huang, Jinliang Zheng, Boxiao Liu, Jia Wang, Osamu Yoshie, Yu Liu, Hongsheng Li

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Dataset and models are available at https://github.com/jihaonew/MM-Instruct

Explore the Limits of Omni-modal Pretraining at Scale

Yiyuan Zhang, Handong Li, Jing Liu, Xiangyu Yue

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Project Website: https://invictus717.github.io/MiCo/

Exploring the Spectrum of Visio-Linguistic Compositionality and Recognition

Youngtaek Oh, Pyunghwan Ahn, Jinhyung Kim, Gwangmo Song, Soonyoung Lee, In So Kweon, Junmo Kim

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted to CVPRW 2024 on 'What is Next in Multimodal Foundation Models?'. Code: https://github.com/ytaek-oh/vl_compo

AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction

Zhen Xing, Qi Dai, Zejia Weng, Zuxuan Wu, Yu-Gang Jiang

专题命中 VLM训练与架构 ：MLLM(abstract)；分类 cs.CV、cs.AI、cs.LG

FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learning

Yuwei Fu, Haichao Zhang, Di Wu, Wei Xu, Benoit Boulet

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments ICML 2024

EditWorld: Simulating World Dynamics for Instruction-Following Image Editing

Ling Yang, Bohan Zeng, Jiaming Liu, Hong Li, Minghao Xu, Wentao Zhang, Shuicheng Yan

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Project: https://github.com/YangLing0818/EditWorld

FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion

Zehan Wang, Ziang Zhang, Xize Cheng, Rongjie Huang, Luping Liu, Zhenhui Ye, Haifeng Huang, Yang Zhao, Tao Jin, Peng Gao, Zhou Zhao

专题命中 VLM训练与架构 ：InternVL(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted by ICML 2024. The code and checkpoints will be released at https://github.com/zehanwang01/FreeBind

Frozen Transformers in Language Models Are Effective Visual Encoder Layers

Ziqi Pang, Ziyang Xie, Yunze Man, Yu-Xiong Wang

专题命中 VLM训练与架构 ：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments ICLR 2024 Spotlight. 23 pages, 13 figures. Code at https://github.com/ziqipang/LM4VisualEncoding

MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors

Yuan Tang, Xu Han, Xianzhi Li, Qiao Yu, Yixue Hao, Long Hu, Min Chen

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 17 pages, 9 figures

DOCCI: Descriptions of Connected and Contrasting Images

Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, Su Wang, Jason Baldridge

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.AI、cs.LG

OmniFusion Technical Report

Elizaveta Goncharova, Anton Razzhigaev, Matvey Mikhalchuk, Maxim Kurkin, Irina Abdullaeva, Matvey Skripkin, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 17 pages, 4 figures, 9 tables, 2 appendices

VeCLIP: Improving CLIP Training via Visual-enriched Captions

Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, Meng Cao

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments CV/ML

How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary Investigation

Zhongyi Han, Guanglin Zhou, Rundong He, Jindong Wang, Tailin Wu, Yilong Yin, Salman Khan, Lina Yao, Tongliang Liu, Kun Zhang

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments added the investigation of Gemini. 66 pages, 41 figures

Talk Through It: End User Directed Manipulation Learning

Carl Winge, Adam Imdieke, Bahaa Aldeeb, Dongyeop Kang, Karthik Desingh

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)

MultiWay-Adapater: Adapting large-scale multi-modal models for scalable image-text retrieval

Zijun Long, George Killick, Richard McCreadie, Gerardo Aragon Camarasa

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI、cs.LG

InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance

Pengyu Wang, Dong Zhang, Linyang Li, Chenkun Tan, Xinghao Wang, Ke Ren, Botian Jiang, Xipeng Qiu

专题命中 VLM训练与架构 ：LLaVA(abstract)；multimodal large language model(abstract)

Improving fine-grained understanding in image-text pre-training

Ioana Bica, Anastasija Ilić, Matthias Bauer, Goker Erdogan, Matko Bošnjak, Christos Kaplanis, Alexey A. Gritsenko, Matthias Minderer, Charles Blundell, Razvan Pascanu, Jovana Mitrović

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 26 pages

Training Diffusion Models with Reinforcement Learning

Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, Sergey Levine

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 23 pages, 16 figures

Reinforcement Learning from Diffusion Feedback: Q* for Image Search

Aboli Marathe

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Brain

Dota Tianai Dong, Mariya Toneva

专题命中 VLM训练与架构 ：grounding(abstract)；分类 cs.CV、cs.AI、cs.LG

Three Towers: Flexible Contrastive Learning with Pretrained Image Models

Jannik Kossen, Mark Collier, Basil Mustafa, Xiao Wang, Xiaohua Zhai, Lucas Beyer, Andreas Steiner, Jesse Berent, Rodolphe Jenatton, Efi Kokiopoulou

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted for publication at NeurIPS 2023

Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling

Yaqing Wang, Jialin Wu, Tanmaya Dabral, Jiageng Zhang, Geoff Brown, Chun-Ta Lu, Frederick Liu, Yi Liang, Bo Pang, Michael Bendersky, Radu Soricut

专题命中 VLM训练与架构 ：vision language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards

Alexandre Ramé, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, Matthieu Cord

专题命中 VLM训练与架构 ：grounding(abstract)；分类 cs.CV、cs.AI、cs.LG

Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts

Mayug Maniparambil, Chris Vorster, Derek Molloy, Noel Murphy, Kevin McGuinness, Noel E. O'Connor

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Paper accepted at ICCV-W 2023. V2 contains additional comparisons with concurrent works

Vision Language Transformers: A Survey

Clayton Fields, Casey Kennington

专题命中 VLM训练与架构 ：vision language model(abstract)；分类 cs.CV、cs.AI、cs.LG

PhD Thesis: Exploring the role of (self-)attention in cognitive and computer vision architecture

Mohit Vaishnav

专题命中 VLM训练与架构 ：visual reasoning(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments PhD Thesis, 152 pages, 32 figures, 6 tables

TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment

Jianwei Yang, Yonatan Bisk, Jianfeng Gao

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted by ICCV 2021

Learning Compact Vision Tokens for Efficient Large Multimodal Models

Hao Tang, Chengchao Shen

专题命中 VLM训练与架构 ：LLaVA(abstract,comments)；分类 cs.CV、cs.AI

Comments The source code and trained weights are available at https://github.com/visresearch/LLaVA-STF

MR-IQA-2: Faithful Image Quality Reflection via Fine-Grained Credit Assignment

MR-IQA-2：通过细粒度信用分配实现真实的图像质量反映

Yuan li, Youyuan Lin, Chenhui Chu, Shin'ya Nishida

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

AI总结 本研究针对现有盲图像质量评估（IQA）中推理真实性不足的问题，提出MR-IQA-2框架，通过解耦推理与评分的细粒度信用分配，在IQA基准上实现了与人类评分的竞争性对齐，还能提供更丰富的视觉理解。