视觉大模型 / VLM - arXivDaily 专题

2401.01326 2024-01-17 cs.CL cs.AI cs.LG 62%

An Autoregressive Text-to-Graph Framework for Joint Entity and Relation Extraction

Urchade Zaratiana, Nadi Tomeh, Pierre Holat, Thierry Charnois

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.AI、cs.LG

Comments AAAI 2024 (camera ready version)

2401.02173 2024-01-05 cs.CV cs.AI 62%

Prompt Decoupling for Text-to-Image Person Re-identification

Weihao Li, Lei Tan, Pingyang Dai, Yan Zhang

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2308.10253 2023-12-29 cs.CV cs.CL cs.LG 62%

StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data

Yanda Li, Chi Zhang, Gang Yu, Zhibin Wang, Bin Fu, Guosheng Lin, Chunhua Shen, Ling Chen, Yunchao Wei

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV、cs.LG

Comments Project page: https://github.com/icoz69/StableLLAVA

2311.18260 2023-12-22 eess.IV cs.CL cs.CV cs.LG 62%

Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation

Ryutaro Tanno, David G. T. Barrett, Andrew Sellergren, Sumedh Ghaisas, Sumanth Dathathri, Abigail See, Johannes Welbl, Karan Singhal, Shekoofeh Azizi, Tao Tu, Mike Schaekermann, Rhys May, Roy Lee, SiWai Man, Zahra Ahmed, Sara Mahdavi, Yossi Matias, Joelle Barral, Ali Eslami, Danielle Belgrave, Vivek Natarajan, Shravya Shetty, Pushmeet Kohli, Po-Sen Huang, Alan Karthikesalingam, Ira Ktena

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

2312.11171 2023-12-19 cs.CV cs.AI 62%

UniDCP: Unifying Multiple Medical Vision-language Tasks via Dynamic Cross-modal Learnable Prompts

Chenlu Zhan, Yufei Zhang, Yu Lin, Gaoang Wang, Hongwei Wang

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2311.10933 2023-11-21 cs.AI cs.CL cs.CV cs.HC 62%

Representing visual classification as a linear combination of words

Shobhit Agarwal, Yevgeniy R. Semenov, William Lotter

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments To be published in the Proceedings of the 3rd Machine Learning for Health symposium, Proceedings of Machine Learning Research (PMLR)

2303.12513 2023-11-07 cs.CV cs.CL cs.LG 62%

Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding

Morris Alper, Michael Fiman, Hadar Averbuch-Elor

专题命中 VLM训练与架构：visual reasoning(abstract)；分类 cs.CV、cs.LG

Comments Accepted to CVPR 2023. Project webpage: https://isbertblind.github.io/

2311.01064 2023-11-03 cs.CV cs.LG 62%

Multimodal Foundation Models for Zero-shot Animal Species Recognition in Camera Trap Images

Zalan Fabian, Zhongqi Miao, Chunyuan Li, Yuanhan Zhang, Ziwei Liu, Andrés Hernández, Andrés Montes-Rojas, Rafael Escucha, Laura Siabatto, Andrés Link, Pablo Arbeláez, Rahul Dodhia, Juan Lavista Ferres

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments 18 pages, 9 figures

2310.16148 2023-10-26 cs.CV cs.AI 62%

Yin Yang Convolutional Nets: Image Manifold Extraction by the Analysis of Opposites

Augusto Seben da Rosa, Frederico Santos de Oliveira, Anderson da Silva Soares, Arnaldo Candido Junior

专题命中 VLM训练与架构：vision language model(abstract)；分类 cs.CV、cs.AI

Comments 12 pages, 5 tables and 6 figures

2310.15145 2023-10-24 cs.RO cs.AI cs.LG 62%

Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning

Jingyun Yang, Max Sobol Mark, Brandon Vu, Archit Sharma, Jeannette Bohg, Chelsea Finn

专题命中 VLM训练与架构：vision language model(abstract)；分类 cs.AI、cs.LG

2306.13856 2023-10-24 cs.CV cs.AI 62%

Learning-to-Rank Meets Language: Boosting Language-Driven Ordering Alignment for Ordinal Classification

Rui Wang, Peipei Li, Huaibo Huang, Chunshui Cao, Ran He, Zhaofeng He

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted by NeurIPS 2023

2211.15398 2023-09-06 cs.CV cs.LG 62%

Leveraging per Image-Token Consistency for Vision-Language Pre-training

Yunhao Gou, Tom Ko, Hansi Yang, James Kwok, Yu Zhang, Mingxuan Wang

专题命中 VLM训练与架构：VLM(abstract)；分类 cs.CV、cs.LG

Comments Accepted by CVPR 2023

2308.09970 2023-08-22 cs.CL cs.AI cs.LG 62%

Tackling Vision Language Tasks Through Learning Inner Monologues

Diji Yang, Kezhen Chen, Jinmeng Rao, Xiaoyuan Guo, Yawen Zhang, Jie Yang, Yi Zhang

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.AI、cs.LG

2307.07397 2023-07-17 cs.CV cs.LG 62%

Improving Zero-Shot Generalization for CLIP with Synthesized Prompts

Zhengbo Wang, Jian Liang, Ran He, Nan Xu, Zilei Wang, Tieniu Tan

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted by ICCV 2023

2212.10549 2023-07-06 cs.CL cs.CV cs.LG 62%

Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment

Rohan Pandey, Rulin Shao, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments ACL 2023

2305.04160 2023-05-23 cs.CL cs.AI cs.CV eess.AS 62%

X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages

Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, Bo Xu

专题命中 VLM训练与架构：visual language model(abstract)；分类 cs.CV、cs.AI

2305.01239 2023-05-03 cs.CV cs.AI 62%

DRPT: Disentangled and Recurrent Prompt Tuning for Compositional Zero-Shot Learning

Xiaocheng Lu, Ziming Liu, Song Guo, Jingcai Guo, Fushuo Huo, Sikai Bai, Tao Han

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2304.14204 2023-04-28 cs.AI cs.CV 62%

Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining

Bingqian Lin, Zicong Chen, Mingjie Li, Haokun Lin, Hang Xu, Yi Zhu, Jianzhuang Liu, Wenjia Cai, Lei Yang, Shen Zhao, Chenfei Wu, Ling Chen, Xiaojun Chang, Yi Yang, Lei Xing, Xiaodan Liang

专题命中 VLM训练与架构：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Project page: https://github.com/chenzcv7/MOTOR

2204.03574 2023-04-25 cs.LG cs.CL cs.CV 62%

Learning to Compose Soft Prompts for Compositional Zero-Shot Learning

Nihal V. Nayak, Peilin Yu, Stephen H. Bach

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments ICLR 2023

2208.05140 2023-04-13 eess.IV cs.CL cs.CV cs.LG 62%

Self-supervised Multi-modal Training from Uncurated Image and Reports Enables Zero-shot Oversight Artificial Intelligence in Radiology

Sangjoon Park, Eun Sun Lee, Kyung Sook Shin, Jeong Eun Lee, Jong Chul Ye

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

2303.02995 2023-03-07 cs.CV cs.CL cs.LG 62%

HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, Yongfeng Zhang

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted at ICLR 2023

2203.13366 2023-01-04 cs.IR cs.AI cs.CL cs.LG 62%

Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)

Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, Yongfeng Zhang

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.AI、cs.LG

Comments Accepted to ACM RecSys 2022. The arXiv version is more comprehensive

2111.10974 2022-12-29 cs.CV cs.AI cs.CL 62%

Many Heads but One Brain: Fusion Brain -- a Competition and a Single Multimodal Multitask Architecture

Daria Bakshandaeva, Denis Dimitrov, Vladimir Arkhipkin, Alex Shonenkov, Mark Potanin, Denis Karachev, Andrey Kuznetsov, Anton Voronov, Vera Davydova, Elena Tutubalina, Aleksandr Petiushko

专题命中 VLM训练与架构：visual question answering(abstract)；分类 cs.CV、cs.AI

2211.08025 2022-11-16 cs.LG cs.CL cs.CV 62%

FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformers

Jinyu Chen, Wenchao Xu, Song Guo, Junxiao Wang, Jie Zhang, Haozhao Wang

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

2201.12596 2022-09-15 cs.CV cs.AI cs.MM 62%

MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning

Zejun Li, Zhihao Fan, Huaixiao Tou, Jingjing Chen, Zhongyu Wei, Xuanjing Huang

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ACM MM22

2105.12210 2021-06-01 cs.LG cs.CV cs.NE 62%

The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design

George Philipp

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.CV、cs.LG

Comments This work is based on the PhD thesis with the same name, author, year and institution. Both works may be cited interchangeably

2103.15679 2021-03-30 cs.CV cs.LG 62%

Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers

Hila Chefer, Shir Gur, Lior Wolf

专题命中 VLM训练与架构：visual question answering(abstract)；分类 cs.CV、cs.LG

2011.09530 2020-11-20 cs.CV cs.AI eess.IV 62%

Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language

Hassan Akbari, Hamid Palangi, Jianwei Yang, Sudha Rao, Asli Celikyilmaz, Roland Fernandez, Paul Smolensky, Jianfeng Gao, Shih-Fu Chang

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.CV、cs.AI

2006.14744 2020-07-28 cs.CL cs.CV cs.LG 62%

Graph Optimal Transport for Cross-Domain Alignment

Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li, Lawrence Carin, Jingjing Liu

专题命中 VLM训练与架构：visual question answering(abstract)；分类 cs.CV、cs.LG

Journal ref ICML 2020

1907.07804 2020-07-06 cs.LG cs.CL cs.CV cs.NE stat.ML 62%

OmniNet: A unified architecture for multi-modal multi-task learning

Subhojeet Pramanik, Priyanka Agrawal, Aman Hussain

专题命中 VLM训练与架构：visual question answering(abstract)；分类 cs.CV、cs.LG

Comments Source code available at: https://github.com/subho406/OmniNet

1. VLM训练与架构 5067 篇

An Autoregressive Text-to-Graph Framework for Joint Entity and Relation Extraction

Prompt Decoupling for Text-to-Image Person Re-identification

StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data

Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation

UniDCP: Unifying Multiple Medical Vision-language Tasks via Dynamic Cross-modal Learnable Prompts

Representing visual classification as a linear combination of words

Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding

Multimodal Foundation Models for Zero-shot Animal Species Recognition in Camera Trap Images

Yin Yang Convolutional Nets: Image Manifold Extraction by the Analysis of Opposites

Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning

Learning-to-Rank Meets Language: Boosting Language-Driven Ordering Alignment for Ordinal Classification

Leveraging per Image-Token Consistency for Vision-Language Pre-training

Tackling Vision Language Tasks Through Learning Inner Monologues

Improving Zero-Shot Generalization for CLIP with Synthesized Prompts

Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment

X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages

DRPT: Disentangled and Recurrent Prompt Tuning for Compositional Zero-Shot Learning

Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining

Learning to Compose Soft Prompts for Compositional Zero-Shot Learning

Self-supervised Multi-modal Training from Uncurated Image and Reports Enables Zero-shot Oversight Artificial Intelligence in Radiology

HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)

Many Heads but One Brain: Fusion Brain -- a Competition and a Single Multimodal Multitask Architecture

FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformers

MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning

The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design

Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers

Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language

Graph Optimal Transport for Cross-Domain Alignment

OmniNet: A unified architecture for multi-modal multi-task learning