视觉大模型 / VLM - arXivDaily 专题

2411.00871 2024-11-05 cs.LG cs.AI q-bio.MN 62%

LLaMo: Large Language Model-based Molecular Graph Assistant

Jinyoung Park, Minseong Bae, Dohwan Ko, Hyunwoo J. Kim

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.AI、cs.LG

Comments NeurIPS 2024

2409.15657 2024-10-31 cs.AI cs.CL cs.LG 62%

M$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning

Taowen Wang, Yiyang Liu, James Chenhao Liang, junhan zhao, Yiming Cui, Yuning Mao, Shaoliang Nie, Jiahao Liu, Fuli Feng, Zenglin Xu, Cheng Han, Lifu Huang, Qifan Wang, Dongfang Liu

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.AI、cs.LG

Comments EMNLP 2024

2410.19079 2024-10-30 cs.CV cs.LG 62%

BIFRÖST: 3D-Aware Image compositing with Language Instructions

Lingxiao Li, Kaixiong Gong, Weihong Li, Xili Dai, Tao Chen, Xiaojun Yuan, Xiangyu Yue

专题命中 VLM训练与架构：MLLM(abstract)；分类 cs.CV、cs.LG

Comments NeurIPS 2024, Code Available: https://github.com/lingxiao-li/Bifrost

2410.20165 2024-10-29 cs.CV cs.AI 62%

Diff-CXR: Report-to-CXR generation through a disease-knowledge enhanced diffusion model

Peng Huang, Bowen Guo, Shuyu Liang, Junhu Fu, Yuanyuan Wang, Yi Guo

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2407.12315 2024-10-29 cs.CV cs.AI cs.HC cs.IR 62%

ModalChorus: Visual Probing and Alignment of Multi-modal Embeddings via Modal Fusion Map

Yilin Ye, Shishi Xiao, Xingchen Zeng, Wei Zeng

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments VIS 2024

2410.15780 2024-10-22 cs.CV cs.AI 62%

An Efficient System for Automatic Map Storytelling -- A Case Study on Historical Maps

Ziyi Liu, Claudio Affolter, Sidi Wu, Yizi Chen, Lorenz Hurni

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2406.01662 2024-10-18 cs.CV cs.AI 62%

Few-Shot Classification of Interactive Activities of Daily Living (InteractADL)

Zane Durante, Robathan Harries, Edward Vendrow, Zelun Luo, Yuta Kyuragi, Kazuki Kozuka, Li Fei-Fei, Ehsan Adeli

专题命中 VLM训练与架构：VLM(abstract)；分类 cs.CV、cs.AI

2410.08469 2024-10-17 cs.LG cs.CL cs.CV 62%

Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIP

Eunji Kim, Kyuhong Shim, Simyung Chang, Sungroh Yoon

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted at EMNLP 2024 Findings

2410.05997 2024-10-10 eess.AS cs.CV cs.LG cs.SD 62%

An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment

Hugo Malard, Michel Olvera, Stéphane Lathuiliere, Slim Essid

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV、cs.LG

2410.05938 2024-10-10 cs.CV cs.AI 62%

EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment

Yifei Xing, Xiangyuan Lan, Ruiping Wang, Dongmei Jiang, Wenjun Huang, Qingfang Zheng, Yaowei Wang

专题命中 VLM训练与架构：MLLM(abstract)；分类 cs.CV、cs.AI

2407.12871 2024-10-10 cs.CL cs.AI cs.LG 62%

MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation

Xiaohan Wang, Dian Li, Yilin Zhao, Sinbadliu, Hui Wang

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.AI、cs.LG

Comments 9 pages, 4 figures

2410.05361 2024-10-10 cs.LG cs.AI cs.SD eess.AS 62%

RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Prediction

Yuwei Zhang, Tong Xia, Aaqib Saeed, Cecilia Mascolo

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.AI、cs.LG

2410.03038 2024-10-08 cs.LG cs.CV 62%

CPFD: Confidence-aware Privileged Feature Distillation for Short Video Classification

Jinghao Shi, Xiang Shen, Kaili Zhao, Xuedong Wang, Vera Wen, Zixuan Wang, Yifan Wu, Zhixin Zhang

专题命中 VLM训练与架构：VLM(abstract)；分类 cs.CV、cs.LG

Comments Camera ready for CIKM 2024

2410.03334 2024-10-07 cs.CV cs.AI 62%

An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation

Ahmed Abdulaal, Hugo Fry, Nina Montaña-Brown, Ayodeji Ijishakin, Jack Gao, Stephanie Hyland, Daniel C. Alexander, Daniel C. Castro

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2407.08044 2024-09-30 cs.CL cs.AI cs.LG 62%

RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization

Xijie Huang, Zechun Liu, Shih-Yang Liu, Kwang-Ting Cheng

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.AI、cs.LG

Comments EMNLP 2024 Findings, Codes: https://github.com/HuangOwen/RoLoRA, Models: https://huggingface.co/collections/ScarletAce/rolora-66f5f228a90681c7c4512b28

2409.17634 2024-09-27 cs.CV cs.AI 62%

P4Q: Learning to Prompt for Quantization in Visual-language Models

Huixin Sun, Runqi Wang, Yanjing Li, Xianbin Cao, Xiaolong Jiang, Yao Hu, Baochang Zhang

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2407.00634 2024-09-25 cs.CV cs.LG 62%

Tarsier: Recipes for Training and Evaluating Large Video Description Models

Jiawei Wang, Liping Yuan, Yuchen Zhang, Haomiao Sun

专题命中 VLM训练与架构：VLM(abstract)；分类 cs.CV、cs.LG

2409.14163 2024-09-24 cs.CV cs.CL cs.LG 62%

PromptTA: Prompt-driven Text Adapter for Source-free Domain Generalization

Haoran Zhang, Shuanghao Bai, Wanqi Zhou, Jingwen Fu, Badong Chen

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

2409.11059 2024-09-19 cs.CV cs.LG 62%

OneEncoder: A Lightweight Framework for Progressive Alignment of Modalities

Bilal Faye, Hanane Azzag, Mustapha Lebbah

专题命中 VLM训练与架构：visual question answering(abstract)；分类 cs.CV、cs.LG

2409.10921 2024-09-18 cs.CV cs.AI 62%

KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graph

Yanbei Jiang, Krista A. Ehinger, Jey Han Lau

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted at IJCAI 2024

2406.18790 2024-09-13 cs.CV cs.AI 62%

MUMU: Bootstrapping Multimodal Image Generation from Text-to-Image Data

William Berman, Alexander Peysakhovich

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2312.10692 2024-09-04 cs.CV cs.AI 62%

Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusion

Xiao Wang, Jiandong Jin, Chenglong Li, Jin Tang, Cheng Zhang, Wei Wang

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted by IEEE TCSVT 2024, Camera Ready Version

2408.16667 2024-08-30 cs.LG cs.AI cs.CL cs.MA 62%

Iterative Graph Alignment

Fangyuan Yu, Hardeep Singh Arora, Matt Johnson

专题命中 VLM训练与架构：VLM(abstract)；分类 cs.AI、cs.LG

Comments 12 pages, 4 figures

2408.14023 2024-08-27 cs.CV cs.AI 62%

Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos

Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, Hui Wang

专题命中 VLM训练与架构：MLLM(abstract)；分类 cs.CV、cs.AI

Comments 10 pages, 5 figures

2408.07037 2024-08-14 cs.CV cs.AI 62%

PathInsight: Instruction Tuning of Multimodal Datasets and Models for Intelligence Assisted Diagnosis in Histopathology

Xiaomin Wu, Rui Xu, Pengchen Wei, Wenkang Qin, Peixiang Huang, Ziheng Li, Lin Luo

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV、cs.AI

Comments 10 pages, 2 figures

2408.03574 2024-08-08 cs.CV cs.CL cs.LG 62%

Teach CLIP to Develop a Number Sense for Ordinal Regression

Yao Du, Qiang Zhai, Weihang Dai, Xiaomeng Li

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted by ECCV 2024

2408.00290 2024-08-02 cs.CV cs.AI 62%

Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network

Bin Cheng, Jiaxuan Lu

专题命中 VLM训练与架构：MLLM(abstract)；分类 cs.CV、cs.AI

2403.19584 2024-07-31 cs.CV cs.AI 62%

Img2Loc: Revisiting Image Geolocalization using Multi-modality Foundation Models and Image-based Retrieval-Augmented Generation

Zhongliang Zhou, Jielu Zhang, Zihan Guan, Mengxuan Hu, Ni Lao, Lan Mu, Sheng Li, Gengchen Mai

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV、cs.AI

2406.04413 2024-07-25 cs.CV cs.AI 62%

Efficient 3D-Aware Facial Image Editing via Attribute-Specific Prompt Learning

Amandeep Kumar, Muhammad Awais, Sanath Narayan, Hisham Cholakkal, Salman Khan, Rao Muhammad Anwer

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted at ECCV, 2024. Amandeep Kumar and Muhammad Awais are joint first authors. More details are available at https://awaisrauf.github.io/3d_face_editing

2310.08320 2024-07-24 cs.LG cs.CL cs.CR cs.CV 62%

Defending Our Privacy With Backdoors

Dominik Hintersdorf, Lukas Struppek, Daniel Neider, Kristian Kersting

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted at ECAI 2024

1. VLM训练与架构 5067 篇

LLaMo: Large Language Model-based Molecular Graph Assistant

M$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning

BIFRÖST: 3D-Aware Image compositing with Language Instructions

Diff-CXR: Report-to-CXR generation through a disease-knowledge enhanced diffusion model

ModalChorus: Visual Probing and Alignment of Multi-modal Embeddings via Modal Fusion Map

An Efficient System for Automatic Map Storytelling -- A Case Study on Historical Maps

Few-Shot Classification of Interactive Activities of Daily Living (InteractADL)

Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIP

An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment

EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment

MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation

RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Prediction

CPFD: Confidence-aware Privileged Feature Distillation for Short Video Classification

An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation

RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization

P4Q: Learning to Prompt for Quantization in Visual-language Models

Tarsier: Recipes for Training and Evaluating Large Video Description Models

PromptTA: Prompt-driven Text Adapter for Source-free Domain Generalization

OneEncoder: A Lightweight Framework for Progressive Alignment of Modalities

KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graph

MUMU: Bootstrapping Multimodal Image Generation from Text-to-Image Data

Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusion

Iterative Graph Alignment

Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos

PathInsight: Instruction Tuning of Multimodal Datasets and Models for Intelligence Assisted Diagnosis in Histopathology

Teach CLIP to Develop a Number Sense for Ordinal Regression

Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network

Img2Loc: Revisiting Image Geolocalization using Multi-modality Foundation Models and Image-based Retrieval-Augmented Generation

Efficient 3D-Aware Facial Image Editing via Attribute-Specific Prompt Learning

Defending Our Privacy With Backdoors