视觉大模型 / VLM

SHAPE : Self-Improved Visual Preference Alignment by Iteratively Generating Holistic Winner

Kejia Chen, Jiawen Zhang, Jiacong Hu, Jiazhen Yang, Jian Lou, Zunlei Feng, Mingli Song

专题命中 VLM训练与架构 ：visual language model(abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

Aligning Large Language Models and Geometric Deep Models for Protein Representation

Dong Shu, Bingbing Duan, Kai Guo, Kaixiong Zhou, Jiliang Tang, Mengnan Du

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.AI、cs.LG

Comments 37 pages, 10 figures

PinLanding: Content-First Keyword Landing Page Generation via Multi-Modal AI for Web-Scale Discovery

Faye Zhang, Jasmine Wan, Qianyu Cheng, Jinfeng Rao

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI、cs.LG

Improving Adversarial Transferability in MLLMs via Dynamic Vision-Language Alignment Attack

Chenhe Gu, Jindong Gu, Andong Hua, Yao Qin

专题命中 VLM训练与架构 ：LLaVA(abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.LG

Comments arXiv admin note: text overlap with arXiv:2403.09766

Object-centric Binding in Contrastive Language-Image Pretraining

Rim Assouel, Pietro Astolfi, Florian Bordes, Michal Drozdzal, Adriana Romero-Soriano

专题命中 VLM训练与架构 ：vision language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

LLaSA: Large Language and Structured Data Assistant

Yao Xu, Shizhu He, Jiabei Chen, Zeng Xiangrong, Bingning Wang, Guang Liu, Jun Zhao, Kang Liu

专题命中 VLM训练与架构 ：vision-language model(abstract)；grounding(abstract)；分类 cs.AI、cs.LG

Comments NAACL 2025 Main

PerPO: Perceptual Preference Optimization via Discriminative Rewarding

Zining Zhu, Liang Zhao, Kangheng Lin, Jinze Yang, En Yu, Chenglong Liu, Haoran Wei, Jianjian Sun, Zheng Ge, Xiangyu Zhang

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.AI、cs.LG

Personalized Preference Fine-tuning of Diffusion Models

Meihua Dang, Anikait Singh, Linqi Zhou, Stefano Ermon, Jiaming Song

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.LG

OpenLKA: an open dataset of lane keeping assist from market autonomous vehicles

Yuhang Wang, Abdulaziz Alhuraish, Shengming Yuan, Shuyi Wang, Hao Zhou

专题命中 VLM训练与架构 ：vision language model(abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Baichuan-Omni Technical Report

Yadong Li, Haoze Sun, Mingan Lin, Tianpeng Li, Guosheng Dong, Tao Zhang, Bowen Ding, Wei Song, Zhenglin Cheng, Yuqi Huo, Song Chen, Xu Li, Da Pan, Shusen Zhang, Xin Wu, Zheng Liang, Jun Liu, Tao Zhang, Keer Lu, Yaqi Zhao, Yanjun Shen, Fan Yang, Kaicheng Yu, Tao Lin, Jianhua Xu, Zenan Zhou, Weipeng Chen

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

SCBench: A Sports Commentary Benchmark for Video LLMs

Kuangzhi Ge, Lingjun Chen, Kevin Zhang, Yulin Luo, Tianyu Shi, Liaoyuan Fan, Xiang Li, Guanqun Wang, Shanghang Zhang

专题命中 VLM训练与架构 ：LLaVA(abstract)；InternVL(abstract)；分类 cs.CV、cs.AI

PAR: Prompt-Aware Token Reduction Method for Efficient Large Multimodal Models

Yingen Liu, Fan Wu, Ruihui Li, Zhuo Tang, Kenli Li

专题命中 VLM训练与架构 ：visual question answering(abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments 10 pages, 5 figures,3 tables

BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions

Anas Awadalla, Le Xue, Manli Shu, An Yan, Jun Wang, Senthil Purushwalkam, Sheng Shen, Hannah Lee, Oscar Lo, Jae Sung Park, Etash Guha, Silvio Savarese, Ludwig Schmidt, Yejin Choi, Caiming Xiong, Ran Xu

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

MIRAGE: Multimodal Identification and Recognition of Annotations in Indian General Prescriptions

Tavish Mankash, V. S. Chaithanya Kota, Anish De, Praveen Prakash, Kshitij Jadhav

专题命中 VLM训练与架构 ：LLaVA(abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments 5 pages, 9 figures, 3 tables, submitted to ISBI 2025

BendVLM: Test-Time Debiasing of Vision-Language Embeddings

Walter Gerych, Haoran Zhang, Kimia Hamidieh, Eileen Pan, Maanas Sharma, Thomas Hartvigsen, Marzyeh Ghassemi

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)

Michael Saxon, Fatima Jahara, Mahsa Khoshnoodi, Yujie Lu, Aditya Sharma, William Yang Wang

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments NeurIPS 2024 Spotlight

Teach Multimodal LLMs to Comprehend Electrocardiographic Images

Ruoqi Liu, Yuelin Bai, Xiang Yue, Ping Zhang

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

ViT3D Alignment of LLaMA3: 3D Medical Image Report Generation

Siyou Li, Beining Xu, Yihao Luo, Dong Nie, Le Zhang

专题命中 VLM训练与架构 ：visual question answering(abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

Democratizing MLLMs in Healthcare: TinyLLaVA-Med for Efficient Healthcare Diagnostics in Resource-Constrained Settings

Aya El Mir, Lukelo Thadei Luoga, Boyuan Chen, Muhammad Abdullah Hanif, Muhammad Shafique

专题命中 VLM训练与架构 ：LLaVA(abstract)；MLLM(abstract)；分类 cs.AI、cs.LG

Decoding Style: Efficient Fine-Tuning of LLMs for Image-Guided Outfit Recommendation with Preference

Najmeh Forouzandehmehr, Nima Farrokhsiar, Ramin Giahi, Evren Korpeoglu, Kannan Achan

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.AI、cs.LG

Comments CIKM 2024

Multi-modal Relation Distillation for Unified 3D Representation Learning

Huiqun Wang, Yiping Bao, Panwang Pan, Zeming Li, Xiao Liu, Ruijie Yang, Di Huang

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ECCV2024

CROME: Cross-Modal Adapters for Efficient Multimodal LLM

Sayna Ebrahimi, Sercan O. Arik, Tejas Nama, Tomas Pfister

专题命中 VLM训练与架构 ：visual question answering(abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.LG

D-Rax: Domain-specific Radiologic assistant leveraging multi-modal data and eXpert model predictions

Hareem Nisar, Syed Muhammad Anwar, Zhifan Jiang, Abhijeet Parida, Ramon Sanchez-Jacob, Vishwesh Nath, Holger R. Roth, Marius George Linguraru

专题命中 VLM训练与架构 ：vision language model(abstract)；LLaVA(abstract)；分类 cs.AI、cs.LG

Comments accepted to the MICCAI 2024 Second International Workshop on Foundation Models for General Medical AI

UMBRAE: Unified Multimodal Brain Decoding

Weihao Xia, Raoul de Charette, Cengiz Öztireli, Jing-Hao Xue

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments ECCV 2024. Project: https://weihaox.github.io/UMBRAE

MATE: Meet At The Embedding -- Connecting Images with Long Texts

Young Kyun Jang, Junmo Kang, Yong Jae Lee, Donghyun Kim

专题命中 VLM训练与架构 ：vision language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?

Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong, Qinglan Huang, Canyu Chen, Qinghao Ye, Zhihong Zhu, Yuqing Zhang, Jiawei Zhou, Zhuokai Zhao, Rafael Rafailov, Chelsea Finn, Huaxiu Yao

专题命中 VLM训练与架构 ：VLM(abstract)；LLaVA(abstract)；分类 cs.CV、cs.LG

Comments 42 pages, 13 figures, 33 tables

RAVEN: Multitask Retrieval Augmented Vision-Language Learning

Varun Nagaraj Rao, Siddharth Choudhary, Aditya Deshpande, Ravi Kumar Satzoda, Srikar Appalaraju

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

MammothModa: Multi-Modal Large Language Model

Qi She, Junwen Pan, Xin Wan, Rui Zhang, Dawei Lu, Kai Huang

专题命中 VLM训练与架构 ：LLaVA(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments Technical report

Directed Domain Fine-Tuning: Tailoring Separate Modalities for Specific Training Tasks

Daniel Wen, Nafisa Hussain

专题命中 VLM训练与架构 ：visual language model(abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression

Zilun Zhang, Yutao Sun, Tiancheng Zhao, Leigang Sha, Ruochen Xu, Kyusong Lee, Jianwei Yin

专题命中 VLM训练与架构 ：LLaVA(abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI