多模态大模型

Neural Parameter Search for Slimmer Fine-Tuned Models and Better Transfer

Guodong Du, Zitao Fang, Jing Li, Junlin Li, Runhua Jiang, Shuyang Yu, Yifei Guo, Yangneng Chen, Sim Kuan Goh, Ho-Kin Tang, Daojing He, Honghai Liu, Min Zhang

机构 * Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳）） ； Xiamen University Malaysia（厦门大学马来西亚分校） 纠错

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CL、cs.AI

Comments Accepted by ACL2025 Main

Learning Joint ID-Textual Representation for ID-Preserving Image Synthesis

Zichuan Liu, Liming Jiang, Qing Yan, Yumin Jia, Hao Kang, Xin Lu

机构 * ByteDance Intelligent Creation（字节跳动智能创作） 纠错

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

HAMF: A Hybrid Attention-Mamba Framework for Joint Scene Context Understanding and Future Motion Representation Learning

Xiaodong Mei, Sheng Wang, Jie Cheng, Yingbing Chen, Dan Xu

机构 * Hong Kong University of Science and Technology（香港科技大学） 纠错

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments In submission

SpikeCLIP: A Contrastive Language-Image Pretrained Spiking Neural Network

Changze Lv, Tianlong Li, Wenhao Liu, Yufei Gu, Jianhan Xu, Cenyuan Zhang, Muling Wu, Xiaoqing Zheng, Xuanjing Huang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL

ModRWKV: Transformer Multimodality in Linear Time

Jiale Kang, Ziyin Yue, Qingyu Yin, Jiang Rui, Weile Li, Zening Lu, Zhouran Ji

机构 * RWKVOS ； Zhejiang University（浙江大学） ； The Hong Kong University of Science and Technology（香港科学与技术大学） 纠错

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularization

Iñigo Pikabea, Iñaki Lacunza, Oriol Pareras, Carlos Escolano, Aitor Gonzalez-Agirre, Javier Hernando, Marta Villegas

机构 * Barcelona Supercomputing Center（巴塞罗那超级计算中心） ； Universitat Politècnica de Catalunya（加泰罗尼亚理工大学） 纠错

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments v2: Expanded model merging experiments. Fix duplicated subsection on limitations

Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation

Tiansheng Wen, Yifei Wang, Zequn Zeng, Zhong Peng, Yudi Su, Xinyang Liu, Bo Chen, Hongwei Liu, Stefanie Jegelka, Chenyu You

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ICML2025

Advancing Food Nutrition Estimation via Visual-Ingredient Feature Fusion

Huiyan Qi, Bin Zhu, Chong-Wah Ngo, Jingjing Chen, Ee-Peng Lim

机构 * Singapore Management University（新加坡管理大学） ； Fudan University（复旦大学） 纠错

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Accepted for publication in ACM International Conference on Multimedia Retrieval 2025

Efficient Bilinear Attention-based Fusion for Medical Visual Question Answering

Zhilin Zhang, Jie Wang, Zhanghao Qin, Ruiqi Zhu, Xiaoliang Gong

机构 * Tandon School of Engineering, New York University（纽约大学工程学院） ； School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院） ； College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院） 纠错

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV、cs.AI

Comments To be published in 2025 International Joint Conference on Neural Networks (IJCNN)

VladVA: Discriminative Fine-tuning of LVLMs

Yassine Ouali, Adrian Bulat, Alexandros Xenos, Anestis Zaganidis, Ioannis Maniadis Metaxas, Brais Martinez, Georgios Tzimiropoulos

机构 * Samsung AI Cambridge（三星AI剑桥） ； Technical University of Iasi（伊阿西技术大学） ； Queen Mary University of London（伦敦女王学院） 纠错

专题命中 多模态训练与对齐 ：image-text(abstract)；分类 cs.CV、cs.AI

Comments Published at CVPR 2025

Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgetting

Yijie Hong, Xiaofei Yin, Xinzhong Wang, Yi Tu, Ya Guo, Sufeng Duan, Weiqiang Wang, Lingyong Fang, Depeng Wang, Huijia Zhu

机构 * Shanghai Jiao Tong University（上海交通大学） ； Ant Security Lab, Ant Group（蚂蚁集团安全实验室） 纠错

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 13 pages, 3 figures

FILA: Fine-Grained Vision Language Models

Shiding Zhu, Wenhui Dong, Jun Song, Yingbo Wang, Yanan Guo, Bo Zheng

机构 * Zhejiang University（浙江大学） ； Taobao & Tmall Group of Alibaba（阿里巴巴集团） ； USTC（中国科学技术大学） 纠错

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments 9 pages, 4 figures, accepted to ICLR 2025 workshop

PyViT-FUSE: A Foundation Model for Multi-Sensor Earth Observation Data

Manuel Weber, Carly Beneke

机构 * EarthDaily Analytics 纠错

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments 11 pages, 13 figures, Published at ICLR 2025 - Machine Learning for Remote Sensing (ML4RS) Workshop

Cognitive Memory in Large Language Models

Lianlei Shan, Shixian Luo, Zezhou Zhu, Yu Yuan, Yong Wu

机构 * Li Auto（利亚 Auto） 纠错

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 37 pages, 9 figures

PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models

Nusrat Jahan Prottasha, Upama Roy Chowdhury, Shetu Mohanto, Tasfia Nuzhat, Abdullah As Sami, Md Shamol Ali, Md Shohanur Islam Sobuj, Hafijur Raman, Md Kowsher, Ozlem Ozmen Garibay

机构 * University of Central Florida（佛罗里达中央大学） ； Khulna University of Engineering & Technology（库尔纳工程与技术大学） ； Delineate Inc.（Delineate公司） ； Universiti Tenaga Nasional（国家能源大学） ； University of South Florida（佛罗里达州立大学） ； Daffodil International University（达夫迪尔国际大学） ； Anymate Me（Anymate Me公司） 纠错

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments PEFT Survey paper

Harmony: A Unified Framework for Modality Incremental Learning

Yaguang Song, Xiaoshan Yang, Dongmei Jiang, Yaowei Wang, Changsheng Xu

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.AI、cs.MM

Enhancing Multi-task Learning Capability of Medical Generalist Foundation Model via Image-centric Multi-annotation Data

Xun Zhu, Fanbin Mo, Zheng Zhang, Jiaxi Wang, Yiming Shi, Ming Wu, Chuang Zhang, Miao Li, Ji Wu

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

FocalLens: Instruction Tuning Enables Zero-Shot Conditional Image Representations

Cheng-Yu Hsieh, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Chun-Liang Li, Ranjay Krishna, Oncel Tuzel, Hadi Pouransari

专题命中 多模态训练与对齐 ：image-text(abstract)；分类 cs.CV、cs.CL

Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusion

Junkai Zhang, Bin Li, Shoujun Zhou, Yue Du

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV、cs.AI

Face-LLaVA: Facial Expression and Attribute Understanding through Instruction Tuning

Ashutosh Chaubey, Xulang Guan, Mohammad Soleymani

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Project Page: https://face-llava.github.io

From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models

Chejian Xu, Wei Ping, Peng Xu, Zihan Liu, Boxin Wang, Mohammad Shoeybi, Bo Li, Bryan Catanzaro

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning

Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Binbin Hu, Ziqi Liu, Wen Zhang, Huajun Chen

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CL、cs.AI

Comments ICLR 2025 Camera-ready Version. Fix a caption typo in the current version

VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learning

Xianwei Zhuang, Yuxin Xie, Yufan Deng, Dongchao Yang, Liming Liang, Jinghan Ru, Yuguo Yin, Yuexian Zou

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Code is available at: https://github.com/VARGPT-family/VARGPT-v1.1. arXiv admin note: text overlap with arXiv:2501.12327

Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant

Alan Dao, Dinh Bach Vu, Huy Hoang Ha

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、eess.AS

Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentation

Ting Liu, Siyuan Li

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV、cs.AI

Comments accepted to CVPR2025

BiPVL-Seg: Bidirectional Progressive Vision-Language Fusion with Global-Local Alignment for Medical Image Segmentation

Rafi Ibn Sultan, Hui Zhu, Chengyin Li, Dongxiao Zhu

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV、cs.AI

Advancements in Natural Language Processing: Exploring Transformer-Based Architectures for Text Understanding

Tianhao Wu, Yu Wang, Ngoc Quach

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments This paper has been accepted by the 5th International Conference on Artificial Intelligence and Industrial Technology Applications (AIITA 2025)

Human Motion Instruction Tuning

Lei Li, Sen Jia, Jianhao Wang, Zhongyu Jiang, Feng Zhou, Ju Dai, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by CVPR 2025

Bridging Writing Manner Gap in Visual Instruction Tuning by Creating LLM-aligned Instructions

Dong Jing, Nanyi Fei, Zhiwu Lu

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CL、cs.AI

TransFusionOdom: Interpretable Transformer-based LiDAR-Inertial Fusion Odometry Estimation

Leyuan Sun, Guanqun Ding, Yue Qiu, Yusuke Yoshiyasu, Fumio Kanehiro

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Submitted to IEEE Sensors Journal with some modifications. This work has been submitted to the IEEE for possible publication

Journal ref JSEN.2023.3302401