多模态大模型

TouchUp-G: Improving Feature Representation through Graph-Centric Finetuning

Jing Zhu, Xiang Song, Vassilis N. Ioannidis, Danai Koutra, Christos Faloutsos

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments SIGIR 2024

ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understanding

Zhengzhuo Xu, Bowen Qu, Yiyan Qi, Sinan Du, Chengjin Xu, Chun Yuan, Jian Guo

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

COAP: Memory-Efficient Training with Correlation-Aware Gradient Projection

Jinqi Xiao, Shen Sang, Tiancheng Zhi, Jing Liu, Qing Yan, Yuqian Zhang, Linjie Luo, Bo Yuan

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2025

Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning

Weitai Kang, Haifeng Huang, Yuzhang Shang, Mubarak Shah, Yan Yan

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 8 pages

An empirical study of LLaMA3 quantization: from LLMs to MLLMs

Wei Huang, Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, Michele Magno

专题命中 多模态训练与对齐 ：multi-modal(abstract)；MLLM(abstract)

Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos

Mehmet Saygin Seyfioglu, Wisdom O. Ikezogwo, Fatemeh Ghezloo, Ranjay Krishna, Linda Shapiro

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)

Subba Reddy Oota, Zijiao Chen, Manish Gupta, Raju S. Bapi, Gael Jobard, Frederic Alexandre, Xavier Hinaut

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 61 pages, 22 figures

Journal ref Published in Transactions on Machine Learning Research (12/2024)

Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities

Adriel Saporta, Aahlad Puli, Mark Goldstein, Rajesh Ranganath

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NeurIPS 2024

TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment

Wei Li, Hehe Fan, Yongkang Wong, Mohan Kankanhalli, Yi Yang

专题命中 多模态训练与对齐 ：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NeurIPS 2024 (Spotlight)

Targetless Extrinsic Calibration of Stereo Cameras, Thermal Cameras, and Laser Sensors in the Wild

Taimeng Fu, Huai Yu, Wen Yang, Yaoyu Hu, Sebastian Scherer

专题命中 多模态训练与对齐 ：multi-modal(abstract)；cross-modal(abstract)

Comments This work has been submitted to the IEEE for possible publication

Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments

Haritheja Etukuru, Norihito Naka, Zijin Hu, Seungjae Lee, Julian Mehu, Aaron Edsinger, Chris Paxton, Soumith Chintala, Lerrel Pinto, Nur Muhammad Mahi Shafiullah

专题命中 多模态训练与对齐 ：multi-modal(abstract)；MLLM(abstract)

Comments Project website https://robotutilitymodels.com

Self-Supervised Visual Preference Alignment

Ke Zhu, Zheng Ge, Liang Zhao, Xiangyu Zhang

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments MM2024 oral

High Efficiency Image Compression for Large Visual-Language Models

Binzhe Li, Shurun Wang, Shiqi Wang, Yan Ye

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Improving Alignment and Robustness with Circuit Breakers

Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, Dan Hendrycks

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Code and models are available at https://github.com/GraySwanAI/circuit-breakers

Do Vision and Language Models Share Concepts? A Vector Space Alignment Study

Jiaang Li, Yova Kementchedjhieva, Constanza Fierro, Anders Søgaard

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 12 pages, long paper accepted by TACL

Whiteboard-of-Thought: Thinking Step-by-Step Across Modalities

Sachit Menon, Richard Zemel, Carl Vondrick

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project website: whiteboard.cs.columbia.edu/

Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression

Zilun Zhang, Yutao Sun, Tiancheng Zhao, Leigang Sha, Ruochen Xu, Kyusong Lee, Jianwei Yin

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Federated Document Visual Question Answering: A Pilot Study

Khanh Nguyen, Dimosthenis Karatzas

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

LLaMA-Excitor: General Instruction Tuning via Indirect Feature Interaction

Bo Zou, Chao Yang, Yu Qiao, Chengbin Quan, Youjian Zhao

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments This paper is accepted by CVPR 2024

LayoutLLM: Large Language Model Instruction Tuning for Visually Rich Document Understanding

Masato Fujitake

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments LREC-COLING 2024

GmGM: a Fast Multi-Axis Gaussian Graphical Model

Bailey Andrew, David Westhead, Luisa Cutillo

专题命中 多模态训练与对齐 ：multimodal(abstract)；multi-modal(abstract)

Comments 8 pages (33 additional in supplementary material), 19 figures, accepted at AIStats 2024

Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA

Wentao Mo, Yang Liu

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments To be published in AAAI 24

A Survey of Reasoning with Foundation Models

Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, Yue Wu, Wenhai Wang, Junsong Chen, Zhangyue Yin, Xiaozhe Ren, Jie Fu, Junxian He, Wu Yuan, Qi Liu, Xihui Liu, Yu Li, Hao Dong, Yu Cheng, Ming Zhang, Pheng Ann Heng, Jifeng Dai, Ping Luo, Jingdong Wang, Ji-Rong Wen, Xipeng Qiu, Yike Guo, Hui Xiong, Qun Liu, Zhenguo Li

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 20 Figures, 160 Pages, 750+ References, Project Page https://github.com/reasoning-survey/Awesome-Reasoning-Foundation-Models

Improving Radiology Summarization with Radiograph and Anatomy Prompts

Jinpeng Hu, Zhihong Chen, Yang Liu, Xiang Wan, Tsung-Hui Chang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 11 pages, ACL2023 Findings

Soft Alignment of Modality Space for End-to-end Speech Translation

Yuhao Zhang, Kaiqi Kou, Bei Li, Chen Xu, Chunliang Zhang, Tong Xiao, Jingbo Zhu

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted to ICASSP2024

Learning Speech Representation From Contrastive Token-Acoustic Pretraining

Chunyu Qiang, Hao Li, Yixin Tian, Ruibo Fu, Tao Wang, Longbiao Wang, Jianwu Dang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted by ICASSP 2024

APoLLo: Unified Adapter and Prompt Learning for Vision Language Models

Sanjoy Chowdhury, Sayan Nag, Dinesh Manocha

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at EMNLP 2023 (Main track)

Meaning Representations from Trajectories in Autoregressive Models

Tian Yu Liu, Matthew Trager, Alessandro Achille, Pramuditha Perera, Luca Zancato, Stefano Soatto

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Unified Coarse-to-Fine Alignment for Video-Text Retrieval

Ziyang Wang, Yi-Lin Sung, Feng Cheng, Gedas Bertasius, Mohit Bansal

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ICCV 2023

Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers

Dahun Kim, Anelia Angelova, Weicheng Kuo

专题命中 多模态训练与对齐 ：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2023 Highlight - https://github.com/mcahny/rovit ; adds LAION-2B result