多模态大模型

From Text to Pixel: Advancing Long-Context Understanding in MLLMs

Yujie Lu, Xiujun Li, Tsu-Jui Fu, Miguel Eckstein, William Yang Wang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL

CluMo: Cluster-based Modality Fusion Prompt for Continual Learning in Visual Question Answering

Yuliang Cai, Mohammad Rostami

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning

Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, Natasha Jaques

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments weirdlabuw.github.io/vpl

Cross-aware Early Fusion with Stage-divided Vision and Language Transformer Encoders for Referring Image Segmentation

Yubin Cho, Hyunwoo Yu, Suk-ju Kang

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Published in IEEE Transactions on Multimedia (TMM)

UrFound: Towards Universal Retinal Foundation Models via Knowledge-Guided Masked Modeling

Kai Yu, Yang Zhou, Yang Bai, Zhi Da Soh, Xinxing Xu, Rick Siow Mong Goh, Ching-Yu Cheng, Yong Liu

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Multistain Pretraining for Slide Representation Learning in Pathology

Guillaume Jaume, Anurag Vaidya, Andrew Zhang, Andrew H. Song, Richard J. Chen, Sharifa Sahai, Dandan Mo, Emilio Madrigal, Long Phi Le, Faisal Mahmood

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments ECCV'24

Are Bigger Encoders Always Better in Vision Large Models?

Bozhou Li, Hao Liang, Zimo Meng, Wentao Zhang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL

EVA-02: A Visual Representation for Neon Genesis

Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, Yue Cao

专题命中 多模态训练与对齐 ：image-text(abstract)；分类 cs.CV、cs.CL

Comments v2: Fix some known issues & typos. v1: To Asuka. Code & Models: https://github.com/baaivision/EVA/tree/master/EVA-02

Journal ref Image and Vision Computing. Volume 149, September 2024, 105171

Advancing Vietnamese Visual Question Answering with Transformer and Convolutional Integration

Ngoc Son Nguyen, Van Son Nguyen, Tung Le

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted at the journal of Computers & Electrical Engineering (Received 8 March 2024, Revised 8 June 2024, Accepted 10 July 2024)

Journal ref Computers and Electrical Engineering 119 (2024) 109474

Rethinking RGB-D Fusion for Semantic Segmentation in Surgical Datasets

Muhammad Abdullah Jamal, Omid Mohareri

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Enhancing Inertial Hand based HAR through Joint Representation of Language, Pose and Synthetic IMUs

Vitor Fortes Rey, Lala Shakti Swarup Ray, Xia Qingxin, Kaishun Wu, Paul Lukowicz

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments ISWC 2024

DOPRA: Decoding Over-accumulation Penalization and Re-allocation in Specific Weighting Layer

Jinfeng Wei, Xiaofeng Zhang

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CL、cs.AI

Region-centric Image-Language Pretraining for Open-Vocabulary Detection

Dahun Kim, Anelia Angelova, Weicheng Kuo

专题命中 多模态训练与对齐 ：image-text(abstract)；分类 cs.CV、cs.AI

Comments ECCV 2024, project page at https://github.com/google-research/google-research/tree/master/fvlm/dito

Understanding and Improving Transfer Learning of Deep Models via Neural Collapse

Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez-Granda, Zhihui Zhu, Qing Qu

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments First two authors contributed equally. Accepted at TMLR

PG-Attack: A Precision-Guided Adversarial Attack Framework Against Vision Foundation Models for Autonomous Driving

Jiyuan Fu, Zhaoyu Chen, Kaixun Jiang, Haijing Guo, Shuyong Gao, Wenqiang Zhang

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.MM

Comments First-Place in the CVPR 2024 Workshop Challenge: Black-box Adversarial Attacks on Vision Foundation Models

Common Sense Reasoning for Deepfake Detection

Yue Zhang, Ben Colman, Xiao Guo, Ali Shahriyari, Gaurav Bharaj

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Stream State-tying for Sign Language Recognition

Jiyong Ma, Wen Gao, Chunli Wang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Directed Domain Fine-Tuning: Tailoring Separate Modalities for Specific Training Tasks

Daniel Wen, Nafisa Hussain

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Zoom and Shift are All You Need

Jiahao Qin

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments 8 pages, 3 figures

LangCell: Language-Cell Pre-training for Cell Identity Understanding

Suyuan Zhao, Jiahuan Zhang, Yushuai Wu, Yizhen Luo, Zaiqing Nie

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CL、cs.AI

Comments Accpeted by ICML 2024, code released

Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion

Ishaan Singh Rawal, Alexander Matyasko, Shantanu Jaiswal, Basura Fernando, Cheston Tan

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Accepted at ICML 2024

Attribute-Aware Implicit Modality Alignment for Text Attribute Person Search

Xin Wang, Fangfang Liu, Zheng Li, Caili Guo

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

MedFLIP: Medical Vision-and-Language Self-supervised Fast Pre-Training with Masked Autoencoder

Lei Li, Tianfang Zhang, Xinglin Zhang, Jiaqi Liu, Bingqi Ma, Yan Luo, Tao Chen

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL

X-VILA: Cross-Modality Alignment for Large Language Model

Hanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, Hongxu Yin

专题命中 多模态训练与对齐 ：any-to-any(abstract)；分类 cs.CV、cs.CL

Comments Technical Report

Enhanced Object Tracking by Self-Supervised Auxiliary Depth Estimation Learning

Zhenyu Wei, Yujie He, Zhanchuan Cai

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Safety Alignment for Vision Language Models

Zhendong Liu, Yuanbi Nie, Yingshui Tan, Xiangyu Yue, Qiushi Cui, Chongjun Wang, Xiaoyong Zhu, Bo Zheng

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments 23 pages, 15 figures

Text-centric Alignment for Multi-Modality Learning

Yun-Da Tsai, Ting-Yu Yen, Pei-Fu Guo, Zhe-Yan Li, Shou-De Lin

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL

Transcriptomics-guided Slide Representation Learning in Computational Pathology

Guillaume Jaume, Lukas Oldenburg, Anurag Vaidya, Richard J. Chen, Drew F. K. Williamson, Thomas Peeters, Andrew H. Song, Faisal Mahmood

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments CVPR'24, Oral

Are aligned neural networks adversarially aligned?

Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, Ludwig Schmidt

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models

Shizhe Diao, Rui Pan, Hanze Dong, Ka Shun Shum, Jipeng Zhang, Wei Xiong, Tong Zhang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments Published in NAACL 2024 Demo Track