多模态大模型

Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Driving

Ran Tian, Boyi Li, Xinshuo Weng, Yuxiao Chen, Edward Schmerling, Yue Wang, Boris Ivanovic, Marco Pavone

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.AI

Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining

Qi Zhang, Tianqi Du, Haotian Huang, Yifei Wang, Yisen Wang

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CL

SemanticFormer: Holistic and Semantic Traffic Scene Representation for Trajectory Prediction using Knowledge Graphs

Zhigang Sun, Zixu Wang, Lavdim Halilaj, Juergen Luettin

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV

Comments 8 pages, 7 figures, has been accepted for publication in the IEEE Robotics and Automation Letters (RA-L)

Learning Modality Knowledge Alignment for Cross-Modality Transfer

Wenxuan Ma, Shuang Li, Lincan Cai, Jingxuan Kang

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV

Comments ICML 2024

BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuning

Ruyang Liu, Chen Li, Yixiao Ge, Ying Shan, Thomas H. Li, Ge Li

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV

InFiConD: Interactive No-code Fine-tuning with Concept-based Knowledge Distillation

Jinbin Huang, Wenbin He, Liang Gou, Liu Ren, Chris Bryan

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.AI

DMF-Net: Image-Guided Point Cloud Completion with Dual-Channel Modality Fusion and Shape-Aware Upsampling Transformer

Aihua Mao, Yuxuan Tang, Jiangtao Huang, Ying He

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV

Semantic Enhanced Few-shot Object Detection

Zheng Wang, Yingjie Gao, Qingjie Liu, Yunhong Wang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV

Comments Accepted by ICIP 2024

GFM4MPM: Towards Geospatial Foundation Models for Mineral Prospectivity Mapping

Angel Daruna, Vasily Zadorozhnyy, Georgina Lukoczki, Han-Pang Chiu

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

Comments 12 pages, 16 figures, 7 tables

Relational Learning in Pre-Trained Models: A Theory from Hypergraph Recovery Perspective

Yang Chen, Cong Fang, Zhouchen Lin, Bing Liu

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.AI

CrossFuse: A Novel Cross Attention Mechanism based Infrared and Visible Image Fusion Approach

Hui Li, Xiao-Jun Wu

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV

Comments 11 pages, 16 fuigures

WirelessLLM: Empowering Large Language Models Towards Wireless Intelligence

Jiawei Shao, Jingwen Tong, Qiong Wu, Wei Guo, Zijian Li, Zehong Lin, Jun Zhang

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.AI

Unveiling Incomplete Modality Brain Tumor Segmentation: Leveraging Masked Predicted Auto-Encoder and Divergence Learning

Zhongao Sun, Jiameng Li, Yuhan Wang, Jiarong Cheng, Qing Zhou, Chun Li

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRI

Yulong Liu, Yongqiang Ma, Guibo Zhu, Haodong Jing, Nanning Zheng

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

Comments A versatile brain decoding method learning from cross-subject fMRI data

CoMoFusion: Fast and High-quality Fusion of Infrared and Visible Image with Consistency Model

Zhiming Meng, Hui Li, Zeyang Zhang, Zhongwei Shen, Yunlong Yu, Xiaoning Song, Xiaojun Wu

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

MINT: Boosting Audio-Language Model via Multi-Target Pre-Training and Instruction Tuning

Hang Zhao, Yifei Xin, Zhesong Yu, Bilei Zhu, Lu Lu, Zejun Ma

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 eess.AS

Learning 1D Causal Visual Representation with De-focus Attention Networks

Chenxin Tao, Xizhou Zhu, Shiqian Su, Lewei Lu, Changyao Tian, Xuan Luo, Gao Huang, Hongsheng Li, Yu Qiao, Jie Zhou, Jifeng Dai

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

GrootVL: Tree Topology is All You Need in State Space Model

Yicheng Xiao, Lin Song, Shaoli Huang, Jiangshan Wang, Siyu Song, Yixiao Ge, Xiu Li, Ying Shan

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV

Comments The code is available at https://github.com/EasonXiao-888/GrootVL

One-Shot Federated Learning with Bayesian Pseudocoresets

Tim d'Hondt, Mykola Pechenizkiy, Robert Peharz

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.AI

Comments 10 pages

SparseDrive: End-to-End Autonomous Driving via Sparse Scene Representation

Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang, Haoran Wu, Sifa Zheng

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

WTTFNet: A Weather-Time-Trajectory Fusion Network for Pedestrian Trajectory Prediction in Urban Complex

Ho Chun Wu, Esther Hoi Shan Lau, Paul Yuen, Kevin Hung, John Kwok Tai Chui, Andrew Kwok Fai Lui

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV

Comments 12 pages, 7 figures

Evaluating Zero-Shot GPT-4V Performance on 3D Visual Question Answering Benchmarks

Simranjit Singh, Georgios Pavlakos, Dimitrios Stamoulis

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

Comments Accepted at 1st Workshop on Multimodalities for 3D Scenes CVPR 2024

Feasibility and benefits of joint learning from MRI databases with different brain diseases and modalities for segmentation

Wentian Xu, Matthew Moffat, Thalia Seale, Ziyun Liang, Felix Wagner, Daniel Whitehouse, David Menon, Virginia Newcombe, Natalie Voets, Abhirup Banerjee, Konstantinos Kamnitsas

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

Comments Accepted to MIDL 2024

Journal ref Proceedings of Machine Learning Research, MIDL 2024

Efficient Remote Sensing with Harmonized Transfer Learning and Modality Alignment

Tengjun Huang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV

Comments Accepted by the Twelfth International Conference on Learning Representations (ICLR) Workshop

Adapting LLaMA Decoder to Vision Transformer

Jiahao Wang, Wenqi Shao, Mengzhao Chen, Chengyue Wu, Yong Liu, Taiqiang Wu, Kaipeng Zhang, Songyang Zhang, Kai Chen, Ping Luo

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV

Comments 23 pages, 11 figures

Multimodality Invariant Learning for Multimedia-Based New Item Recommendation

Haoyue Bai, Le Wu, Min Hou, Miaomiao Cai, Zhuangzhuang He, Yuyang Zhou, Richang Hong, Meng Wang

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.AI

Towards Precision Healthcare: Robust Fusion of Time Series and Image Data

Ali Rasekh, Reza Heidari, Amir Hosein Haji Mohammad Rezaie, Parsa Sharifi Sedeh, Zahra Ahmadi, Prasenjit Mitra, Wolfgang Nejdl

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV

ContextualFusion: Context-Based Multi-Sensor Fusion for 3D Object Detection in Adverse Operating Conditions

Shounak Sural, Nishad Sahu, Ragunathan Rajkumar

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV

Comments 8 pages, 8 figures

Masked Image Modelling for retinal OCT understanding

Theodoros Pissas, Pablo Márquez-Neila, Sebastian Wolf, Martin Zinkernagel, Raphael Sznitman

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV

GLaD: Synergizing Molecular Graphs and Language Descriptors for Enhanced Power Conversion Efficiency Prediction in Organic Photovoltaic Devices

Thao Nguyen, Tiara Torres-Flores, Changhyun Hwang, Carl Edwards, Ying Diao, Heng Ji

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.AI

Comments In progress