多模态大模型

A Comprehensive Overview of Large Language Models

Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, Ajmal Mian

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CL

Task Aware Modulation using Representation Learning: An Approach for Few Shot Learning in Environmental Systems

Arvind Renganathan, Rahul Ghosh, Ankush Khandelwal, Vipin Kumar

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.AI

Breaking Modality Gap in RGBT Tracking: Coupled Knowledge Distillation

Andong Lu, Jiacong Zhao, Chenglong Li, Yun Xiao, Bin Luo

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

Comments Accepted by ACM MM2024

KNN Transformer with Pyramid Prompts for Few-Shot Learning

Wenhao Li, Qiangchang Wang, Peng Zhao, Yilong Yin

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV

Comments 10 pages, 5 figures, accepted by ACM Multimedia 2024

Toward Guidance-Free AR Visual Generation via Condition Contrastive Alignment

Huayu Chen, Hang Su, Peize Sun, Jun Zhu

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

DAT: Dialogue-Aware Transformer with Modality-Group Fusion for Human Engagement Estimation

Jia Li, Yangchen Yu, Yin Chen, Yu Zhang, Peng Jia, Yunbo Xu, Ziqiang Li, Meng Wang, Richang Hong

专题命中 多模态训练与对齐 ：audio-visual(abstract)；分类 cs.CV

Comments 1st Place on the NoXi Base dataset in the Multi-Domain Engagement Estimation Challenge held by MultiMediate 24, accepted by ACM Multimedia 2024. The source code is available at \url{https://github.com/MSA-LMC/DAT}

Mitigating Dialogue Hallucination for Large Vision Language Models via Adversarial Instruction Tuning

Dongmin Park, Zhaofang Qian, Guangxing Han, Ser-Nam Lim

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

Video-LLaVA: Learning United Visual Representation by Alignment Before Projection

Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, Li Yuan

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

Fully Aligned Network for Referring Image Segmentation

Yong Liu, Ruihao Xu, Yansong Tang

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV

EEG-EMG FAConformer: Frequency Aware Conv-Transformer for the fusion of EEG and EMG

ZhengXiao He, Minghong Cai, Letian Li, Siyuan Tian, Ren-Jie Dai

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.AI

Emu3: Next-Token Prediction is All You Need

Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tiejun Huang, Zhongyuan Wang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV

Comments Project Page: https://emu.baai.ac.cn

SimTxtSeg: Weakly-Supervised Medical Image Segmentation with Simple Text Cues

Yuxin Xie, Tao Zhou, Yi Zhou, Geng Chen

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV

Comments accepted by MICCAI 2024

Automatic Registration of SHG and H&E Images with Feature-based Initial Alignment and Intensity-based Instance Optimization: Contribution to the COMULIS Challenge

Marek Wodzinski, Henning Müller

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

Probabilistically Aligned View-unaligned Clustering with Adaptive Template Selection

Wenhua Dong, Xiao-Jun Wu, Zhenhua Feng, Sara Atito, Muhammad Awais, Josef Kittler

专题命中 多模态训练与对齐 ：image-text(abstract)；分类 cs.CV

Comments 12 pages, 6 figures

GraspMamba: A Mamba-based Language-driven Grasp Detection Framework with Hierarchical Feature Learning

Huy Hoang Nguyen, An Vuong, Anh Nguyen, Ian Reid, Minh Nhat Vu

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV

Comments 8 pages. Project page: https://airvlab.github.io/grasp-anything/

Accelerated Multi-Contrast MRI Reconstruction via Frequency and Spatial Mutual Learning

Qi Chen, Xiaohan Xing, Zhen Chen, Zhiwei Xiong

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV

Comments Accepted as a poster by Medical Image Computing and Computer Assisted Intervention (MICCAI) 2024

OneEncoder: A Lightweight Framework for Progressive Alignment of Modalities

Bilal Faye, Hanane Azzag, Mustapha Lebbah

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV

Mamba-YOLO-World: Marrying YOLO-World with Mamba for Open-Vocabulary Detection

Haoxuan Wang, Qingdong He, Jinlong Peng, Hao Yang, Mingmin Chi, Yabiao Wang

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

Identifying Influential nodes in Brain Networks via Self-Supervised Graph-Transformer

Yanqing Kang, Di Zhu, Haiyang Zhang, Enze Shi, Sigang Yu, Jinru Wu, Xuhui Wang, Xuan Liu, Geng Chen, Xi Jiang, Tuo Zhang, Shu Zhang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.AI

PoseEmbroider: Towards a 3D, Visual, Semantic-aware Human Pose Representation

Ginger Delmas, Philippe Weinzaepfel, Francesc Moreno-Noguer, Grégory Rogez

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

Comments Published in ECCV 2024

RCBEVDet++: Toward High-accuracy Radar-Camera Fusion 3D Perception Network

Zhiwei Lin, Zhe Liu, Yongtao Wang, Le Zhang, Ce Zhu

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

Comments The extended work of RCBEVDet (CVPR2024)

SSFam: Scribble Supervised Salient Object Detection Family

Zhengyi Liu, Sheng Deng, Xinrui Wang, Linbo Wang, Xianyong Fang, Bin Tang

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

Comments Accepted by TMM 2024

CLFT: Camera-LiDAR Fusion Transformer for Semantic Segmentation in Autonomous Driving

Junyi Gu, Mauro Bellone, Tomáš Pivoňka, Raivo Sell

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV

Comments Accepted to IEEE Transactions on Intelligent Vehicles

Why mamba is effective? Exploit Linear Transformer-Mamba Network for Multi-Modality Image Fusion

Chenguang Zhu, Shan Gao, Huafeng Chen, Guangqian Guo, Chaowei Wang, Yaoxing Wang, Chen Shu Lei, Quanjiang Fan

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV

SG-MIM: Structured Knowledge Guided Efficient Pre-training for Dense Prediction

Sumin Son, Hyesong Choi, Dongbo Min

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning

Weijian Huang, Cheng Li, Hong-Yu Zhou, Hao Yang, Jiarun Liu, Yong Liang, Hairong Zheng, Shaoting Zhang, Shanshan Wang

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

Journal ref Nature Communications 15, 7620 (2024)

BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation

Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus, Song Han

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

Comments ICRA 2023. The first two authors contributed equally to this work. Project page: https://bevfusion.mit.edu

Cross Fusion RGB-T Tracking with Bi-directional Adapter

Zhirong Zeng, Xiaotao Liu, Meng Sun, Hongyu Wang, Jing Liu

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

BOX3D: Lightweight Camera-LiDAR Fusion for 3D Object Detection and Localization

Mario A. V. Saucedo, Nikolaos Stathoulopoulos, Vidya Sumathy, Christoforos Kanellakis, George Nikolakopoulos

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV

Comments Presented in MED 2024

Text-guided Foundation Model Adaptation for Long-Tailed Medical Image Classification

Sirui Li, Li Lin, Yijin Huang, Pujin Cheng, Xiaoying Tang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV

Comments Accepted by IEEE ISBI 2024