多模态大模型 - arXivDaily 专题

2311.12793 2023-11-29 cs.CV 83%

ShareGPT4V: Improving Large Multi-Modal Models with Better Captions

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, Dahua Lin

专题命中多模态训练与对齐：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Project: https://ShareGPT4V.github.io

2203.04838 2023-11-27 cs.CV cs.RO eess.IV 83%

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Jiaming Zhang, Huayao Liu, Kailun Yang, Xinxin Hu, Ruiping Liu, Rainer Stiefelhagen

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted to IEEE Transactions on Intelligent Transportation Systems (T-ITS). The source code of CMX is publicly available at https://github.com/huaaaliu/RGBX_Semantic_Segmentation

2311.14315 2023-11-27 cs.AI 83%

Robust Domain Misinformation Detection via Multi-modal Feature Alignment

Hui Liu, Wenya Wang, Hao Sun, Anderson Rocha, Haoliang Li

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.AI

Comments Accepted by TIFS 2023

2311.03106 2023-11-07 cs.CV 83%

Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding

Shengkai Sun, Daizong Liu, Jianfeng Dong, Xiaoye Qu, Junyu Gao, Xun Yang, Xun Wang, Meng Wang

专题命中多模态训练与对齐：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments Accepted by ACM MM 2023. The code is available at https://github.com/HuiGuanLab/UmURL

2310.16754 2023-10-31 cs.CV 83%

CAD -- Contextual Multi-modal Alignment for Dynamic AVQA

Asmar Nadeem, Adrian Hilton, Robert Dawes, Graham Thomas, Armin Mustafa

专题命中多模态训练与对齐：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV

Comments Accepted to IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2024

2310.14143 2023-10-24 cs.CV cs.LG 83%

MMTF-DES: A Fusion of Multimodal Transformer Models for Desire, Emotion, and Sentiment Analysis of Social Media Data

Abdul Aziz, Nihad Karim Chowdhury, Muhammad Ashad Kabir, Abu Nowshed Chy, Md. Jawad Siddique

专题命中多模态训练与对齐：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments 28 pages, 4 figures

2310.12100 2023-10-19 cs.CL cs.AI cs.CV cs.LG cs.MM 83%

Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling

Yaqing Wang, Jialin Wu, Tanmaya Dabral, Jiageng Zhang, Geoff Brown, Chun-Ta Lu, Frederick Liu, Yi Liang, Bo Pang, Michael Bendersky, Radu Soricut

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2309.17104 2023-10-04 cs.CV 83%

Prototype-guided Cross-modal Completion and Alignment for Incomplete Text-based Person Re-identification

Tiantian Gong, Guodong Du, Junsheng Wang, Yongkang Ding, Liyan Zhang

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Sorry, some collaborators do not agree to publish it on Arxiv, so please withdraw this paper

Journal ref ACM International Conference on Multimedia 2023

2304.00379 2023-10-02 eess.IV cs.CV 83%

Improved Multimodal Fusion for Small Datasets with Auxiliary Supervision

Gregory Holste, Douwe van der Wal, Hans Pinckaers, Rikiya Yamashita, Akinori Mitani, Andre Esteva

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments IEEE ISBI 2023 (see http://2023.biomedicalimaging.org/en/)

2309.13266 2023-09-26 cs.RO cs.AI 83%

Robust Navigation with Cross-Modal Fusion and Knowledge Transfer

Wenzhe Cai, Guangran Cheng, Lingyue Kong, Lu Dong, Changyin Sun

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.AI

Comments Accepted by ICRA 2023

2309.12855 2023-09-25 eess.IV cs.CV cs.LG 83%

Cross-Modal Translation and Alignment for Survival Analysis

Fengtao Zhou, Hao Chen

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted by ICCV2023

2205.02357 2023-09-19 cs.CL cs.AI cs.CV cs.LG cs.MM 83%

Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion

Xiang Chen, Ningyu Zhang, Lei Li, Shumin Deng, Chuanqi Tan, Changliang Xu, Fei Huang, Luo Si, Huajun Chen

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by SIGIR 2022. Fix a severe bug

2308.09369 2023-08-21 cs.CV 83%

Single Frame Semantic Segmentation Using Multi-Modal Spherical Images

Suresh Guttikonda, Jason Rambach

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted at WACV 2024

2308.07732 2023-08-16 cs.CV 83%

UniTR: A Unified and Efficient Multi-Modal Transformer for Bird's-Eye-View Representation

Haiyang Wang, Hao Tang, Shaoshuai Shi, Aoxue Li, Zhenguo Li, Bernt Schiele, Liwei Wang

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by ICCV2023

2305.10469 2023-08-08 cs.CV 83%

Object Segmentation by Mining Cross-Modal Semantics

Zongwei Wu, Jingjing Wang, Zhuyun Zhou, Zhaochong An, Qiuping Jiang, Cédric Demonceaux, Guolei Sun, Radu Timofte

专题命中多模态训练与对齐：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments ACM MM 2023

2208.00919 2023-08-07 cs.CV 83%

Benchmarking Visual-Inertial Deep Multimodal Fusion for Relative Pose Regression and Odometry-aided Absolute Pose Regression

Felix Ott, Nisha Lakshmana Raichur, David Rügamer, Tobias Feigl, Heiko Neumann, Bernd Bischl, Christopher Mutschler

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Under review

2302.10909 2023-07-31 cs.LG cs.AI 83%

Multi-modal Machine Learning in Engineering Design: A Review and Future Directions

Binyang Song, Rui Zhou, Faez Ahmed

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.AI

2307.13205 2023-07-26 cs.MM 83%

Text-oriented Modality Reinforcement Network for Multimodal Sentiment Analysis from Unaligned Multimodal Sequences

Yuxuan Lei, Dingkang Yang, Mingcheng Li, Shunli Wang, Jiawei Chen, Lihua Zhang

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.MM

Comments Accepted by CICAI 2023 (Finalist of Best Student Paper Award)

2306.15946 2023-06-29 cs.CV 83%

Knowledge-Enhanced Hierarchical Information Correlation Learning for Multi-Modal Rumor Detection

Jiawei Liu, Jingyi Xie, Fanrui Zhang, Qiang Zhang, Zheng-jun Zha

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2203.01311 2023-06-29 cs.LG cs.AI cs.CL cs.CV cs.MM 83%

High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning

Paul Pu Liang, Yiwei Lyu, Xiang Fan, Jeffrey Tsaw, Yudong Liu, Shentong Mo, Dani Yogatama, Louis-Philippe Morency, Ruslan Salakhutdinov

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments TMLR 2023, Code available at https://github.com/pliang279/HighMMT

2306.03810 2023-06-07 cs.CV cs.RO 83%

X-Align++: cross-modal cross-view alignment for Bird's-eye-view segmentation

Shubhankar Borse, Senthil Yogamani, Marvin Klingner, Varun Ravi, Hong Cai, Abdulaziz Almuzairee, Fatih Porikli

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted for publication at Springer Machine Vision and Applications Journal. The Version of Record of this article is published in Machine Vision and Applications Journal, and is available online at https://doi.org/10.1007/s00138-023-01400-7. arXiv admin note: substantial text overlap with arXiv:2210.06778

2306.02050 2023-06-07 cs.LG cs.CV 83%

Provable Dynamic Fusion for Low-Quality Multimodal Data

Qingyang Zhang, Haitao Wu, Changqing Zhang, Qinghua Hu, Huazhu Fu, Joey Tianyi Zhou, Xi Peng

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by ICML 2023

2304.06306 2023-05-16 cs.CV 83%

Efficient Multimodal Fusion via Interactive Prompting

Yaowei Li, Ruijie Quan, Linchao Zhu, Yi Yang

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Camera-ready version for CVPR2023

2305.07713 2023-05-16 cs.CV 83%

Multi-Modal 3D Object Detection by Box Matching

Zhe Liu, Xiaoqing Ye, Zhikang Zou, Xinwei He, Xiao Tan, Errui Ding, Jingdong Wang, Xiang Bai

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2305.06179 2023-05-12 cs.CV cs.RO 83%

A Multi-modal Approach to Single-modal Visual Place Classification

Tomoya Iwasaki, Kanji Tanaka, Kenta Tsukahara

专题命中多模态训练与对齐：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments 7 pages, 6 figures, 1 table

2304.13130 2023-04-27 cs.CV 83%

Hypernymization of named entity-rich captions for grounding-based multi-modal pretraining

Giacomo Nebbia, Adriana Kovashka

专题命中多模态训练与对齐：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV

2212.05699 2023-04-13 cs.IR cs.MM 83%

Multimodal Matching-aware Co-attention Networks with Mutual Knowledge Distillation for Fake News Detection

Linmei Hu, Ziwang Zhao, Weijian Qi, Xuemeng Song, Liqiang Nie

专题命中多模态训练与对齐：multimodal(title,abstract)；image-text(abstract)；分类 cs.MM

2304.00719 2023-04-04 cs.CV 83%

Multi-Modal Representation Learning with Text-Driven Soft Masks

Jaeyoo Park, Bohyung Han

专题命中多模态训练与对齐：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments CVPR 2023

2303.08600 2023-03-16 cs.CV 83%

MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving

Jiale Li, Hang Dai, Hao Han, Yong Ding

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted to CVPR 2023 (preprint)

2303.03595 2023-03-15 cs.CV 83%

LoGoNet: Towards Accurate 3D Object Detection with Local-to-Global Cross-Modal Fusion

Xin Li, Tao Ma, Yuenan Hou, Botian Shi, Yuchen Yang, Youquan Liu, Xingjiao Wu, Qin Chen, Yikang Li, Yu Qiao, Liang He

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted by CVPR2023

1. 多模态训练与对齐 6903 篇

ShareGPT4V: Improving Large Multi-Modal Models with Better Captions

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Robust Domain Misinformation Detection via Multi-modal Feature Alignment

Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding

CAD -- Contextual Multi-modal Alignment for Dynamic AVQA

MMTF-DES: A Fusion of Multimodal Transformer Models for Desire, Emotion, and Sentiment Analysis of Social Media Data

Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling

Prototype-guided Cross-modal Completion and Alignment for Incomplete Text-based Person Re-identification

Improved Multimodal Fusion for Small Datasets with Auxiliary Supervision

Robust Navigation with Cross-Modal Fusion and Knowledge Transfer

Cross-Modal Translation and Alignment for Survival Analysis

Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion

Single Frame Semantic Segmentation Using Multi-Modal Spherical Images

UniTR: A Unified and Efficient Multi-Modal Transformer for Bird's-Eye-View Representation

Object Segmentation by Mining Cross-Modal Semantics

Benchmarking Visual-Inertial Deep Multimodal Fusion for Relative Pose Regression and Odometry-aided Absolute Pose Regression

Multi-modal Machine Learning in Engineering Design: A Review and Future Directions

Text-oriented Modality Reinforcement Network for Multimodal Sentiment Analysis from Unaligned Multimodal Sequences

Knowledge-Enhanced Hierarchical Information Correlation Learning for Multi-Modal Rumor Detection

High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning

X-Align++: cross-modal cross-view alignment for Bird's-eye-view segmentation

Provable Dynamic Fusion for Low-Quality Multimodal Data

Efficient Multimodal Fusion via Interactive Prompting

Multi-Modal 3D Object Detection by Box Matching

A Multi-modal Approach to Single-modal Visual Place Classification

Hypernymization of named entity-rich captions for grounding-based multi-modal pretraining

Multimodal Matching-aware Co-attention Networks with Mutual Knowledge Distillation for Fake News Detection

Multi-Modal Representation Learning with Text-Driven Soft Masks

MSeg3D: Multi-modal 3D Semantic Segmentation for Autonomous Driving

LoGoNet: Towards Accurate 3D Object Detection with Local-to-Global Cross-Modal Fusion