多模态大模型

MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter

Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, Tat-Seng Chua

专题命中 多模态训练与对齐 ：cross-modal(title,abstract)；分类 cs.CL、cs.MM

Comments EMNLP main conference. 9 pages

3DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding

Zeju Li, Chao Zhang, Xiaoyan Wang, Ruilong Ren, Yifan Xu, Ruifei Ma, Xiangde Liu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.MM

Comments 9 pages, 5 figures

Fine-grained Disentangled Representation Learning for Multimodal Emotion Recognition

Haoqin Sun, Shiwan Zhao, Xuechen Wang, Wenjia Zeng, Yong Chen, Yong Qin

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.MM、eess.AS

Comments Accepted by ICASSP 2024

Bi-directional Adapter for Multi-modal Tracking

Bing Cao, Junliang Guo, Pengfei Zhu, Qinghua Hu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted by AAAI 2024. Code is available at https://github.com/SparkTempest/BAT

Recognizing Unseen Objects via Multimodal Intensive Knowledge Graph Propagation

Likang Wu, Zhi Li, Hongke Zhao, Zhefeng Wang, Qi Liu, Baoxing Huai, Nicholas Jing Yuan, Enhong Chen

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments arXiv admin note: text overlap with arXiv:1805.11724 by other authors

Joint-Individual Fusion Structure with Fusion Attention Module for Multi-Modal Skin Cancer Classification

Peng Tang, Xintong Yan, Yang Nan, Xiaobin Hu, Xiaobin Hu, Bjoern H Menzee. Sebastian Krammer, Tobias Lasser

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments submitted to Pattern Recognition journal before 2022

Conditional Prompt Tuning for Multimodal Fusion

Ruixiang Jiang, Lingbo Liu, Changwen Chen

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments under review

Towards Transferable Multi-modal Perception Representation Learning for Autonomy: NeRF-Supervised Masked AutoEncoder

Xiaohao Xu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Continual Instruction Tuning for Large Multimodal Models

Jinghan He, Haiyun Guo, Ming Tang, Jinqiao Wang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Self-MI: Efficient Multimodal Fusion via Self-Supervised Multi-Task Learning with Auxiliary Mutual Information Maximization

Cam-Van Thi Nguyen, Ngoc-Hoa Thi Nguyen, Duc-Trong Le, Quang-Thuy Ha

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.MM

Comments Accepted at The 37th Pacific Asia Conference on Language, Information and Computation (PACLIC 37)

Multimodal Stress Detection Using Facial Landmarks and Biometric Signals

Majid Hosseini, Morteza Bodaghi, Ravi Teja Bhupatiraju, Anthony Maida, Raju Gottumukkala

专题命中 多模态训练与对齐 ：multimodal(title)；multi-modal(abstract)；分类 cs.CV、cs.AI

Comments 16 pages, 8 figures

A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations

Hui Ma, Jian Wang, Hongfei Lin, Bo Zhang, Yijia Zhang, Bo Xu

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.AI、cs.MM

Comments 13 pages, 10 figures. Accepted by IEEE Transactions on Multimedia (TMM)

Journal ref IEEE Transactions on Multimedia (Early Access), 27 April 2023

DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models

Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, Sibei Yang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments 24 pages, 13 figures, to be published in NeurIPS 2023

MISSRec: Pre-training and Transferring Multi-modal Interest-aware Sequence Representation for Recommendation

Jinpeng Wang, Ziyun Zeng, Yunxiao Wang, Yuting Wang, Xingyu Lu, Tianxiang Li, Jun Yuan, Rui Zhang, Hai-Tao Zheng, Shu-Tao Xia

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.AI、cs.MM

Comments Accepted to ACM MM 2023. Data and code are available

ConceptFusion: Open-set Multimodal 3D Mapping

Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, Antonio Torralba

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments RSS 2023. Project page: https://concept-fusion.github.io Explainer video: https://www.youtube.com/watch?v=rkXgws8fiDs Code: https://github.com/concept-fusion/concept-fusion

Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining

Emanuele Bugliarello, Aida Nematzadeh, Lisa Anne Hendricks

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments EMNLP 2023

Universal Multi-modal Entity Alignment via Iteratively Fusing Modality Similarity Paths

Bolin Zhu, Xiaoze Liu, Xin Mao, Zhuo Chen, Lingbing Guo, Tao Gui, Qi Zhang

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

DRIN: Dynamic Relation Interactive Network for Multimodal Entity Linking

Shangyu Xing, Fei Zhao, Zhen Wu, Chunhui Li, Jianbing Zhang, Xinyu Dai

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.MM

Comments Accepted by ACM MM 2023

SeMAnD: Self-Supervised Anomaly Detection in Multimodal Geospatial Datasets

Daria Reshetova, Swetava Ganguli, C. V. Krishnakumar Iyer, Vipul Pandey

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Extended version of the accepted research track paper at the 31st ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM SIGSPATIAL 2023), Hamburg, Germany. 11 pages, 8 figures, 6 tables

Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection

Yichao Cao, Qingfei Tang, Feng Yang, Xiu Su, Shan You, Xiaobo Lu, Chang Xu

专题命中 多模态训练与对齐 ：cross-modal(title)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments ICCV2023

SDVRF: Sparse-to-Dense Voxel Region Fusion for Multi-modal 3D Object Detection

Binglu Ren, Jianqin Yin

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning

Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Russell Howes, Vasu Sharma, Puxin Xu, Hovhannes Tamoyan, Oron Ashual, Uriel Singer, Shang-Wen Li, Susan Zhang, Richard James, Gargi Ghosh, Yaniv Taigman, Maryam Fazel-Zarandi, Asli Celikyilmaz, Luke Zettlemoyer, Armen Aghajanyan

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Exchanging-based Multimodal Fusion with Transformer

Renyu Zhu, Chengcheng Han, Yong Qian, Qiushi Sun, Xiang Li, Ming Gao, Xuezhi Cao, Yunsen Xian

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding

Hao Feng, Zijian Wang, Jingqun Tang, Jinghui Lu, Wengang Zhou, Houqiang Li, Can Huang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Reinforcement Learning Based Multi-modal Feature Fusion Network for Novel Class Discovery

Qiang Li, Qiuyang Ma, Weizhi Nie, Anan Liu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.AI、cs.MM

Few-shot Multimodal Sentiment Analysis based on Multimodal Probabilistic Fusion Prompts

Xiaocui Yang, Shi Feng, Daling Wang, Pengfei Hong, Soujanya Poria

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.MM

Comments 9 pages, 2 figures, 7 tables. It has been accepted ACM MM 2023

MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid

Zhuo Chen, Jiaoyan Chen, Wen Zhang, Lingbing Guo, Yin Fang, Yufeng Huang, Yichi Zhang, Yuxia Geng, Jeff Z. Pan, Wenting Song, Huajun Chen

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Comments ACM Multimedia 2023 Accpeted, Repo: https://github.com/zjukg/MEAformer

Journal ref ACM MM 2023

ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning

Liang Zhao, En Yu, Zheng Ge, Jinrong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chunrui Han, Xiangyu Zhang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments 15 pages, 8 figures

TriFormer: A Multi-modal Transformer Framework For Mild Cognitive Impairment Conversion Prediction

Linfeng Liu, Junyan Lyu, Siyu Liu, Xiaoying Tang, Shekhar S. Chandra, Fatima A. Nasrallah

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Revisiting Modality Imbalance In Multimodal Pedestrian Detection

Arindam Das, Sudip Das, Ganesh Sistu, Jonathan Horgan, Ujjwal Bhattacharya, Edward Jones, Martin Glavin, Ciarán Eising

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments 5 pages, 3 figure, 4 tables

Journal ref In proceedings of the IEEE 2023 International Conference on Image Processing