多模态大模型

SCOUT: A Situated and Multi-Modal Human-Robot Dialogue Corpus

Stephanie M. Lukin, Claire Bonial, Matthew Marge, Taylor Hudson, Cory J. Hayes, Kimberly A. Pollard, Anthony Baker, Ashley N. Foots, Ron Artstein, Felix Gervits, Mitchell Abrams, Cassidy Henry, Lucia Donatelli, Anton Leuski, Susan G. Hill, David Traum, Clare R. Voss

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CL

Comments 14 pages, 7 figures

Journal ref 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) https://aclanthology.org/2024.lrec-main.1259/

QARM: Quantitative Alignment Multi-Modal Recommendation at Kuaishou

Xinchen Luo, Jiangxia Cao, Tianyu Sun, Jinkai Yu, Rui Huang, Wei Yuan, Hezheng Lin, Yichen Zheng, Shiyao Wang, Qigen Hu, Changqing Qiu, Jiaqi Zhang, Xu Zhang, Zhiheng Yan, Jingming Zhang, Simin Zhang, Mingxing Wen, Zhaojie Liu, Kun Gai, Guorui Zhou

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.AI

Comments Work in progress

Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Model

Ting Liu, Liangtao Shi, Richang Hong, Yue Hu, Quanjun Yin, Linfeng Zhang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV

Comments 8 pages, 4figures

MDA: An Interpretable and Scalable Multi-Modal Fusion under Missing Modalities and Intrinsic Noise Conditions

Lin Fan, Yafei Ou, Cenyang Zheng, Pengyu Dai, Tamotsu Kamishima, Masayuki Ikebe, Kenji Suzuki, Xun Gong

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV

A Multimodal Fusion Network For Student Emotion Recognition Based on Transformer and Tensor Product

Ao Xiang, Zongqing Qi, Han Wang, Qin Yang, Danqing Ma

专题命中 多模态训练与对齐 ：multimodal(title)；multi-modal(abstract)；分类 cs.CV

Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Models

Wei Wang, Zhaowei Li, Qi Xu, Linfeng Li, YiQing Cai, Botian Jiang, Hang Song, Xingcan Hu, Pengyu Wang, Li Xiao

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV

CoMiX: Cross-Modal Fusion with Deformable Convolutions for HSI-X Semantic Segmentation

Xuming Zhang, Xingfa Gu, Qingjiu Tian, Lorenzo Bruzzone

专题命中 多模态训练与对齐 ：cross-modal(title)；multimodal(abstract)；分类 cs.CV

RNA-GPT: Multimodal Generative System for RNA Sequence Understanding

Yijia Xiao, Edward Sun, Yiqiao Jin, Wei Wang

专题命中 多模态训练与对齐 ：multimodal(title)；multi-modal(abstract)；分类 cs.AI

Comments Machine Learning for Structural Biology Workshop, NeurIPS 2024

AstroM$^3$: A self-supervised multimodal model for astronomy

Mariia Rizhko, Joshua S. Bloom

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.AI

Multimodal Instruction Tuning with Hybrid State Space Models

Jianing Zhou, Han Li, Shuai Zhang, Ning Xie, Ruijie Wang, Xiaohan Nie, Sheng Liu, Lingyun Wang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV

Material Property Prediction with Element Attribute Knowledge Graphs and Multimodal Representation Learning

Chao Huang, Chunyan Chen, Ling Shi, Chen Chen

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.AI

Adapting Segment Anything Model to Multi-modal Salient Object Detection with Semantic Feature Fusion Guidance

Kunpeng Wang, Danying Lin, Chenglong Li, Zhengzheng Tu, Bin Luo

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV

Comments 10 pages, 9 figures

Multi-modal Iterative and Deep Fusion Frameworks for Enhanced Passive DOA Sensing via a Green Massive H2AD MIMO Receiver

Jiatong Bai, Minghao Chen, Wankai Tang, Yifan Li, Cunhua Pan, Yongpeng Wu, Feng Shu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.AI

Predicting Stroke through Retinal Graphs and Multimodal Self-supervised Learning

Yuqing Huang, Bastian Wittmann, Olga Demler, Bjoern Menze, Neda Davoudi

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV

Comments Accepted as oral paper at ML-CDS workshop, MICCAI 2024

Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applications

Daan Schouten, Giulia Nicoletti, Bas Dille, Catherine Chia, Pierpaolo Vendittelli, Megan Schuurmans, Geert Litjens, Nadieh Khalili

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.AI

Comments 28 pages

Graph-Based Multi-Modal Sensor Fusion for Autonomous Driving

Depanshu Sani, Saket Anand

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV

Comments An extended abstract accepted at Young Researchers' Symposium, ICVGIP '24. This extended abstract contains the following: 1. Short summary of our work, SAGA-KF, accepted at ICPR'24. 2. A proposal that was awarded the Qualcomm Innovation Fellowship'24

Multimodal Super-Resolution: Discovering hidden physics and its application to fusion plasmas

Azarakhsh Jalalvand, SangKyeun Kim, Jaemin Seo, Qiming Hu, Max Curie, Peter Steiner, Andrew Oakleigh Nelson, Yong-Su Na, Egemen Kolemen

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.AI

Multi-modal biometric authentication: Leveraging shared layer architectures for enhanced security

Vatchala S, Yogesh C, Yeshwanth Govindarajan, Krithik Raja M, Vishal Pramav Amirtha Ganesan, Aashish Vinod A, Dharun Ramesh

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV

Fuse It or Lose It: Deep Fusion for Multimodal Simulation-Based Inference

Marvin Schmitt, Leona Odole, Stefan T. Radev, Paul-Christian Bürkner

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.AI

Multimodal Graph Neural Network for Recommendation with Dynamic De-redundancy and Modality-Guided Feature De-noisy

Feng Mo, Lin Xiao, Qiya Song, Xieping Gao, Eryao Liang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.MM

A Visual Question Answering Method for SAR Ship: Breaking the Requirement for Multimodal Dataset Construction and Model Fine-Tuning

Fei Wang, Chengcheng Chen, Hongyu Chen, Yugang Chang, Weiming Zeng

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV

MambaReg: Mamba-Based Disentangled Convolutional Sparse Coding for Unsupervised Deformable Multi-Modal Image Registration

Kaiang Wen, Bin Xie, Bin Duan, Yan Yan

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV

H2G2-Net: A Hierarchical Heterogeneous Graph Generative Network Framework for Discovery of Multi-Modal Physiological Responses

Haidong Gu, Nathan Gaw, Yinan Wang, Chancellor Johnstone, Christine Beauchene, Sophia Yuditskaya, Hrishikesh Rao, Chun-An Chou

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.AI

Comments Paper accepted in Human-Centric Representation Learning workshop at AAAI 2024 (https://hcrl-workshop.github.io/2024/)

PathoGen-X: A Cross-Modal Genomic Feature Trans-Align Network for Enhanced Survival Prediction from Histopathology Images

Akhila Krishna, Nikhil Cherian Kurian, Abhijeet Patil, Amruta Parulekar, Amit Sethi

专题命中 多模态训练与对齐 ：cross-modal(title,abstract)；分类 cs.CV

Multimodal Fusion on Low-quality Data: A Comprehensive Survey

Qingyang Zhang, Yake Wei, Zongbo Han, Huazhu Fu, Xi Peng, Cheng Deng, Qinghua Hu, Cai Xu, Jie Wen, Di Hu, Changqing Zhang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.AI

Comments Feel free to comment on our manuscript: qingyangzhang@tju$.$edu$.$cn

JEMA: A Joint Embedding Framework for Scalable Co-Learning with Multimodal Alignment

Joao Sousa, Roya Darabi, Armando Sousa, Frank Brueckner, Luís Paulo Reis, Ana Reis

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV

Comments 26 pages, 14 figures

Enhanced Survival Prediction in Head and Neck Cancer Using Convolutional Block Attention and Multimodal Data Fusion

Aiman Farooq, Utkarsh Sharma, Deepak Mishra

专题命中 多模态训练与对齐 ：multimodal(title)；multi-modal(abstract)；分类 cs.CV

Comments Accepted to [ACCV 2024 Workshop]

DeepLight: Reconstructing High-Resolution Observations of Nighttime Light With Multi-Modal Remote Sensing Data

Lixian Zhang, Runmin Dong, Shuai Yuan, Jinxiao Zhang, Mengxuan Chen, Juepeng Zheng, Haohuan Fu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV

Comments This paper has been accepted in IJCAI 2024

BEVPose: Unveiling Scene Semantics through Pose-Guided Multi-Modal BEV Alignment

Mehdi Hosseinzadeh, Ian Reid

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV

Comments Accepted for presentation at the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024. Project page: https://m80hz.github.io/bevpose/

Evaluating Cost-Accuracy Trade-offs in Multimodal Search Relevance Judgements

Silvia Terragni, Hoang Cuong, Joachim Daiber, Pallavi Gudipati, Pablo N. Mendes

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL

Journal ref CIKM MMSR 2024