多模态大模型

Transforming LLMs into Cross-modal and Cross-lingual Retrieval Systems

Frank Palma Gomez, Ramon Sanabria, Yun-hsuan Sung, Daniel Cer, Siddharth Dalmia, Gustavo Hernandez Abrego

专题命中 跨模态检索 ：cross-modal(title)；multi-modal(abstract)；分类 cs.CL、eess.AS

Research on Optimization of Natural Language Processing Model Based on Multimodal Deep Learning

Dan Sun, Yaxin Liang, Yining Yang, Yuhan Ma, Qishi Zhan, Erdi Gao

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Don't Buy it! Reassessing the Ad Understanding Abilities of Contrastive Multimodal Models

A. Bavaresco, A. Testoni, R. Fernández

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted to the main conference ACL 2024

RNG: Reducing Multi-level Noise and Multi-grained Semantic Gap for Joint Multimodal Aspect-Sentiment Analysis

Yaxin Liu, Yan Zhou, Ziming Li, Jinchuan Zhang, Yu Shang, Chenyang Zhang, Songlin Hu

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by ICME 2024

Enhancing Semantics in Multimodal Chain of Thought via Soft Negative Sampling

Guangmin Zheng, Jin Wang, Xiaobing Zhou, Xuejie Zhang

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by LREC-COLING 2024

2M-NER: Contrastive Learning for Multilingual and Multimodal NER with Language and Modal Fusion

Dongsheng Wang, Xiaoqin Feng, Zeming Liu, Chuan Wang

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 20 pages

Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval

Young Kyun Jang, Donghyun Kim, Zihang Meng, Dat Huynh, Ser-Nam Lim

专题命中 跨模态检索 ：multi-modal(title)；image-text(abstract)；分类 cs.CV、cs.AI

Comments 15 pages

PDF-MVQA: A Dataset for Multimodal Information Retrieval in PDF-based Visual Question Answering

Yihao Ding, Kaixuan Ren, Jiabin Huang, Siwen Luo, Soyeon Caren Han

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted by IJCAI 2024

Compressible and Searchable: AI-native Multi-Modal Retrieval System with Learned Image Compression

Jixiang Luo

专题命中 跨模态检索 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

SELECTOR: Heterogeneous graph network with convolutional masked autoencoder for multimodal robust prediction of cancer survival

Liangrui Pan, Yijun Peng, Yan Li, Xiang Wang, Wenjuan Liu, Liwen Xu, Qingchun Liang, Shaoliang Peng

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted on Computers in Biology and Medicine

Hierarchical Multimodal Pre-training for Visually Rich Webpage Understanding

Hongshen Xu, Lu Chen, Zihan Zhao, Da Ma, Ruisheng Cao, Zichen Zhu, Kai Yu

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search

Yifei Yuan, Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke, Wai Lam

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted to WWW24

Large Multi-Modal Models (LMMs) as Universal Foundation Models for AI-Native Wireless Systems

Shengzhe Xu, Christo Kurisummoottil Thomas, Omar Hashash, Nikhil Muralidhar, Walid Saad, Naren Ramakrishnan

专题命中 跨模态检索 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Cross-Modal Coordination Across a Diverse Set of Input Modalities

Jorge Sánchez, Rodrigo Laguna

专题命中 跨模态检索 ：cross-modal(title,abstract)；分类 cs.CV、cs.MM

CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognition

Cheng Peng, Ke Chen, Lidan Shou, Gang Chen

专题命中 跨模态检索 ：multi-modal(title,abstract)；分类 cs.AI、cs.MM

Debiasing Multimodal Sarcasm Detection with Contrastive Learning

Mengzhao Jia, Can Xie, Liqiang Jing

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CL、cs.MM

Cross-Modal Information-Guided Network using Contrastive Learning for Point Cloud Registration

Yifan Xie, Jihua Zhu, Shiqi Li, Pengcheng Shi

专题命中 跨模态检索 ：cross-modal(title,abstract)；分类 cs.CV、cs.AI

Comments 8 pages, accepted by RAL 2023

FOCAL: Contrastive Learning for Multimodal Time-Series Sensing Signals in Factorized Orthogonal Latent Space

Shengzhong Liu, Tomoyoshi Kimura, Dongxin Liu, Ruijie Wang, Jinyang Li, Suhas Diggavi, Mani Srivastava, Tarek Abdelzaher

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.AI、cs.MM

Comments Code available at: [github](https://github.com/tomoyoshki/focal)

Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering

Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, Bill Byrne

专题命中 跨模态检索 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Comments To appear at NeurIPS 2023. This is the camera-ready version. We fixed some numbers and added more experiments to address reviewers' comments

Generating Images with Multimodal Language Models

Jing Yu Koh, Daniel Fried, Ruslan Salakhutdinov

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments NeurIPS 2023. Project page: http://jykoh.com/gill

AspectMMKG: A Multi-modal Knowledge Graph with Aspect-aware Entities

Jingdan Zhang, Jiaan Wang, Xiaodan Wang, Zhixu Li, Yanghua Xiao

专题命中 跨模态检索 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by CIKM 2023

Multimodal Contrastive Learning and Tabular Attention for Automated Alzheimer's Disease Prediction

Weichen Huang

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

CoVLR: Coordinating Cross-Modal Consistency and Intra-Modal Structure for Vision-Language Retrieval

Yang Yang, Zhongtian Fu, Xiangyu Wu, Wenjie Li

专题命中 跨模态检索 ：cross-modal(title,abstract)；分类 cs.CV、cs.MM

Comments I apologize for my operational mistake, which has resulted in the absence of a revised version of the manuscript. Furthermore, I am concerned that the submission process of this paper may potentially lead to conflicts. Therefore, I kindly request the withdrawal of the manuscript

Unifying Two-Stream Encoders with Transformers for Cross-Modal Retrieval

Yi Bin, Haoxuan Li, Yahui Xu, Xing Xu, Yang Yang, Heng Tao Shen

专题命中 跨模态检索 ：cross-modal(title,abstract)；分类 cs.CV、cs.MM

Comments Accepted at ACM Multimedia 2023

Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generation

Qian Yang, Qian Chen, Wen Wang, Baotian Hu, Min Zhang

专题命中 跨模态检索 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted by ACM Multimedia 2023

Prototypical Contrastive Transfer Learning for Multimodal Language Understanding

Seitaro Otsuki, Shintaro Ishikawa, Komei Sugiura

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted for presentation at IROS23

Masked Vision and Language Pre-training with Unimodal and Multimodal Contrastive Losses for Medical Visual Question Answering

Pengfei Li, Gang Liu, Jinlong He, Zixu Zhao, Shenjun Zhong

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments accepted by MICCAI2023

Multimodal Prompt Retrieval for Generative Visual Question Answering

Timothy Ossowski, Junjie Hu

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answering

Alireza Salemi, Mahta Rafiee, Hamed Zamani

专题命中 跨模态检索 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

M3PT: A Multi-Modal Model for POI Tagging

Jingsong Yang, Guanzhou Han, Deqing Yang, Jingping Liu, Yanghua Xiao, Xiang Xu, Baohua Wu, Shenghua Ni

专题命中 跨模态检索 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted by KDD 2023