多模态大模型 - arXivDaily 专题

2404.01958 2024-04-03 cs.LG 82%

MESEN: Exploit Multimodal Data to Design Unimodal Human Activity Recognition with Few Labels

Lilin Xu, Chaojie Gu, Rui Tan, Shibo He, Jiming Chen

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)

Comments Accepted to the 21th ACM Conference on Embedded Networked Sensor Systems (SenSys 2023)

2307.16847 2024-02-20 cs.LG 82%

CroSSL: Cross-modal Self-Supervised Learning for Time-series through Latent Masking

Shohreh Deldari, Dimitris Spathis, Mohammad Malekzadeh, Fahim Kawsar, Flora Salim, Akhil Mathur

专题命中跨模态检索：cross-modal(title,abstract)；multimodal(abstract)

Comments Accepted in WSDM24. Short version presented in ML4MHD @ICML23

2305.01447 2024-01-31 cs.MM cs.CL cs.CV cs.DB cs.IR 82%

Multimodal Neural Databases

Giovanni Trappolini, Andrea Santilli, Emanuele Rodolà, Alon Halevy, Fabrizio Silvestri

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Journal ref SIGIR 2023: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

2312.12094 2023-12-21 q-bio.BM 82%

CrossBind: Collaborative Cross-Modal Identification of Protein Nucleic-Acid-Binding Residues

Linglin Jing, Sheng Xu, Yifan Wang, Yuzhe Zhou, Tao Shen, Zhigang Ji, Hui Fang, Zhen Li, Siqi Sun

专题命中跨模态检索：cross-modal(title,abstract)；multi-modal(abstract)

Comments Accepted to AAAI-24

2312.07435 2023-12-13 cs.CV cs.AI cs.CL cs.LG 82%

Cross-modal Contrastive Learning with Asymmetric Co-attention Network for Video Moment Retrieval

Love Panta, Prashant Shrestha, Brabeem Sapkota, Amrita Bhattarai, Suresh Manandhar, Anand Kumar Sah

专题命中跨模态检索：cross-modal(title)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

2310.13267 2023-10-23 cs.CL cs.CV cs.LG cs.SD eess.AS 82%

On the Language Encoder of Contrastive Cross-modal Models

Mengjie Zhao, Junya Ono, Zhi Zhong, Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Wei-Hsiang Liao, Takashi Shibuya, Hiromi Wakaki, Yuki Mitsufuji

专题命中跨模态检索：cross-modal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

2305.12767 2023-10-17 cs.CV cs.AI cs.CL 82%

D$^2$TV: Dual Knowledge Distillation and Target-oriented Vision Modeling for Many-to-Many Multimodal Summarization

Yunlong Liang, Fandong Meng, Jiaan Wang, Jinan Xu, Yufeng Chen, Jie Zhou

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments EMNLP 2023 Findings

2308.01919 2023-08-07 cs.MM cs.AI cs.CV 82%

Emotion recognition based on multi-modal electrophysiology multi-head attention Contrastive Learning

Yunfei Guo, Tao Zhang, Wu Huang

专题命中跨模态检索：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments arXiv admin note: text overlap with arXiv:2208.00877 by other authors

2307.09484 2023-07-24 q-bio.BM cs.CE cs.LG physics.chem-ph 82%

MolFM: A Multimodal Molecular Foundation Model

Yizhen Luo, Kai Yang, Massimo Hong, Xing Yi Liu, Zaiqing Nie

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)

Comments 31 pages, 15 figures, and 15 tables

2302.10632 2023-07-19 cs.IR 82%

Multi-Modal Self-Supervised Learning for Recommendation

Wei Wei, Chao Huang, Lianghao Xia, Chuxu Zhang

专题命中跨模态检索：multi-modal(title,abstract)；cross-modal(abstract)

Comments This paper has been published as a full paper at WWW 2023

2301.13823 2023-06-16 cs.CL cs.AI cs.CV cs.LG 82%

Grounding Language Models to Images for Multimodal Inputs and Outputs

Jing Yu Koh, Ruslan Salakhutdinov, Daniel Fried

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Published in ICML 2023. Project page: https://jykoh.com/fromage

2304.00020 2023-05-17 cs.LG cs.AI cs.CL cs.CV 82%

SemiMemes: A Semi-supervised Learning Approach for Multimodal Memes Analysis

Pham Thai Hoang Tung, Nguyen Tan Viet, Ngo Tien Anh, Phan Duy Hung

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ICCCI 2023

2304.11095 2023-04-24 cs.LG cs.AI cs.CL cs.CV cs.IR 82%

Is Cross-modal Information Retrieval Possible without Training?

Hyunjin Choi, Hyunjae Lee, Seongho Joe, Youngjune L. Gwon

专题命中跨模态检索：cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Journal ref Advances in Information Retrieval: 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, Proceedings, Part II

2211.12764 2023-03-23 cs.CV cs.AI cs.CL 82%

VoP: Text-Video Co-operative Prompt Tuning for Cross-Modal Retrieval

Siteng Huang, Biao Gong, Yulin Pan, Jianwen Jiang, Yiliang Lv, Yuyuan Li, Donglin Wang

专题命中跨模态检索：cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by CVPR 2023

2209.13764 2022-09-29 cs.IR 82%

Mr. Right: Multimodal Retrieval on Representation of ImaGe witH Text

Cheng-An Hsieh, Cheng-Ping Hsieh, Pu-Jen Cheng

专题命中跨模态检索：multimodal(title,abstract)；image-text(abstract)

Comments Dataset available at https://github.com/hsiehjackson/Mr.Right

2201.00693 2022-07-29 cs.IR cs.AI cs.CL cs.CV 82%

Multimodal Entity Tagging with Multimodal Knowledge Base

Hao Peng, Hang Li, Lei Hou, Juanzi Li, Chao Qiao

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 11 pages, 4 figures

2207.11482 2022-07-26 cs.CV cs.AI cs.LG cs.MM 82%

Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss

Riccardo Franceschini, Enrico Fini, Cigdem Beyan, Alessandro Conti, Federica Arrigoni, Elisa Ricci

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted to 26th International Conference on Pattern Recognition (ICPR) 2022

2204.10172 2022-04-22 eess.AS cs.AI cs.CL 82%

Gated Multimodal Fusion with Contrastive Learning for Turn-taking Prediction in Human-robot Dialogue

Jiudong Yang, Peiying Wang, Yi Zhu, Mingchao Feng, Meng Chen, Xiaodong He

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted by ICASSP 2022

1910.08549 2019-10-22 cs.CL cs.AI cs.CV cs.HC cs.LG 82%

Towards Learning Cross-Modal Perception-Trace Models

Achim Rettinger, Viktoria Bogdanova, Philipp Niemann

专题命中跨模态检索：cross-modal(title)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

1905.12203 2019-05-30 cs.LG stat.ML 82%

Weakly-paired Cross-Modal Hashing

Xuanwu Liu, Jun Wang, Guoxian Yu, Carlotta Domeniconi, Xiangliang Zhang

专题命中跨模态检索：cross-modal(title,abstract)；multi-modal(abstract)

1808.09648 2019-05-28 cs.CL cs.AI cs.CV 82%

Adapting Visual Question Answering Models for Enhancing Multimodal Community Q&A Platforms

Avikalp Srivastava, Hsin Wen Liu, Sumio Fujita

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Submitted for review at CIKM 2019

1803.00682 2019-05-24 cs.IR 82%

Learning Decorrelated Hashing Codes for Multimodal Retrieval

Dayong Tian

专题命中跨模态检索：multimodal(title,abstract)；image-text(abstract)

1905.04450 2019-05-14 cs.LG stat.ML 82%

Ranking-based Deep Cross-modal Hashing

Xuanwu Liu, Guoxian Yu, Carlotta Domeniconi, Jun Wang, Yazhou Ren, Maozu Guo

专题命中跨模态检索：cross-modal(title,abstract)；multi-modal(abstract)

1902.00644 2019-02-05 cs.IR 82%

Joint Cluster Unary Loss for Efficient Cross-Modal Hashing

Shifeng Zhang, Jianmin Li, Bo Zhang

专题命中跨模态检索：cross-modal(title,abstract)；multimodal(abstract)

1810.11303 2018-10-29 cs.IR 82%

Investigating non-classical correlations between decision fused multi-modal documents

Dimitris Gkoumas, Sagar Uprety, Dawei Song

专题命中跨模态检索：multi-modal(title,abstract)；multimodal(abstract)

1804.04318 2018-04-26 cs.CV cs.AI cs.MM 82%

Cross-Modal Retrieval with Implicit Concept Association

Yale Song, Mohammad Soleymani

专题命中跨模态检索：cross-modal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

1711.06794 2018-03-02 cs.CV cs.AI cs.CL 82%

Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering

Pan Lu, Hongsheng Li, Wei Zhang, Jianyong Wang, Xiaogang Wang

专题命中跨模态检索：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments To appear in AAAI 2018

1708.04776 2017-08-17 cs.CV cs.CL cs.MM 82%

Modality-specific Cross-modal Similarity Measurement with Recurrent Attention Network

Yuxin Peng, Jinwei Qi, Yuxin Yuan

专题命中跨模态检索：cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments 13 pages, submitted to IEEE Transactions on Image Processing

1702.01229 2017-07-11 cs.LG stat.ML 82%

Simple to Complex Cross-modal Learning to Rank

Minnan Luo, Xiaojun Chang, Zhihui Li, Liqiang Nie, Alexander G. Hauptmann, Qinghua Zheng

专题命中跨模态检索：cross-modal(title,abstract)；multi-modal(abstract)

Comments 14 pages; Accepted by Computer Vision and Image Understanding

1504.04818 2016-05-24 cs.IR 82%

Composite Correlation Quantization for Efficient Multimodal Retrieval

Mingsheng Long, Yue Cao, Jianmin Wang, Philip S. Yu

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)

1. 跨模态检索 3463 篇

MESEN: Exploit Multimodal Data to Design Unimodal Human Activity Recognition with Few Labels

CroSSL: Cross-modal Self-Supervised Learning for Time-series through Latent Masking

Multimodal Neural Databases

CrossBind: Collaborative Cross-Modal Identification of Protein Nucleic-Acid-Binding Residues

Cross-modal Contrastive Learning with Asymmetric Co-attention Network for Video Moment Retrieval

On the Language Encoder of Contrastive Cross-modal Models

D$^2$TV: Dual Knowledge Distillation and Target-oriented Vision Modeling for Many-to-Many Multimodal Summarization

Emotion recognition based on multi-modal electrophysiology multi-head attention Contrastive Learning

MolFM: A Multimodal Molecular Foundation Model

Multi-Modal Self-Supervised Learning for Recommendation

Grounding Language Models to Images for Multimodal Inputs and Outputs

SemiMemes: A Semi-supervised Learning Approach for Multimodal Memes Analysis

Is Cross-modal Information Retrieval Possible without Training?

VoP: Text-Video Co-operative Prompt Tuning for Cross-Modal Retrieval

Mr. Right: Multimodal Retrieval on Representation of ImaGe witH Text

Multimodal Entity Tagging with Multimodal Knowledge Base

Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss

Gated Multimodal Fusion with Contrastive Learning for Turn-taking Prediction in Human-robot Dialogue

Towards Learning Cross-Modal Perception-Trace Models

Weakly-paired Cross-Modal Hashing

Adapting Visual Question Answering Models for Enhancing Multimodal Community Q&A Platforms

Learning Decorrelated Hashing Codes for Multimodal Retrieval

Ranking-based Deep Cross-modal Hashing

Joint Cluster Unary Loss for Efficient Cross-Modal Hashing

Investigating non-classical correlations between decision fused multi-modal documents

Cross-Modal Retrieval with Implicit Concept Association

Co-attending Free-form Regions and Detections with Multi-modal Multiplicative Feature Embedding for Visual Question Answering

Modality-specific Cross-modal Similarity Measurement with Recurrent Attention Network

Simple to Complex Cross-modal Learning to Rank

Composite Correlation Quantization for Efficient Multimodal Retrieval