多模态大模型 - arXivDaily 专题

1912.12014 2019-12-30 cs.CL cs.CV 81%

Visual Agreement Regularized Training for Multi-Modal Machine Translation

Pengcheng Yang, Boxing Chen, Pei Zhang, Xu Sun

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted by AAAI 2020

1804.05788 2019-11-11 cs.AI cs.CV cs.HC 81%

Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning

Samarth Tripathi, Sarthak Tripathi, Homayoon Beigi

专题命中多模态评测：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.AI

1910.13212 2019-10-30 cs.LG cs.CL cs.SD eess.AS stat.ML 81%

Privacy Enhanced Multimodal Neural Representations for Emotion Recognition

Mimansa Jaiswal, Emily Mower Provost

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments 8 pages

1704.08424 2019-09-10 stat.ML cs.AI cs.CL cs.LG 81%

Multimodal Word Distributions

Ben Athiwaratkun, Andrew Gordon Wilson

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments This paper also appears at ACL 2017

1908.08992 2019-08-27 cs.CV cs.AI cs.LG eess.SP stat.ML 81%

MEx: Multi-modal Exercises Dataset for Human Activity Recognition

Anjana Wijekoon, Nirmalie Wiratunga, Kay Cooper

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.CV、cs.AI

1804.01486 2019-08-21 cs.CL cs.AI stat.ML 81%

Clinical Concept Embeddings Learned from Massive Sources of Multimodal Medical Data

Andrew L. Beam, Benjamin Kompa, Allen Schmaltz, Inbar Fried, Griffin Weber, Nathan P. Palmer, Xu Shi, Tianxi Cai, Isaac S. Kohane

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CL、cs.AI

1906.10198 2019-06-26 cs.CL cs.SD eess.AS 81%

Multimodal and Multi-view Models for Emotion Recognition

Gustavo Aguilar, Viktor Rozgić, Weiran Wang, Chao Wang

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments ACL 2019

1906.01815 2019-06-06 cs.CL cs.CV 81%

Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)

Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, Soujanya Poria

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted at ACL 2019

1811.11365 2019-05-28 cs.CV cs.CL 81%

Unsupervised Multi-modal Neural Machine Translation

Yuanhang Su, Kai Fan, Nguyen Bach, C. -C. Jay Kuo, Fei Huang

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted to CVPR 2019

1904.04745 2019-04-10 cs.CV cs.CL 81%

Cross-Modal Self-Attention Network for Referring Image Segmentation

Linwei Ye, Mrigank Rochan, Zhi Liu, Yang Wang

专题命中多模态评测：cross-modal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted to CVPR2019

1904.01356 2019-04-03 cs.CV cs.CL 81%

Aiding Intra-Text Representations with Visual Context for Multimodal Named Entity Recognition

Omer Arshad, Ignazio Gallo, Shah Nawaz, Alessandro Calefati

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL

1803.07427 2019-02-13 cs.CL cs.CV cs.IR 81%

Multimodal Sentiment Analysis: Addressing Key Issues and Setting up the Baselines

Soujanya Poria, Navonil Majumder, Devamanyu Hazarika, Erik Cambria, Alexander Gelbukh, Amir Hussain

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments IEEE Intelligence Systems. arXiv admin note: substantial text overlap with arXiv:1707.09538

1811.12276 2018-12-05 cs.CL cs.AI cs.LG 81%

Improving Hospital Mortality Prediction with Medical Named Entities and Multimodal Learning

Mengqi Jin, Mohammad Taha Bahadori, Aaron Colak, Parminder Bhatia, Busra Celikkaya, Ram Bhakta, Selvan Senthivel, Mohammed Khalilia, Daniel Navarro, Borui Zhang, Tiberiu Doman, Arun Ravi, Matthieu Liger, Taha Kass-hout

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Machine Learning for Health (ML4H) Workshop at NeurIPS 2018 arXiv:1811.07216

1809.03695 2018-09-12 cs.CL cs.AI 81%

Evaluating Multimodal Representations on Sentence Similarity: vSTS, Visual Semantic Textual Similarity Dataset

Oier Lopez de Lacalle, Aitor Soroa, Eneko Agirre

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CL、cs.AI

Journal ref ICCV17: second workshop on Closing the Loop Between Vision and Language. Venice, Italy. 2017

1809.00812 2018-09-05 cs.CL cs.CV 81%

RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes

Semih Yagcioglu, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments EMNLP 2018

1705.10369 2018-04-18 cs.LG cs.CL cs.CV cs.IT cs.MA math.IT 81%

Emergent Communication in a Multi-Modal, Multi-Step Referential Game

Katrina Evtimova, Andrew Drozdov, Douwe Kiela, Kyunghyun Cho

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Comments Published as a conference paper at ICLR 2018. 12 pages

1610.00163 2017-09-26 stat.ML cs.AI cs.CV 81%

X-CNN: Cross-modal Convolutional Neural Networks for Sparse Datasets

Petar Veličković, Duo Wang, Nicholas D. Lane, Pietro Liò

专题命中多模态评测：cross-modal(title,abstract)；分类 cs.CV、cs.AI

Comments To appear in the 7th IEEE Symposium Series on Computational Intelligence (IEEE SSCI 2016), 8 pages, 6 figures. Minor revisions, in response to reviewers' comments

1705.04350 2017-07-10 cs.CL cs.CV 81%

Imagination improves Multimodal Translation

Desmond Elliott, Ákos Kádár

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Clarified main contributions, minor correction to Equation 8, additional comparisons in Table 2, added more related work

1705.02694 2017-05-09 cs.HC cs.AI cs.CV cs.LG 81%

Multimodal Affect Analysis for Product Feedback Assessment

Amol S Patwardhan, Gerald M Knapp

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments 10 pages, ISERC 2013, IIE Annual Conference. Proceedings. Institute of Industrial Engineers

1611.08321 2016-11-28 cs.LG cs.CL cs.CV 81%

Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images

Junhua Mao, Jiajing Xu, Yushi Jing, Alan Yuille

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Appears in NIPS 2016. The datasets introduced in this work will be gradually released on the project page

2608.18317 2026-08-20 cs.CV cs.RO 新提交 80%

Reproducible Multimodal Affordance Prediction

可复现的多模态可供性预测

Tommaso Apicella, Alessio Xompero, Andrea Cavallaro

机构 * Istituto Italiano di Tecnologia（意大利技术研究院）； EPFL（洛桑联邦理工学院）

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV

AI总结针对可供性预测方法评估难的问题，本文提出Affordance Sheet以规范任务表述等信息，实现可供性模型的可复现基准测试与现实场景可靠评估。

Comments Paper accepted to Workshop on Human-Centered Multimodal Intelligence in the Wild (HCMIW) in European Conference on Computer Vision (ECCV) 2026; 18 pages, 3 figures, 7 tables. Project webpage at https://apicis.github.io/aff-sheet

2607.17452 2026-07-21 cs.CL 新提交 80%

How Reliable Are Multimodal Signals of Conversational State? Evidence from Remote Dyadic Collaborative Tasks

对话状态的多模态信号有多可靠？来自远程二元协作任务的证据

Tahiya Chowdhury

机构 * Colby College（科尔比学院）

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CL

AI总结研究探讨从多模态行为测量对话状态的特征可靠性，提出三维评估框架用于视频会议二元对话特征评估，发现语言特征预测佳但跨任务通用性差，声学可靠性受说话者身份影响，交互特征是唯一可靠信号，强调相关评估对对话系统特征选择的重要性。

Comments Accepted, to appear in Proceedings of ACM International Conference on Multimodal Interaction 2026, 13 pages, 6 figures, 4 tables

2607.11192 2026-07-16 cs.CV 版本更新 80%

GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents

GDP.pdf：针对专业PDF文档的基础多模态推理基准测试

Suhaas Garre, Emily Ritchie, Sushant Mehta, Edwin Chen

机构 * Surge AI

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV

AI总结该研究针对专业PDF文档构建多模态推理基准测试GDP.pdf，由专业人员编写问题-文档对，通过严格筛选保留问题，有详细评分标准和能力分类。评估七个前沿模型，发现多数错误源于特定模式，公开了完整基准测试。

Comments 9 pages. v2: results updated to July 2026 leaderboard (17 models). Accepted at the 2nd Workshop on Knowledge-Intensive Multimodal Reasoning (KnowledgeMR) at CVPR 2026 (non-archival), under the former title "PDFParse: A Benchmark for Grounded Multimodal Reasoning over Professional PDF Documents". Dataset: https://huggingface.co/datasets/surgeai/GDP.pdf ; Code: https://github.com/surge-ai/gdp-pdf

2603.09643 2026-04-17 cs.ET cs.AI 80%

MM-tau-p$^2$: Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings

MM-tau-p$^2$: 人格自适应提示用于双控制设置中多模态代理的鲁棒性评估

Anupam Purwar, Aditya Choudhary

机构 * Sprinklr AI

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.AI；multimodal(comments)

AI总结本文提出MM-tau-p$^2$基准，通过12个新指标评估多模态代理在双控制环境中的鲁棒性，结合用户输入解决查询，并展示即使使用前沿LLM，多模态鲁棒性等指标仍需考虑。

Comments A benchmark for evaluating multimodal both voice and text LLM agents in dualcontrol settings. We introduce persona adaptive prompting and 12 new metrics to assess robustness safety efficiency and recovery in customer support scenarios

2511.21364 2025-11-27 cs.LG cs.CV 80%

BanglaMM-Disaster: A Multimodal Transformer-Based Deep Learning Framework for Multiclass Disaster Classification in Bangla

BanglaMM-Disaster: 一种基于Transformer的多模态深度学习框架，用于孟加拉语多类灾害分类

Ariful Islam, Md Rifat Hossen, Md. Mahmudul Arif, Abdullah Al Noman, Md Arifur Rahman

机构 * Department of Computer Science and Engineering（计算机科学与工程系）； Chittagong University of Engineering and Technology（奇特格隆工程与技术大学）； Department of Electronics and Telecommunication Engineering（电子与电信工程系）； Wilmington University（维明顿大学）； College of Graduate and Professional Studies（研究生与专业研究学院）； Trine University（特林大学）

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV

AI总结 BanglaMM-Disaster通过结合文本和视觉数据，提出了一种多模态深度学习框架，用于孟加拉语多类灾害分类，提升了灾害响应效率。

Comments Presented at the 2025 IEEE International Conference on Signal Processing, Information, Communication and Systems (SPICSCON), November 21-22, 2025, University of Rajshahi, Bangladesh. 6 pages, 9 disaster classes, multimodal dataset with 5,037 samples

2511.02794 2025-11-05 cs.AI cs.MA 80%

When One Modality Sabotages the Others: A Diagnostic Lens on Multimodal Reasoning

Chenyu Zhang, Minsol Kim, Shohreh Ghorbani, Jingyao Wu, Rosalind Picard, Patricia Maes, Paul Pu Liang

机构 * Harvard University（哈佛大学）； MIT Media Lab（麻省理工学院媒体实验室）

专题命中多模态评测：multimodal(title,abstract)；分类 cs.AI

Comments Accepted at the Multimodal Algorithmic Reasoning (MAR) Workshop, NeurIPS 2025

2507.07048 2025-10-07 cs.CV 80%

Comprehensive Evaluation of Large Multimodal Models for Nutrition Analysis: A New Benchmark Enriched with Contextual Metadata

Bruce Coburn, Jiangpeng He, Megan E. Rollo, Satvinder S. Dhaliwal, Deborah A. Kerr, Fengqing Zhu

机构 * Purdue University（普渡大学）； Indiana University（印第安纳大学）； Curtin University（Curtin大学）

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV

Comments The extended full version of the accepted paper in 2025 IEEE BHI conference with title: Evaluating Large Multimodal Models for Nutrition Analysis: A New Benchmark Enriched with Contextual Metadata. Dataset is available at: https://skynet.ecn.purdue.edu/~coburn6/ACETADA/

2505.18111 2025-05-26 cs.CV 80%

Adapting SAM 2 for Visual Object Tracking: 1st Place Solution for MMVPR Challenge Multi-Modal Tracking

Cheng-Yen Yang, Hsiang-Wei Huang, Pyong-Kun Kim, Chien-Kai Kuo, Jui-Wei Chang, Kwang-Ju Kim, Chung-I Huang, Jenq-Neng Hwang

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.CV

Comments Accepted by ICPR Multi-Modal Visual Pattern Recognition Workshop

2407.17492 2024-10-30 physics.chem-ph cs.AI cs.LG 80%

Unraveling Molecular Structure: A Multimodal Spectroscopic Dataset for Chemistry

Marvin Alberts, Oliver Schilter, Federico Zipoli, Nina Hartrampf, Teodoro Laino

专题命中多模态评测：multimodal(title,abstract)；分类 cs.AI

Comments 29 pages, submited to conference, code available at: https://github.com/rxn4chemistry/multimodal-spectroscopic-dataset

2403.15408 2024-03-26 eess.SP cs.AI cs.LG 80%

Multi-modal Heart Failure Risk Estimation based on Short ECG and Sampled Long-Term HRV

Sergio González, Abel Ko-Chun Yi, Wan-Ting Hsieh, Wei-Chao Chen, Chun-Li Wang, Victor Chien-Chia Wu, Shang-Hung Chang

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.AI

Journal ref S. González, A. K.-C. Yi, W.-T. Hsieh, W.-C. Chen, C.-L. Wang, V. C.-C. Wu, S.-H. Chang, Multi-modal heart failure risk estimation based on short ECG and sampled long-term HRV, Information Fusion 107 (2024) 102337

1. 多模态评测 9170 篇

Visual Agreement Regularized Training for Multi-Modal Machine Translation

Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning

Privacy Enhanced Multimodal Neural Representations for Emotion Recognition

Multimodal Word Distributions

MEx: Multi-modal Exercises Dataset for Human Activity Recognition

Clinical Concept Embeddings Learned from Massive Sources of Multimodal Medical Data

Multimodal and Multi-view Models for Emotion Recognition

Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)

Unsupervised Multi-modal Neural Machine Translation

Cross-Modal Self-Attention Network for Referring Image Segmentation

Aiding Intra-Text Representations with Visual Context for Multimodal Named Entity Recognition

Multimodal Sentiment Analysis: Addressing Key Issues and Setting up the Baselines

Improving Hospital Mortality Prediction with Medical Named Entities and Multimodal Learning

Evaluating Multimodal Representations on Sentence Similarity: vSTS, Visual Semantic Textual Similarity Dataset

RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes

Emergent Communication in a Multi-Modal, Multi-Step Referential Game

X-CNN: Cross-modal Convolutional Neural Networks for Sparse Datasets

Imagination improves Multimodal Translation

Multimodal Affect Analysis for Product Feedback Assessment

Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images

Reproducible Multimodal Affordance Prediction

How Reliable Are Multimodal Signals of Conversational State? Evidence from Remote Dyadic Collaborative Tasks

GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents

MM-tau-p$^2$: Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings

BanglaMM-Disaster: A Multimodal Transformer-Based Deep Learning Framework for Multiclass Disaster Classification in Bangla

When One Modality Sabotages the Others: A Diagnostic Lens on Multimodal Reasoning

Comprehensive Evaluation of Large Multimodal Models for Nutrition Analysis: A New Benchmark Enriched with Contextual Metadata

Adapting SAM 2 for Visual Object Tracking: 1st Place Solution for MMVPR Challenge Multi-Modal Tracking

Unraveling Molecular Structure: A Multimodal Spectroscopic Dataset for Chemistry

Multi-modal Heart Failure Risk Estimation based on Short ECG and Sampled Long-Term HRV