多模态大模型 - arXivDaily 专题

2501.16643 2025-03-19 cs.CL cs.AI cs.SD eess.AS 82%

An LLM Benchmark for Addressee Recognition in Multi-modal Multi-party Dialogue

Koji Inoue, Divesh Lala, Mikey Elmers, Keiko Ochi, Tatsuya Kawahara

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments This paper has been accepted for presentation at International Workshop on Spoken Dialogue Systems Technology 2025 (IWSDS 2025) and represents the author's version of the work

2503.13399 2025-03-18 cs.CV cs.AI cs.CL cs.LG q-bio.CB 82%

MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research

James Burgess, Jeffrey J Nirschl, Laura Bravo-Sánchez, Alejandro Lozano, Sanket Rajan Gupte, Jesus G. Galaz-Montoya, Yuhui Zhang, Yuchang Su, Disha Bhowmik, Zachary Coman, Sarina M. Hasan, Alexandra Johannesson, William D. Leineweber, Malvika G Nair, Ridhi Yarlagadda, Connor Zuraski, Wah Chiu, Sarah Cohen, Jan N. Hansen, Manuel D Leonetti, Chad Liu, Emma Lundberg, Serena Yeung-Levy

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2025 (Conference on Computer Vision and Pattern Recognition) Project page at https://jmhb0.github.io/microvqa Benchmark at https://huggingface.co/datasets/jmhb/microvqa

2503.10627 2025-03-14 cs.CV cs.AI cs.CL 82%

SciVerse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMMs on Multi-modal Scientific Problems

Ziyu Guo, Ray Zhang, Hao Chen, Jialin Gao, Dongzhi Jiang, Jiaze Wang, Pheng-Ann Heng

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Initially released in September 2024. Project page: https://sciverse-cuhk.github.io

2503.01064 2025-03-04 cs.CL cs.AI cs.CV 82%

Scientific Reasoning: Assessment of Multimodal Generative LLMs

Florian Dreyer, Ekaterina Kolos, Daria Matiash

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2411.16955 2025-03-04 cs.LG cond-mat.mtrl-sci 82%

Probing the limitations of multimodal language models for chemistry and materials research

Nawaf Alampara, Mara Schilling-Wilhelmi, Martiño Ríos-García, Indrajeet Mandal, Pranav Khetarpal, Hargun Singh Grover, N. M. Anoop Krishnan, Kevin Maik Jablonka

专题命中多模态评测：multimodal(title,abstract)；cross-modal(abstract)

2502.20504 2025-03-03 cs.CL cs.AI cs.CV 82%

A Thousand Words or An Image: Studying the Influence of Persona Modality in Multimodal LLMs

Julius Broomfield, Kartik Sharma, Srijan Kumar

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2502.17540 2025-02-26 cs.CV cs.AI cs.CL 82%

PosterSum: A Multimodal Benchmark for Scientific Poster Summarization

Rohit Saxena, Pasquale Minervini, Frank Keller

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments This paper includes a dataset of research posters with abstracts. We provide two cited examples ( arXiv:2211.11880 and arXiv:2210.07571 ) to illustrate reference summaries

2502.15422 2025-02-24 cs.CL cs.AI cs.CV 82%

Evaluating Multimodal Generative AI with Korean Educational Standards

Sanghee Park, Geewook Kim

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 18 pages; To appear at NAACL 2025 Main Conference (Project page: https://github.com/naver-ai/KoNET )

2502.13832 2025-02-20 cs.HC 82%

ArtMentor: AI-Assisted Evaluation of Artworks to Explore Multimodal Large Language Models Capabilities

Chanjin Zheng, Zengyi Yu, Yilin Jiang, Mingzi Zhang, Xunuo Lu, Jing Jin, Liteng Gao

专题命中多模态评测：multimodal(title,abstract)；MLLM(abstract)

Comments 18 pages, 12 figures. Accepted by CHI 2025

2501.05767 2025-02-19 cs.CL cs.AI cs.CV 82%

Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models

You Li, Heyu Huang, Chi Chen, Kaiyu Huang, Chao Huang, Zonghao Guo, Zhiyuan Liu, Jinan Xu, Yuhua Li, Ruixuan Li, Maosong Sun

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 21 pages, 8 figures

2502.09621 2025-02-14 cs.CV cs.AI cs.CL 82%

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency

Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, Bo Zhang, Chaoyou Fu, Peng Gao, Hongsheng Li

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project Page: https://mmecot.github.io/

2502.08916 2025-02-14 cs.CV cs.AI cs.CL cs.MA 82%

PathFinder: A Multi-Modal Multi-Agent System for Medical Diagnostic Decision-Making Applied to Histopathology

Fatemeh Ghezloo, Mehmet Saygin Seyfioglu, Rustin Soraki, Wisdom O. Ikezogwo, Beibin Li, Tejoram Vivekanandan, Joann G. Elmore, Ranjay Krishna, Linda Shapiro

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2406.11230 2025-02-12 cs.LG cs.AI cs.CL cs.CV 82%

Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models

Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu, Hao Wang

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at NAACL 2025 Main

2406.05862 2025-01-14 cs.CL cs.AI cs.CV 82%

II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models

Ziqiang Liu, Feiteng Fang, Xi Feng, Xinrun Du, Chenhao Zhang, Zekun Wang, Yuelin Bai, Qixuan Zhao, Liyang Fan, Chengguang Gan, Hongquan Lin, Jiaming Li, Yuansheng Ni, Haihong Wu, Yaswanth Narsupalli, Zhigang Zheng, Chengming Li, Xiping Hu, Ruifeng Xu, Xiaojun Chen, Min Yang, Jiaheng Liu, Ruibo Liu, Wenhao Huang, Ge Zhang, Shiwen Ni

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 100 pages, 82 figures, add citations

2407.09413 2025-01-14 cs.CL cs.AI cs.CV 82%

SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers

Shraman Pramanick, Rama Chellappa, Subhashini Venugopalan

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NeurIPS 2024, Datasets & Benchmarks track

2412.12606 2024-12-18 cs.AI cs.CL cs.CV 82%

Multi-Dimensional Insights: Benchmarking Real-World Personalization in Large Multimodal Models

YiFan Zhang, Shanglin Lei, Runqi Qiao, Zhuoma GongQue, Xiaoshuai Song, Guanting Dong, Qiuna Tan, Zhe Wei, Peiqing Yang, Ye Tian, Yadong Xue, Xiaofei Wang, Honggang Zhang

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 33 pages, 33 figures, Work in progress

2412.05536 2024-12-10 eess.IV cs.AI cs.CL cs.CV 82%

Comprehensive Evaluation of Multimodal AI Models in Medical Imaging Diagnosis: From Data Augmentation to Preference-Based Comparison

Cailian Ruan, Chengyue Huang, Yahe Yang

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2412.05255 2024-12-09 cs.AI cs.CL cs.CV cs.MA 82%

TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft

Qian Long, Zhi Li, Ran Gong, Ying Nian Wu, Demetri Terzopoulos, Xiaofeng Gao

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2412.02527 2024-12-04 astro-ph.IM astro-ph.GA astro-ph.SR 82%

The Multimodal Universe: Enabling Large-Scale Machine Learning with 100TB of Astronomical Scientific Data

The Multimodal Universe Collaboration, Jeroen Audenaert, Micah Bowles, Benjamin M. Boyd, David Chemaly, Brian Cherinka, Ioana Ciucă, Miles Cranmer, Aaron Do, Matthew Grayling, Erin E. Hayes, Tom Hehir, Shirley Ho, Marc Huertas-Company, Kartheik G. Iyer, Maja Jablonska, Francois Lanusse, Henry W. Leung, Kaisey Mandel, Juan Rafael Martínez-Galarza, Peter Melchior, Lucas Meyer, Liam H. Parker, Helen Qu, Jeff Shen, Michael J. Smith, Connor Stone, Mike Walmsley, John F. Wu

专题命中多模态评测：multimodal(title,abstract)；multi-modal(abstract)

Comments Accepted at NeurIPS Datasets and Benchmarks track

2308.02490 2024-12-03 cs.AI cs.CL cs.CV cs.LG 82%

MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, Lijuan Wang

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ICML 2024. Code, data and leaderboard: https://github.com/yuweihao/MM-Vet

2411.18932 2024-12-02 cs.CL cs.AI cs.CV 82%

ScratchEval: Are GPT-4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challenges

Rao Fu, Ziyang Luo, Hongzhan Lin, Zhen Ye, Jing Ma

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2411.17760 2024-11-28 cs.CL cs.AI cs.CV cs.LG 82%

Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach

Shijian Deng, Wentian Zhao, Yu-Jhe Li, Kun Wan, Daniel Miranda, Ajinkya Kale, Yapeng Tian

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2409.12959 2024-11-28 cs.CV cs.AI cs.CL cs.IR 82%

MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines

Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Jiayi Lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Chaoyou Fu, Guanglu Song, Peng Gao, Yu Liu, Chunyuan Li, Hongsheng Li

专题命中多模态评测：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project Page: https://mmsearch.github.io

2411.14797 2024-11-25 cs.LG cs.AI cs.CL cs.CV 82%

Continual SFT Matches Multimodal RLHF with Negative Supervision

Ke Zhu, Yu Wang, Yanpeng Sun, Qiang Chen, Jiangjiang Liu, Gang Zhang, Jingdong Wang

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2406.20098 2024-11-19 cs.CV cs.AI cs.CL 82%

Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs

Sukmin Yun, Haokun Lin, Rusiru Thushara, Mohammad Qazim Bhat, Yongxin Wang, Zutao Jiang, Mingkai Deng, Jinhong Wang, Tianhua Tao, Junbo Li, Haonan Li, Preslav Nakov, Timothy Baldwin, Zhengzhong Liu, Eric P. Xing, Xiaodan Liang, Zhiqiang Shen

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NeurIPS 2024 Datasets and Benchmarks Camera-ready Version. Website at https://mbzuai-llm.github.io/webpage2code/

2401.11944 2024-11-05 cs.CL cs.AI cs.CV 82%

CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark

Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, Haoran Zhang, Xingwei Qu, Junjie Wang, Ruibin Yuan, Yizhi Li, Zekun Wang, Yudong Liu, Yu-Hsuan Tsai, Fengji Zhang, Chenghua Lin, Wenhao Huang, Jie Fu

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2409.20012 2024-11-04 cs.CL cs.AI cs.MM 82%

Towards Robust Multimodal Sentiment Analysis with Incomplete Data

Haoyu Zhang, Wenbin Wang, Tianshu Yu

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Accepted to NeurIPS 2024

2410.18096 2024-10-25 cs.IR cs.AI cs.CL cs.CV 82%

$M^3EL$: A Multi-task Multi-topic Dataset for Multi-modal Entity Linking

Fang Wang, Shenglin Yin, Xiaoying Bai, Minghao Hu, Tianwei Yan, Yi Liang

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2410.16503 2024-10-23 cs.AI cs.CL cs.CV 82%

Allo-AVA: A Large-Scale Multimodal Conversational AI Dataset for Allocentric Avatar Gesture Animation

Saif Punjwani, Larry Heck

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2408.08464 2024-10-23 cs.CR 82%

$\textit{MMJ-Bench}$: A Comprehensive Study on Jailbreak Attacks and Defenses for Multimodal Large Language Models

Fenghua Weng, Yue Xu, Chengyan Fu, Wenjie Wang

专题命中多模态评测：multimodal(title,abstract)；MLLM(abstract)

1. 多模态评测 9150 篇

An LLM Benchmark for Addressee Recognition in Multi-modal Multi-party Dialogue

MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research

SciVerse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMMs on Multi-modal Scientific Problems

Scientific Reasoning: Assessment of Multimodal Generative LLMs

Probing the limitations of multimodal language models for chemistry and materials research

A Thousand Words or An Image: Studying the Influence of Persona Modality in Multimodal LLMs

PosterSum: A Multimodal Benchmark for Scientific Poster Summarization

Evaluating Multimodal Generative AI with Korean Educational Standards

ArtMentor: AI-Assisted Evaluation of Artworks to Explore Multimodal Large Language Models Capabilities

Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models

MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency

PathFinder: A Multi-Modal Multi-Agent System for Medical Diagnostic Decision-Making Applied to Histopathology

Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models

II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models

SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers

Multi-Dimensional Insights: Benchmarking Real-World Personalization in Large Multimodal Models

Comprehensive Evaluation of Multimodal AI Models in Medical Imaging Diagnosis: From Data Augmentation to Preference-Based Comparison

TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft

The Multimodal Universe: Enabling Large-Scale Machine Learning with 100TB of Astronomical Scientific Data

MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities

ScratchEval: Are GPT-4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challenges

Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach

MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines

Continual SFT Matches Multimodal RLHF with Negative Supervision

Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs

CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark

Towards Robust Multimodal Sentiment Analysis with Incomplete Data

$M^3EL$: A Multi-task Multi-topic Dataset for Multi-modal Entity Linking

Allo-AVA: A Large-Scale Multimodal Conversational AI Dataset for Allocentric Avatar Gesture Animation

$\textit{MMJ-Bench}$: A Comprehensive Study on Jailbreak Attacks and Defenses for Multimodal Large Language Models