FinMTM: A Multi-Turn Multimodal Benchmark for Financial Reasoning and Agent Evaluation
HiThink Research(HiThink研究机构);Wuhan University(武汉大学);Zhejiang University(浙江大学);Nanyang Technological University(南洋理工大学);Shanghai Institute of Technology(上海理工大学)
The financial domain poses substantial challenges for vision-language models (VLMs) due to specialized chart formats and knowledge-intensive reasoning requirements. However, existing financial benchmarks are largely single-turn and rely on a narrow set of question formats, limiting comprehensive evaluation in realistic application scenarios. To address this gap, we propose FinMTM, a multi-turn multimodal benchmark that expands diversity along both data and task dimensions. On the data side, we curate and annotate 11{,}133 bilingual (Chinese and English) financial QA pairs grounded in financial visuals, including candlestick charts, statistical plots, and report figures. On the task side, FinMTM covers single- and multiple-choice questions, multi-turn open-ended dialogues, and agent-based tasks. We further design task-specific evaluation protocols, including a set-overlap scoring rule for multiple-choice questions, a weighted combination of turn-level and session-level scores for multi-turn dialogues, and a composite metric that integrates planning quality with final outcomes for agent tasks. Extensive experimental evaluation of 22 VLMs reveal their limitations in fine-grained visual perception, long-context reasoning, and complex agent workflows.
FCMBench: The First Large-scale Financial Credit Multimodal Benchmark for Real-world Applications
AI Lab, Qifu Technology(奇富科技AI实验室);College of Future Information Technology, Fudan University(复旦大学未来信息学院);School of Future Technology, South China University of Technology(华南理工大学未来技术学院);Pazhou Lab(琶洲实验室)
SAGE: An Expert-Annotated South Asian GI Endoscopy Dataset for Multimodal Learning and Hallucination Analysis
Nepal Applied Mathematics and Informatics Institute for Research(尼泊尔应用数学与信息技术研究所);GastroIntestinal Department, Dhulikhel Hospital(杜尔基hel医院消化内科);Univesity of Lausanne(洛桑大学);University of West Virginia(西弗吉尼亚大学);University of Utah(犹他大学);University of Aberdeen(阿伯丁大学)
FinCriticalED: A Visual Benchmark for Financial Fact-Level OCR
Columbia University(哥伦比亚大学);Stevens Institute of Technology(史蒂文斯理工学院);Georgia Institute of Technology(佐治亚理工学院);New York University(纽约大学);University of Minnesota(明尼苏达大学);Halmstad University(哈尔姆斯塔德大学);Harvard University(哈佛大学);University of Manchester(曼彻斯特大学)
DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detection
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University at Buffalo, State University of New York(纽约州立大学布法罗分校);Nanjing University of Posts and Telecommunications(南京邮电大学)
Fusing Perceptual Vision Experts with Multimodal LLMs for Explainable Plant Disease Diagnosis: From Benchmark Imagery to Real-World Robotic Field Validation
Cornell University(康奈尔大学);University of the Peloponnese(伯罗奔尼撒大学);Agricultural University of Athens(雅典农业大学)