arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.03130cs.CVcs.CE
收藏

FinMTM:面向金融推理和智能体评估的多轮多模态基准

FinMTM: A Multi-Turn Multimodal Benchmark for Financial Reasoning and Agent Evaluation

HiThink Research(HiThink研究机构);Wuhan University(武汉大学);Zhejiang University(浙江大学);Nanyang Technological University(南洋理工大学);Shanghai Institute of Technology(上海理工大学)

Chenxi Zhang, Ziliang Gan, Liyun Zhu, Youwei Pang, Qing Zhang, Rongjunchen Zhang

AI总结:

FinMTM提出一个多轮多模态基准,用于评估金融推理和智能体任务,通过多样化数据和任务设计,揭示了VLMs在视觉感知、推理和复杂工作流中的局限性。

摘要:

金融领域对视觉语言模型(VLMs)提出了重大挑战,由于专门的图表格式和知识密集型推理需求。然而,现有的金融基准大多为单轮,并且依赖于狭窄的问题格式集,限制了在现实应用场景中的全面评估。为了解决这一差距,我们提出了FinMTM,一个多轮多模态基准,扩展了数据和任务两个维度的多样性。在数据方面,我们整理并标注了11,133对双语(中英文)的金融问答对,这些问答对基于金融视觉内容,包括K线图、统计图表和报告图表。在任务方面,FinMTM涵盖了单选和多选问题、多轮开放式对话以及基于智能体的任务。我们进一步设计了任务特定的评估协议,包括用于多选问题的集重评分规则,用于多轮对话的加权组合评分(轮级和会话级评分),以及整合规划质量和最终结果的综合指标。对22个VLMs的广泛实验评估揭示了它们在细粒度视觉感知、长上下文推理和复杂智能体工作流中的局限性。

英文摘要:

The financial domain poses substantial challenges for vision-language models (VLMs) due to specialized chart formats and knowledge-intensive reasoning requirements. However, existing financial benchmarks are largely single-turn and rely on a narrow set of question formats, limiting comprehensive evaluation in realistic application scenarios. To address this gap, we propose FinMTM, a multi-turn multimodal benchmark that expands diversity along both data and task dimensions. On the data side, we curate and annotate 11{,}133 bilingual (Chinese and English) financial QA pairs grounded in financial visuals, including candlestick charts, statistical plots, and report figures. On the task side, FinMTM covers single- and multiple-choice questions, multi-turn open-ended dialogues, and agent-based tasks. We further design task-specific evaluation protocols, including a set-overlap scoring rule for multiple-choice questions, a weighted combination of turn-level and session-level scores for multi-turn dialogues, and a composite metric that integrates planning quality with final outcomes for agent tasks. Extensive experimental evaluation of 22 VLMs reveal their limitations in fine-grained visual perception, long-context reasoning, and complex agent workflows.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑