当表格失控:评估多模态模型在法语金融文档上的表现
When Tables Go Crazy: Evaluating Multimodal Models on French Financial Documents
AI总结:
提出Scribe Finance基准,评估多模态模型在法语金融文档上的文本、表格、图表及多轮对话理解能力,发现模型在图表和多轮对话中表现脆弱。
摘要:
视觉语言模型(VLM)在许多文档理解任务上表现良好,但它们在专业非英语领域的可靠性仍未充分探索。这一差距在金融领域尤为关键,因为金融文档混合了密集的监管文本、数值表格和视觉图表,且提取错误可能产生实际后果。我们引入了Scribe Finance,这是首个用于评估法语金融文档理解的多模态基准。该数据集包含1,204个经过专家验证的问题,涵盖文本提取、表格理解、图表解释和多轮对话推理,数据来自真实的投资说明书、KID和PRIIP。我们使用LLM-as-judge协议评估了六个开源VLM(8B-124B参数)。虽然模型在文本和表格任务上表现强劲(85-90%准确率),但在图表解释上表现不佳(34-62%)。最值得注意的是,多轮对话揭示了一个严重的失败模式:早期错误在轮次间传播,导致准确率降至约50%,无论模型大小如何。这些结果表明,当前的VLM在定义明确的提取任务上有效,但在交互式、多步骤的金融分析中仍然脆弱。Scribe Finance提供了一个具有挑战性的基准,用于衡量和推动这一高风险场景的进展。
英文摘要:
Vision-language models (VLMs) perform well on many document understanding tasks, yet their reliability in specialized, non-English domains remains underexplored. This gap is especially critical in finance, where documents mix dense regulatory text, numerical tables, and visual charts, and where extraction errors can have real-world consequences. We introduce Scribe Finance, the first multimodal benchmark for evaluating French financial document understanding. The dataset contains 1,204 expert-validated questions spanning text extraction, table comprehension, chart interpretation, and multi-turn conversational reasoning, drawn from real investment prospectuses, KIDs, and PRIIPs. We evaluate six open-weight VLMs (8B-124B parameters) using an LLM-as-judge protocol. While models achieve strong performance on text and table tasks (85-90% accuracy), they struggle with chart interpretation (34-62%). Most notably, multi-turn dialogue reveals a sharp failure mode: early mistakes propagate across turns, driving accuracy down to roughly 50% regardless of model size. These results show that current VLMs are effective for well-defined extraction tasks but remain brittle in interactive, multi-step financial analysis. Scribe Finance offers a challenging benchmark to measure and drive progress in this high-stakes setting.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。