arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.20503cs.CL
收藏

保护多模态大语言模型免受误导性可视化的影响

Protecting multimodal large language models against misleading visualizations

Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE(普遍知识处理实验室(UKP实验室)、计算机科学系,德累斯顿技术大学及应用网络安全国家研究中心ATHENE);Department of Electrical Engineering, KU Leuven(电气工程系,鲁文大学);Department of Computer Science, KU Leuven(计算机科学系,鲁文大学)

Jonathan Tonglet, Tinne Tuytelaars, Marie-Francine Moens, Iryna Gurevych

AI总结:

研究发现多模态大语言模型在面对误导性可视化时回答准确性显著下降,提出两种有效方法提升其处理能力,同时保持非误导性可视化下的准确性。

摘要:

可视化在日益数据驱动的世界中扮演着关键角色。多模态大语言模型(MLLMs)在自动图表理解方面的研究已迅速发展,标准基准上的表现持续改进。然而,为了可靠性,MLLMs必须对误导性可视化具有鲁棒性,即那些扭曲底层数据的图表,导致读者得出不准确的结论。本文揭示了一个重要漏洞:MLLM在误导性可视化上的问答(QA)准确性平均降至随机基线水平。为解决此问题,我们首次比较了六种推理方法,以提高MLLM在误导性可视化上的问答性能,而不影响非误导性可视化上的准确性。我们发现两种方法,基于表格的问答和重绘可视化,是有效的,改进幅度高达19.6个百分点。我们公开了代码和数据。

英文摘要:

Visualizations play a pivotal role in daily communication in an increasingly data-driven world. Research on multimodal large language models (MLLMs) for automated chart understanding has accelerated massively, with steady improvements on standard benchmarks. However, for MLLMs to be reliable, they must be robust to misleading visualizations, i.e., charts that distort the underlying data, leading readers to draw inaccurate conclusions. Here, we uncover an important vulnerability: MLLM question-answering (QA) accuracy on misleading visualizations drops on average to the level of the random baseline. To address this, we provide the first comparison of six inference-time methods to improve QA performance on misleading visualizations, without compromising accuracy on non-misleading ones. We find that two methods, table-based QA and redrawing the visualization, are effective, with improvements of up to 19.6 percentage points. We make our code and data available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑