Protecting multimodal large language models against misleading visualizations
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE(普遍知识处理实验室(UKP实验室)、计算机科学系,德累斯顿技术大学及应用网络安全国家研究中心ATHENE);Department of Electrical Engineering, KU Leuven(电气工程系,鲁文大学);Department of Computer Science, KU Leuven(计算机科学系,鲁文大学)
Jonathan Tonglet, Tinne Tuytelaars, Marie-Francine Moens, Iryna Gurevych
Visualizations play a pivotal role in daily communication in an increasingly data-driven world. Research on multimodal large language models (MLLMs) for automated chart understanding has accelerated massively, with steady improvements on standard benchmarks. However, for MLLMs to be reliable, they must be robust to misleading visualizations, i.e., charts that distort the underlying data, leading readers to draw inaccurate conclusions. Here, we uncover an important vulnerability: MLLM question-answering (QA) accuracy on misleading visualizations drops on average to the level of the random baseline. To address this, we provide the first comparison of six inference-time methods to improve QA performance on misleading visualizations, without compromising accuracy on non-misleading ones. We find that two methods, table-based QA and redrawing the visualization, are effective, with improvements of up to 19.6 percentage points. We make our code and data available.
Is this chart lying to me? Automating the detection of misleading visualizations
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE(普遍知识处理实验室(UKP实验室)、计算机科学系、图腾斯大学(TU Darmstadt)及应用网络安全国家研究中心ATHENE);Department of Electrical Engineering, KU Leuven(电子工程系、鲁文大学);Department of Computer Science, KU Leuven(计算机科学系、鲁文大学)
RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild
School of Computing (SoC), National University of Singapore (NUS)(新加坡国立大学计算机学院(SoC));National University of Singapore (NUS)(新加坡国立大学);Department of Electrical and Computer Engineering (ECE), National University of Singapore (NUS)(新加坡国立大学电子与计算机工程系(ECE))
On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI
MICS, CentraleSupélec - Université Paris-Saclay, France(MICS,中央圣艾尔布兰大学-巴黎萨克雷大学,法国);Google DeepMind, London, UK(谷歌DeepMind,伦敦,英国);CONICET, Universidad de Buenos Aires, Argentina(CONICET,布宜诺斯艾利斯大学,阿根廷)
Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);Tsinghua University(清华大学);Ant Group(蚂蚁集团);Alibaba Group(阿里巴巴集团)
VizRAG: Enhancing Retrieval-Augmented Generation with Hypergraph Visualization
Southern University of Science and Technology(南方科技大学);Hong Kong University of Science and Technology(香港科技大学);Huawei Research(华为研究院);Beihang University(北京航空航天大学)
Faithful Chart Generation for Multimodal Deep Research: Frame-Evidence Co-Adaptation
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);University of Amsterdam(阿姆斯特丹大学)