多模态数据理解：理解视觉-文本信息链如何影响数据解释

中文摘要 AI 辅助

可视化和文本常常共同作用以支持有效的数据通信。尽管有这种常见范式，但我们对这些模态的相互作用如何影响人们的数据理解知之甚少。我们提出了一种新颖的实验范式来研究多模态数据理解，即人们从多模态视觉和文本数据中理解信息的过程，涵盖众包和出声思考环境。我们的方法采用两条顺序链来呈现多模态信息，一条可视化优先链和一条文本优先链，要求人们反复描述所呈现的数据。通过比较人们在不同链上的数据理解变化，我们可以评估每种模态的信息贡献以及它们如何塑造后续理解。我们发现可视化优先链促进了基于假设驱动发现的探索性理解，而文本优先链产生了类似于框架效应的验证性理解，其中可视化用于强化和确认从文本中得出的观察结果。我们的发现为多模态信息整合提供了实证见解，对设计更有效的数据驱动通信具有启示意义。

英文摘要

Visualizations and text often work together to support effective data communication. Despite this common paradigm, we know little about how the interplay of these modalities affects people's data comprehension. We present a novel experimental paradigm to investigate multimodal data comprehension---the process of people comprehending information from multimodal visual and textual data---across both crowdsourced and think-aloud environments. Our methodology employs two sequential chains for presenting multimodal information---a visualization-first chain and a text-first chain---asking people to describe the data presented iteratively. By comparing how people's data comprehension changes across the chain, we can assess the information contribution of each modality and how they shape subsequent comprehension. We found that the visualization-first chain facilitates exploratory comprehension with hypothesis-driven discovery, whereas the text-first chain yields confirmatory comprehension akin to framing effects where visualizations serve to reinforce and confirm observations drawn from text. Our findings provide empirical insights into multimodal information integration, with implications for designing more effective data-driven communication.

发表机构

• University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
• University of California San Diego（加利福尼亚大学圣迭戈分校）

机构由 AI 辅助整理，请以论文原文为准。