arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.15280cs.HC
收藏

在可刷新触觉显示器上支持多模态数据交互：一种将触觉与对话式AI结合的架构

Supporting Multimodal Data Interaction on Refreshable Tactile Displays: An Architecture to Combine Touch and Conversational AI

Monash University（莫纳什大学）；Yonsei University（延世大学）

Samuel Reinders, Munazza Zaib, Matthew Butler, Bongshin Lee, Ingrid Zukerman, Lizhen Qu, Kim Marriott

备注：
Accepted to be presented at IEEE PacificVis 2026; Best Paper Award (VisNotes track); Replacement: external DOI added
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出了一种结合触觉与对话式AI的架构，用于在可刷新触觉显示器上实现多模态数据交互，通过融合触觉输入与语音语言，提升视障人士的数据可视化体验。

摘要：

将对话式AI与可刷新触觉显示器（RTDs）相结合，为视障或低视力人群提供了创建可访问数据可视化的重要潜力。为了支持研究人员和开发者使用RTDs构建可访问的数据可视化，我们提出了一个多模态数据交互架构以及一个开源的参考实现。我们的系统是首个将触觉输入与RTD上的对话代理结合的系统，能够实现将触觉上下文与 spoken language（口语语言）融合的指示性查询，例如“这些点之间的趋势是什么？”该架构解决了关键的技术挑战，包括RTD上的触觉传感、视觉到触觉的编码、将触觉上下文与对话式AI整合，以及多模态输出的同步。我们的贡献是双方面的：（1）一个将RTD硬件、外部触觉传感和对话式AI整合在一起的架构，以实现多模态数据交互；（2）一个开源的参考实现，证明了其可行性。这项工作为未来多模态可访问数据可视化的研究提供了技术基础。

英文摘要：

Combining conversational AI with refreshable tactile displays (RTDs) offers significant potential for creating accessible data visualization for people who are blind or have low vision (BLV). To support researchers and developers building accessible data visualizations with RTDs, we present a multimodal data interaction architecture along with an open-source reference implementation. Our system is the first to combine touch input with a conversational agent on an RTD, enabling deictic queries that fuse touch context with spoken language, such as "what is the trend between these points?" The architecture addresses key technical challenges, including touch sensing on RTDs, visual-to-tactile encoding, integrating touch context with conversational AI, and synchronizing multimodal output. Our contributions are twofold: (1) a technical architecture integrating RTD hardware, external touch sensing, and conversational AI to enable multimodal data interaction; and (2) an open-source reference implementation demonstrating its feasibility. This work provides a technical foundation to support future research in multimodal accessible data visualization.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑