Penquiry：一种利用大语言模型的基于笔的交互式原位问答系统
Penquiry: A Pen-based Interactive In-situ Q&A System Leveraging LLMs
浏览论文内容
中文总结 AI 辅助
Penquiry利用笔在数字学习材料上直接提问，通过内容捕捉和问题自动补全解决指称与表达障碍，显著降低提问开销，为基于笔的原位AI交互提供新蓝图。
中文摘要 AI 辅助
基于笔的数字设备仍然是主动、认知参与学习的首选媒介。与此同时，大语言模型（LLMs）已成为自主学习中不可或缺的工具，使学生能够澄清概念。然而，基于笔的工作流程的流畅、空间特性与LLMs对离散、键盘密集型输入的要求之间存在根本性的交互鸿沟。我们提出了Penquiry，一种原位问答系统，通过使学习者能够直接用笔在数字学习材料上提问来弥合这一鸿沟。我们描述了这种多模态转换中的两个主要交互挑战：指称障碍（Referential Barrier），它阻碍了将细粒度视觉元素锚定到查询上下文中；以及表达障碍（Expressive Barrier），它迫使学习者将各种非文本意图（如方程和图表）转化为僵硬的打字句子。为解决这些问题，Penquiry引入了一个中介层，具有内容捕捉（Content Snapping）功能以实现明确引用，以及问题自动补全（Question Autocompletion）功能以将稀疏的墨迹关键词扩展为丰富的语义查询。通过两项迭代用户研究（每项研究N=16），我们证明与传统的界面相比，Penquiry显著降低了提问的认知和物理开销，为基于笔的原位AI交互提供了新的蓝图。
英文摘要
Pen-based digital devices remain a preferred medium for active, cognitively engaging study. Concurrently, Large Language Models (LLMs) have become indispensable for self-directed learning, enabling students to clarify concepts. However, a fundamental interaction gap exists between the fluid, spatial nature of pen-based workflows and the discrete, keyboard-heavy requirements of LLMs. We present Penquiry, an in-situ question-and-answer system that bridges this gap by enabling learners to pose questions directly on digital study materials via a pen. We characterize two primary interaction challenges in this multimodal transition: a Referential Barrier, which hinders grounding fine-grained visual elements into the query context, and an Expressive Barrier, which forces learners to translate diverse, non-textual intents--such as equations and diagrams--into rigid, typed sentences. To resolve these, Penquiry introduces a mediation layer featuring Content Snapping for unambiguous referencing and Question Autocompletion to expand sparse ink keywords into rich semantic queries. Through two iterative user studies (N = 16 per study), we demonstrate that Penquiry significantly reduces the cognitive and physical overhead of inquiry compared to traditional interfaces, providing a new blueprint for pen-based, in-situ AI interaction
发表机构
- Seoul National University（首尔大学）
- Pohang University of Science and Technology（浦项科技大学）
- Hankuk University of Foreign Studies（韩国外国语大学）
机构由 AI 辅助整理，请以论文原文为准。
