arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.03401cs.HCcs.AIcs.CV
收藏

LLMs能否进行注意力推理?多模态课堂行为的零样本分析

Can LLMs Reason About Attention? Towards Zero-Shot Analysis of Multimodal Classroom Behavior

Virginia Tech(弗吉尼亚理工大学);University of Virginia(弗吉尼亚大学);University of Pennsylvania(宾夕法尼亚大学)

Nolan Platt, Sehrish Nizamani, Alp Tural, Elif Tural, Saad Nizamani, Andrew Katz, Yoonje Lee, Nada Basit

AI总结:

本文提出一个隐私保护的分析流程,利用OpenPose和Gaze-LLE提取学生注意力信息,并通过QwQ-32B-Reasoning进行零样本分析,探讨LLMs在多模态行为理解中的潜力与局限。

摘要:

理解学生参与度通常需要耗时的手动观察或侵入性记录,引发隐私问题。本文提出一个隐私保护的流程,通过分析课堂视频提取学生注意力信息,不存储任何可识别视频。系统运行于单个GPU,使用OpenPose进行骨骼提取和Gaze-LLE进行视觉注意力估计。原始视频帧在姿态提取后立即删除,仅保留几何坐标(存储为JSON),符合FERPA规定。提取的姿态和注视数据通过QwQ-32B-Reasoning处理,进行跨讲座段的零样本分析。教师通过网页仪表板查看注意力热图和行为摘要。初步发现LLMs在多模态行为理解中可能有前景,但仍在课堂布局的空间推理方面存在困难。本文讨论了这些限制,并概述了在教育分析中改进LLM空间理解的方向。

英文摘要:

Understanding student engagement usually requires time-consuming manual observation or invasive recording that raises privacy concerns. We present a privacy-preserving pipeline that analyzes classroom videos to extract insights about student attention, without storing any identifiable footage. Our system runs on a single GPU, using OpenPose for skeletal extraction and Gaze-LLE for visual attention estimation. Original video frames are deleted immediately after pose extraction, thus only geometric coordinates (stored as JSON) are retained, ensuring compliance with FERPA. The extracted pose and gaze data is processed by QwQ-32B-Reasoning, which performs zero-shot analysis of student behavior across lecture segments. Instructors access results through a web dashboard featuring attention heatmaps and behavioral summaries. Our preliminary findings suggest that LLMs may show promise for multimodal behavior understanding, although they still struggle with spatial reasoning about classroom layouts. We discuss these limitations and outline directions for improving LLM spatial comprehension in educational analytics contexts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑