研究儿童在混合现实环境中的具身学习(他们在其中协作模拟科学过程),需要分析复杂的多模态数据,以解释其学习与协调行为。学习科学家已经开发了用于分析此类数据的 Interaction Analysis(IA,互动分析)方法,但这要求研究人员观看数小时视频,以提取并解释学生的学习模式。本研究旨在简化研究人员的任务,使用机器学习和多模态学习分析来支持 IA 流程。本研究结合机器学习算法与多模态分析,以支持并简化研究人员的工作,使其通过学生在模拟场景中的动作、凝视和情感反应,全面理解学生的科学参与。为促进有效的研究人员—AI 协作,我们提出一项初步案例研究,以确定在时间轴上可视化呈现学生状态、行动、凝视、情感和移动的可行性。案例研究聚焦于学生学习光合作用的特定科学场景。该时间轴使我们能够考察由多模态分析和互动分析识别出的关键学习时刻之间的一致性,并揭示关于学生时间性学习进展的洞见。
英文摘要:
Investigating children's embodied learning in mixed-reality environments, where they collaboratively simulate scientific processes, requires analyzing complex multimodal data to interpret their learning and coordination behaviors. Learning scientists have developed Interaction Analysis (IA) methodologies for analyzing such data, but this requires researchers to watch hours of videos to extract and interpret students' learning patterns. Our study aims to simplify researchers' tasks, using Machine Learning and Multimodal Learning Analytics to support the IA processes. Our study combines machine learning algorithms and multimodal analyses to support and streamline researcher efforts in developing a comprehensive understanding of students' scientific engagement through their movements, gaze, and affective responses in a simulated scenario. To facilitate an effective researcher-AI partnership, we present an initial case study to determine the feasibility of visually representing students' states, actions, gaze, affect, and movement on a timeline. Our case study focuses on a specific science scenario where students learn about photosynthesis. The timeline allows us to investigate the alignment of critical learning moments identified by multimodal and interaction analysis, and uncover insights into students' temporal learning progressions.
SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents
Data Intelligence for Scientific Innovation Lab, Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心科学智能创新数据实验室);Beihang University(北京航空航天大学)
Multi-Step Reasoning for Embodied Question Answering via Tool Augmentation
Beijing Key Laboratory of Intelligent Information Technology(北京智能信息科技重点实验室);Beijing Institute of Technology(北京理工大学);Guangdong Laboratory of Machine Perception and Intelligent Computing(广东机器感知与智能计算实验室);Shenzhen MSU-BIT University(深圳MSU-BIT大学);School of Mechanical Engineering(机械工程学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Scalable Co-Optimization of Morphology and Control in Embodied Machines
Cornell University(康奈尔大学);University of Wyoming(怀俄明大学);University of Vermont(佛蒙特大学);NASA Ames/SGT Inc(美国国家航空航天局艾姆斯研究中心/SGT公司);Columbia University(哥伦比亚大学)
Embodied Instruction Following in Unknown Environments
School of Intelligent Engineering and Automation, Beijing University of Posts and Telecommunications(北京邮电大学智能工程与自动化学院);School of Electrical and Electronic Engineering, Nanyang Technological University(南洋理工大学电子与电气工程学院);Department of Automation, Tsinghua University(清华大学自动化学院)
Runaway is Ashamed, But Helpful: On the Early-Exit Behavior of Large Language Model-based Agents in Embodied Environments
Southeast University(东南大学);The University of Sydney(悉尼大学);Southeast University Shenzhen Research Institute(东南大学深圳研究院);Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学计算机与智能信息学院,深圳,中国);College of Computing and Data Science at Nanyang Technological University, Singapore 639798(南洋理工大学计算机与数据科学学院,新加坡)
Suzume-chan: Your Personal Navigator as an Embodied Information Hub
Doctoral Program in Informatics, University of Tsukuba(信息学博士项目,东京大学);Doctoral Program in Nursing Science, University of Tsukuba(护理科学博士项目,东京大学);Institute of Library, Information and Media Science, University of Tsukuba(图书馆、信息与媒体科学研究所,东京大学)