基于多模态计算机视觉的九孔柱测试中眼手协调量化流程
A multimodal computer-vision based pipeline for eye-hand coordination quantification during the Nine-Hole Peg Test
- NYU Langone Health（纽约大学朗格尼医疗中心）
- NYU Tandon School of Engineering（纽约大学坦登工程学院）
- Kessler Foundation（凯斯勒基金会）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本研究提出多模态计算机视觉流程，在未修改的九孔柱测试中同步分析注视、手部运动与物体状态，以事件级量化眼手协调，扩展了传统完成时间指标，为神经疾病研究提供可复现平台。
AI中文摘要：
九孔柱测试（9-HPT）是多发性硬化症（MS）中广泛使用的临床手部灵巧度测量方法，但其主要结果指标——任务完成时间，仅能提供关于导致表现受损的视觉运动机制的有限信息。仪器化方法可以提供更丰富的运动学信息，但通常需要对任务进行修改或使用物理仪器，这可能改变自然的手-物体交互。我们开发了一种多模态计算机视觉流程，能够在未修改的9-HPT期间实现注视、手部运动学和物体状态的同步分析。使用Pupil Labs Invisible眼镜记录注视，使用WiLoR重建三维手部关键点，并使用定制的YOLO26x-seg模型检测和跟踪钉板、孔、钉子和收集箱。眼动追踪、手部和物体数据在视频帧级别进行时间同步，并通过帧特定的投影变换在空间上配准到共同的以钉板为中心的坐标系。该流程进一步整合了孔状态转换、手部区域分类和指尖运动学，以识别拾取（PICK）和放置（PLACE）事件，并分割手部运输和静止阶段，同时从眼动追踪系统中导出注视和扫视标签。自动检测结果经过视觉审查并在必要时进行修正。最终框架提供了同步的、事件分辨的注视行为、手部运动、物体交互和眼手协调测量，同时保持标准化的9-HPT管理。该方法将9-HPT扩展到完成时间之外，并为研究MS及其他神经系统疾病中灵巧度受损的视觉运动机制提供了可复现的平台。
英文摘要：
The Nine-Hole Peg Test (9-HPT) is a widely used clinical measure of manual dexterity in multiple sclerosis (MS), but its primary outcome, task completion time, provides limited information about the visuomotor mechanisms contributing to impaired performance. Instrumented approaches can provide richer kinematic information but often require modifications to the task or physical instrumentation that may alter natural hand-object interaction. We developed a multimodal computer-vision pipeline that enables synchronized analysis of gaze, hand kinematics, and object state during an unmodified 9-HPT. Gaze was recorded using Pupil Labs Invisible glasses, three-dimensional hand landmarks were reconstructed using WiLoR, and a custom YOLO26x-seg model detected and tracked the pegboard, holes, pegs, and collection bin. Eye-tracking, hand, and object data were temporally synchronized at the video-frame level and spatially registered to a common pegboard-centered coordinate system using frame-specific projective transformations. The pipeline further integrated hole-state transitions, hand-zone classification, and fingertip kinematics to identify PICK and PLACE events and segment hand transport and stationary phases, while fixation and saccade labels were derived from the eye-tracking system. Automated detections were visually reviewed and corrected when necessary. The resulting framework provides synchronized, event-resolved measures of gaze behavior, hand movement, object interaction, and eye-hand coordination while preserving standardized 9-HPT administration. This approach extends the 9-HPT beyond completion time and provides a reproducible platform for investigating the visuomotor mechanisms underlying dexterity impairment in MS and other neurological disorders.
