发表机构
School of Computing, Binghamton University; SUNY Upstate Medical University（宾汉姆顿大学计算学院; 纽约州立大学上州医科大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出ReMiX-MAE框架，构建SMP数据集，在仅用RGB的疼痛评估任务中，其性能优于仅用RGB的基线，且在数据有限的临床场景中迁移能力更优。
AI 中文摘要
临床自动化疼痛评估受限于两类问题：一是临床标注的面部视频数据稀缺，且标签质量差（常为序列级自我报告）；二是RGB信号中的疼痛线索可能细微或接近中性，而热成像与深度信号虽具信息价值却难以常规部署。为解决这些挑战，本文提出ReMiX-MAE（Reconstructing Missing Channel Cross-Modal Masked Autoencoder，即重构缺失通道跨模态掩码自编码器），这是一种自监督多模态掩码预训练框架，可从同步的RGB、热成像及深度视频中学习可迁移的面部表征，并明确训练对缺失模态的鲁棒性，从而支持仅基于RGB的部署。为填补临床面部疼痛数据的缺口（需具备视频级自我报告与纵向治疗轨迹），本文构建了交感介导疼痛（Sympathetic Mediated Pain，SMP）数据集，包含多次就诊的前后配对记录。在仅用RGB部署的场景下，本文通过直接特征提取与从RGB解码的伪多模态特征两种方式评估ReMiX-MAE，结果显示其在SMP数据集上始终优于仅用RGB的掩码自编码器基线，且伪多模态特征在具有挑战性的五分类场景中能带来额外性能提升；在外部数据集上，ReMiX-MAE相比仅用RGB的基线表现出更鲁棒、标签效率更高的迁移能力，凸显其在数据有限的临床场景中的优势。
英文摘要
Automated pain assessment in real clinics is limited by scarce clinically grounded facial video data with weak labels (often sequence-level self-report) and by the fact that pain cues can be subtle or near-neutral in RGB, while thermal and depth signals are informative yet impractical to deploy routinely. To address these challenges, we propose ReMiX-MAE (Reconstructing Missing Channel Cross-Modal Masked Autoencoder), a self-supervised multimodal masked pretraining framework that learns transferable facial representations from synchronized RGB, thermal, and depth videos and explicitly trains robustness to missing modalities, enabling RGB-only deployment. To fill the gap of clinically grounded facial pain data with video-level self-report and longitudinal treatment trajectories, we collect the Sympathetic Mediated Pain (SMP) dataset with paired pre- and post-recordings across multiple visits. Under RGB-only deployment, we evaluate ReMiX-MAE using both direct feature extraction and pseudo-multimodal features decoded from RGB. ReMiX-MAE consistently outperforms an RGB-only masked autoencoder baseline on SMP, with pseudo-multimodal features providing additional gains in the challenging five-class setting. Across external datasets, ReMiX-MAE further shows more robust and label-efficient transfer than RGB-only baselines, highlighting its advantage in data-limited clinical settings.
