BEAST3D: 通过高斯泼溅从多视角视频进行动物行为分析与神经编码
BEAST3D: Animal behavioral analysis and neural encoding from multi-view video via Gaussian splatting
AI总结：
提出BEAST3D自监督预训练框架，利用未标注的多视角视频通过3D高斯泼溅重建和动物分割，学习3D视觉表征，有效应用于新视角合成、多视角姿态估计和神经编码。
摘要：
多视角视频记录越来越多地用于捕捉实验环境中动物的3D运动，但从这些记录中提取丰富的3D表示仍然具有挑战性。有监督的姿态估计需要大量手动标注，而在通用场景数据集上训练的通用3D重建模型无法适用于实验室实验的专业图像和稀疏视角设置。我们通过BEAST3D解决了这些限制，这是一个自监督预训练框架，从未标注的、已校准的多视角视频中学习3D视觉表示。BEAST3D使用视觉变换器预测3D高斯泼溅，通过可微渲染重建保留视角，同时将动物从背景中分割出来。BEAST3D通过直接以已知相机参数为条件，仅用四个视角即可重建3D结构——这与通用模型不同，后者必须从实验室环境中很少有的密集重叠视角估计相机几何。通过在四个物种上的全面评估，我们证明BEAST3D产生丰富的、视角不变的特征，这些特征有效地迁移到三个下游任务：新视角合成（验证了学习到的3D表示的质量）、多视角姿态估计（提供了行为分析中广泛使用的稀疏关键点轨迹）和神经编码（将3D行为特征与同时记录的神经活动相关联）。因此，BEAST3D建立了一个利用现代多视角实验室记录中3D结构的行为分析多功能框架。
英文摘要：
Multi-view video recordings are increasingly used to capture the 3D movements of animals in experimental settings, yet extracting rich 3D representations from these recordings remains challenging. Supervised pose estimation requires extensive manual annotation, while general-purpose 3D reconstruction models trained on generic scene datasets fail on the specialized imagery and sparse-view setting of laboratory experiments. We address these limitations with BEAST3D, a self-supervised pretraining framework that learns 3D visual representations from unlabeled, calibrated multi-view video. BEAST3D uses a vision transformer to predict 3D Gaussian splats that reconstruct held-out views through differentiable rendering, while simultaneously segmenting the animal from the background. BEAST3D reconstructs 3D structure with as few as four views by conditioning directly on known camera parameters--unlike general-purpose models, which must estimate camera geometry from dense overlapping viewpoints that are seldom available in lab settings. Through comprehensive evaluation across four species, we demonstrate that BEAST3D produces rich, viewpoint-invariant features that transfer effectively to three downstream tasks: novel view synthesis, which validates the quality of the learned 3D representations; multi-view pose estimation, which provides the sparse keypoint trajectories widely used in behavioral analysis; and neural encoding, which relates 3D behavioral features to simultaneously recorded neural activity. BEAST3D thus establishes a versatile framework for behavioral analysis that leverages 3D structure in modern multi-view laboratory recordings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
