TrackFish3D：基于多视角视频的鱼群自监督3D跟踪
TrackFish3D: Self-Supervised 3D Tracking of Schooling Fish from Multi-view Videos
浏览论文内容
中文总结 AI 辅助
TrackFish3D提出几何驱动的自监督框架，利用多视角几何和对比学习实现鱼群密集3D跟踪，无需身份标注，显著提升跟踪准确率并泛化至鸟类跟踪。
中文摘要 AI 辅助
量化鱼类集体行为需要精确的轨迹，然而由于频繁的遮挡、视觉上相似的个体以及长期缺乏身份标注，多视角3D跟踪仍然具有挑战性。我们提出了TrackFish3D，一个几何驱动的自监督框架，用于密集多相机鱼群3D跟踪。TrackFish3D不依赖于基于外观的重新识别或手动标注的身份，而是将校准后的多视角几何转化为监督信号：三角测量和重投影一致性提供伪关联，而几何编码器和全局关联变换器学习每个帧内的全对全跨视角对应关系。为了使这些关联具有身份感知性，TrackFish3D引入了一个自监督对比目标，在嵌入空间中分离共同可见的个体，并配备了一个时间预测器，以保持身份并桥接跨帧的短暂遮挡。由此产生的模型在未标注的视频上训练一次，并直接应用于未见过的测试视频，无需跨视角身份标签、时间标注、3D真值、外观特征或测试时优化。在我们的基准测试中，TrackFish3D将3D多目标跟踪准确率从最强基线的87.7%提升到95.8%。在3D-ZeF斑马鱼基准上，它达到了81.1%的MOTA，而最佳几何基线为77.4%。TrackFish3D还能泛化到鱼类之外，在真实世界的鸟类跟踪上取得了强劲的结果。
英文摘要
Quantifying collective fish behavior requires accurate trajectories, yet multi-view 3D tracking remains challenging due to frequent occlusions, visually similar individuals, and the long-standing scarcity of identity annotations. We present TrackFish3D, a geometry-driven self-supervised framework for dense multi-camera 3D tracking of schooling fish. Instead of relying on appearance-based re-identification or manually annotated identities, TrackFish3D turns calibrated multi-view geometry into supervision: triangulation and reprojection consistency provide pseudo-associations, while a geometric encoder and global association transformer learn all-to-all cross-view correspondence within each frame. To make these associations identity-aware, TrackFish3D introduces a self-supervised contrastive objective that separates co-visible individuals in the embedding space, together with a temporal predictor that preserves identities and bridges short occlusions across frames. The resulting model is trained once on unlabeled footage and applied directly to unseen test videos, requiring no cross-view identity labels, temporal annotations, 3D ground truth, appearance features, or test-time optimization. On our benchmark, TrackFish3D improves 3D Multi-Object Tracking Accuracy from 87.7% for the strongest baseline to 95.8%. On the 3D-ZeF zebrafish benchmark, it achieves 81.1% MOTA, compared with 77.4% for the best geometric baseline. TrackFish3D also generalizes beyond fish, achieving strong results on real-world bird tracking.
发表机构
- The University of Hong Kong（香港大学）
- Centre for Transformative Garment Production（转型服装生产中心）
- Hong Kong University of Science and Technology（香港科技大学）
- Texas A&M University（德克萨斯A&M大学）
机构由 AI 辅助整理，请以论文原文为准。
