Video2SwimFish：一种从真实鱼类视频重建可控鱼类模型与生物运动学的自动化流水线

中文摘要 AI 辅助

我们提出了Video2SwimFish，一个用于从真实鱼类视频构建可控鱼类资产的自动化流水线和基准，服务于水下具身人工智能。给定一条鱼的同步多视角视频，该流水线从VLM选择的规范帧重建度量缩放的可变形网格，通过VLM演员-评论家循环生成适应个体形态的内部关节，并从观察到的鱼中线曲率中提取生物运动流形（BLM）。BLM提供了一个由真实鱼类运动约束的低维动作空间，使得可以为每条重建的鱼学习个体游泳策略。我们发布了两个配对数据集：来自6个物种的120条鱼的同步顶视和前视记录，以及从中衍生的可控资产和个体游泳策略。由于每个资产都与其来源动物相关联，该数据集支持一个基准，该基准不仅评估任务成功，还评估在轨迹跟随、从视频的无奖励游泳行为迁移以及下游案例研究（模拟BlueROV水下机器人捕获其中一个资产）中对个体在轨迹形状、身体曲率和尾拍频率方面的保真度。我们发现任务成功和运动保真度不一定同时提高：实现最高任务完成度的方法不是实现最高运动保真度的方法，我们将忠实再现个体动物运动学视为社区的一个开放挑战。项目网站：此https URL。

英文摘要

We present Video2SwimFish, an automated pipeline and benchmark for building controllable fish assets from real-fish videos for underwater embodied AI. Given synchronized multi-view videos of an individual fish, the pipeline reconstructs a metrically scaled deformable mesh from a VLM-selected canonical frame, generates internal articulation adapted to that individual's morphology through a VLM actor-critic loop, and extracts a Biological Locomotion Manifold (BLM) from the fish's observed midline curvature. The BLM provides a low-dimensional action space bounded by real-fish motion, enabling an individual swimming policy to be learned for each reconstructed fish. We release two paired datasets: synchronized top- and front-view recordings of 120 individual fish across 6 species, and the controllable assets and individual swimming policies derived from them. Because every asset is tied to the animal it came from, the dataset supports a benchmark that evaluates locomotion learning not only on task success but on fidelity to that individual in trajectory shape, body curvature, and tail-beat frequency, across trajectory following, reward-free swimming behavior transfer from video, and a downstream case study in which a simulated BlueROV underwater robot captures one of the assets. We find that task success and locomotion fidelity do not necessarily improve together: the method achieving the highest task completion is not the method achieving the highest locomotion fidelity, and we identify faithful reproduction of individual animal locomotion as an open challenge for the community. Project website: https://hangongchen.github.io/video2swimfish-web/.

发表机构

• North Carolina A&T State University（北卡罗来纳农工州立大学）
• University of Maryland, College Park（马里兰大学帕克分校）

机构由 AI 辅助整理，请以论文原文为准。