arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.00837cs.CV
收藏

AniMer:使用族感知 Transformer 的动物姿态与形状估计

AniMer: Animal Pose and Shape Estimation Using Family Aware Transformer

Southern University of Science and Technology(南方科技大学);China Mobile Research Institute(中国移动研究院);The Hong Kong University of Science and Technology(香港科技大学);The University of Hong Kong(香港大学);Tsinghua University(清华大学)

Jin Lyu, Tianyi Zhu, Yi Gu, Li Lin, Pujin Cheng, Yebin Liu, Xiaoying Tang, Liang An

AI总结:

AniMer 提出族感知 Transformer 与动物族监督对比学习,并构建大规模合成数据集 CtrlAni3D,以提升多物种四足动物的姿态与形状估计精度,在多个数据集上优于现有方法。

摘要:

动物行为和生物力学的定量分析需要跨物种的精确动物姿态与形状估计,这对动物福利和生物学研究具有重要意义。然而,以往方法较小的网络容量和有限的多物种数据集使这一问题尚未得到充分探索。为此,本文提出 AniMer,利用族感知 Transformer 估计动物姿态与形状,提升多种四足动物科的重建精度。AniMer 的一个关键思想是集成了高容量 Transformer 骨干网络和动物族监督对比学习方案,在单一框架内统一对不同四足动物形状的判别性理解。为了实现有效训练,我们汇总了大多数可用的开源四足动物数据集,包括带有 3D 或 2D 标签的数据。为了提高 3D 标注数据的多样性,我们引入 CtrlAni3D,这是一个通过新的基于扩散的条件图像生成流程创建的大规模合成数据集。CtrlAni3D 包含约 10k 张具有像素对齐 SMAL 标签的图像。我们总共获得 41.3k 张用于训练和验证的标注图像。因此,族感知 Transformer 网络与大规模数据集的结合使 AniMer 不仅在 Animal3D 和 CtrlAni3D 等 3D 数据集上优于现有方法,而且在分布外的 Animal Kingdom 数据集上也表现更优。消融研究进一步证明了我们的网络设计和 CtrlAni3D 在提升 AniMer 面向自然场景应用性能方面的有效性。AniMer 的项目主页为 https://luoxue-star.github.io/AniMer_project_page/。

英文摘要:

Quantitative analysis of animal behavior and biomechanics requires accurate animal pose and shape estimation across species, and is important for animal welfare and biological research. However, the small network capacity of previous methods and limited multi-species dataset leave this problem underexplored. To this end, this paper presents AniMer to estimate animal pose and shape using family aware Transformer, enhancing the reconstruction accuracy of diverse quadrupedal families. A key insight of AniMer is its integration of a high-capacity Transformer-based backbone and an animal family supervised contrastive learning scheme, unifying the discriminative understanding of various quadrupedal shapes within a single framework. For effective training, we aggregate most available open-sourced quadrupedal datasets, either with 3D or 2D labels. To improve the diversity of 3D labeled data, we introduce CtrlAni3D, a novel large-scale synthetic dataset created through a new diffusion-based conditional image generation pipeline. CtrlAni3D consists of about 10k images with pixel-aligned SMAL labels. In total, we obtain 41.3k annotated images for training and validation. Consequently, the combination of a family aware Transformer network and an expansive dataset enables AniMer to outperform existing methods not only on 3D datasets like Animal3D and CtrlAni3D, but also on out-of-distribution Animal Kingdom dataset. Ablation studies further demonstrate the effectiveness of our network design and CtrlAni3D in enhancing the performance of AniMer for in-the-wild applications. The project page of AniMer is https://luoxue-star.github.io/AniMer_project_page/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑