Lester：通过视频对象分割与跟踪实现转描动画
Lester: rotoscope animation through video object segmentation and tracking
AI总结：
本文提出Lester方法，利用SAM和DeAOT进行视频对象分割与跟踪，结合Douglas-Peucker算法简化轮廓，自动生成复古风格2D动画，在时序一致性上优于扩散模型且比3D姿态估计更实用。
摘要：
本文介绍了Lester，一种从视频自动合成复古风格2D动画的新方法。该方法将这一挑战主要作为对象分割和跟踪问题来处理。视频帧使用Segment Anything Model (SAM)进行处理，生成的掩码通过DeAOT（一种用于半监督视频对象分割的分层传播方法）在后续帧中进行跟踪。掩码轮廓的几何形状通过Douglas-Peucker算法进行简化。最后，可选择性地添加面部特征、像素化和基础阴影效果。结果表明，该方法具有出色的时序一致性，能够正确处理具有不同姿势和外观、动态镜头、局部镜头和多样背景的视频。与基于扩散模型的视频到视频翻译流程相比，所提出的方法提供了一种更简单且确定性的途径，后者存在时序一致性问题，且无法很好地处理像素化和模式化输出。该方法也比基于3D人体姿态估计的技术实用得多，后者需要定制的3D模型，且在可处理的场景类型上非常有限。
英文摘要：
This article introduces Lester, a novel method to automatically synthetise retro-style 2D animations from videos. The method approaches the challenge mainly as an object segmentation and tracking problem. Video frames are processed with the Segment Anything Model (SAM) and the resulting masks are tracked through subsequent frames with DeAOT, a method of hierarchical propagation for semi-supervised video object segmentation. The geometry of the masks' contours is simplified with the Douglas-Peucker algorithm. Finally, facial traits, pixelation and a basic shadow effect can be optionally added. The results show that the method exhibits an excellent temporal consistency and can correctly process videos with different poses and appearances, dynamic shots, partial shots and diverse backgrounds. The proposed method provides a more simple and deterministic approach than diffusion models based video-to-video translation pipelines, which suffer from temporal consistency problems and do not cope well with pixelated and schematic outputs. The method is also much most practical than techniques based on 3D human pose estimation, which require custom handcrafted 3D models and are very limited with respect to the type of scenes they can process.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
