JOKR：用于无监督跨域运动重定向的联合关键点表示
JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting
摘要：
通过使用深度神经网络，视频中的无监督运动重定向任务已经取得了实质性的进展。虽然早期的工作集中在特定的对象先验，如人脸或身体，最近的工作考虑了无监督的情况。然而，当源视频和目标视频具有不同的形状时，当前方法失败。为了缓解这个问题，我们引入了JOKR -一种JOint关键点表示，它可以捕获源视频和目标视频共同的运动，而不需要任何对象先验或数据汇集。通过采用域混淆项，我们强制两个视频的无监督关键点表示是不可区分的。这鼓励了两个域所共有的运动部分之间的分离，以及它们独特的外观和运动，使得能够生成捕获一个域的运动同时描绘另一个域的风格的视频。为了实现对象具有不同比例或方向的情况，我们在JOKR之间应用学习的仿射变换。这增强了仿射不变的表示，并在实践中扩大了各种可能的重定向对。这种几何驱动的表示能够实现进一步的直观控制，例如时间一致性和手动编辑。通过全面的实验，我们证明了我们的方法的适用性，不同的挑战性的跨域视频对。我们评估我们的方法定性和定量，并证明我们的方法处理各种跨域的情况下，如不同的动物，不同的花，和人类。我们还展示了优越的时间一致性和视觉质量相比，国家的最先进的替代品，通过统计指标和用户研究。源代码和视频可以在https://rmokady.github.io/JOKR/上找到。
英文摘要：
The task of unsupervised motion retargeting in videos has seen substantial advancements through the use of deep neural networks. While early works concentrated on specific object priors such as a human face or body, recent work considered the unsupervised case. When the source and target videos, however, are of different shapes, current methods fail. To alleviate this problem, we introduce JOKR - a JOint Keypoint Representation that captures the motion common to both the source and target videos, without requiring any object prior or data collection. By employing a domain confusion term, we enforce the unsupervised keypoint representations of both videos to be indistinguishable. This encourages disentanglement between the parts of the motion that are common to the two domains, and their distinctive appearance and motion, enabling the generation of videos that capture the motion of the one while depicting the style of the other. To enable cases where the objects are of different proportions or orientations, we apply a learned affine transformation between the JOKRs. This augments the representation to be affine invariant, and in practice broadens the variety of possible retargeting pairs. This geometry-driven representation enables further intuitive control, such as temporal coherence and manual editing. Through comprehensive experimentation, we demonstrate the applicability of our method to different challenging cross-domain video pairs. We evaluate our method both qualitatively and quantitatively, and demonstrate that our method handles various cross-domain scenarios, such as different animals, different flowers, and humans. We also demonstrate superior temporal coherency and visual quality compared to state-of-the-art alternatives, through statistical metrics and a user study. Source code and videos can be found at https://rmokady.github.io/JOKR/ .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
