AutoFoley：使用深度学习为无声视频人工合成同步音轨
AutoFoley: Artificial Synthesis of Synchronized Sound Tracks for Silent Videos with Deep Learning
摘要：
在电影制作中，福利艺术家负责创建一个覆盖配乐，帮助电影为观众带来活力。这就要求艺术家首先确定声音，这将提高听众的经验，从而加强导演的意图，为给定的场景。在本文中，我们介绍了AutoFoley，一个全自动的深度学习工具，可用于合成视频的代表性音轨。AutoFoley可用于没有与视频相关联的相应音频文件的应用程序中，或者需要识别关键场景并提供合成的强化音轨的情况下。合成声道的一个重要性能标准是与输入视频时间同步，这提供了合成声音的逼真和可信的描绘。与现有的声音预测和生成架构不同，我们的算法能够通过将插值技术和时间关系网络（TRN）结合起来，精确识别快速移动的视频剪辑中的动作和帧间关系。我们采用了一个强大的多尺度递归神经网络（RNN）与卷积神经网络（CNN），以更好地理解复杂的输入输出关联随着时间的推移。为了评估AutoFoley，我们创建并引入了一个大规模的音频视频数据集，其中包含电影中经常用作Foley效果的各种声音。我们的实验表明，合成的声音逼真地描绘了准确的时间同步的相关视觉输入。AutoFoley的人类定性测试显示，超过73%的测试对象认为生成的音轨是原始的，这是声音合成中跨模态研究的一个值得注意的进步。
英文摘要：
In movie productions, the Foley Artist is responsible for creating an overlay soundtrack that helps the movie come alive for the audience. This requires the artist to first identify the sounds that will enhance the experience for the listener thereby reinforcing the Directors's intention for a given scene. In this paper, we present AutoFoley, a fully-automated deep learning tool that can be used to synthesize a representative audio track for videos. AutoFoley can be used in the applications where there is either no corresponding audio file associated with the video or in cases where there is a need to identify critical scenarios and provide a synthesized, reinforced soundtrack. An important performance criterion of the synthesized soundtrack is to be time-synchronized with the input video, which provides for a realistic and believable portrayal of the synthesized sound. Unlike existing sound prediction and generation architectures, our algorithm is capable of precise recognition of actions as well as inter-frame relations in fast moving video clips by incorporating an interpolation technique and Temporal Relationship Networks (TRN). We employ a robust multi-scale Recurrent Neural Network (RNN) associated with a Convolutional Neural Network (CNN) for a better understanding of the intricate input-to-output associations over time. To evaluate AutoFoley, we create and introduce a large scale audio-video dataset containing a variety of sounds frequently used as Foley effects in movies. Our experiments show that the synthesized sounds are realistically portrayed with accurate temporal synchronization of the associated visual inputs. Human qualitative testing of AutoFoley show over 73% of the test subjects considered the generated soundtrack as original, which is a noteworthy improvement in cross-modal research in sound synthesis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
