arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.02762cs.CV
收藏

STRIVE:视频中的场景文本替换

STRIVE: Scene Text Replacement In Videos

NEC Laboratories, America(美国NEC实验室);Palo Alto Research Center(帕洛阿尔托研究中心);Amazon(亚马逊);Stanford University(斯坦福大学)

Vijay Kumar B G, Jeyasri Subramanian, Varnith Chordia, Eugene Bart, Shaobo Fang, Kelly Guan, Raja Bala

摘要:

我们提出了使用深度风格转移和学习的光度变换来替换视频中的场景文本。在静态图像文本替换的最新进展的基础上,我们提出了在保留原始视频的外观和运动特征的同时改变文本的扩展。与静态图像文本替换的问题相比,我们的方法解决了视频引入的额外挑战,即由变化的照明引起的效果,运动模糊,相机对象姿态随时间的不同变化,以及时间一致性的保持。我们把这个问题分成三个步骤。首先,使用时空变换器网络将所有帧中的文本归一化为正面姿势。其次,使用最先进的静止图像文本替换方法在单个参考帧中替换文本。最后,使用一种新颖的学习图像变换网络将新文本从参考帧转移到剩余帧,该网络以时间一致的方式捕获照明和模糊效果。在合成和具有挑战性的真实视频上的结果显示了逼真的文本传输,具有竞争力的定量和定性性能,以及相对于替代方案的优越推理速度。我们介绍了新的合成和现实世界的数据集与配对的文本对象。据我们所知,这是深度视频文本替代的第一次尝试。

英文摘要:

We propose replacing scene text in videos using deep style transfer and learned photometric transformations.Building on recent progress on still image text replacement,we present extensions that alter text while preserving the appearance and motion characteristics of the original video.Compared to the problem of still image text replacement,our method addresses additional challenges introduced by video, namely effects induced by changing lighting, motion blur, diverse variations in camera-object pose over time,and preservation of temporal consistency. We parse the problem into three steps. First, the text in all frames is normalized to a frontal pose using a spatio-temporal trans-former network. Second, the text is replaced in a single reference frame using a state-of-art still-image text replacement method. Finally, the new text is transferred from the reference to remaining frames using a novel learned image transformation network that captures lighting and blur effects in a temporally consistent manner. Results on synthetic and challenging real videos show realistic text trans-fer, competitive quantitative and qualitative performance,and superior inference speed relative to alternatives. We introduce new synthetic and real-world datasets with paired text objects. To the best of our knowledge this is the first attempt at deep video text replacement.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑