证词的新维度：用反射场范例重新照亮视频
A New Dimension in Testimony: Relighting Video with Reflectance Field Exemplars
摘要：
我们提出了一种基于学习的方法来估计4D反射场的一个人给定的视频片段照明下，一个平面照明的环境下，同一主题。对于训练数据，我们一次使用一盏灯来照亮对象，并以各种姿势和视角捕获反射场数据。我们估计输入视频片段的照明环境，并使用主体的反射场来创建由输入照明环境照亮的主体的合成图像。然后，我们训练一个深度卷积神经网络，从合成图像中回归反射场。我们还使用了一个可微分渲染器，通过将relit图像与输入视频帧进行匹配来为网络提供反馈。这种半监督训练方案允许神经网络处理数据集中看不见的姿势以及补偿照明估计误差。我们评估我们的方法对真实的大屠杀幸存者的视频片段，并表明我们的方法优于国家的最先进的方法在现实主义和速度。
英文摘要：
We present a learning-based method for estimating 4D reflectance field of a person given video footage illuminated under a flat-lit environment of the same subject. For training data, we use one light at a time to illuminate the subject and capture the reflectance field data in a variety of poses and viewpoints. We estimate the lighting environment of the input video footage and use the subject's reflectance field to create synthetic images of the subject illuminated by the input lighting environment. We then train a deep convolutional neural network to regress the reflectance field from the synthetic images. We also use a differentiable renderer to provide feedback for the network by matching the relit images with the input video frames. This semi-supervised training scheme allows the neural network to handle unseen poses in the dataset as well as compensate for the lighting estimation error. We evaluate our method on the video footage of the real Holocaust survivors and show that our method outperforms the state-of-the-art methods in both realism and speed.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
