Tragic Talkers：一个用于视听机器学习研究的古典声光场数据集
Tragic Talkers: A Shakespearean Sound- and Light-Field Dataset for Audio-Visual Machine Learning Research
摘要：
3D视听制作旨在为消费者提供身临其境的互动体验。然而，忠实地再现真实世界的3D场景仍然是一项具有挑战性的任务。这部分是由于缺乏可用的数据集，使视听研究在这方面。在大多数现有的多视图数据集中，伴随的音频被忽略。同样，空间音频研究的数据集主要提供单峰内容，当包括视觉数据时，质量远远不能满足标准的生产需求。我们提出了“悲剧的谈话者”，一个视听数据集组成的摘录从“罗密欧与朱丽叶”戏剧捕捉麦克风阵列和多个共同定位的摄像机的光场视频。Tragic Talkers为基于对象的媒体（OBM）制作提供了理想的内容。它旨在涵盖各种传统的谈话场景，如独白，两人对话，以及相当大的运动和遮挡的互动，产生从总共22个不同的角度和两个16元件麦克风阵列捕获的30个序列。此外，我们还提供了语音活动标签，每个摄像机视图的2D人脸边界框，2D姿势检测关键点，演员嘴巴的3D跟踪数据以及对话transmittance。我们相信社区将受益于此数据集，因为它可以帮助多学科研究。数据集的可能用途进行了讨论。
英文摘要：
3D audio-visual production aims to deliver immersive and interactive experiences to the consumer. Yet, faithfully reproducing real-world 3D scenes remains a challenging task. This is partly due to the lack of available datasets enabling audio-visual research in this direction. In most of the existing multi-view datasets, the accompanying audio is neglected. Similarly, datasets for spatial audio research primarily offer unimodal content, and when visual data is included, the quality is far from meeting the standard production needs. We present "Tragic Talkers", an audio-visual dataset consisting of excerpts from the "Romeo and Juliet" drama captured with microphone arrays and multiple co-located cameras for light-field video. Tragic Talkers provides ideal content for object-based media (OBM) production. It is designed to cover various conventional talking scenarios, such as monologues, two-people conversations, and interactions with considerable movement and occlusion, yielding 30 sequences captured from a total of 22 different points of view and two 16-element microphone arrays. Additionally, we provide voice activity labels, 2D face bounding boxes for each camera view, 2D pose detection keypoints, 3D tracking data of the mouth of the actors, and dialogue transcriptions. We believe the community will benefit from this dataset as it can assist multidisciplinary research. Possible uses of the dataset are discussed.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
