从孤立岛屿到盘古大陆：统一人类动作理解的语义空间
From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understanding
AI总结：
本文提出统一的动作语义空间，将多个异构数据集对齐为统一数据库“盘古大陆”，并设计新模型实现物理空间到语义空间的映射，显著提升动作理解与迁移学习性能。
摘要：
动作理解长期以来备受关注，它可以被形式化为从物理空间到语义空间的映射。通常，研究者根据各自特定的选择构建数据集来定义类别，并分别推动基准的发展。由于语义鸿沟和不同的类别粒度（例如，数据集A中的“做家务”与数据集B中的“洗盘子”），这些数据集彼此不兼容，如同“孤立岛屿”。我们认为需要一个更具原则性的语义空间来集中社区力量，并联合使用所有数据集以追求可泛化的动作学习。为此，我们基于动词分类层级设计了一个结构化的动作语义空间，覆盖大量动作。通过将先前数据集的类别对齐到我们的语义空间，我们将（图像/视频/骨架/动作捕捉）数据集汇聚到一个统一标签体系下的统一数据库中，即将“孤立岛屿”连接成“盘古大陆”。相应地，我们提出了一种从物理空间映射到语义空间的新模型，以充分利用盘古大陆。在大量实验中，我们的新系统展现出显著优势，尤其是在迁移学习方面。我们的代码和数据将在 https://mvig-rhos.com/pangea 公开。
英文摘要：
Action understanding has attracted long-term attention. It can be formed as the mapping from the physical space to the semantic space. Typically, researchers built datasets according to idiosyncratic choices to define classes and push the envelope of benchmarks respectively. Datasets are incompatible with each other like "Isolated Islands" due to semantic gaps and various class granularities, e.g., do housework in dataset A and wash plate in dataset B. We argue that we need a more principled semantic space to concentrate the community efforts and use all datasets together to pursue generalizable action learning. To this end, we design a structured action semantic space given verb taxonomy hierarchy and covering massive actions. By aligning the classes of previous datasets to our semantic space, we gather (image/video/skeleton/MoCap) datasets into a unified database in a unified label system, i.e., bridging "isolated islands" into a "Pangea". Accordingly, we propose a novel model mapping from the physical space to semantic space to fully use Pangea. In extensive experiments, our new system shows significant superiority, especially in transfer learning. Our code and data will be made public at https://mvig-rhos.com/pangea.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
