MP-Mat：基于多平面表示的3D与实例感知人体抠图与编辑框架
MP-Mat: A 3D-and-Instance-Aware Human Matting and Editing Framework with Multiplane Representation
AI总结：
提出MP-Mat框架，通过几何与实例双层面多平面表示实现3D感知的多人抠图，并支持高效可控的图像编辑，在零样本编辑任务中大幅超越专业方法。
摘要：
人体实例抠图旨在估计图像中每个人体实例的alpha遮罩，这一任务具有挑战性，因为在复杂情况下，需要将属于多个实例的混合像素沿着毛发和细薄边界结构进行分离，容易失败。在本工作中，我们通过引入MP-Mat来解决这一问题，这是一个新颖的具有多平面表示的3D与实例感知抠图框架，其中多平面概念从两个不同视角设计：场景几何层面和实例层面。具体来说，我们首先构建特征级多平面表示，基于深度差异将场景分割为多个平面。这种方法使场景表示具有3D感知性，并可作为根据3D位置不同来分离实例的有效线索，从而提高可解释性和边界处理能力，尤其在遮挡区域。然后，我们引入另一种多平面表示，从实例层面视角分割场景，并用遮罩和颜色来表示每个实例。我们还将背景视为一个特殊实例，这一点常被现有方法忽略。这种实例级表示有助于前景和背景内容感知，并可用于图像编辑等其他下游任务。一旦构建完成，该表示可被重复使用，以实现高效的可控实例级图像编辑。大量实验验证了MP-Mat在抠图任务中的明显优势。我们还展示了其在图像编辑任务中的优越性，这是现有以抠图为主的方法探索不足的领域，我们的方法在零样本推理下甚至以较大幅度超越了经过训练的专业图像编辑技术。代码已在https://github.com/JiaoSiyi/MPMat.git开源。
英文摘要：
Human instance matting aims to estimate an alpha matte for each human instance in an image, which is challenging as it easily fails in complex cases requiring disentangling mingled pixels belonging to multiple instances along hairy and thin boundary structures. In this work, we address this by introducing MP-Mat, a novel 3D-and-instance-aware matting framework with multiplane representation, where the multiplane concept is designed from two different perspectives: scene geometry level and instance level. Specifically, we first build feature-level multiplane representations to split the scene into multiple planes based on depth differences. This approach makes the scene representation 3D-aware, and can serve as an effective clue for splitting instances in different 3D positions, thereby improving interpretability and boundary handling ability especially in occlusion areas. Then, we introduce another multiplane representation that splits the scene in an instance-level perspective, and represents each instance with both matte and color. We also treat background as a special instance, which is often overlooked by existing methods. Such an instance-level representation facilitates both foreground and background content awareness, and is useful for other down-stream tasks like image editing. Once built, the representation can be reused to realize controllable instance-level image editing with high efficiency. Extensive experiments validate the clear advantage of MP-Mat in matting task. We also demonstrate its superiority in image editing tasks, an area under-explored by existing matting-focused methods, where our approach under zero-shot inference even outperforms trained specialized image editing techniques by large margins. Code is open-sourced at https://github.com/JiaoSiyi/MPMat.git}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
