探索与描述：3D环境中的具身视觉描述
Explore and Tell: Embodied Visual Captioning in 3D Environments
AI总结：
针对单张图像视角不佳阻碍细粒度场景理解的问题，提出具身视觉描述任务与级联具身描述模型CaBOT，构建ET-Cap数据集支撑研究，实验证明模型表现优于基线。
摘要：
尽管当前的视觉描述模型已取得令人瞩目的性能，但它们通常假设图像拍摄效果良好且能提供场景的完整视图。然而在现实场景中，单张图像可能无法提供良好的视角，这会阻碍细粒度的场景理解。为克服这一局限，我们提出了一项名为Embodied Captioning（具身描述）的新任务，该任务为视觉描述模型配备导航能力，使其能够主动探索场景，降低次优视角带来的视觉歧义。具体而言，从一个随机视角出发，智能体必须在环境中导航，从不同视角收集信息，并生成一段描述场景中所有物体的全面段落。为支撑这项任务，我们使用Kubric模拟器构建了ET-Cap数据集，该数据集包含10K个物体杂乱摆放的3D场景，且每个场景配有三段标注段落。我们提出了Cascade Embodied Captioning模型（CaBOT，级联具身描述模型）来解决这一任务，该模型由导航器和描述生成器组成。导航器预测在环境中应采取的动作，而描述生成器则基于整个导航轨迹生成段落描述。大量实验表明，我们的模型的表现优于其他精心设计的基线模型。我们的数据集、代码和模型可在https://aim3-ruc.github.io/ExploreAndTell获取。
英文摘要：
While current visual captioning models have achieved impressive performance, they often assume that the image is well-captured and provides a complete view of the scene. In real-world scenarios, however, a single image may not offer a good viewpoint, hindering fine-grained scene understanding. To overcome this limitation, we propose a novel task called Embodied Captioning, which equips visual captioning models with navigation capabilities, enabling them to actively explore the scene and reduce visual ambiguity from suboptimal viewpoints. Specifically, starting at a random viewpoint, an agent must navigate the environment to gather information from different viewpoints and generate a comprehensive paragraph describing all objects in the scene. To support this task, we build the ET-Cap dataset with Kubric simulator, consisting of 10K 3D scenes with cluttered objects and three annotated paragraphs per scene. We propose a Cascade Embodied Captioning model (CaBOT), which comprises of a navigator and a captioner, to tackle this task. The navigator predicts which actions to take in the environment, while the captioner generates a paragraph description based on the whole navigation trajectory. Extensive experiments demonstrate that our model outperforms other carefully designed baselines. Our dataset, codes and models are available at https://aim3-ruc.github.io/ExploreAndTell.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
