开放词汇时空语义表示的组合语义
Compositional Semantics for Open Vocabulary Spatio-semantic Representations
AI总结：
提出潜在组合语义嵌入z*作为可查询时空语义记忆的知识表示，证明其存在性、最优性及可发现性，并引入充分相似性推理方法提升重叠语义推理性能。
摘要：
视觉语言模型（VLM）将环境感知转换为LLM可解释的视觉语言语义。然而，完成复杂任务通常需要对当前感知之外的信息进行推理。我们提出潜在组合语义嵌入z*作为可查询时空语义记忆的基于学习的原则性知识表示。我们在数学上证明z*总是可以找到，并且最优z*是任何集合Z的质心。我们推导了估计相关和不相关语义可分离性的概率界限。我们证明z*可以通过迭代梯度下降从视觉外观和单一描述中发现。我们在包括CLIP和SBERT的四个嵌入空间上实验验证了我们的发现。结果表明，z*可以表示由SBERT编码的多达10个语义，以及理想均匀分布的高维嵌入的多达100个语义。我们引入了三个具有重叠语义的新数据集，以表明在常规非重叠注释上训练的常见VLM能够发现z*。我们提出的充分相似性推理方法克服了传统推理的根本局限性，并将更高层次的重叠语义推理性能平均提高了19.63 mIoU。
英文摘要：
Vision-language models (VLMs) transform environment percepts into vision-language semantics interpretable by LLMs. However, completing complex tasks often requires reasoning about information beyond what is currently perceived. We propose latent compositional semantic embeddings z* as a principled learning-based knowledge representation for queryable spatio-semantic memories. We mathematically prove that z* can always be found, and that the optimal z* is the centroid for any set Z. We derive a probabilistic bound for estimating separability of related and unrelated semantics. We prove that z* is discoverable from visual appearance and singular descriptions by iterative gradient descent. We experimentally verify our findings on four embedding spaces including CLIP and SBERT. Our results show that z* can represent up to 10 semantics encoded by SBERT, and up to 100 semantics for ideal uniformly distributed high-dimensional embeddings. We introduce three new datasets with overlapping semantics to show that common VLMs trained on conventional nonoverlapping annotations discover z*. Our novel sufficient similarity inference method overcomes fundamental limitations of conventional inference, and improves higher-level overlapping semantic inference performance by 19.63 mIoU on average.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
