多模态查询引导的目标定位
Multimodal Query-guided Object Localization
AI总结：
提出一种基于手绘草图和文本描述的多模态查询引导目标定位方法，通过跨模态注意力机制和正交投影评分技术解决查询与自然图像间的领域差距及信息融合挑战。
摘要：
考虑在单样本查询引导的目标定位场景中，既没有目标图像也没有目标类别名称作为查询。在这种情况下，手绘草图可以作为查询的选择。然而，仅使用手绘粗略草图作为查询可能会在目标定位中产生歧义，例如，笔记本电脑的草图可能被误认为沙发。另一方面，类别的语言定义，例如“一种小型便携式计算机，小到可以放在膝上使用”，与草图查询结合，能提供更好的视觉和语义线索用于目标定位。在本研究中，我们提出了一种在具有挑战性的开放集设置下的多模态查询引导目标定位方法。具体而言，我们使用来自两种模态的查询，即手绘草图和目标描述（也称为 gloss），来执行目标定位。多模态查询引导的目标定位是一项具有挑战性的任务，特别是当查询与自然图像之间存在较大的领域差距，以及由于结合查询中存在的互补和最小信息的挑战。例如，手绘粗略草图包含目标的抽象形状信息，而文本描述通常捕获给定目标类别的部分语义信息。为了解决上述挑战，我们提出了一种新颖的跨模态注意力方案，该方案引导区域提议网络生成与输入查询相关的目标提议，以及一种新颖的基于正交投影的提议评分技术，该技术根据查询对每个提议进行评分，从而得出最终的定位结果。
英文摘要：
Consider a scenario in one-shot query-guided object localization where neither an image of the object nor the object category name is available as a query. In such a scenario, a hand-drawn sketch of the object could be a choice for a query. However, hand-drawn crude sketches alone, when used as queries, might be ambiguous for object localization, e.g., a sketch of a laptop could be confused for a sofa. On the other hand, a linguistic definition of the category, e.g., a small portable computer small enough to use in your lap" along with the sketch query, gives better visual and semantic cues for object localization. In this work, we present a multimodal query-guided object localization approach under the challenging open-set setting. In particular, we use queries from two modalities, namely, hand-drawn sketch and description of the object (also known as gloss), to perform object localization. Multimodal query-guided object localization is a challenging task, especially when a large domain gap exists between the queries and the natural images, as well as due to the challenge of combining the complementary and minimal information present across the queries. For example, hand-drawn crude sketches contain abstract shape information of an object, while the text descriptions often capture partial semantic information about a given object category. To address the aforementioned challenges, we present a novel cross-modal attention scheme that guides the region proposal network to generate object proposals relevant to the input queries and a novel orthogonal projection-based proposal scoring technique that scores each proposal with respect to the queries, thereby yielding the final localization results. ...
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
