Talking Points：描述与定位像素
Talking Points: Describing and Localizing Pixels
AI总结：
提出一个由点描述器和点定位器组成的像素级关键点定位框架，利用合成数据集LlamaPointInPart和GRPO优化，实现语言引导的精确像素定位，性能优于基线。
摘要：
视觉-语言模型在跨模态理解方面取得了显著成功。然而，这些模型仍局限于对象级或区域级的定位，缺乏通过自然语言进行像素级精确关键点理解的能力。我们提出了一种用于像素级定位的新框架。该框架由两个互补的组件组成：一个点描述器（Point Descriptor），用于生成单个关键点的丰富、上下文相关的描述；以及一个点定位器（Point Localizer），用于根据这些描述回归出精确的像素坐标。与以往依赖模板化提示或关键点名称的工作不同，我们的方法生成自由形式、从粗到细的描述，将关键点置于其视觉上下文中。由于没有现成的数据集来训练这样的系统，我们引入了LlamaPointInPart，这是一个精心策划的数据集，包含20K+个图像-关键点-描述三元组，由多个视觉-语言模型合成，捕捉从场景级上下文到关键点周围视觉特征的多尺度信息。为了实现跨类别泛化，我们通过GRPO在AP-10K上优化点描述器，使用冻结的点定位器作为奖励模型，以生成最大化定位准确性的描述。为了评估我们的结果，我们建立了一个新的评估协议。我们不是将方法生成的文本描述与真实值进行比较，而是使用定位器来确定预测点与真实点之间的接近程度。实验表明，在LlamaPointInPart上，我们的方法优于基线模型。我们框架的双向特性应能促进未来在关键点引导的图像理解和语言引导的精确定位中的应用。我们的代码和数据集可在https://github.com/matanr/Talking_Points公开获取。
英文摘要：
Vision-language models have achieved remarkable success in cross-modal understanding. Yet, these models remain limited to object-level or region-level grounding, lacking the capability for pixel-precise keypoint comprehension through natural language. We introduce a novel framework for pixel level grounding. The framework consists of two complementary components: a Point Descriptor that generates rich, contextual descriptions of individual keypoints, and a Point Localizer that regresses precise pixel coordinates from these descriptions. Unlike prior work that relies on templated prompts or keypoint names, our approach produces free-form, coarse-to-fine descriptions that situate keypoints within their visual context. Since there is no available dataset to train such a system, we introduce LlamaPointInPart, a carefully curated dataset of 20K+ image-keypoint-description triplets synthesized from multiple vision-language models, capturing multi-scale information from scene-level context to visual features around the keypoint. For cross-category generalization, we optimize the Point Descriptor on AP-10K via GRPO, using the frozen Point Localizer as a reward model to produce descriptions that maximize localization accuracy. To evaluate our results we establish a new evaluation protocol. Instead of comparing the text description produced by our method to the ground truth, we use the localizer to determine how close is the predicted point generated to the ground truth point. Experiments demonstrate superior performance compared to baseline models on LlamaPointInPart.The bidirectional nature of our framework should enable future applications in both keypoint-guided image understanding and language-guided precise localization. Our code and dataset are publicly available at https://github.com/matanr/Talking_Points.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
