弥合粗粒与细粒识别：一种混合方法用于交互教育游戏中的开放多粒度物体识别
Bridging Coarse and Fine Recognition: A Hybrid Approach for Open-Ended Multi-Granularity Object Recognition in Interactive Educational Games
AI总结：
本文提出HyMOR混合框架，结合MLLM和CLIP模型，解决开放多粒度物体识别中的粗粒与细粒任务差距问题，通过TBO数据集实验验证其在多模态内容生成和互动学习中的有效性。
摘要：
近年来，多模态大语言模型（MLLMs）的进步使开放性物体识别成为可能，但它们在细粒任务上表现不佳。相比之下，CLIP风格模型在细粒识别方面表现出色，但缺乏对通用物体类别的广泛覆盖。为弥合这一差距，我们提出了HyMOR，一种混合多粒度开放性物体识别框架，整合了MLLM与CLIP模型。在HyMOR中，MLLM负责开放性和粗粒度物体识别，而CLIP模型专门用于识别特定领域对象，如动物和植物。这种混合设计使模型能够准确理解多个语义粒度，为下游多模态内容生成和互动游戏提供稳健的感知基础。为支持内容丰富和教育场景的评估，我们引入了TBO（TextBook Objects）数据集，包含20,942张标注有8,816个物体类别的图像，这些类别提取自教科书。大量实验表明，HyMOR在CLIP的细粒识别上将差距缩小至0.2%，同时在基准MLLM上将通用物体识别率提高2.5%，以平均Sentence-BERT（SBert）相似度衡量。总体而言，HyMOR在所有评估数据集上的平均SBert提高了23.2%，凸显其在多模态游戏内容生成和互动学习应用中的有效性。
英文摘要：
Recent advances in Multimodal Large Language Models (MLLMs) have enabled open-ended object recognition, yet they struggle with fine-grained tasks. In contrast, CLIP-style models excel at fine-grained recognition but lack broad coverage of general object categories. To bridge this gap, we propose \textbf{HyMOR}, a \textbf{Hy}brid \textbf{M}ulti-granularity open-ended \textbf{O}bject \textbf{R}ecognition framework that integrates an MLLM with a CLIP model. In HyMOR, the MLLM performs open-ended and coarse-grained object recognition, while the CLIP model specializes in fine-grained identification of domain-specific objects such as animals and plants. This hybrid design enables accurate object understanding across multiple semantic granularities, serving as a robust perceptual foundation for downstream multi-modal content generation and interactive gameplay. To support evaluation in content-rich and educational scenarios, we introduce TBO (TextBook Objects), a dataset containing 20,942 images annotated with 8,816 object categories extracted from textbooks. Extensive experiments demonstrate that HyMOR narrows the fine-grained recognition gap with CLIP to 0.2\% while improving general object recognition by 2.5\% over a baseline MLLM, measured by average Sentence-BERT (SBert) similarity. Overall, HyMOR achieves a 23.2\% improvement in average SBert across all evaluated datasets, highlighting its effectiveness in enabling accurate perception for multi-modal game content generation and interactive learning applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
