arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.11503cs.CVcs.CL
收藏

GeoGPT4V:迈向具备几何图像生成能力的几何多模态大语言模型

GeoGPT4V: Towards Geometric Multi-modal Large Language Models with Geometric Image Generation

University of Science and Technology of China(中国科学技术大学);Alibaba Group(阿里巴巴集团)

Shihao Cai, Keqin Bao, Hangyu Guo, Jizhi Zhang, Jun Song, Bo Zheng

AI总结:

针对多模态模型难以利用几何图像且现有数据存在难学习、图文不对齐的问题,本文利用 GPT-4/GPT-4V 生成对齐数据并构建 GeoGPT4V,显著提升多个模型在 MathVista 和 MathVision 上的几何表现。

摘要:

大语言模型已在数学问题求解中得到广泛应用。然而,在通常需要视觉辅助以更好理解的几何问题中,即使是当前最先进的多模态模型,仍难以有效利用图像信息。高质量数据对于增强多模态模型的几何能力至关重要,但现有开源数据集及相关工作要么对模型直接学习而言过于困难,要么存在文本与图像不对齐的问题。为解决这一问题,我们提出一种新型流程,利用 GPT-4 和 GPT-4V 生成文本与图像对齐、相对基础的几何问题,以促进模型学习。我们构建了一个包含 4.9K 道几何问题的数据集,并将其与 19K 条开源数据结合,形成 GeoGPT4V 数据集。实验结果表明,GeoGPT4V 数据集显著提升了多种模型在 MathVista 和 MathVision 基准上的几何性能。代码可在 https://github.com/Lanyu0303/GeoGPT4V_Project 获取。

英文摘要:

Large language models have seen widespread adoption in math problem-solving. However, in geometry problems that usually require visual aids for better understanding, even the most advanced multi-modal models currently still face challenges in effectively using image information. High-quality data is crucial for enhancing the geometric capabilities of multi-modal models, yet existing open-source datasets and related efforts are either too challenging for direct model learning or suffer from misalignment between text and images. To overcome this issue, we introduce a novel pipeline that leverages GPT-4 and GPT-4V to generate relatively basic geometry problems with aligned text and images, facilitating model learning. We have produced a dataset of 4.9K geometry problems and combined it with 19K open-source data to form our GeoGPT4V dataset. Experimental results demonstrate that the GeoGPT4V dataset significantly improves the geometry performance of various models on the MathVista and MathVision benchmarks. The code is available at https://github.com/Lanyu0303/GeoGPT4V_Project

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑