arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.00367cs.CLcs.CV
收藏

AutomaTikZ:基于 TikZ 的文本引导科学矢量图形合成

AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ

Bielefeld University(比勒费尔德大学);University of Hamburg(汉堡大学);University of Mannheim(曼海姆大学)

Jonas Belouadi, Anne Lauscher, Steffen Eger

AI总结:

本文提出 AutomaTikZ 框架,以 TikZ 语言为中间表示,构建 120k 规模的 DaTikZ 数据集,并微调 LLaMA 及多模态增强模型 CLiMA,实现文本到科学矢量图形的高质量生成,性能超越 GPT-4 和 Claude 2。

摘要:

从文本生成位图图形已获得广泛关注,但对于科学图表而言,矢量图形通常更受青睐。由于矢量图形通常使用底层图形原语进行编码,直接生成它们十分困难。为解决这一问题,我们提出使用 TikZ——一种可编译为矢量图形的知名抽象图形语言——作为科学图表的中间表示。TikZ 提供面向人类的高层命令,从而便于利用任意大型语言模型进行条件语言建模。为此,我们引入了 DaTikZ,这是首个大规模 TikZ 数据集,包含 120k 个与图注对齐的 TikZ 绘图。我们在 DaTikZ 上微调 LLaMA,以及我们的新模型 CLiMA,后者通过多模态 CLIP 嵌入增强 LLaMA。在人工评估和自动评估中,CLiMA 和 LLaMA 在与人类创建图表的相似度方面均优于商业模型 GPT-4 和 Claude 2,其中 CLiMA 还进一步改善了文本与图像的对齐。我们的详细分析表明,所有模型均具有良好的泛化能力,且不易受到记忆效应的影响。然而,与人类以及我们的模型相比,GPT-4 和 Claude 2 倾向于生成更简单的图表。我们将框架 AutomaTikZ 以及模型权重和数据集公开发布。

英文摘要:

Generating bitmap graphics from text has gained considerable attention, yet for scientific figures, vector graphics are often preferred. Given that vector graphics are typically encoded using low-level graphics primitives, generating them directly is difficult. To address this, we propose the use of TikZ, a well-known abstract graphics language that can be compiled to vector graphics, as an intermediate representation of scientific figures. TikZ offers human-oriented, high-level commands, thereby facilitating conditional language modeling with any large language model. To this end, we introduce DaTikZ, the first large-scale TikZ dataset consisting of 120k TikZ drawings aligned with captions. We fine-tune LLaMA on DaTikZ, as well as our new model CLiMA, which augments LLaMA with multimodal CLIP embeddings. In both human and automatic evaluation, CLiMA and LLaMA outperform commercial GPT-4 and Claude 2 in terms of similarity to human-created figures, with CLiMA additionally improving text-image alignment. Our detailed analysis shows that all models generalize well and are not susceptible to memorization. GPT-4 and Claude 2, however, tend to generate more simplistic figures compared to both humans and our models. We make our framework, AutomaTikZ, along with model weights and datasets, publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑