使用具有序列级训练的深度神经网络将数学公式图像转换为LaTeX序列
Translating Math Formula Images to LaTeX Sequences Using Deep Neural Networks with Sequence-level Training
摘要:
在本文中,我们提出了一个深度神经网络模型,该模型具有编码器-解码器架构,可以将数学公式的图像转换为LaTeX标记序列。编码器是一个卷积神经网络(CNN),它将图像转换为一组特征图。为了更好地捕捉数学符号的空间关系,在将特征图展开为向量之前,使用2D位置编码来增强特征图。解码器是一个堆叠的双向长短期记忆(LSTM)模型,集成了软注意力机制,作为语言模型将编码器输出转换为LaTeX令牌序列。神经网络的训练分两步进行。第一步是使用最大似然估计(MLE)作为目标函数的令牌级训练。在令牌级训练完成后,基于强化学习的策略梯度算法,采用序列级训练目标函数来优化整体模型。我们的设计还通过在序列级训练期间关闭解码器中的反馈回路来克服曝光偏差问题,即,在每个时间步馈入所述预测令牌而不是所述地面实况令牌。该模型在IM 2LATEX-100 K数据集上进行训练和评估,并在基于序列和基于图像的评估指标上显示出最先进的性能。
英文摘要:
In this paper we propose a deep neural network model with an encoder-decoder architecture that translates images of math formulas into their LaTeX markup sequences. The encoder is a convolutional neural network (CNN) that transforms images into a group of feature maps. To better capture the spatial relationships of math symbols, the feature maps are augmented with 2D positional encoding before being unfolded into a vector. The decoder is a stacked bidirectional long short-term memory (LSTM) model integrated with the soft attention mechanism, which works as a language model to translate the encoder output into a sequence of LaTeX tokens. The neural network is trained in two steps. The first step is token-level training using the Maximum-Likelihood Estimation (MLE) as the objective function. At completion of the token-level training, the sequence-level training objective function is employed to optimize the overall model based on the policy gradient algorithm from reinforcement learning. Our design also overcomes the exposure bias problem by closing the feedback loop in the decoder during sequence-level training, i.e., feeding in the predicted token instead of the ground truth token at every time step. The model is trained and evaluated on the IM2LATEX-100K dataset and shows state-of-the-art performance on both sequence-based and image-based evaluation metrics.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。