RL-CSDia：计算机科学图的表示学习
RL-CSDia: Representation Learning of Computer Science Diagrams
摘要：
近年来，计算机视觉的研究主要集中在表达真实世界场景的自然图像上。他们在视觉问答等各种任务上都有出色的表现。图表是教育领域中经常出现的一种特殊的视觉表达形式，对于学习者理解多模态知识具有重要意义。目前对图的研究主要集中在生物学、地理学等自然学科，其表达方式仍与自然图像相似。另一类图如来自计算机科学的图是由包含复杂拓扑和关系的图形组成的，对这类图的研究还是空白。数据的稀缺性和语义的混乱性是图形理解的主要挑战，这主要体现在表达的多样性上。在本文中，我们构建了一个新的数据集的图形图命名为计算机科学图（CSDia）。它包含1,200多个图表和详尽的对象和关系注释。考虑到图中的各种表达式所引起的视觉噪声，我们引入了图的拓扑结构来解析拓扑结构。在此基础上，提出了图分析网（DiagraphParsingNet，DPN），从拓扑、视觉特征和文本三个方面对图进行描述，并将该模型应用于图的分类任务中，以评价图的理解能力。实验结果表明了该方法在图形理解中的有效性。
英文摘要：
Recent studies on computer vision mainly focus on natural images that express real-world scenes. They achieve outstanding performance on diverse tasks such as visual question answering. Diagram is a special form of visual expression that frequently appears in the education field and is of great significance for learners to understand multimodal knowledge. Current research on diagrams preliminarily focuses on natural disciplines such as Biology and Geography, whose expressions are still similar to natural images. Another type of diagrams such as from Computer Science is composed of graphics containing complex topologies and relations, and research on this type of diagrams is still blank. The main challenges of graphic diagrams understanding are the rarity of data and the confusion of semantics, which are mainly reflected in the diversity of expressions. In this paper, we construct a novel dataset of graphic diagrams named Computer Science Diagrams (CSDia). It contains more than 1,200 diagrams and exhaustive annotations of objects and relations. Considering the visual noises caused by the various expressions in diagrams, we introduce the topology of diagrams to parse topological structure. After that, we propose Diagram Parsing Net (DPN) to represent the diagram from three branches: topology, visual feature, and text, and apply the model to the diagram classification task to evaluate the ability of diagrams understanding. The results show the effectiveness of the proposed DPN on diagrams understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
