用于解析和总结信息图的合成训练图标提议
Synthetically Trained Icon Proposals for Parsing and Summarizing Infographics
AI总结:
本文提出一种合成数据生成策略,利用互联网抓取的图标增强信息图背景,训练图标提议机制,以38%精确率和34%召回率定位图标,并结合图标分类与文本提取实现多模态总结应用。
摘要:
信息图广泛应用于新闻、商业和教育媒体中,是手工制作的,旨在有效传达关于复杂且往往抽象主题的信息,如“保护环境的方法”和“理解金融危机”。信息图由风格和语义多样的视觉和文本元素组成,给计算机视觉带来了新的挑战。虽然自动文本提取在信息图上效果良好,但在自然图像上训练的计算机视觉方法无法识别信息图中的独立视觉元素,即“图标”。为了弥合这一表示差距,我们提出了一种合成数据生成策略:我们将来自Visually29K数据集的信息图中的背景补丁与从互联网抓取的图标进行增强,这些图标用作图标提议机制的训练数据。在一个包含1K张标注信息图的测试集上,图标定位的精确率为38%,召回率为34%(在自然图像上训练的最佳模型精确率为14%,召回率为7%)。将我们的图标提议与图标分类和文本提取相结合,我们提出了一种多模态总结应用。该应用以信息图为输入,自动生成文本标签和视觉标签,分别代表信息图主题的文本和视觉特征。
英文摘要:
Widely used in news, business, and educational media, infographics are handcrafted to effectively communicate messages about complex and often abstract topics including `ways to conserve the environment' and `understanding the financial crisis'. Composed of stylistically and semantically diverse visual and textual elements, infographics pose new challenges for computer vision. While automatic text extraction works well on infographics, computer vision approaches trained on natural images fail to identify the stand-alone visual elements in infographics, or `icons'. To bridge this representation gap, we propose a synthetic data generation strategy: we augment background patches in infographics from our Visually29K dataset with Internet-scraped icons which we use as training data for an icon proposal mechanism. On a test set of 1K annotated infographics, icons are located with 38% precision and 34% recall (the best model trained with natural images achieves 14% precision and 7% recall). Combining our icon proposals with icon classification and text extraction, we present a multi-modal summarization application. Our application takes an infographic as input and automatically produces text tags and visual hashtags that are textually and visually representative of the infographic's topics respectively.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。