Communicating complex system designs or scientific processes through text alone is inefficient and prone to ambiguity. A system that automatically generates scientific architecture diagrams from text with high semantic fidelity can be useful in multiple applications like enterprise architecture visualization, AI-driven software design, and educational content creation. Hence, in this paper, we focus on leveraging language models to perform semantic understanding of the input text description to generate intermediate code that can be processed to generate high-fidelity architecture diagrams. Unfortunately, no clean large-scale open-access dataset exists, implying lack of any effective open models for this task. Hence, we contribute a comprehensive dataset, \system, comprising scientific architecture images, their corresponding textual descriptions, and associated DOT code representations. Leveraging this resource, we fine-tune a suite of small language models, and also perform in-context learning using GPT-4o. Through extensive experimentation, we show that \system{} models significantly outperform existing baseline models like DiagramAgent and perform at par with in-context learning-based generations from GPT-4o. We make the code, data and models publicly available.
Draw with Thought: Unleashing Multimodal Reasoning for Scientific Diagram Generation
Nanjing University of Information Science \& Technology Nanjing China;East China Normal University Shanghai China;The Hong Kong University of Science;Brown University Providence America;Southwest Jiaotong University Chengdu China;Nanjing University of Information Science \& Technology;East China Normal University;Brown University;Southwest Jiaotong University
提出无需训练的框架Draw with Thought,通过认知思维链推理将科学图表重建为可编辑的mxGraph XML代码,并发布Plot2XML基准,实验证明该方法能生成高保真且结构有效的重建结果。
ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?
University of Twente(特文特大学);University of Technology Nuremberg (UTN)(纽伦堡工业大学(UTN));University of Sheffield(谢菲尔德大学);Harbin Institute of Technology(哈尔滨工业大学);University of Mannheim(曼海姆大学);Technische Universität Dresden(德累斯顿工业大学)
AIGS: Generating Science from AI-Powered Automated Falsification
Department of Computer Science & Technology, Tsinghua University(清华大学计算机科学与技术系);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院)
SciXGen: A Scientific Paper Dataset for Context-Aware Text Generation
The University of Tokyo(东京大学);Tokyo Institute of Technology(东京工业大学);National Institute of Advanced Industrial Science Technology(日本产业技术综合研究所)
在科学论文中生成文本不仅需要捕获给定输入中包含的内容,还需要经常获取称为\textit{context}的外部信息。我们提出了一个新的任务,即科学领域的\textbf{context-aware text generation},旨在开发语境在生成文本中的贡献,从而推动了科学文本的生成。为此,我们为Conte \textbf{X} t-Aware Text \textbf{Gen} eration(SciXGen)提出了一个新的具有挑战性的大规模\textbf{Sci}有效论文数据集,由注释良好的205,304篇论文组成,其中充分引用了广泛使用的对象(例如,表、图、算法)。我们使用最先进的技术,对我们新构建的SciXGen数据集在生成描述和段落方面的功效进行了全面的基准测试。我们的数据集和基准将公开提供,希望能促进科学文本生成研究。
Evaluating Compliance with Visualization Guidelines in Diagrams for Scientific Publications Using Large Vision Language Models
Department of Computer Science, University of Applied Sciences and Arts Dortmund(应用科学与艺术大学多特蒙德计算机科学系);Institute for Medical Informatics, Biometry and Epidemiology (IMIBE)(医学信息学、生物统计与流行病学研究所);Institute for Artificial Intelligence in Medicine (IKIM), University Hospital Essen(医学人工智能研究所,埃森大学医院)
ReviewRL: Towards Automated Scientific Review with RL
Tsinghua University(清华大学);University of Washington(华盛顿大学);Shanghai AI Laboratory(上海人工智能实验室);Peking University(北京大学);Harbin Engineering University(哈尔滨工程大学);Beijing Institute of Technology(北京理工大学)
Generating Knowledge Graphs by Employing Natural Language Processing and Machine Learning Techniques within the Scholarly Domain
University of Cagliari(卡利亚里大学);FIZ Karlsruhe - Leibniz Institute for Information Infrastructure(FIZ卡尔斯鲁厄——莱布尼茨信息基础设施研究所);Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);The Open University(开放大学);CNRS(法国国家科学研究中心);University Paris 13(巴黎第十三大学)
Do Large Multimodal Models Solve Caption Generation for Scientific Figures? Lessons Learned from SciCap Challenge 2023
Pennsylvania State University(宾夕法尼亚州立大学);National Tsing Hua University(国立清华大学);AllSci(奥尔赛公司);MetaMetrics Inc.(美塔麦特里克公司);Adobe Research(奥多比研究院);Institute of Information Science, Academia Sinica(中央研究院资讯科学研究所)