Large language models (LLMs) have demonstrated strong capabilities across various language tasks, notably through instruction-tuning methods. However, LLMs face challenges in visualizing complex, real-world data through charts and plots. Firstly, existing datasets rarely cover a full range of chart types, such as 3D, volumetric, and gridded charts. Secondly, supervised fine-tuning methods do not fully leverage the intricate relationships within rich datasets, including text, code, and figures. To address these challenges, we propose a hierarchical pipeline and a new dataset for chart generation. Our dataset, Text2Chart31, includes 31 unique plot types referring to the Matplotlib library, with 11.1K tuples of descriptions, code, data tables, and plots. Moreover, we introduce a reinforcement learning-based instruction tuning technique for chart generation tasks without requiring human feedback. Our experiments show that this approach significantly enhances the model performance, enabling smaller models to outperform larger open-source models and be comparable to state-of-the-art proprietary models in data visualization tasks. We make the code and dataset available at https://github.com/fatemehpesaran310/Text2Chart31.
Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model
York University（约克大学）
信息可视化（如条形图和折线图）在探索数据和传达见解方面非常受欢迎。对某些人来说，解释和理解这种可视化可能具有挑战性，例如那些视力受损或视觉素养低的人。在这项工作中，我们引入了一个新的数据集，并提出了一个神经模型自动生成图表的自然语言摘要。生成的摘要提供了对图表的解释，并传达了在该图表中发现的关键见解。我们的神经模型是通过扩展数据到文本生成任务的最先进模型开发的，该模型利用了基于transformer的编码器-解码器架构。我们发现，我们的方法在内容选择指标上远远优于基础模型（55.42% vs. 8.49%），并生成了更多信息，简洁和连贯的摘要。
What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning
ShanghaiTech University（上海科技大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Meituan（美团）；Alibaba Group（阿里巴巴集团）；The Hong Kong University of Science and Technology（香港科技大学）
SciEval: A Benchmark for Automatic Evaluation of K-12 Science Instructional Materials
Department of Computer Science and Engineering, University at Buffalo, USA（宾夕法尼亚大学布法罗分校计算机科学与工程系）；Department of Teaching and Learning, Washington State University, USA（华盛顿州立大学教学与学习系）；College of Chemistry, Beijing Normal University, China（北京师范大学化学学院）；College of AI, Cyber and Computing, University of Texas at San Antonio, USA（德克萨斯大学圣安东尼奥分校人工智能、网络与计算学院）
Charting Empirical Laws for LLM Fine-Tuning in Scientific Multi-Discipline Learning
The University of Sydney（悉尼大学）；Shanghai AI Laboratory（上海人工智能实验室）；Edith Cowan University（埃德温·考文大学）；University of Oxford（牛津大学）；The Chinese University of Hong Kong（香港中文大学）；Beihang University（北航大学）
