Can Large Language Models Understand Symbolic Graphics Programs?
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Cambridge（剑桥大学）；MIT（麻省理工学院）
Zeju Qiu, Weiyang Liu, Haiwen Feng, Zhen Liu, Tim Z. Xiao, Katherine M. Collins, Joshua B. Tenenbaum, Adrian Weller, Michael J. Black, Bernhard Schölkopf
Against the backdrop of enthusiasm for large language models (LLMs), there is a growing need to scientifically assess their capabilities and shortcomings. This is nontrivial in part because it is difficult to find tasks which the models have not encountered during training. Utilizing symbolic graphics programs, we propose a domain well-suited to test multiple spatial-semantic reasoning skills of LLMs. Popular in computer graphics, these programs procedurally generate visual data. While LLMs exhibit impressive skills in general program synthesis and analysis, symbolic graphics programs offer a new layer of evaluation: they allow us to test an LLM's ability to answer semantic questions about the images or 3D geometries without a vision encoder. To semantically understand the symbolic programs, LLMs would need to possess the ability to "imagine" and reason how the corresponding graphics content would look with only the symbolic description of the local curvatures and strokes. We use this task to evaluate LLMs by creating a large benchmark for the semantic visual understanding of symbolic graphics programs, built procedurally with minimal human effort. Particular emphasis is placed on transformations of images that leave the image level semantics invariant while introducing significant changes to the underlying program. We evaluate commercial and open-source LLMs on our benchmark to assess their ability to reason about visual output of programs, finding that LLMs considered stronger at reasoning generally perform better. Lastly, we introduce a novel method to improve this ability -- Symbolic Instruction Tuning (SIT), in which the LLM is finetuned with pre-collected instruction data on symbolic graphics programs. Interestingly, we find that SIT not only improves LLM's understanding on symbolic programs, but it also improves general reasoning ability on various other benchmarks.
Symbolic Graphics Programming with Large Language Models
The Chinese University of Hong Kong（香港中文大学）；Westlake University（西湖大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoning
University of California, Berkeley（加州大学伯克利分校）；Carnegie Mellon University（卡内基梅隆大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Impossible, Inc.（Impossible公司）
ABC-Eval: Benchmarking Large Language Models on Symbolic Music Understanding and Instruction Following
Graduate School of Informatics, Kyoto University, Japan（京都大学信息科学研究生院）；College of Computer Science and Artificial Intelligence, Fudan University, China（复旦大学计算机科学与人工智能学院）；Graduate School of Engineering, Kyoto University, Japan（京都大学工学研究科）
Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanism
Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University（上海交通大学自然科学院）；School of Mathematical Sciences, Shanghai Jiao Tong University（上海交通大学数学学院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Huawei Noah’s Ark Lab（华为诺亚实验室）；Key Laboratory of Marine Intelligent Equipment and System, Ministry of Education（教育部海洋智能装备与系统重点实验室）
LLM-ABBA: Understanding time series via symbolic approximation
Sorbonne Université, CNRS, LIP6（索邦大学、法国国家科学研究中心、LIP6实验室）；Department of Numerical Mathematics, Charles University（数值数学系、查尔斯大学）；Department of Cybernetics, Czech Technical University in Prague（控制论系、布拉格捷克技术大学）
Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning
State Key Laboratory of Novel Software Technology, Nanjing University, China（新型软件技术国家重点实验室，南京大学，中国）；School of Artificial Intelligence, Nanjing University, China（人工智能学院，南京大学，中国）；School of Intelligence Science and Technology, Nanjing University, China（智能科学与技术学院，南京大学，中国）
Towards Symbolic XAI -- Explanation Through Human Understandable Logical Relationships Between Features
Berlin Institute for the Foundations of Learning（柏林学习与数据基础研究所）；Machine Learning Group, Technical University of Berlin（柏林技术大学机器学习组）；Department of Artificial Intelligence, Korea University（韩国大学人工智能系）；Department of Mathematics and Computer Science, Free University of Berlin（柏林自由大学数学与计算机科学系）
Extracting Symbolic Sequences from Visual Representations via Self-Supervised Learning
Universidad Nacional Autónoma de México（墨西哥国立自治大学）；Posgrado en Ciencia e Ingeniería de la Computación（计算机科学与工程研究生课程）；Instituto de Investigaciones en Matemáticas Aplicadas y en Sistemas（应用数学与系统研究所）
