As large language models have demonstrated impressive performance in many domains, recent works have adopted language models (LMs) as controllers of visual modules for vision-and-language tasks. While existing work focuses on equipping LMs with visual understanding, we propose two novel interpretable/explainable visual programming frameworks for text-to-image (T2I) generation and evaluation. First, we introduce VPGen, an interpretable step-by-step T2I generation framework that decomposes T2I generation into three steps: object/count generation, layout generation, and image generation. We employ an LM to handle the first two steps (object/count generation and layout generation), by finetuning it on text-layout pairs. Our step-by-step T2I generation framework provides stronger spatial control than end-to-end models, the dominant approach for this task. Furthermore, we leverage the world knowledge of pretrained LMs, overcoming the limitation of previous layout-guided T2I works that can only handle predefined object classes. We demonstrate that our VPGen has improved control in counts/spatial relations/scales of objects than state-of-the-art T2I generation models. Second, we introduce VPEval, an interpretable and explainable evaluation framework for T2I generation based on visual programming. Unlike previous T2I evaluations with a single scoring model that is accurate in some skills but unreliable in others, VPEval produces evaluation programs that invoke a set of visual modules that are experts in different skills, and also provides visual+textual explanations of the evaluation results. Our analysis shows that VPEval provides a more human-correlated evaluation for skill-specific and open-ended prompts than widely used single model-based evaluation. We hope that our work encourages future progress on interpretable/explainable generation and evaluation for T2I models.
MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems
National University of Singapore(新加坡国立大学);The University of Hong Kong(香港大学);Nanyang Technological University(南洋理工大学);Hong Kong Baptist University(香港浸会大学)
Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection
School of Software, Beihang University(北京航空航天大学软件学院);School of Computer Science, Peking University(北京大学计算机学院);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Galbot(盖尔博特(Galbot))
LLM-based Visual Code Completion for Aerospace Geometric Design
Faculty of Engineering and Physical Sciences, University of Southampton(南安普顿大学工程与物理科学学院);School of Electronics and Computer Science, University of Southampton(南安普顿大学电子与计算机科学学院)
Teaching Programming in the Age of Generative AI: Insights from Literature, Pedagogical Proposals, and Student Perspectives
Departamento de Sistemas de Información - Universidad del Bío-Bío(信息系统系 - 生物-生物大学);Escuela de Informática y Telecomunicaciones - DuocUC(信息与电信工程学院 - DuocUC)
Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding
The Future Network of Intelligence Institute(未来智能网络研究院);The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳));School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳)理工学院);IHPC, A*STAR, Singapore(新加坡科研局智能系统研究所);The University of Hong Kong(香港大学)
Automated Design of Salient Object Detection Algorithms with Brain Programming
CICESE Research Center(CICESE研究中心);Universidad Anahuac Querétaro(阿纳瓦克克雷塔罗大学);Universidad Nacional Autónoma de México(墨西哥国立自治大学);Universidad Autónoma de San Luis Potosí(圣路易斯波托西自治大学);Universidad Autónoma de Tlaxcala(特拉斯卡拉自治大学)