Pyramid Coder：用于组合式视觉问答的分层代码生成器
Pyramid Coder: Hierarchical Code Generator for Compositional Visual Question Answering
AI总结：
本文提出PyramidCoder，一种无需训练的分层提示框架，通过查询重述、代码生成和答案聚合三个层级，利用冻结LLM提升程序化视觉问答的准确率。
摘要：
视觉问答（VQA）是基于视觉输入对自然语言问题提供准确答案的任务。程序化视觉问答（PVQA）模型近期备受关注。这类模型利用大型语言模型（LLMs）生成可执行程序，以解决需要复杂视觉推理的问题。然而，让LLMs理解图像处理模块的用法并生成相关代码仍存在挑战。为克服这些挑战，本文提出PyramidCoder，一种面向PVQA模型的新型提示框架。PyramidCoder包含三个层级，各司其职：查询重述、代码生成和答案聚合。值得注意的是，PyramidCoder在每个层级仅使用单个冻结的LLM和预定义提示，无需额外训练，并确保在多种LLM架构间的灵活性。与最先进的PVQA模型相比，本方法在GQA数据集上准确率至少提升0.5%，在VQAv2数据集上提升1.4%，在NLVR2数据集上提升2.9%。
英文摘要：
Visual question answering (VQA) is the task of providing accurate answers to natural language questions based on visual input. Programmatic VQA (PVQA) models have been gaining attention recently. These use large language models (LLMs) to formulate executable programs that address questions requiring complex visual reasoning. However, there are challenges in enabling LLMs to comprehend the usage of image processing modules and generate relevant code. To overcome these challenges, this paper introduces PyramidCoder, a novel prompting framework for PVQA models. PyramidCoder consists of three hierarchical levels, each serving a distinct purpose: query rephrasing, code generation, and answer aggregation. Notably, PyramidCoder utilizes a single frozen LLM and pre-defined prompts at each level, eliminating the need for additional training and ensuring flexibility across various LLM architectures. Compared to the state-of-the-art PVQA model, our approach improves accuracy by at least 0.5% on the GQA dataset, 1.4% on the VQAv2 dataset, and 2.9% on the NLVR2 dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
