面向视觉推理任务中非可微视觉编程框架的分步蒸馏学习策略
A Stepwise Distillation Learning Strategy for Non-differentiable Visual Programming Frameworks on Visual Reasoning Tasks
AI总结：
针对非可微视觉编程框架性能弱、泛化差且无法微调的问题，提出分步蒸馏策略SDVP，将小型任务专用模型知识蒸馏到大型VLM中，在多个基准上实现显著性能提升并保留跨任务能力。
摘要：
近年来，视觉编程（Visual Programming，VProg）凭借其可解释性与跨任务通用性，已成为视觉推理（visual reasoning，VR）任务的重要框架。然而，即便调用强大的预训练视觉语言模型（Vision-Language models，VLMs）作为视觉子模块，VProg在特定VR任务上的性能仍远逊于训练良好的任务专用网络。尽管调用任务专用模型可进一步提升VProg在特定VR任务上的性能，但会大幅削弱VProg的跨任务泛化能力。此外，VProg的非可微特性使其无法通过在特定VR任务上直接微调来进一步提升性能。为解决这些问题，我们提出SDVP——一种适用于多种VR任务下非可微VProg的分步蒸馏学习策略。具体而言，SDVP将VProg中已训练良好的小型任务专用模型针对分解后视觉子任务的能力，分步蒸馏到对应视觉子模块调用的规模大得多的VLMs中。此外，将小型任务专用模型的知识蒸馏到预训练的大型VLMs中而非替换后者，有助于保留VProg的跨任务能力。在不同VProg框架上开展的广泛全面的实验结果表明，SDVP在特定VR基准上取得了显著性能提升：针对VisProg，在GQA上提升2.4%、在NLVRv2上提升6.2%；针对ViperGPT，在GQA上提升6.5%、在NLVRv2上提升4.0%，同时还能让VProg在未见过的及过往的VR任务上保持可观性能。
英文摘要：
Recently, Visual Programming (VProg) has emerged as a significant framework for visual reasoning (VR) tasks due to its interpretability and cross-task generality. However, even with invoking powerful pre-trained Vision-Language models (VLMs) as visual sub-modules, the performance of VProg on specific VR tasks is markedly inferior compared to well-trained task-specific networks. Although invoking task-specific models can further enhance the performance of VProg on specific VR tasks, it greatly diminishes the cross-task generalization ability of VProg. Besides, the non-differentiable nature of VProg prevents direct fine-tuning on specific VR tasks for further performance improvement. Attempt to address these issues, we propose SDVP, a Stepwise Distillation learning strategy for non-differentiable VPorg across various VR tasks. Specifically, our SDVP stepwise distills the capabilities of existing, well-trained small task-specific models for decomposed visual sub-tasks in VProg into the much larger VLMs invoked by corresponding visual sub-modules. Besides, distilling the knowledge of little-size task-specific models into pre-trained larger VLMs rather than replacing them helps keep the cross-task abilities of VProgs. Extensive and comprehensive experimental results on different VProg frameworks demonstrate that our SDVP obtains significant performance gains on specific VR benchmarks, i.e., GQA (+2.4\%) and NLVRv2 (+6.2\%) for VisProg and GQA (+6.5\%) and NLVRv2 (+4.0\%) for ViperGPT, and also maintains a promising performance for VProg on unseen and previous VR tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
