改进的大语言模型数据高效感知增强基线
Improved Baselines for Data-efficient Perceptual Augmentation of LLMs
AI总结：
本文对多种感知骨干网络连接机制进行了广泛实验评估，在低数据设置下发现了一种新机制，在多任务上取得接近最优性能且训练时间减少4倍。
摘要：
大型语言模型（LLMs）的能力近期已发展至前所未有的水平，为广泛领域的各种新应用铺平了道路。在计算机视觉中，当与预训练的视觉骨干网络结合时，LLMs可用于引导图像描述和视觉问答等视觉-语言任务。尽管已有不同的方法被探索用于将LLMs与处理视觉或音频数据等“感知骨干网络”相连接，但这些方法往往针对不同的任务、不同的数据集，并使用不同的感知骨干网络和语言模型，从而阻碍了对连接机制的直接比较。为了弥补方法之间缺乏可比性的问题，我们对不同的连接机制进行了广泛的实验评估，涵盖多个任务（包括图像、视频和音频描述以及视觉问答）、数据集和骨干网络，并特别关注低数据设置。我们发现，使用现有机制获得了优于现有技术结果的性能提升，并确定了一种新的连接机制，该机制在不同任务上产生（接近）最优结果，同时训练时间减少了4倍。
英文摘要：
The abilities of large language models (LLMs) have recently progressed to unprecedented levels, paving the way to novel applications in a wide variety of areas. In computer vision, LLMs can be used to prime vision-language tasks such image captioning and visual question answering when coupled with pre-trained vision backbones. While different approaches have been explored to interface LLMs with ``perceptual backbones'' that process, e.g., visual or audio data, they are often explored for different tasks, different datasets, and using different perceptual backbones and language models, hindering direct comparison of the interfacing mechanisms. To remedy this lack of comparability between methods, we present an extensive experimental evaluation of different interfacing mechanisms, across multiple tasks (including image, video, and audio captioning as well as visual question answering), datasets and backbones, paying special attention to low-data settings. We find improved performance using existing mechanisms over state-of-the-art results, and identify a new interfacing mechanism that yields (near) optimal results across different tasks, while obtaining a 4x reduction in training time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
