arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.03194cs.CV

CityLLaVA:面向城市场景的视觉语言模型高效微调

CityLLaVA: Efficient Fine-Tuning for VLMs in City Scenario

Alibaba Group(阿里巴巴集团)

Zhizhao Duan, Hao Cheng, Duo Xu, Xi Wu, Xiangxie Zhang, Xi Ye, Zhen Xie

AI总结:

本文提出CityLLaVA框架,通过边界框视觉预处理、问答序列构建、块扩展微调和顺序提问预测增强,高效微调视觉语言模型用于城市场景交通安全分析,并以33.4308分取得排行榜第一。

摘要:

在广阔且动态的城市环境中,交通安全描述与分析在从保险查勘到事故预防等应用中发挥着关键作用。本文提出了CityLLaVA,一个面向城市场景设计的视觉语言模型(VLM)新型微调框架。CityLLaVA通过以下方式增强模型理解能力和预测精度:(1)利用边界框进行最优视觉数据预处理,包括在训练和测试阶段进行视频最佳视角选择和视觉提示工程;(2)构建简洁的问答序列并设计文本提示以改进指令理解;(3)实施块扩展以高效微调大规模VLM;(4)通过独特的基于顺序提问的预测增强来提升预测精度。我们的方法展现出顶级性能,取得了33.4308的基准分数,在排行榜上位居第一。代码见:https://github.com/alibaba/AICITY2024_Track2_AliOpenTrek_CityLLaVA

英文摘要:

In the vast and dynamic landscape of urban settings, Traffic Safety Description and Analysis plays a pivotal role in applications ranging from insurance inspection to accident prevention. This paper introduces CityLLaVA, a novel fine-tuning framework for Visual Language Models (VLMs) designed for urban scenarios. CityLLaVA enhances model comprehension and prediction accuracy through (1) employing bounding boxes for optimal visual data preprocessing, including video best-view selection and visual prompt engineering during both training and testing phases; (2) constructing concise Question-Answer sequences and designing textual prompts to refine instruction comprehension; (3) implementing block expansion to fine-tune large VLMs efficiently; and (4) advancing prediction accuracy via a unique sequential questioning-based prediction augmentation. Demonstrating top-tier performance, our method achieved a benchmark score of 33.4308, securing the leading position on the leaderboard. The code can be found: https://github.com/alibaba/AICITY2024_Track2_AliOpenTrek_CityLLaVA

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑