arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.08443cs.CVcs.MM
收藏

POINTS1.5: 构建面向真实世界应用的视觉语言模型

POINTS1.5: Building a Vision-Language Model towards Real World Applications

Pattern Recognition Center(模式识别中心);WeChat AI(微信人工智能);Tencent Inc(腾讯公司)

Yuan Liu, Le Tian, Xiao Zhou, Xinyu Gao, Kavio Yu, Yang Yu, Jie Zhou

AI总结:

该研究推出增强版视觉语言模型POINTS1.5,通过采用动态高分辨率NaViT视觉编码器、新增双语支持及优化视觉指令微调数据集过滤方法,在参数规模较小的情况下实现了更优的真实应用性能。

摘要:

视觉语言模型近年来取得了显著进展,在光学字符识别、复杂图表分析等一系列任务上展现出优异性能。基于这一发展趋势,我们推出了一款新型视觉语言模型POINTS1.5,旨在在各类真实世界应用中实现出色表现。POINTS1.5是POINTS1.0的增强版本,融合了多项关键创新:其一,我们将原本采用固定图像分辨率的CLIP视觉编码器,替换为支持原生动态高分辨率的NaViT(原生视觉Transformer)风格视觉编码器,使POINTS1.5无需将图像拆分为图块即可处理任意分辨率的图像。其二,我们为POINTS1.5新增了双语支持,显著提升了其中文处理能力。由于面向视觉语言模型的开源中文数据集十分稀缺,我们从互联网收集了大量图像,并结合人工与自动方法对其进行标注。其三,我们提出了一套针对视觉指令微调数据集的严格过滤方法,对所有过滤方法进行了全面评估,选取效果最优的方法得到最终的视觉指令微调集。得益于这些创新,POINTS1.5的性能显著优于POINTS1.0,在一系列真实世界应用中均展现出强劲表现。值得注意的是,POINTS1.5-7B在训练token数不足40亿的情况下,在OpenCompass榜单的100亿参数以下模型中排名第一。

英文摘要:

Vision-language models have made significant strides recently, demonstrating superior performance across a range of tasks, e.g. optical character recognition and complex diagram analysis. Building on this trend, we introduce a new vision-language model, POINTS1.5, designed to excel in various real-world applications. POINTS1.5 is an enhancement of POINTS1.0 and incorporates several key innovations: i) We replace the original CLIP vision encoder, which had a fixed image resolution, with a NaViT-style vision encoder that supports native dynamic high resolution. This allows POINTS1.5 to process images of any resolution without needing to split them into tiles. ii) We add bilingual support to POINTS1.5, significantly enhancing its capability in Chinese. Due to the scarcity of open-source Chinese datasets for vision-language models, we collect numerous images from the Internet and annotate them using a combination of manual and automatic methods. iii) We propose a set of rigorous filtering methods for visual instruction tuning datasets. We comprehensively evaluate all these filtering methods, and choose the most effective ones to obtain the final visual instruction tuning set. Thanks to these innovations, POINTS1.5 significantly outperforms POINTS1.0 and demonstrates strong performance across a range of real-world applications. Notably, POINTS1.5-7B is trained on fewer than 4 billion tokens and ranks first on the OpenCompass leaderboard among models with fewer than 10 billion parameters

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑