Vision-language models (VLMs) are achieving increasingly strong performance on multimodal tasks. However, reasoning capabilities remain limited particularly for smaller VLMs, while those of large-language models (LLMs) have seen numerous improvements. We propose a technique to transfer capabilities from LLMs to VLMs. On the recently introduced ChartQA, our method obtains state-of-the-art performance when applied on the PaLI3-5B VLM by \citet{chen2023pali3}, while also enabling much better performance on PlotQA and FigureQA.
We first improve the chart representation by continuing the pre-training stage using an improved version of the chart-to-table translation task by \citet{liu2023deplot}. We then propose constructing a 20x larger dataset than the original training set. To improve general reasoning capabilities and improve numerical operations, we synthesize reasoning traces using the table representation of charts. Lastly, our model is fine-tuned using the multitask loss introduced by \citet{hsieh2023distilling}.
Our variant ChartPaLI-5B outperforms even 10x larger models such as PaLIX-55B without using an upstream OCR system, while keeping inference time constant compared to the PaLI3-5B baseline. When rationales are further refined with a simple program-of-thought prompt \cite{chen2023program}, our model outperforms the recently introduced Gemini Ultra and GPT-4V.
Encoded but Not Routed: Explaining the Table-Chart Gap in Scientific Claim Verification
The University of Tokyo（东京大学）；NII LLMC（日本信息处理学会LLMC）；National Institute of Informatics（日本信息处理研究所）；University of Göttingen（哥廷根大学）；Inria, LS2N, Nantes Université（法国国家信息与自动化技术研究院（Inria）、LS2N实验室、南特大学）
Towards General Purpose Medical AI: Continual Learning Medical Foundation Model
Beijing University of Posts and Telecommunications（北京邮电大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；West China Hospital, Sichuan University（四川大学华西医院）；Shanghai Jiao Tong University（上海交通大学）；Sichuan University Pittsburgh Institute（四川大学匹兹堡学院）
How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbations
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China（教育部知识驱动人机智能工程研究中心）；International Center of Future Science, Jilin University（吉林大学未来科学国际合作中心）
Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models
University of British Columbia（不列颠哥伦比亚大学）；Zuse School（Zuse学校）；Qatar Computing Research Institute (QCRI)（卡塔尔计算研究所）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
Transferable Decoding with Visual Entities for Zero-Shot Image Captioning
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Tencent（腾讯）；Shanghai Jiao Tong University（上海交通大学）；Southern University of Science and Technology（南方科技大学）；The University of Hong Kong（香港大学）
