The goal of vision-language modeling is to allow models to tie language understanding with visual inputs. The aim of this paper is to evaluate and align the Visual Language Model (VLM) called Multimodal Augmentation of Generative Models through Adapter-based finetuning (MAGMA) with human values. MAGMA is a VLM that is capable of image captioning and visual question-answering. We will evaluate its alignment in three different scenarios. To begin, we assess MAGMA's out-of-the-box alignment through the checkpoint provided by Hugging Face. Then, we measure if few-shot learning manages to improve the results. Finally, we finetune the model on aligned examples and evaluate its behavior.
Aligning Thoughts with Answers: Probability Rewards to Tame Thinking Drift
National University of Singapore(新加坡国立大学);A*STAR Institute of Advanced Intelligence and Computing, Singapore(新加坡科技研究局先进智能与计算研究所);South China University of Technology(华南理工大学);University of Science and Technology of China(中国科学技术大学);Google DeepMind(谷歌DeepMind)
AlpaCare:Instruction-tuned Large Language Models for Medical Application
University of California, Santa Barbara(加州大学圣巴巴拉分校);Chinese Academy of Medical Sciences and Peking Union Medical College(中国医学科学院北京协和医学院);University of Maryland, College Park(马里兰大学帕克分校)
Enhancing Vision-Language Model with Unmasked Token Alignment
Institute for AI Industry Research (AIR)(人工智能产业研究院);Tsinghua University(清华大学);Sensetime Research(商汤科技研究院);CPII under InnoHK(InnoHK下属CPII);Shanghai AI Laboratory(上海人工智能实验室)
Aligning Machine and Human Visual Representations across Abstraction Levels
Google DeepMind Machine Learning Group(谷歌DeepMind机器学习组);Technische Universität Berlin(技术大学柏林);BIFOLD;Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究所);Max Planck Institute for Human Cognitive and Brain Sciences(人类认知与脑科学Max Planck研究所);Max Planck Institute for Human Development(人类发展Max Planck研究所);TUD Dresden University of Technology(德累斯顿技术大学);Anthropic;Department of Artificial Intelligence, Korea University(人工智能系,韩国大学);Max Planck Institute for Informatics(信息Max Planck研究所)