Large vision-language models (VLMs) have become state-of-the-art for many computer vision tasks, with in-context learning (ICL) as a popular adaptation strategy for new ones. But can VLMs learn novel concepts purely from visual demonstrations, or are they limited to adapting to the output format of ICL examples? We propose a new benchmark we call Spatial Visual Ambiguity Tasks (SVAT) that challenges state-of-the-art VLMs to learn new visuospatial tasks in-context. We find that VLMs fail to do this zero-shot, and sometimes continue to fail after finetuning. However, adding simpler data to the training by curriculum learning leads to improved ICL performance.
Beijing Institute of Technology（北京理工大学）；Peking University（北京大学）；Yangtze Delta Region Academy of Beijing Institute of Technology（北京理工大学长三角研究院）；BIGAI（北京通用人工智能研究院）
Decompose and Recompose: Reasoning New Skills from Existing Abilities for Cross-Task Robotic Manipulation
School of Artificial Intelligence, College of Intelligence and Computing, Tianjin University, China（人工智能学院，智能与计算学院，天津大学，中国）；School of Computer Science and Information Engineering, Hefei University of Technology, China（计算机科学与信息工程学院，合肥工业大学，中国）
本文提出Decompose and Recompose框架，通过分解现有任务演示为可解释的技能-动作对，实现跨任务机械操作的零样本泛化。
Fairness in Multi-modal Medical Diagnosis with Demonstration Selection
Arizona State University（亚利桑那州立大学）；University of Rochester（罗切斯特大学）；University of Virginia（弗吉尼亚大学）；UCL（伦敦大学学院）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Central Florida（佛罗里达中央大学）
Singapore Management University（新加坡管理大学）；Southwest Jiaotong University（西南交通大学）；Singapore University of Technology and Design（新加坡科技设计大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
