本文提出 Self-Imagine 方法，利用单个多模态模型将文本问题转为 HTML 图像再结合原问题进行推理，无需额外训练，在多个数学和通用推理任务上显著提升性能。
摘要：
视觉-语言模型（VLMs）在处理复杂的基于文本的问题时，其潜力往往未被充分利用，尤其是当这些问题可以从视觉表示中受益时。与人类解决复杂文本问题的能力相呼应，即（1）从问题中创建视觉图表，以及（2）推断解决该问题需要采取的步骤，我们提出了 Self-Imagine。我们利用单一的视觉-语言模型（VLM）使用 HTML 生成问题的结构化表示，然后将该 HTML 渲染为图像，最后使用同一个 VLM 同时基于问题和图像来回答问题。我们的方法不需要任何额外的训练数据或训练过程。我们使用最先进的（LLAVA-1.5 和 GEMINI PRO）VLM，在三个数学任务和九个通用推理任务上评估了我们的方法。我们的方法提升了 LLAVA-1.5 和 GEMINI PRO 在所有数学任务上的性能（平均 GSM8K：+3.1%；ASDIV：+3.2%；SVAMP：+6.9%），并在大多数通用推理任务上平均提升了 3.2% 到 6.0%。
英文摘要：
The potential of Vision-Language Models (VLMs) often remains underutilized in handling complex text-based problems, particularly when these problems could benefit from visual representation. Resonating with humans' ability to solve complex text-based problems by (1) creating a visual diagram from the problem and (2) deducing what steps they need to take to solve it, we propose Self-Imagine. We leverage a single Vision-Language Model (VLM) to generate a structured representation of the question using HTML, then render the HTML as an image, and finally use the same VLM to answer the question using both the question and the image. Our approach does not require any additional training data or training. We evaluate our approach on three mathematics tasks and nine general-purpose reasoning tasks using state-of-the-art (LLAVA-1.5 and GEMINI PRO) VLMs. Our approach boosts the performance of LLAVA-1.5 and GEMINI PRO on all math tasks (on average GSM8K: +3.1%; ASDIV: +3.2%; SVAMP: +6.9%) and the majority of the general-purpose reasoning tasks by 3.2% to 6.0% on average.
DeepImagine: Clinical Trial Outcome Prediction via Stepwise Local Counterfactual Imaginations
Laboratory for Emerging Intelligence, University of California, San Diego（新兴智能实验室，加州大学圣地亚哥分校）；Department of Dermatology, University of California, San Diego（皮肤科系，加州大学圣地亚哥分校）；University of Chicago（芝加哥大学）；Elsevier（艾尔斯特出版社）
UC Santa Barbara（加州大学圣巴巴拉分校）；UC Santa Cruz（加州大学圣克鲁兹分校）
Automatic evaluations for natural language generation (NLG) conventionally rely on token-level or embedding-level comparisons with text references. This differs from human language processing, for which visual imagination often improves comprehension. In this work, we propose ImaginE, an imagination-based automatic evaluation metric for natural language generation. With the help of StableDiffusion, a state-of-the-art text-to-image generator, we automatically generate an image as the embodied imagination for the text snippet and compute the imagination similarity using contextual embeddings. Ex…
Inria - Flowers team（法国国家信息与自动化研究所花朵团队）；Université de Bordeaux（波尔多大学）；Inserm U1093（法国国家健康与医学研究院U1093）；Cloud Temple（云庙）；Paris（巴黎）；ENSTA ParisTech（巴黎高等科技学院）；Université de Dijon（第戎大学）
Developmental machine learning studies how artificial agents can model the way children learn open-ended repertoires of skills. Such agents need to create and represent goals, select which ones to pursue and learn to achieve them. Recent approaches have considered goal spaces that were either fixed and hand-defined or learned using generative models of states. This limited agents to sample goals within the distribution of known effects. We argue that the ability to imagine out-of-distribution goals is key to enable creative discoveries and open-ended learning. Children do so by leveraging the …
Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigation
Fudan University（复旦大学）；Shanghai AI Laboratory（上海人工智能实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；MAIS, Institute of Automation of Chinese Academy of Sciences（中国科学院自动化研究所MAIS）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Science and Technology of China（中国科学技术大学）；TeleAI, China Telecom Corp Ltd（中国电信TeleAI）
VLM-SAFE: Vision-Language Model-Guided Safety-Aware Reinforcement Learning with World Models for Autonomous Driving
Lyles School of Civil and Construction Engineering, Purdue University（普渡大学莱尔斯土木与建筑工程学院）；Department of Civil and Environmental Engineering, University of Wisconsin-Madison（威斯康星大学麦迪逊分校土木与环境工程系）；Google（谷歌）
Nanyang Technological University（南洋理工大学）；ByteDance AI Lab（字节跳动人工智能实验室）
There are common semantics shared across text and images. Given a sentence in a source language, whether depicting the visual scene helps translation into a target language? Existing multimodal neural machine translation methods (MNMT) require triplets of bilingual sentence - image for training and tuples of source sentence - image for inference. In this paper, we propose ImagiT, a novel machine translation method via visual imagination. ImagiT first learns to generate visual representation from the source sentence, and then utilizes both source sentence and the "imagined representation" to pr…
Human thinking requires the brain to understand the meaning of language expression and to properly organize the thoughts flow using the language. However, current natural language processing models are primarily limited in the word probability estimation. Here, we proposed a Language guided imagination (LGI) network to incrementally learn the meaning and usage of numerous words and syntaxes, aiming to form a human-like machine thinking process. LGI contains three subsystems: (1) vision system that contains an encoder to disentangle the input or imagined scenarios into abstract population repre…
As humans, we can modify our assumptions about a scene by imagining alternative objects or concepts in our minds. For example, we can easily anticipate the implications of the sun being overcast by rain clouds (e.g., the street will get wet) and accordingly prepare for that. In this paper, we introduce a new task/dataset called Commonsense Reasoning for Counterfactual Scene Imagination (CoSIm) which is designed to evaluate the ability of AI systems to reason about scene change imagination. In this task/dataset, models are given an image and an initial question-response pair about the image. Ne…
Learning structured representations of visual scenes is currently a major bottleneck to bridging perception with reasoning. While there has been exciting progress with slot-based models, which learn to segment scenes into sets of objects, learning configurational properties of entire groups of objects is still under-explored. To address this problem, we introduce Constellation, a network that learns relational abstractions of static visual scenes, and generalises these abstractions over sensory particularities, thus offering a potential basis for abstract relational reasoning. We further show …
Artificial Phantasia: Emergent Mental Imagery in Large Language Models
Khoury College of Computer Sciences, Northeastern University（东北大学克劳利计算机科学学院）；Department of Psychology, Northeastern University（东北大学心理学系）；Department of Philosophy, Northeastern University（东北大学哲学系）
Dream to Recall: Imagination-Guided Experience Retrieval for Memory-Persistent Vision-and-Language Navigation
National Key Laboratory of Human Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University（西安交通大学人机混合增强智能全国重点实验室）；School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（上海交通大学自动化与智能感知学院）
Language-Conditioned World Modeling for Visual Navigation
University of Washington（华盛顿大学）；National University of Singapore（新加坡国立大学）；Clemson University（克莱姆森大学）；Drexel University（德雷塞尔大学）；Microsoft Research（微软研究院）
Learning Vision-Language-Action World Models for Autonomous Driving
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院教育部人工智能重点实验室）；Central Research Institute, Huawei（华为中央研究院）
Microsoft Research Asia（微软亚洲研究院）；Nanjing University（南京大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Wuhan University（武汉大学）；University of Technology Sydney（悉尼科技大学）；Tsinghua University（清华大学）
