Multi-modal program synthesis refers to the task of synthesizing programs (code) from their specification given in different forms, such as a combination of natural language and examples. Examples provide a precise but incomplete specification, and natural language provides an ambiguous but more "complete" task description. Machine-learned pre-trained models (PTMs) are adept at handling ambiguous natural language, but struggle with generating syntactically and semantically precise code. Program synthesis techniques can generate correct code, often even from incomplete but precise specifications, such as examples, but they are unable to work with the ambiguity of natural languages. We present an approach that combines PTMs with component-based synthesis (CBS): PTMs are used to generate candidates programs from the natural language description of the task, which are then used to guide the CBS procedure to find the program that matches the precise examples-based specification. We use our combination approach to instantiate multi-modal synthesis systems for two programming domains: the domain of regular expressions and the domain of CSS selectors. Our evaluation demonstrates the effectiveness of our domain-agnostic approach in comparison to a state-of-the-art specialized system, and the generality of our approach in providing multi-modal program synthesis from natural language and examples in different programming domains.
Differentiable Modal Synthesis for Physical Modeling of Planar String Sound and Motion Simulation
Seoul National University（首尔大学）；Artificial Intelligence Institute of Seoul National University（首尔大学人工智能学院）；Interdisciplinary Program in Artificial Intelligence（人工智能跨学科项目）
Synesthesia of Vehicles: Tactile Data Synthesis from Visual Inputs
Dept. of Transportation Science, Beihang Univ.（北京航空航天大学交通运输科学系）；State Key Lab of Intelligent Transportation System, Beihang Univ.（北京航空航天大学智能交通系统国家重点实验室）；Hangzhou International Innovation Institute, Beihang Univ.（杭州国际创新研究院）；China Software Testing Center(Ministry of Industry and Information Technology Software and Integrated Circuit Promotion Center)（中国软件测试中心（工业和信息化部软件与集成电路促进中心））
KAUST（阿卜杜拉国王科技大学）；iFlyTek-Surrey Joint Research Center on Artificial Intelligence（讯飞-萨里人工智能联合研究中心）；University of Surrey（萨里大学）；Meta；Surrey Institute for People-Centred Artificial Intelligence（萨里以人为本人工智能研究所）
Few-shot (FS) and zero-shot (ZS) learning are two different approaches for scaling temporal action detection (TAD) to new classes. The former adapts a pretrained vision model to a new task represented by as few as a single video per class, whilst the latter requires no training examples by exploiting a semantic description of the new class. In this work, we introduce a new multi-modality few-shot (MMFS) TAD problem, which can be considered as a marriage of FS-TAD and ZS-TAD by leveraging few-shot support videos and new class names jointly. To tackle this problem, we further introduce a novel M…
OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis
Shanghai AI Laboratory（上海人工智能实验室）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；Shanghai Jiaotong University（上海交通大学）；Tsinghua University（清华大学）
Though significant progress has been made for speaker-dependent Video-to-Speech (VTS) synthesis, little attention is devoted to multi-speaker VTS that can map silent video to speech, while allowing flexible control of speaker identity, all in a single system. This paper proposes a novel multi-speaker VTS system based on cross-modal knowledge transfer from voice conversion (VC), where vector quantization with contrastive predictive coding (VQCPC) is used for the content encoder of VC to derive discrete phoneme-like acoustic units, which are transferred to a Lip-to-Index (Lip2Ind) network to inf…
FedMed-GAN: Federated Domain Translation on Unsupervised Cross-Modality Brain Image Synthesis
Southern University of Science and Technology（南方科技大学）；University of Surrey（萨里大学）；Tencent（腾讯）；University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；Bielefeld University（比勒费尔德大学）
LeMat-Synth: a multi-modal toolbox to curate broad synthesis procedure databases from scientific literature
Massachusetts Institute of Technology（麻省理工学院）；Entalpic（Entalpic公司）；URM MAP；LIRIS（LIRIS实验室）；Georgia Tech（佐治亚理工学院）；University of California Santa Barbara（加州大学圣巴巴拉分校）；Chalmers University of Technology（查尔姆斯理工大学）；University of Gothenburg（哥德堡大学）；ETH Zurich（苏黎世联邦理工学院）；EPFL（洛桑联邦理工学院）；Johns Hopkins University（约翰斯·霍普金斯大学）；Hugging Face（Hugging Face公司）；Imperial College London（帝国理工学院）
Magnetic resonance imaging (MRI) is being increasingly utilized to assess, diagnose, and plan treatment for a variety of diseases. The ability to visualize tissue in varied contrasts in the form of MR pulse sequences in a single scan provides valuable insights to physicians, as well as enabling automated systems performing downstream analysis. However many issues like prohibitive scan time, image corruption, different acquisition protocols, or allergies to certain contrast materials may hinder the process of acquiring multiple sequences for a patient. This poses challenges to both physicians a…
