Benchmarking and Improving Compositional Generalization of Multi-aspect Controllable Text Generation
University of Science and Technology of China（中国科学技术大学）；City University of Hong Kong（香港城市大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Nanyang Technological University（南洋理工大学）
Tianqi Zhong, Zhaoyi Li, Quan Wang, Linqi Song, Ying Wei, Defu Lian, Zhendong Mao
Compositional generalization, representing the model's ability to generate text with new attribute combinations obtained by recombining single attributes from the training data, is a crucial property for multi-aspect controllable text generation (MCTG) methods. Nonetheless, a comprehensive compositional generalization evaluation benchmark of MCTG is still lacking. We propose CompMCTG, a benchmark encompassing diverse multi-aspect labeled datasets and a crafted three-dimensional evaluation protocol, to holistically evaluate the compositional generalization of MCTG approaches. We observe that existing MCTG works generally confront a noticeable performance drop in compositional testing. To mitigate this issue, we introduce Meta-MCTG, a training framework incorporating meta-learning, where we enable models to learn how to generalize by simulating compositional generalization scenarios in the training phase. We demonstrate the effectiveness of Meta-MCTG through achieving obvious improvement (by at most 3.64%) for compositional testing performance in 94.4% cases.
Text style transfer aims to controllably generate text with targeted stylistic changes while maintaining core meaning from the source sentence constant. Many of the existing style transfer benchmarks primarily focus on individual high-level semantic changes (e.g. positive to negative), which enable controllability at a high level but do not offer fine-grained control involving sentence structure, emphasis, and content of the sentence. In this paper, we introduce a large-scale benchmark, StylePTB, with (1) paired sentences undergoing 21 fine-grained stylistic changes spanning atomic lexical, sy…
The Hebrew University of Jerusalem（耶路撒冷希伯来大学）；Allen Institute for AI（艾伦人工智能研究院）；Bar-Ilan University（巴伊兰大学）；Tel-Aviv University（特拉维夫大学）；Stanford University（斯坦福大学）
While neural language models often perform surprisingly well on natural language understanding (NLU) tasks, their strengths and limitations remain poorly understood. Controlled synthetic tasks are thus an increasingly important resource for diagnosing model behavior. In this work we focus on story understanding, a core competency for NLU systems. However, the main synthetic resource for story understanding, the bAbI benchmark, lacks such a systematic mechanism for controllable task generation. We develop Dyna-bAbI, a dynamic framework providing fine-grained control over task generation in bAbI…
University of California, Santa Barbara（加州大学圣巴巴拉分校）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；Google（谷歌）
Large-scale diffusion models have achieved state-of-the-art results on text-to-image synthesis (T2I) tasks. Despite their ability to generate high-quality yet creative images, we observe that attribution-binding and compositional capabilities are still considered major challenging issues, especially when involving multiple objects. In this work, we improve the compositional skills of T2I models, specifically more accurate attribute binding and better image compositions. To do this, we incorporate linguistic structures with the diffusion guidance process based on the controllable properties of …
While deep generative models have become the leading methods for algorithmic composition, it remains a challenging problem to control the generation process because the latent variables of most deep-learning models lack good interpretability. Inspired by the content-style disentanglement idea, we design a novel architecture, under the VAE framework, that effectively learns two interpretable latent factors of polyphonic music: chord and texture. The current model focuses on learning 8-beat long piano composition segments. We show that such chord-texture disentanglement provides a controllable g…
HACHIMI: Scalable and Controllable Student Persona Generation via Orchestrated Agents
East China Normal University（华东师范大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Shanghai Innovation Institute（上海创新研究院）
MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education
AI Singapore, National University of Singapore（新加坡国立大学新加坡人工智能）；School of Computing, National University of Singapore（新加坡国立大学计算学院）；Institute of Advanced Intelligence and Computing, A*STAR（新加坡科技研究局先进智能与计算研究所）
