DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models
University of Chinese Academy of Sciences（中国科学院大学）；Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室）；ByteDance Inc.（字节跳动公司）
Ying Zhou, Xinyao Wang, Yulei Niu, Yaojie Shen, Lexin Tang, Fan Chen, Ben He, Le Sun, Longyin Wen
备注：
21 pages, 9 figures, Accepted by ACL 2025, Findings
Recent advancements in large language models (LLMs) have significantly enhanced their knowledge and generative capabilities, leading to a surge of interest in leveraging LLMs for high-quality data synthesis. However, synthetic data generation via prompting LLMs remains challenging due to LLMs' limited understanding of target data distributions and the complexity of prompt engineering, especially for structured formatted data. To address these issues, we introduce DiffLM, a controllable data synthesis framework based on variational autoencoder (VAE), which further (1) leverages diffusion models to reserve more information of original distribution and format structure in the learned latent distribution and (2) decouples the learning of target distribution knowledge from the LLM's generative objectives via a plug-and-play latent feature injection module. As we observed significant discrepancies between the VAE's latent representations and the real data distribution, the latent diffusion module is introduced into our framework to learn a fully expressive latent distribution. Evaluations on seven real-world datasets with structured formatted data (i.e., Tabular, Code, and Tool data) demonstrate that DiffLM generates high-quality data, with performance on downstream tasks surpassing that of real data by 2%-7% in certain cases. Data and code are available at https://github.com/bytedance/DiffLM.
Pain in 3D: Generating Controllable Synthetic Faces for Automated Pain Assessment
Vector Institute（向量研究所）；KITE Research Institute（KITE研究机构）；University Health Network（大学健康网络）；Department of Computer Science, University of Toronto（多伦多大学计算机科学系）；Department of Medical Imaging, University of Toronto（多伦多大学医学成像系）；Institute of Biomedical Engineering, University of Toronto（多伦多大学生物医学工程研究所）
SALIENT: Frequency-Aware Paired Diffusion for Controllable Long-Tail CT Detection
Trauma Radiology AI Laboratory (TRAIL)（创伤放射AI实验室）；Department of Radiology and Nuclear Medicine（放射科与核医学科）；University of Maryland School of Medicine（马里兰大学医学院）
H-LDM: Hierarchical Latent Diffusion Models for Controllable and Interpretable PCG Synthesis from Clinical Metadata
School of Cyber Engineering Xidian University Xi'an, China（电子科技大学信息工程学院西安中国）；School of Telecommunications Engineering Xidian University Xi'an, China（电子科技大学电信工程学院西安中国）
REGEN: Reference-Guided Synthetic Multivariate Time Series Generation for Forecasting
Birla AI Labs, Office of Ananya Birla（Birla AI实验室，Ananya Birla办公室）；Birla Institute of Technology and Science, Pilani（Birla理工学院与科学学院，Pilani）；Kalinga Institute of Industrial Technology, Bhubaneswar（Kalinga工业技术学院，Bhubaneswar）
HACHIMI: Scalable and Controllable Student Persona Generation via Orchestrated Agents
East China Normal University（华东师范大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Shanghai Innovation Institute（上海创新研究院）
DocDjinn: Controllable Synthetic Document Generation with VLMs and Handwriting Diffusion
RheinMain University of Applied Sciences（莱茵-美因应用科学大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）；DeepReader GmbH（DeepReader公司）；Insiders Technologies GmbH（Insiders Technologies公司）；National University of Sciences and Technology（国家科学与技术大学）
Adapting generic speech recognition models to specific individuals is a challenging problem due to the scarcity of personalized data. Recent works have proposed boosting the amount of training data using personalized text-to-speech synthesis. Here, we ask two fundamental questions about this strategy: when is synthetic data effective for personalization, and why is it effective in those cases? To address the first question, we adapt a state-of-the-art automatic speech recognition (ASR) model to target speakers from four benchmark datasets representative of different speaker types. We show that…
Our ability to sample realistic natural images, particularly faces, has advanced by leaps and bounds in recent years, yet our ability to exert fine-tuned control over the generative process has lagged behind. If this new technology is to find practical uses, we need to achieve a level of control over generative networks which, without sacrificing realism, is on par with that seen in computer graphics and character animation. To this end we propose ConfigNet, a neural face model that allows for controlling individual aspects of output images in semantically meaningful ways and that is a signifi…
KAUST（阿卜杜拉国王科技大学）；Adobe Research（奥多比研究院）；University College London（伦敦大学学院）
Scalable generation of furniture layouts is essential for many applications in virtual reality, augmented reality, game development and synthetic data generation. Many existing methods tackle this problem as a sequence generation problem which imposes a specific ordering on the elements of the layout making such methods impractical for interactive editing or scene completion. Additionally, most methods focus on generating layouts unconditionally and offer minimal control over the generated layouts. We propose COFS, an architecture based on standard transformer architecture blocks from language…
Methods for designing organic materials with desired properties have high potential impact across fields such as medicine, renewable energy, petrochemical engineering, and agriculture. However, using generative modeling to design substances with desired properties is difficult because candidate compounds must satisfy multiple constraints, including synthetic accessibility and other metrics that are intuitive to domain experts but challenging to quantify. We propose C5T5, a novel self-supervised pretraining method that enables transformers to make zero-shot select-and-replace edits, altering or…
