Automated design synthesis has the potential to revolutionize the modern engineering design process and improve access to highly optimized and customized products across countless industries. Successfully adapting generative Machine Learning to design engineering may enable such automated design synthesis and is a research subject of great importance. We present a review and analysis of Deep Generative Machine Learning models in engineering design. Deep Generative Models (DGMs) typically leverage deep networks to learn from an input dataset and synthesize new designs. Recently, DGMs such as feedforward Neural Networks (NNs), Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and certain Deep Reinforcement Learning (DRL) frameworks have shown promising results in design applications like structural optimization, materials design, and shape synthesis. The prevalence of DGMs in engineering design has skyrocketed since 2016. Anticipating continued growth, we conduct a review of recent advances to benefit researchers interested in DGMs for design. We structure our review as an exposition of the algorithms, datasets, representation methods, and applications commonly used in the current literature. In particular, we discuss key works that have introduced new techniques and methods in DGMs, successfully applied DGMs to a design-related domain, or directly supported the development of DGMs through datasets or auxiliary methods. We further identify key challenges and limitations currently seen in DGMs across design fields, such as design creativity, handling constraints and objectives, and modeling both form and functional performance simultaneously. In our discussion, we identify possible solution pathways as key areas on which to target future work.
Generative VS non-Generative Models in Engineering Shape Optimization
University of Strathclyde（斯特拉斯克莱德大学）；Nazarbayev University（纳扎尔巴耶夫大学）；BAR Technologies（BAR科技公司）；Foundation for Research & Technology Hellas (FORTH)（希腊研究与技术基金会）
Deep generative models are a class of techniques that train deep neural networks to model the distribution of training samples. Research has fragmented into various interconnected approaches, each of which make trade-offs including run-time, diversity, and architectural restrictions. In particular, this compendium covers energy-based models, variational autoencoders, generative adversarial networks, autoregressive models, normalizing flows, in addition to numerous hybrid approaches. These techniques are compared and contrasted, explaining the premises behind each and how they are interrelated,…
National Institute of Technology Calicut（卡利卡特国家理工学院）
The microstructure of material strongly influences its mechanical properties and the microstructure itself is influenced by the processing conditions. Thus, establishing a Process-Structure-Property relationship is a crucial task in material design and is of interest in many engineering applications. We develop a GAN (Generative Adversarial Network) to synthesize microstructures based on given processing conditions. This approach is devoid of feature engineering, needs little domain awareness, and can be applied to a wide variety of material systems. Results show that our GAN model can produce…
Generative AI Models for Different Steps in Architectural Design: A Literature Review
School of Architecture, Tianjin University（天津大学建筑学院）；Graduate School of Advanced Science and Technology, Japan Advanced Institute of Science and Technology（日本先进科学技术学院先进科学技术研究院）
University of Saskatchewan（萨斯喀彻温大学）；Philips Canada（飞利浦加拿大）
Generative adversarial networks have gained a lot of attention in the computer vision community due to their capability of data generation without explicitly modelling the probability density function. The adversarial loss brought by the discriminator provides a clever way of incorporating unlabeled samples into training and imposing higher order consistency. This has proven to be useful in many cases, such as domain adaptation, data augmentation, and image-to-image translation. These properties have attracted researchers in the medical imaging community, and we have seen rapid adoption in man…
A Review of Human Emotion Synthesis Based on Generative Technology
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；MICS Thrust, The Hong Kong University of Science and Technology (GZ)（香港科技大学（广州）MICS学科）；School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China（电子科技大学深圳高等研究院）；School of Information Technology, Carleton University（卡尔顿大学信息技术学院）
Institute for AI in Medicine (IKIM), University Medicine Essen（埃森大学医学人工智能研究所（IKIM））；Computer Vision and Learning Lab (CVL), Heidelberg University（海德堡大学计算机视觉与学习实验室（CVL））
Deep neural networks are commonly used for medical purposes such as image generation, segmentation, or classification. Besides this, they are often criticized as black boxes as their decision process is often not human interpretable. Encouraging the latent representation of a generative model to be disentangled offers new perspectives of control and interpretability. Understanding the data generation process could help to create artificial medical data sets without violating patient privacy, synthesizing different data modalities, or discovering data generating characteristics. These character…
University of California, Santa Barbara（加州大学圣巴巴拉分校）；Carnegie Mellon University Silicon Valley（卡内基梅隆大学硅谷校区）
In the past few years, consumer review sites have become the main target of deceptive opinion spam, where fictitious opinions or reviews are deliberately written to sound authentic. Most of the existing work to detect the deceptive reviews focus on building supervised classifiers based on syntactic and lexical patterns of an opinion. With the successful use of Neural Networks on various classification applications, in this paper, we propose FakeGAN a system that for the first time augments and adopts Generative Adversarial Networks (GANs) for a text classification task, in particular, detectin…
This document aims to provide a review on learning with deep generative models (DGMs), which is an highly-active area in machine learning and more generally, artificial intelligence. This review is not meant to be a tutorial, but when necessary, we provide self-contained derivations for completeness. This review has two features. First, though there are different perspectives to classify DGMs, we choose to organize this review from the perspective of graphical modeling, because the learning methods for directed DGMs and undirected DGMs are fundamentally different. Second, we differentiate mode…
National Engineering Research Center for Big Data Software（国家大数据软件工程技术研究中心）；School of Software, Tsinghua University（清华大学软件学院）
Today, many industrial processes are undergoing digital transformation, which often requires the integration of well-understood domain models and state-of-the-art machine learning technology in business processes. However, requirements elicitation and design decision making about when, where and how to embed various domain models and end-to-end machine learning techniques properly into a given business workflow requires further exploration. This paper aims to provide an overview of the requirements engineering process for machine learning applications in terms of cross domain collaborations. W…
A recommender system's basic task is to estimate how users will respond to unseen items. This is typically modeled in terms of how a user might rate a product, but here we aim to extend such approaches to model how a user would write about the product. To do so, we design a character-level Recurrent Neural Network (RNN) that generates personalized product reviews. The network convincingly learns styles and opinions of nearly 1000 distinct authors, using a large corpus of reviews from BeerAdvocate.com. It also tailors reviews to describe specific items, categories, and star ratings. Using a sim…
Context: Mobile app reviews written by users on app stores or social media are significant resources for app developers.Analyzing app reviews have proved to be useful for many areas of software engineering (e.g., requirement engineering, testing). Automatic classification of app reviews requires extensive efforts to manually curate a labeled dataset. When the classification purpose changes (e.g. identifying bugs versus usability issues or sentiment), new datasets should be labeled, which prevents the extensibility of the developed models for new desired classes/tasks in practice. Recent pre-tr…
CSIRO（澳大利亚联邦科学与工业研究组织）；University of New South Wales（新南威尔士大学）
Federated learning is an emerging machine learning paradigm where clients train models locally and formulate a global model based on the local model updates. To identify the state-of-the-art in federated learning and explore how to develop federated learning systems, we perform a systematic literature review from a software engineering perspective, based on 231 primary studies. Our data synthesis covers the lifecycle of federated learning system development that includes background understanding, requirement analysis, architecture design, implementation, and evaluation. We highlight and summar…
Data scientists often develop machine learning models to solve a variety of problems in the industry and academy but not without facing several challenges in terms of Model Development. The problems regarding Machine Learning Development involves the fact that such professionals do not realize that they usually perform ad-hoc practices that could be improved by the adoption of activities presented in the Software Engineering Development Lifecycle. Of course, since machine learning systems are different from traditional Software systems, some differences in their respective development processe…
Prompt engineering paradigms for medical applications: scoping review and recommendations for better practices
Geneva University Hospitals（日内瓦大学医院）；University of Geneva（日内瓦大学）；Université Paris-Saclay（巴黎萨克雷大学）；CNRS（法国国家科学研究中心）；Sorbonne Université（索邦大学）；INSERM（法国国家健康与医学研究院）；Université Sorbonne Paris-Nord（索邦巴黎北大学）
During the last two years there has been a plethora of large generative models such as ChatGPT or Stable Diffusion that have been published. Concretely, these models are able to perform tasks such as being a general question and answering system or automatically creating artistic images that are revolutionizing several sectors. Consequently, the implications that these generative models have in the industry and society are enormous, as several job positions may be transformed. For example, Generative AI is capable of transforming effectively and creatively texts to images, like the DALLE-2 mod…
University of Montréal（蒙特利尔大学）；University of Rouen（鲁昂大学）；Facebook Artificial Intelligence Research（Facebook人工智能研究院）
Sentiment analysis is a common task in natural language processing that aims to detect polarity of a text document (typically a consumer review). In the simplest settings, we discriminate only between positive and negative sentiment, turning the task into a standard binary classification problem. We compare several ma- chine learning approaches to this problem, and combine them to achieve the best possible results. We show how to use for this task the standard generative lan- guage models, which are slightly complementary to the state of the art techniques. We achieve strong results on a well-…
Internet of Things (IoT) has catapulted human ability to control our environments through ubiquitous sensing, communication, computation, and actuation. Over the past few years, IoT has joined forces with Machine Learning (ML) to embed deep intelligence at the far edge. TinyML (Tiny Machine Learning) has enabled the deployment of ML models for embedded vision on extremely lean edge hardware, bringing the power of IoT and ML together. However, TinyML powered embedded vision applications are still in a nascent stage, and they are just starting to scale to widespread real-world IoT deployment. To…
A Research Roadmap for Augmenting Software Engineering Processes and Software Products with Generative AI
University of Naples "Federico II" Napoli Italy；Ruhr Institute for Software Technology (paluno), University of Duisburg-Essen Essen Germany；Karlsruhe Institute of Technology (KIT) Karlsruhe Germany；Gran Sasso Science Institute (GSSI) L'Aquila Italy；University of Naples "Federico II"；Ruhr Institute for Software Technology (paluno), University of Duisburg-Essen；Karlsruhe Institute of Technology (KIT)；Gran Sasso Science Institute (GSSI)
Washington & Lee University（华盛顿与李大学）；William & Mary（威廉与玛丽学院）；George Mason University（乔治梅森大学）
An increasingly popular set of techniques adopted by software engineering (SE) researchers to automate development tasks are those rooted in the concept of Deep Learning (DL). The popularity of such techniques largely stems from their automated feature engineering capabilities, which aid in modeling software artifacts. However, due to the rapid pace at which DL techniques have been adopted, it is difficult to distill the current successes, failures, and opportunities of the current research landscape. In an effort to bring clarity to this crosscutting area of work, from its modern inception to…
