A literature review: Generative adversarial networks for civil structural health monitoring

Structural Health Monitoring (SHM) of civil structures has been constantly evolving with novel methods, advancements in data science, and more accessible technology to address issues related to structural safety, operations, and resiliency. Research and development in the civil SHM field during the last few decades have been progressive due to the increasing use of Artificial Intelligence (AI) methods such as Machine Learning (ML) and Deep Learning (DL). Particularly, Generative Adversarial Networks (GAN), which is a subfamily of Deep Learning has been highly favored in the SHM community within the last couple of years. After its release in 2014, GANs (original GAN and other GAN variants) have been in use for a wide variety of applications in various disciplines, and it has been one of the most popular research topics in the AI-ML domain. While there has not been a review study on the applications of GAN in the civil SHM field, this paper aims to fill this gap by presenting a literature review of the studies that employed GAN specifically in civil SHM applications from 2014 to date, in a condensed format. This study intends to inform SHM practitioners and researchers about GANs and present the highlights of the published work on GANs in the civil SHM field.

1 Introduction

1.1 Civil structural health monitoring

While the environmental or man-made stressors negatively impact the integrity and consequently shorten the remaining useful lives of civil engineering structures, the old and ageing civil structures are getting more vulnerable to such external effects as time passes by. Therefore, there is a growing concern for the health condition of civil structures around the world and as a result, a great deal of research and development is dedicated to this matter.

Currently, in practice, condition assessment of civil structures is mostly implemented via conventional techniques such as hammer tapping, chain dragging to identify voids, and visual inspection for delamination in concrete and other visible items. Yet, the traditional methods can be challenging (e.g., time-consuming, labor-intensive, and subjective) since the conventional methods solely depend on human effort. To overcome such challenges, Structural Health Monitoring (SHM) has been in use [(Aktan et al., 1996; Housner et al., 1997)]. Briefly, SHM is a methodology to examine the health status of civil structures, mostly for large and occasionally medium-sized civil structure systems, by collecting sensorial data with the use of sensors e.g., accelerometers, strain gauges, potentiometers, cameras, lasers, and Non-Destructive Technique/Evaluation (NDT/E) (Gucunski et al., 2010; Scott et al., 2003; Stanley et al., 1995) which use sophisticated tools such as ultrasound, ground penetration radar, infrared camera, and electromagnetic methods to detect local defects. The goal of SHM is to track structural responses to determine the structural condition to support decision-making. The decision could be for a number of purposes such as design verification, damage detection, effective and efficient maintenance and operations. This is achieved by means of tracking changes in geometric or material properties of the structure based on the analyzed sensorial data for the decision-making process. In other words, SHM includes structural assessment, diagnosis and prognosis processes based on the analysis and evaluation of the collected sensorial information (Farrar and Worden, 2007). Usually, the SHM techniques are classified by their application level such as local and global levels (Dong and Catbas, 2021). At the local level, SHM applications aim to diagnose the local damages via NDT/E and Non-Contact Tools or NCT. Identification of the dynamic behavior and diagnosing irregularities in the global movement of the complete structure is considered SHM at the global level. Global SHM can be implemented via fixed systems such as vibration-strain-displacement-based sensorial data collection and possibly NCT [(Li et al., 2016; Avci et al., 2021)]. Figure 1 describes SHM considering their application levels including both at local and global levels. Furthermore, NCT such as computer vision tools including RGB and IR camera and Light Detection and Ranging (LiDAR) has demonstrated many applications both at the local level (e.g., crack, or spalling detection) and global level (e.g., displacement and vibration monitoring) (Catbas and Khuc, 2018; Feng and Feng, 2018; Kaartinen et al., 2022).

FIGURE 1

FIGURE 1. Structural Health Monitoring considering their application levels.

The application levels of SHM could be shown from another perspective as new tools emerge in the field, which is depicted in Figure 2. As the focus shifted to robotic applications over time, researchers have started to implement robot-based methods (Zhu et al., 2012; van Nguyen et al., 2018; McLaughlin et al., 2020). In SHM applications, robotic systems could be utilized at both global and local levels. For instance, an unmanned ground vehicle (UGV) can collect acceleration data to carry out modal identification (global SHM) and at the same time implement image-based defect detection (local SHM) via the vision sensors installed on its platform. If the robot is only equipped with accelerometers, it can only perform global behavior analysis (global SHM). However, since UGV contributes to high spatial resolution in the collected data, it may also capture the local defects in the structure (local SHM). Similarly, an unmanned aerial vehicle (UAV) can implement image-based defect detection applications (local SHM) or displacement measurement (global SHM) via vision sensors installed on its platform by tracking the specific features in the images (Ribeiro et al., 2021). Additionally, while the NDT/E is used to perform local SHM by examining inside of the structural component, the fixed systems (e.g., accelerometers, strain gauges, potentiometers) may only carry out global SHM applications. NCT, on the other hand, is found in more widespread use in both global and local level SHM continuum. As such, while LiDAR, RGB and IR cameras could be used on robots, they can also be used as fixed systems or as NDT tools.

FIGURE 2

FIGURE 2. Global and local levels of SHM from another perspective as new tools emerge in SHM.

1.2 Motivation, objective, and scope

As per the discussion so far, SHM involves a great deal of data-driven applications such as data collection, preprocessing, and analysis. In the last decade, the research and development in the SHM field have significantly benefited from the use of Artificial Intelligence (AI) methods. Especially, it is observed that researchers use AI methods such as Machine Learning (ML) or Deep Learning (DL) to come up with novel and effective solutions to challenging problems. Since the ML and DL methods can help to discover and learn the complex features in data structures, they have been a research trend for the last decade. Particularly, DL algorithms have been the main focus of research due to their end-to-end solving approach, without any need for manual intervention as DL models learn and extract the features from the data, and reveal enhanced learning performance in complex data structures (Janiesch et al., 2021). More recently, the domain adaptation technique has attracted great attention which aims to build DL models to generalize the training domain to the target domain and deal with the discrepancy between different data distributions. Generative Adversarial Networks (GAN) comes out as a subfamily of DL that can be used in domain adaptation applications. As many ML and DL models have been used in many different disciplines, GANs (original GAN and other GAN variants) also have been highly favored since they were first introduced by Goodfellow et al. (2014); and the SHM community has started to implement the idea of GAN in their applications. However, after a thorough literature review by the authors, there has not been a published review study on the use of GANs in civil SHM. The authors are aiming to fill this gap to summarize the progression of GANs in the civil SHM field. Therefore, this paper aims to present a review of civil SHM work involving the use of GANs in an attempt to benefit both academics and practitioners. The study also intends to inform SHM researchers about the capability of GANs and present the major highlights of the published work on GANs in the civil SHM field, in a condensed format. The rest of the study follows this order: Section 2 explains the search methodology used in this review; Section 3 introduces the fundamentals of GANs; Section 4 presents the GAN-related studies in the civil SHM field; Section 5 provides a discussion and recommendations based on the conducted review; Section 6 concludes the paper with summary and conclusions.

2 Research methodology

It is pertinent to provide details on the research methodology adopted for this review paper. The workflow for this methodology is shown in Figure 3. In data collection, Scopus and Google Scholar database search engines are employed and the keywords of “Generative Adversarial Networks AND Structural Health Monitoring” and “GAN AND SHM” are used. In data analysis, the obtained data from the literature are screened and assessed for eligibility to be included in the review paper. Subsequently, the identified studies are classified based on their applications and their research problems. For the selection of studies, the PRISMA flow diagram is used as illustrated in Figure 4. A total of 155 studies (73 from Google Scholar and 82 from Scopus) have been screened to assess the eligibility to be included in this review. The screening process includes eliminating the studies that are not directly related to using GANs for civil SHM purposes and applications. As such, this paper only includes using “GAN or variants” (GANs) for structural damage diagnosis and prognosis based on the sensorial data (1-D and 2-D data) collected from civil structures (excluding roadway surfaces and pavement structures). The civil SHM field was briefly discussed in Section 1.1. In addition, the studies that cannot be accessed due to the use of a different language other than English or website error were also eliminated. After the screening process, the remaining number of studies was 45. As a result, 45 GAN-related studies published in the civil SHM field are included in this paper. The bar chart shown in Figure 5 indicates the number of published studies per year after the data screening process (Figure 4). It is observed that the number of GAN studies in the civil SHM field is in an increasing trend over the years. Although GAN was released in 2014, it has been introduced to the civil SHM field in 2018 with a delay of 4 years. Thus, it can be argued that the applications of GANs are still in the growing phase in the civil SHM field.

FIGURE 3

FIGURE 3. Research methodology workflow.

FIGURE 4

FIGURE 4. The PRISMA flow diagram for records selection according to (Page et al., 2021).

FIGURE 5

FIGURE 5. The number of papers included in this study published on GAN-based SHM after the PRISMA flow diagram presented in Figure 4.

3 Generative adversarial networks

It is pertinent to provide the fundamentals of Generative Adversarial Networks before discussing their applications of it in the civil SHM field. In 2014, Ian Goodfellow and his colleagues introduced a generative-based DL model, Generative Adversarial Networks (Goodfellow et al., 2014). The model is a combination of a generative network ($G_{\theta }$) and a discriminative network ($D_{\phi }$) where $G_{\theta }$ takes a random noise data $z$ and then maximizes the likelihood that the data it generates $G_{\theta }\left(z\right)$ or $\stackrel{\sim }{x}$ is as similar as possible to the real data $x$. Subsequently, $\stackrel{\sim }{x}$ is fed to the discriminative network ($D_{\phi }$) and also the real data sample $x$ is fed to $D_{\phi }$. Then, based on the scores of $D_{\phi }$ on $\stackrel{\sim }{x}$, in other words $D_{\phi }\left(G_{\theta }\left(z\right)\right)$, $G_{\theta }$ updates its weights via backpropagation. Additionally, based on the differences in scores of $D_{\phi }$ on, in other words $D_{\phi }\left(x\right)$, and $D_{\phi }\left(G_{\theta }\left(z\right)\right)$, $D_{\phi }$ updates its weights. In this respect, GAN is thought of as a two-player game where each player tries to deceive the other player. Then, each player learns from the other’s outputs and tries to be better at each turn. The introduced objective function to train GAN in the original paper is shown in Eq. 1. Additionally, the illustration of the GAN as to the data flow mechanism through the generator and discriminator is shown in Figure 6.

${\min }_{\theta }{\max }_{\phi }V\left(G_{\theta },D_{\phi }\right)={\mathrm{E}}_{x\sim pdata\left(x\right)}\left[\mathit{log}{D}_{\phi }\left(x\right)\right]+{\mathrm{E}}_{z\sim pz\left(z\right)}\left[\mathit{log}\left(1-D_{\phi }\left(G_{\theta }\left(z\right)\right)\right)\right](1)$

FIGURE 6

FIGURE 6. Data flow in GAN (Google, 2022).

The authors in Goodfellow et al. (2014) showed that the introduced GAN model demonstrates impressive image generation performance such that the generator was able to generate pictures ($\stackrel{\sim }{x}$) that are very similar to the ground truth images ($x$). After the introduction of GAN, it has been considered a major success in AI and has become one of the main focuses of researchers. Most of the succeeding studies focused on the difficulties of training GAN due to its complex objective function than the typical deep learning networks. Training GAN means finding a unique solution to Nash equilibrium (the balance between generator and discriminator), which may be very difficult to do so. Thus, often the GAN model never converges. As a result, the training suffers from large oscillation in loss values of the generator and discriminator which makes the training highly unstable. One of the most famous problems of training GAN is “mode collapse”. This happens when the generator finds a way to generate data that would look very similar to what the discriminator knows about the real data. In this way, the generator keeps producing the same output to trick the discriminator during the training as this is an unwanted performance from the generator since it reduces the creativity and diversity of the generator. Additionally, at the beginning of the training, the discriminator performs better since it can easily guess the real data from the synthetic data, thus rejecting all the outputs the generator produces. At this point, training could not be successful due to vanishing gradients as the generator does not learn gradients from the discriminator. To overcome or alleviate these types of problems that are faced during the training, researchers proposed some tips (or so-called hacks) [(Goodfellow, 2016; Salimans et al., 2016)].

After the successful adoption of convolutional neural networks in the computer vision field, in one study (Radford et al., 2015), authors proposed using a deeper network built on convolutions for GAN which they named Deep Convolutional GAN (DCGAN). The authors of that study noted that using deep convolutions helped the model to learn the domain features better and also observed that convolutions alleviated the effects of unstable training. The authors also observed that with simple arithmetic operations on the noise vector given to the generator, they can obtain the desired image results. Figure 7A below shows the generated images with DCGAN after five epochs of training with bedroom pictures and Figure 7B shows the generated images with DCGAN after arithmetic operations with noise vector.

FIGURE 7

FIGURE 7. (A) Generated images with DCGAN after five epochs of training with bedroom pictures, (B) generated images after arithmetic operations with noise vector (Radford et al., 2015).

Moreover, in another study (Arjovsky et al., 2017), the authors addressed the difficulty of training GAN. The authors suggested a radical solution that utilizes Wasserstein distance as the distance metric between the real and the generated data. In previous GAN models, binary cross entropy was used as a distance function. However, Wasserstein distance is a distance metric that measures the distance between probability distributions in metric space. Thereby, it provides more meaningful loss values than the loss functions of the original GAN. Consequently, the authors removed the sigmoid function from the discriminator, thus the discriminator gives a scalar value rather than a probability. For each output scalar value, the discriminator yields, the mean of how much the synthetic data is similar to the real data. As a result, the authors converted the name of the discriminator to “critic”, and they named the model Wasserstein GAN (WGAN). The authors observed that using Wasserstein distance stabilizes the training with smoother gradients flow and assists the critic (discriminator) and generator to provide more accurate outputs. They also observed that Wasserstein distance diminished the exploding gradient problem. In order to compute the Wasserstein distance, the 1-Lipschitz function needs to be defined. For that, the authors implemented a simple weight clipping to restrict weights within a particular range that is controlled by a parameter. The authors of that study also mentioned that weight clipping is a “terrible way” to enforce the Lipschitz function because it may cause the model to lower its learning capacity as the user may not predict the limits of the function (Arjovsky et al., 2017). Additionally, the authors noticed that the model is very sensitive to its parameters such as when the weight clipping parameter is picked high, the training time increases significantly or if it is picked small, the vanishing gradients problem occurs. To tackle this problem, in a succeeding study (Gulrajani et al., 2017), the authors proposed to use gradient penalty in the critic’s loss function rather than weight clipping. Furthermore, the authors also demonstrated that batch normalization negatively affects the gradient penalty, therefore instance normalization is used. The study showed that using gradient penalty in critic helped the training more stable and generated higher quality images. The authors named the model WGAN using Gradient Penalty (WGAN-GP).

While WGAN and WGAN-GP attempted to overcome the challenges of training GAN, there have been various kinds of GAN models proposed over the years which each have advantages and disadvantages depending on the use case. Some of the famous GAN variants and the ones that were used in the civil SHM field are discussed in this paragraph. After the release of the first GAN (Goodfellow et al., 2014), the authors in one work (Mirza and Osindero, 2014), introduced a conditional version of GAN which they named CGAN. In CGAN, the generator generates data based on the conditioned instance. In another study (Chen et al., 2016), the authors proposed a GAN that is able to learn disentangled representations in the data domain where the model maximizes the information between the latent variables and the observation. The authors named the model InfoGAN. In a different work (Ledig et al., 2017), the authors aimed at recovering the finer texture of images when they are upscaled. In other words, they upscaled the images with super-resolution texture detail and they named the model super-resolution GAN (SRGAN). In another study (Mao et al., 2016), the authors proposed using least squares (LS) in the loss function for the discriminator and they named the model LSGAN. In that study, the authors observed that using the LS function helps the GAN model generate higher-quality images than regular GAN. The authors in Isola et al. (2016) proposed a condition-based GAN model that may be the first to implement image-to-image translation. They demonstrated the model in various types of image translation cases successfully and named the model Pix2Pix. In Reed et al. (2016), one of the first kinds of AI-based text-to-image synthesis is proposed and the model is named StackGAN. Essentially, the model in this study can generate plausible images of text descriptions automatically. In another work (Kim et al., 2017), the authors studied a model that learns to discover relations between different domains to overcome the challenge of unpaired data. The proposed model, DiscoGAN, is able to transfer style from one domain to another. In a succeeding study (Zhu et al., 2017), the authors presented CycleGAN (Cycle-consistent GAN) to learn the mapping between two different domains to overcome the challenge of the inability of having the paired instance for training. In Karras et al. (2017), the authors proposed the model ProGAN where they similarly aimed at generating higher resolution images but by taking a different approach. The authors demonstrated that this approach expedites and stabilizes the training and produces unprecedented quality. The authors in Akcay et al. (2018) demonstrated a model that identifies anomalies, which is named GANomaly. The model is based on CGAN which learns the generation of image space and the inference of latent space. In another work (Wang et al., 2018), the authors improved the previously introduced SRGAN model which they named Enhanced-SRGAN or ESRGAN. The authors achieved better visual quality in the produced images after improvements to the network architecture and loss functions. In one study (Zhang et al., 2018a), the authors proposed semantic segmentation with GAN—SegGAN in which a deep semantic segmentation network is integrated into GAN for image segmentation tasks. In Karras et al. (2018), the authors presented a style-based generator, named StyleGAN where it achieves to learn high-level attributes of the data domain and enables to interpolate of these attributes to generate desired realistic outputs. The model demonstrates that it is able to generate style-specific outputs based on the selected parameters. The authors in Zhang et al. (2018b) proposed using a self-attention mechanism in GAN which they named SAGAN. Lastly, in another follow-up study (Karras et al., 2019) by the authors of StyleGAN, they modified the model architecture and training methods such as normalizing and regularizing the generator and improving the progressive growing method. The authors observed significant improvements in image generation. Some of the proposed notable GAN models and the ones employed in the civil SHM field are given in Table 1 (the dates state the initial submission dates of the studies on arXiv and online publishing dates on journals and conferences).

TABLE 1

TABLE 1. Notable GAN models over the years and the ones employed in the civil SHM field.

4 Generative adversarial networks in civil structural health monitoring

As shown in the previous section, GAN and its variants are mostly studied in computer vision applications where the researchers demonstrated many successful GAN studies on image data (2-D data). However, it has been also shown that across many other disciplines, GANs can be employed on 1-D data (e.g., signals) (Guo et al., 2020; Kuo et al., 2020; Luo et al., 2020). It is also observed that in the civil SHM field, there are more studies of GAN with 1-D data (26) than 2-D data (19) based on the included studies in this paper (Figure 8). Since the civil SHM field consists of a large number of multidisciplinary activities, numerous amounts of signal and image -based applications can be seen for approaching different SHM problems. Thus, it is observed that the GAN studies included in this paper (45 studies are included as shown in Figure 3) work with different types of GAN models on 1-D and 2-D data.

FIGURE 8

FIGURE 8. The number of papers per year included in this study (GAN-based SHM studies on 1-D and 2-D data).

In this section, studies are explained and discussed in two categories: GANs worked on 1-D and 2-D data. Although this study categorized the GAN studies into two according to the type of data work with, the difference between working with 1-D and 2-D data for GAN or any other deep learning model is very minimal. Essentially the main difference in the implementation is that while all the computations (apart from linear operations) for 2-D data are based on 2-D convolution, the computations for 1-D data are based on the 1-D convolution process. In addition, depending on the application, the computations in preprocessing may change for 1-D and 2-D data kinds. Lastly, it is important to consider that working with a DL model on 2-D data is computationally costlier than working on 1-D data since the increase in the number of parameters enlarges the computation load significantly. Note that the studies presented in Sections 4.1 and 4.2 are introduced according to the order of their initial online publication (or initial shared on arXiv) date starting from 2018 to 2022.

Based on the literature review performed on GANs in the civil SHM field, several common research problems and solutions are identified. There are also other research statements that address different problems and introduce novel solutions. To identify and clarify the position of GANs in the civil SHM field, in terms of the type of GAN applications studied by the researchers, an illustrative figure is made (Figure 9) which shows the classification of the applications of GANs in civil SHM and the corresponding studies with the GAN models used in each study. As the main concept of GAN is to learn data domain and data generation, some studies solely studied data generation [(Kanghyeok and do Hyoung, 2019; Xiong and Chen, 2019; Zhang and Wang, 2019; Tsialiamanis et al., 2020; Xu et al., 2021; Yu et al., 2021; Tsialiamanis et al., 2022a; Heesch et al., 2021; Colombera et al., 2021; Luleci et al., 2022b; Luleci et al., 2023)] (a total of 11 studies) by using original GAN or other GAN variants. Thus, the data generation category is separated from other categories. But also, the data generation category can be the main category that consists of other application types since other applications are in essence based on the data generation concept. The core research problem of data generation applications points out the data scarcity issue in civil SHM problems. Accordingly, the researchers aimed at using GAN to learn the scarce domain and generate additional and similar data samples to tackle the data scarcity issue.

FIGURE 9

FIGURE 9. Classification of the applications of GANs in the civil SHM field.

Lost data reconstruction [(Lei et al., 2021; Fan et al., 2021; Jiang et al., 2022; Zhuang et al., 2022; Hou et al., 2022)] (a total of five studies) is another type of GAN application observed in the literature. The research problem of this application is missing data caused by malfunctions in data acquisition systems such as errors in sensors or transmission. Because of the missing data problem, the data analysis process could lead to misinformative results. Therefore, it is important to estimate and reconstruct (or impute) the lost data to complete the original data. As a result, researchers demonstrated in several studies that using GAN to overcome the missing data problem is feasible and can be beneficial in civil SHM. Among the included studies, the lost data reconstruction application only experiments with 1-D data.

Data augmentation [(Soleimani-Babakamali et al., 2022a; Luleci et al., 2022a; Li et al., 2020; Yao et al., 2020; Gao et al., 2021; Shin et al., 2021; Chen et al., 2022; Ali and Cha, 2022)] (a total of eight studies) is one of the most encountered application types in the literature. The researchers, herein, address the problem of data imbalance caused by the data scarcity phenomenon. Essentially, the data imbalance problem is having fewer samples in one labeled data class than the other class in the training dataset of an AI model. This issue degrades the model’s detection performance badly. Thereby, researchers aimed at increasing (augmenting) the number of samples in the class that has fewer samples by using GANs. Many studies showed that data augmentation via GANs is an appropriate and beneficial method to balance the imbalanced data class in order to improve the performance of the data-driven AI model. Among the included studies, data augmentation is applied to both 1-D and 2-D data types.

Data domain translation [(Tsialiamanis et al., 2022b; Luleci et al., 2022c; Zhang et al., 2020; Yasuno et al., 2020; Bianchi et al., 2021)] (a total of five studies) is seen less frequently than the other applications in the literature but could be very promising and advantageous to many other problems in civil SHM. The core research problem of domain translation applications is learning the distinct mapping between the data domains. However, there are also other research problems that can be forked into it. Among the included studies in this paper, one of the most common research problems is the difficulty of having access to the paired data in the training dataset of an AI model. This challenge is solved by using CycleGAN which learns the domain mapping and then generates the cross-domain pair of the annotated data. The other common research problem investigated by the researchers in the literature is undamaged-to-damaged domain translation which enables to have access to the damaged pair (e.g., damaged acceleration data or concrete crack picture) while having the undamaged pair. This research problem aims to assist in SHM applications and at the same time reduce the data collection effort from the structures. Among the included studies in this paper, data domain translation is applied to both 1-D and 2-D data types.

Anomaly and novelty detection [(Soleimani-Babakamali et al., 2022b; Soleimani-Babakamali et al., 2022c; Mao et al., 2021; Tilon et al., 2020a; Tilon et al., 2020b; Liu et al., 2022)] (a total of six studies) is another GAN application type that is observed in the literature on civil SHM. Essentially, the main research aim is to detect novelties or anomalies in the observed data. While in novelty detection the outliers in the data are eliminated before training and the inference of the model is made to detect the anomalies in new observations, in outlier (or anomaly) detection, the training data contains outliers, and the model is trying to fit in the observed data while ignoring the deviant observations. Anomaly and novelty detection is found very useful for damage detection in the literature using GANomaly or DCGAN in which the discriminator network plays a critical role in detecting the anomalies or novelties. Among the included studies, anomaly and novelty detection is applied to both 1-D and 2-D data types.

In the civil SHM literature, there are also others [(Wang et al., 2019; Rastin et al., 2021; Yuan et al., 2021; Yang et al., 2022; Huang et al., 2020; Liu and Yeoh, 2020; Sathya et al., 2020; Liu et al., 2020; Sun et al., 2022; Dunphy et al., 2022)] (a total of 10 studies) who are observed that do not fit a category and address different problems. It is observed that the subject of “damage detection after increased resolution” has one study (Sathya et al., 2020) in the literature where the researchers use SRGAN to increase the resolution of the images to improve the performance of the damage classifier. On the other hand, SRGAN is also used for another study where the researchers measure the displacement of a structure on RGB images after they increase the resolution of the pictures (Sun et al., 2022). There are three studies on the topic of “data denoising, deblurring, and repair” for which the aim is to denoise the signal data, deblur the image, and repair the data after the identification of anomalies by using GANs [(Wang et al., 2019); (Liu et al., 2020); (Yang et al., 2022)]. There is one study on the topic of “track irregularity estimation” on acceleration data (Yuan et al., 2021), one study on the topic of “damage identification” via acceleration data (Rastin et al., 2021), and three studies on the topic of “Annotation reduction via transfer learning” [(Huang et al., 2020); (Liu and Yeoh, 2020; Dunphy et al., 2022)] which aim to reduce the need of annotating data through transfer-learning are other instances of GAN applications in civil SHM.

4.1 GANs on 1-D data

4.1.1 Studies published in 2018 (1 paper)

One of the very first published papers about GAN presented the application of improved LSGAN for crack detection in railways on Acoustic Emissions (AE) data (Wang et al., 2019) authors of this study aimed at solving the problem of noise interference of wheel-rail mechanical interaction and they employed an improved version of LSGAN where the authors included Mean Squared Error (MSE) to the generator loss as a regularization. The proposed model was tested in two experiments: Gaussian noise and mechanical noise that is obtained from an actual railway. The authors demonstrated that the presented method effectively denoised both the statistical noise and mechanical noise while preserving the details of the crack features in the signal.

4.1.2 Studies published in 2019 (3 papers)

In this study (Kanghyeok and do Hyoung, 2019), the authors addressed the problem of the data imbalance phenomenon for training the DL-based models for damage detection applications. The authors pointed out that the amount of damaged data before and after the damage is rare. Thus, they used GAN for generating damaged acceleration data and then the authors demonstrated that the synthetic data has a very similar pattern to the real data. As a result, they concluded that generating acceleration data with GAN might be used to tackle the data imbalance problem in damage detection applications.

Xiong and Chen (2019) introduced a study where they addressed the challenge of including different structural load models that consider individual human activities for the design process of slender structures such as footbridge and cantilever stand structures. As such, to build a load model, the main features e.g., periodicity and stationarity of data records are first extracted and mathematically modelled. For that, the authors proposed a unified approach that uses CGAN built on Wasserstein distance with gradient penalty to be able to learn these features, which are high-dimensional variables. In the paper, the CGAN model is used to generate individual walking, jumping, and bouncing loads trained. The authors validated the accuracy of the proposed approach by comparing the spectral densities and single degree of freedom system’s responses of the synthetic samples with real samples. As a result, the authors concluded that the approach can be used to simulate various human-induced loads without extracting features artificially.

In another study (Zhang and Wang, 2019), the authors presented a study of deep learning algorithms for structural condition identification with limited monitoring data where they used a Convolutional Neural Network (CNN) for damage detection on the acceleration data collected from a laboratory truss bridge under different damage scenarios. The authors demonstrated that, as expected, when the data is scarce, the accuracy of CNN degrades for damage detection. They concluded the study by proposing to use GAN (as a future study) to supply synthetic data for complementing the missing data in the training dataset which is expected to increase the accuracy of the damage detection model.

4.1.3 Studies published in 2020 (2 papers)

The authors in one study (Tsialiamanis et al., 2020) investigated an application of InfoGAN in SHM where they demonstrated that InfoGAN can capture the damage acceleration responses in a simulated mass-spring application. They induced different extents of stiffness reductions as damage to the system. Then used InfoGAN to learn the different data distributions (damage scenarios). The authors observed that InfoGAN was successfully able to capture the latent variables in the data distributions. The authors concluded that InfoGAN could be used to classify damage classes and at the same time generate synthetic data for the desired class.

A paper in Lei et al. (2021) studies the lost data reconstruction for SHM using DCGAN. The authors addressed the problem of lost data due to sensor fault or transmission failure that could degrade the analysis results (e.g., modal analysis). They trained DCGAN with missing acceleration and strain datasets separately and then tested on a bridge data. The authors were able to reconstruct entire datasets along with the missing parts. The authors also verified the reconstruction of the acceleration data with modal analysis as illustrated in Figure 10.

FIGURE 10

FIGURE 10. Comparison between the mode shapes extracted from the real and reconstructed signals (Lei et al., 2021).

4.1.4 Studies published in 2021 (12 papers)

Another paper (Xu et al., 2021) studies deep-learning-based cable-bridge condition assessment by reconstructing the probability density distribution (PDF) of girder vertical deflection (GVD) and cable tension (CT) using an unsupervised image transformation model where the model consists of convolutional variational autoencoders and GANs. After training the model with PDFs of GVD and CT, the authors observed that the model could generate very similar synthetic PDFs of GVD and CT to their real counterparts.

The authors in Yu et al. (2021) studied a probabilistic weight estimation using physics-constrained GAN where they address the issue of existing bridge weight-in-motion (BWIM) approaches that seldom account for the uncertainty of vehicle weight in terms of the probabilistic distribution of vehicles. The used GAN model, which is constrained by the known physics knowledge, is trained to learn the distributions of vehicle weights from the observed bridge response under the traffic loading. Then, the model is tested on highway bridges. The authors observed that GAN can successfully capture the uncertainty in vehicle weights and can provide a probabilistic meaning of the anticipated vehicle weight.

Another study (Fan et al., 2021) pointed out the importance of lost responses during an SHM application. The authors in this paper employed SegGAN to conduct dynamic response (acceleration) reconstruction. The model is trained on the acceleration dataset with missing parts and then used to reconstruct the whole response. The authors evaluated the model’s generative performance by performing numerical analysis on a steel frame structure to assess the accuracy and noise immunity of SegGAN. The model was also compared with other networks, and it is observed that SegGAN demonstrated outstanding lost response reconstruction results in both time and frequency domains.

In one another study (Tsialiamanis et al., 2022a), the authors pointed out the data scarcity issue and they used CGAN to tackle the challenge. The authors mention that environmental conditions have significant effects on the structure, and it is not possible to know their impacts on the structural parameters. Thus, the authors trained CGAN with different ranges of temperature and humidity parameters to learn the relationship between the environmental conditions and structural parameters e.g., stiffness and damping. The authors succeeded in generating temperature and humidity data based on the parametrized structural parameters.

The authors in this paper (Jiang et al., 2022) study the continuous missing data imputation for the incomplete dataset by completing with GAN. They address missing data problems due to sensor failure or signal transmission which creates a huge challenge for SHM applications. In the study, the authors used GAN for lost strain data generation based on training with the (missing) observed data. The proposed method was tested on a real concrete bridge and demonstrated an excellent performance for lost strain data generation. One of the sensor’s strain data after reconstruction with GAN is shown in Figure 11.

FIGURE 11

FIGURE 11. The completed strain data after reconstruction with GAN (Jiang et al., 2022).

In this work (Soleimani-Babakamali et al., 2022a), the authors studied on data imbalance problem resulting from low-sampled data and employed GAN to overcome this challenge. Essentially, they used Recurrent Neural Networks (RNN) as a novelty classifier of high-dimensional features. The authors, then, employed GAN to augment the imbalanced dataset to improve the classification result of RNN. They tested the proposed methodology on acceleration datasets of two different test setups and significant improvement was observed in the classification results after the data augmentation.

The authors in this paper (Heesch et al., 2021) addressed the importance of needing data when building a data-driven model for SHM applications. For that, the authors employed a generative model built on the StyleGAN2 model architecture to learn the data distribution of ultrasound wave data and eventually generate similar data points to the original dataset. They tested the model on a composite plate and observed that the generated data is very similar to the original pair. The authors also observed that they can manipulate the outputs by controlling the parameters of StyleGAN2.

In another work, Tsialiamanis et al. (2022b) presented a study on nonlinear modal analysis using CycleGAN. The study proposed to use CycleGAN for one-to-one mapping or domain translation from a latent modal space to the natural coordinate space. In doing so, the orthogonality of mode shapes is preserved, which is achieved by a stack of neural networks. The study that was proposed is tested on data from an experimental system that has three degrees of freedom and simulated systems where both datasets (acceleration signals) are incorporated with nonlinearities. The authors concluded that the introduced method demonstrated success in separating the modes as well as good accuracy in providing a nonlinear superposition function.

In another study (Colombera et al., 2021) by the authors, a GAN-based autoencoder for SHM is proposed where DCGAN is used to train on damaged and undamaged acceleration data of a multistory building. Then, the authors concluded that the model was able to generate reasonable data for different damaged and undamaged states of the building.

Rastin et al. (2021) introduced an approach of unsupervised SHM and damage identification. Essentially, the methodology introduced a two-stage damage identification process where the first phase is based on using DCGAN to detect and quantify the damages in the structure, then, in the second phase, the detected damages are localized using CGAN on the acceleration signals. The methodology introduced in this paper is tested on three different datasets and the authors concluded that GANs can successful be employed for damage quantification and localization.

In another work (Yuan et al., 2021), the authors studied a real-time track irregularity estimation using simulated vehicle acceleration responses. They pointed out the importance of accurate and timely estimation of track irregularities in railway maintenance. Therefore, the authors used WGAN-GP for capturing the correlation between the vehicle’s axle box acceleration signal track irregularities to be able to estimate the track irregularities. The authors validated the proposed methodology with two numerical examples successfully.

Luleci et al., in 2021 (Luleci et al., 2022a) addressed the problem of data scarcity in the SHM field which leads to the problem of data imbalance for AI-based data-driven applications in SHM. Thus, they employed the WDCGAN-GP model (or deep convolutional WGAN-GP) to learn the damaged domain (bolt-loosening) of an acceleration dataset collected from a steel grandstand structure. Then, after the successful generation of similar damaged acceleration datasets, the authors used the generated datasets to augment the imbalanced datasets of the damage detection classifier (Deep Convolutional Neural Network—DCNN) where the augmentation ratios vary for different damage detection scenarios. They concluded that it is applicable to augment the training dataset of DCNN with synthetic data as the output of DCNN significantly remains similar in all different augmentation scenarios. In addition, the authors also proposed a concept where this framework could be used on the bridge structures where they presented as GAN-augmented vibration-based damage detection via the DL model (Figure 12).

FIGURE 12

FIGURE 12. GAN augmented vibration-based damage detection via DL model on bridge structures (Luleci et al., 2022a).

4.1.5 Studies published in 2022 (8 papers)

Yang et al. (2022) published a study which emphasizes the importance of data repair after anomaly identification for improving data quality. Therefore, the authors proposed a two-stage approach where it first identifies the data anomalies using Long Short-Term Memory (LSTM), and then CGAN is employed to construct the data by excluding the anomaly data segments with the assist of the LSTM (through the anomaly identification results). The approach that is proposed in this study is tested on experimental data of a bridge Global Positioning System (GPS) monitoring data. The authors concluded that the data cleansing method can accurately identify and repair GPS anomalies.

The authors in Soleimani-Babakamali et al. (2022b) worked with a DCGAN model where the LSTM and DCNN are used as discriminators to do novelty detection. The entire model is trained with Fourier transformation of acceleration data from two different benchmark datasets. During the training, while the generator produces the data, the discriminator tunes the threshold based on the generator’s output, thus the discriminator can classify novel and normal data. The authors identified more than 95% of the novelties in the studied datasets with the proposed approach.

In another study (Zhuang et al., 2022), the authors addressed the issue of sensor or data transmission errors in bridge weigh-on-motion (WIM) systems where the data loss in WIM degrades the following bridge condition assessment applications. In the paper, the authors trained DCGAN with existing complete WIM data and then tested on missing WIM data of a bridge structure to observe the DCGAN model’s reconstructive ability. The authors concluded that the reconstructed data is in good agreement with the actual dataset.

Luleci et al. (2022b) pointed out the problem of data scarcity in SHM and employed the WDCGAN-GP (or Deep Convolutional WGAN-GP) model to tackle the challenge. In that study, they first trained the model with acceleration data whose domain is damaged (bolt-loosening). Then, the authors implemented a damage detection application using DCNN as a classifier. Basically, DCNN was trained on the real undamaged and damaged acceleration datasets and tested for two cases where the first case consists of real undamaged and damaged datasets; the second case consists of real undamaged and synthetic damaged datasets. Results showed that DCNN made very similar prediction results when it is used for both cases which showed that WDCGAN-GP produced very similar damaged acceleration data to the original data.

In another study, the same authors of the previous studies introduced another concept (Luleci et al., 2022c). In that study, the authors used CycleGAN, which is built on convolutions, and adopted the WGAN-GP model. They named the model CycleWDCGAN-GP. The authors used the model to attempt to translate the undamaged acceleration data domain to the damaged acceleration data domain and vice versa. Although they observed successful domain translation at joint 1, which was its training data (Figure 13), the translation of the test data was not satisfactory. The authors concluded that while the study was not fully successful, undamaged-to-damaged domain translation and similar applications could be very advantageous approaches in SHM such as having the damaged acceleration data of a structure while it is in pristine condition or having the undamaged acceleration data while the structure is in damaged condition.

FIGURE 13

FIGURE 13. (A) Frequency domains of real and synthetic (translated from damaged) undamaged acceleration response signal at joint 1, (B) Frequency domains of real and synthetic (translated from undamaged) damaged acceleration response signal at Joint 1 (Luleci et al., 2022c).

The same authors of (Soleimani-Babakamali et al., 2022b) demonstrated a similar framework in this paper (Soleimani-Babakamali et al., 2022c) by incorporating a system reliability approach. The end goal of the study was to implement unsupervised real-time SHM problems without prior information. The generator of DCGAN used in this study was used for tunning the threshold of a discriminator for novelty detection by the discriminator which holds the major success of the study. The authors tested the proposed real-time unsupervised novelty detection application on two different acceleration datasets successfully with few false novelty estimations.

The authors of this study (Hou et al., 2022) addressed the significance of missing data problems due to sensor or data transmission errors in SHM systems which can lower the quality of SHM applications. Therefore, they proposed to use of DCGAN to reconstruct the missing data in displacement and strain datasets. The authors also compared the accuracy of conducting the lost data reconstruction via DCGAN and other conventional methods. They observed that DCGAN stood out as the most successful model for reconstructing the missing data.

The last work (Luleci et al., 2023) of 2022 experimented with a study of GAN for labelled acceleration data generation where the authors employed WGAN that is built on a deep convolutional network (WDCGAN) to produce similar vibration datasets to the original domain. According to the comparison metric results, the authors observed significant similarities between the generated acceleration dataset and the ground truth.

4.2 GANs on 2-D data

4.2.1 Studies published in 2019 (1 paper)

One of the first works of GAN on image data (Sathya et al., 2020) presented an approach of super-resolution for concrete crack prediction via DL models. The authors in that study addressed the challenge of low-resolution and blurry concrete crack pictures in the available crack datasets. Thus, they proposed to use SRGAN with a pre-trained model of VGG19 to increase the resolution of images and then followed by a crack detection procedure via ResNet50. The authors obtained 98.2% of crack detection accuracy and concluded that this approach could be used for image-based concrete crack detection.

4.2.2 Studies published in 2020 (10 papers)

Huang et al. (2020) addressed the difficulty of implementation of the Compressive Sensing (CS) method in image-based SHM applications. CS is a signal processing technique that is used to recover a signal from a sampling rate much smaller than the limit of the Nyquist sampling. The authors introduced a new approach for CS by using a DCGAN model to capture the latent space of the target images. The study employed a DCGAN model to recover (or reconstruct) the compressed crack images for the subsequent automatic crack segmentation application which uses transfer learning. The training outline of DCGAN with crack images is shown in Figure 14. The authors compared the performance of the introduced DCGAN-based recovery framework with three other CS techniques. The results demonstrated that the introduced method has a better performance than the well-established CS methods. The authors also showed that the introduced framework can be applied for motion blurring and occlusion removal applications.

FIGURE 14

FIGURE 14. Training process of DCGAN for crack images (Huang et al., 2020).

Liu and Yeoh (2020) highlighted the challenges of current DL-based defect recognition applications which are relied on a fully supervised manner that requires a large, annotated dataset for training. Thus, the authors proposed to use of a semi-supervised framework to mitigate the challenge. First, they trained DCGAN as a classifier with crack and non-crack images. Then, the weights and biases of the trained DCGAN are transferred to a CNN model. The study results demonstrated that the semi-supervised method achieved a similar accuracy performance of 98% with the usage of 62.5% annotated images compared to using the supervised method (only CNN).

The authors in this paper (Zhang et al., 2020) presented an image-based crack detection approach using CycleGAN. The study pointed out the challenge of having access to the paired dataset for DL model training. Thus, they first trained the CycleGAN model with two different image domains (crack and no crack images) and then tested it on four crack datasets. The authors successfully translated the no-crack-images to the crack-images and vice versa.

Li et al. (2020) introduced a study where it addresses the inability of obtaining large amounts of labeled samples for ML-based classification applications for post-disaster damage assessments. Thus, the authors introduced an Unsupervised Self-attention Domain Adaptation (USADA) which consists of a GAN, a classifier, and a self-attention module. In this USADA module, GAN is responsible for the generation of the source domain, which is the damaged building pictures, the self-attention model maintains the foreground of the generated outputs of the GAN for better sample quality, and the classifier makes damage classification of the generated damaged building images. The classifier is trained with synthetic images along with the original images. The authors concluded that the data augmentation achieved improvement in classification results when the classifier is trained with augmented data over when it is trained with fewer data points.

In this work (Mao et al., 2021), the authors introduced a methodology for anomaly detection using DCGAN and Autoencoders for SHM in which they tackle the challenge of class imbalance and incompleteness of anomalous patterns in the training dataset. The authors tested the methodology on two SHM datasets (where the acceleration data is converted to Gramian Angular Field images) from a bridge structure including normal and anomalous data. They observed that the proposed method can be trained with no labeled data and class imbalance and incompleteness of anomalous patterns of the training dataset can be avoided. The authors also concluded that the methodology can be a very effective approach to extracting anomalies from SHM data.

In another work (Liu et al., 2020), the authors pointed out the building façade inspection challenges. Although the UAV may provide good visual data for the image-based inspection process, some images could still suffer from quality due to motion blur from excessive vibrations of the UAV. Thus, the authors used DCGAN consisting of residual blocks and skip connections to deblur the images. The proposed model is benchmarked with other models, and they observed that the proposed model is able to achieve significant improvements in deblurring the crack images (Figure 15).

FIGURE 15

FIGURE 15. Sample results of deblurring on complex backgrounds (Liu et al., 2020).

In this study (Tilon et al., 2020a), the authors addressed the significance of infrastructure degradation and post-disaster damage detection. They proposed to use the GANomaly model (trained only with healthy labeled images) to detect anomalies from aerial images. The model aims to detect potholes and cracks in road infrastructures as well as building damage in aerial images. They observed that the model can detect damages in urban scenes at a satisfactory level but failed on asphalt roads which will be investigated in the next study.

The authors in this paper (Yasuno et al., 2020), addressed the difficulty of having paired and annotated images for supervised classification applications in SHM. In this work, the authors proposed to use CycleGAN to generate undamaged concrete images from damaged concrete images. In addition, by using this methodology, they also detected the early damages in concrete using the distance of distributions between two separate domains (damaged and undamaged).

In another work (Yao et al., 2020), the authors studied the data imbalance problem in railway fastener fault diagnosis applications. They proposed to use GAN to learn the distribution of rail fastener failure data (damage data). After the successful generation of damage data, the data is utilized to augment the training dataset of the Resnet-based image classifier for damage detection application. The authors concluded that the augmentation improved the fault detection accuracy significantly.

In the last study of 2020 (Tilon et al., 2020b), the authors worked on post-disaster building damage detection from earth observation imagery using the skip connection-based GANomaly model. Essentially, the model employed in this study is trained with images of post-event (after earthquake) imagery of buildings in their damaged state where the images are both taken with satellite and UAV. After the model is tested on the unseen images, the results of anomaly detection are compared with other models and the authors concluded the results as encouraging. In Figure 16, the model is tested on the satellite image of the Santa-Rosa wildfire showing multiple damaged buildings and burnt surroundings overlaid with anomaly scores (polygon shapes represent buildings). As seen, the high anomaly scores are around the buildings where all the greenery is burnt.

FIGURE 16

FIGURE 16. Post-wildfire satellite imagery from the Santa-Rosa wildfire (USA) showing multiple damaged buildings overlaid with anomaly scores and building polygon. The classification, classification threshold and anomaly scores are indicated (TP, True positive) (Tilon et al., 2020b).

4.2.3 Studies published in 2021 (3 papers)

One of the first studies conducted in 2021 (Gao et al., 2021) pointed out the class imbalance issue in vision-based SHM applications. The authors introduced Balanced semi-supervised (BSS-GAN) to overcome the imbalanced class problem. The authors compared the image generation and detection capabilities of BSS-GAN to other models and concluded that BSS-GAN is able to achieve better detection performance over conventional models.

In this study (Bianchi et al., 2021), the authors studied the image-based application of forecasting infrastructure deterioration with inverse GANs. Basically, they employed the StyleGAN2 model to learn the semantic boundaries in the images e.g., corrosion and no-corrosion in the steel structures. Then, the model is tested on the dataset obtained from the Virginia Department of Transportation to generate pictures in the spectrum of corrosion to non-corrosion e.g., translating non-corrosion steel picture to corrosion steel picture. The authors concluded that the introduced method could be very useful for inspectors, engineers, and owners regarding the damage progress. Figure 17 shows the example images of semantic deterioration forecasting where the first image on the left is the real image and the following images are the gradually translated version of the real image to the image that has corresponding corrosion.

FIGURE 17

FIGURE 17. Examples of semantic deterioration forecasting where the first image on the left is the real image, and the following images are the gradually translated version of the real image (Bianchi et al., 2021).

Shin et al. (Shin et al., 2021) addressed the need for a large dataset for building a DL-based damage recognition model for automatic damage recognition in building structures, which is a challenge to collect a sufficient amount of data from degrading building structures. Therefore, the authors developed a data augmentation methodology using DCGAN. The authors experimented with two different scenarios where in the first scenario, a damage recognition classifier is trained on a small dataset; in the second scenario, the classifier is trained on the DCGAN-augmented dataset. The test results demonstrated that when the classifier is trained on the augmented dataset, the average performance of the classifier was increased by 0.16. The authors concluded that the methodology introduced in this study could be a reliable alternative for image recognition applications.

4.2.4 Studies published in 2022 (5 papers)

In this study (Liu et al., 2022), the authors addressed the importance of anomalies in SHM systems during the data collection which can degrade the structural analysis and evaluation of the collected data. Therefore, the authors introduced GANomaly and CNN -based data anomaly detection and classification from the 2-D Gramian angular field and also on acceleration data. The authors obtained very high anomaly detection and classification results.

In the next paper (Chen et al., 2022), a methodology of crack detection based on DCGAN, and DL is proposed. The authors first used DCGAN to generate crack images and then augmented the training dataset of the damage classifier (resnet18). Then, DeepLabv3 is used for the pixel-level segmentation procedure. The authors observed that the damage detection performance improved significantly.

In this work (Ali and Cha, 2022), the authors introduced a methodology named self-attention-based GAN with internal damage segmentation on infrared images. The authors pointed out the challenge of data collection; especially obtaining data for internal damage. To tackle this challenge, SAGAN (incorporating Wasserstein loss, with and without gradient penalty, and convolutions) is employed to generate synthetic infrared images to augment the training dataset of the internal damage segmentation network (IDSNet). The authors observed that the augmentation improved the performance of the IDSNet model significantly which helped the model to achieve state-of-the-art accuracy.

In this study (Sun et al., 2022), the authors took a different approach to using GANs where they studied the vision-based structural displacement problem. In SHM, monitoring the displacement response of structures is very important for the serviceability of the structure. Therefore, the authors in this study used SRGAN to improve the resolution of the image frames extracted from a video sequence. Then, on the high-resolution frames, the authors utilized SIFT method to track the features on the frames to calculate the displacement of the structure. Figure 18 below shows the results of the SIFT feature point detection and matching after increasing the resolution of images with ORI, BIC, and GAN. The left images on the left half are the source images, and the right images on the right half are the destination images. The authors observed significant improvement in the displacement calculation after the increased resolution of the images as increased resolution with GAN enables more feature points to be detected.

FIGURE 18

FIGURE 18. Results of the SIFT feature point detection and matching after increasing the resolution of images with ORI, BIC, and GAN. The left images on the left half are the source images, and the right images on the right half are the destination images (Sun et al., 2022).

Dunphy et al. (Dunphy et al., 2022) discussed the importance of a large amount of annotated data points to train the DL networks, which is not readily available in real life. The study investigated the multiclass crack identification of concrete structures utilizing sparse amounts of annotated images. The DCGAN is trained with the unlabeled concrete crack images. Following the training, the weights and biases of the discriminator part of the DCGAN model were transferred to a CNN model for the damage identification procedure. The authors demonstrated that the methodology introduced in this paper helps a 0%–30% of reduction in the amount of annotated data used for damage identification applications.

5 Discussion: Strengths, limitations, recommendations, opportunities, and current trends for GANs in civil SHM

Most applications in the civil SHM field rely on data-driven methods. Thus, data collection from the civil structures, then analysis and evaluation, and providing supportive information about the health status of the structure to the engineers, owners or stakeholders is critical (decision-making phase). Based on the literature reviewed in this paper that uses GANs for SHM purposes and applications, they can address and assist in the data collection, data analysis, and evaluation phases. The main application of GANs is data domain learning and accordingly, data generation, which is important for the civil SHM field since one of the major problems in the field is data scarcity. Thereby, GANs could play a very critical role in the data scarcity problem, which is also demonstrated by the studies discussed in Section 4. From the studies reviewed in this paper, the summary of the challenges and their corresponding solution approaches using GANs in civil SHM are illustrated in Figure 19. It is important to note that, apart from the specific problems such as data domain translation or data repair, the core issue the applications mostly address is the data scarcity which involves generating the desired data for a particular problem. Furthermore, the application types of GANs shown in Figure 9 address important problems in civil SHM; and with additional research, validation, and development, those applications can be potentially used in practice in the future.

FIGURE 19

FIGURE 19. Summary of challenges and corresponding solution approaches using GANs in civil SHM.

5.1 Strengths and limitations

As GANs are notably difficult to train, establishing a training mechanism for a GAN model could be difficult as vanishing gradients, mode collapse, and failure of convergence are some of the major challenges of GANs. Although GANs can tackle the data scarcity problem, still substantial amount of data is necessary for the training procedure, which is mostly not possible to provide that amount of data to the GAN model. On the other hand, thanks to their superior generative skills, GANs can be employed in discovering hidden feature representations in complex data structures. As such, they can be efficiently used to learn high-dimensional data like Variational Autoencoders (VAE) and with better generative quality. Further improvements can be taken to increase the performance of GANs for learning complex data structures and stabilizing the training process. As such, using novel activation functions such as Mish (Misra, 2019), Gated Linear Units (GLU), Wasserstein Distance (Arjovsky et al., 2017; Gulrajani et al., 2017) as the adversarial loss metric, and Residual Layers (skip-connections) like applied in CycleGAN are found to be beneficial for vanishing gradients, mode collapse, and failure of convergence problems. Moreover, incorporating different structures of neural networks in GANs such as Fully Connected Neural Networks (FCNN), CNN, LSTM, and/or Transformers (attention-based models) showed to work well in GAN architecture. Especially, when working on image-based data (2-D), convolution operations (such as the operations in CNN or DCNN) are more beneficial as it provides information to the model about the vicinity of the pixel that is interested (advantageous for image recognition applications). Contrastingly, linear transformation operations (such as the operations in FCNN or Deep FCNN—DFCNN) are found more useful for learning the signal-based data (1-D) as it gives information about broader features in the data (signal type data are mostly consisting of periodic structures) (Luleci et al., 2022d).

In this literature review, the studies that are reviewed employed GANs to address different problems in civil SHM (discussed in Section 4.1 and Section 4.2), yet eventually the core of all the problems points out the data scarcity issue in the civil SHM field. As such, lost data reconstruction, data augmentation, data translation between cross domains, anomaly and novelty detection, increasing the resolution of images or increasing the sample rate of signal data, and data denoising, deblurring, and repair for anomaly-contained data application types benefitted the usage of GANs in civil SHM. As mentioned throughout this paper, data scarcity is a common issue in civil SHM. Thus, learning sparse and complex data structures using GANs can be challenging. In addition to applying the techniques discussed in the previous paragraph, some other techniques can be considered to tackle the issue of learning sparse and complex data structures using GANs. Building a deeper network is beneficial for learning the domain-invariant representations in complex data structures; however, this increases the chance of the model getting stuck in local minima. Also, a deeper network increases the parameters that are involved which in turn increases the training time due to longer computational requirements. On the other hand, building a shallower network reduces the chance of overfitting in the data, which is a common problem for the deep network; however, shallower networks will not have the similar learning capacity as the deep networks since they will have more general knowledge about the data structure. Therefore, it is important to find a good balance between deep and shallow networks. Adjusting the learning rate of the network is also a very typical way to find the good balance in addition to finding the balance between deep and shallow networks. When the learning rate is increased, the training time typically decreases and causes the model to make large gradient updates which cause the model to have an overall knowledge about the data domain. When the learning rate is decreased, the training time typically increases and causes the model to make very small gradient updates which increases the training time and may cause the model to get stuck at local minima. ADAM and ADAMW (with weight decay regularization) [(Kingma and Ba, 2014); (Loshchilov and Hutter, 2017)] optimizers can be used in the model as they make the model less sensitive to learning parameters.

5.2 Recommendations, opportunities, and current trends

Civil engineering structures can have very complex mechanisms. Accordingly, the dynamic response data collected from these structures could have very complex data domain representations. Employment of other types of generative DL models such as VAEs and especially DMs in collaboration with GANs and Transformers showed remarkable results in the AI field recently. In the coming years, it is expected that there will be more research on their usage in SHM applications. The authors envision that more research and investigation of using GANs, Transformers, and particularly DMs will open the door for estimating (or generating) dynamic responses of civil structures based on the domain knowledge of a different civil structure. It is of authors’ opinion that estimating the structural conditions of civil structures where there is a minimal-to-no amount of data, holds significant importance as the comprehensive, high spatial resolution data collection process is a major challenge in the civil SHM field. As such, having acceleration, displacement, or strain response data from one structure, and estimating the response of other similar structures or even different structures could be immensely beneficial for civil SHM as it would optimize the data collection effort. To achieve that, domain adaptation techniques incorporating zero-shot, one-shot, or few-shot learning and/or adopting the mechanism of text-to-image synthesis by using GAN and/or Transformer (Vaswani et al., 2017) and/or Diffusion Model (DM) [(Dhariwal and Nichol, 2021); (Ho et al., 2020)] could be very useful. For instance, learning the high-level data structure of acceleration response domains from a civil structure using a model that leverages GAN, Transformer, and DM, and subsequently predicting the response of other structures based on supplied auxiliary information of those structures (e.g., structural parameters: damping, mode shapes, geometric properties etc.) fed to the model can make the response estimation achievable. The fact that GANs perform better at continuous data (e.g., signal, image) and transformers are better at learning discrete data (e.g., texts) make the concept more feasible to realize. Especially, recent studies showed that DMs can be superior to GANs [(Dhariwal and Nichol, 2021); (Ho et al., 2020)]. Since civil structures can have a very complex mechanism, it is important for the AI model to be able to intake and learn different data types to gain every possible useful information about the structure. For that reason, a high level of collaboration with the AI field is critically important. For the last several years, the focus has been on Natural Language Processing (NLP) applications such as text-to-image synthesis where DMs play a major role. Accordingly, many models have been developed over the years such as models from Google: Bert (Devlin et al., 2018), Imagen (Saharia et al., 2022), Imagen-video (Ho et al., 2022); from DeepMind: WaveNet (van den Oord et al., 2016), Gopher (Rae et al., 2021); from OpenAI: GPT-3 (Brown et al., 2020), DALL-E (Ramesh et al., 2021), GLIDE (Nichol et al., 2021), from Meta AI: Make-A-Video (Singer et al., 2022). Throughout the development of these models, a multidisciplinary collaboration may not be needed, at least not as much as the development of model Alphafold (Jumper et al., 2021) and Alphafold2 (Cheng et al., 2022) (which aims to predict the formation of complex protein structures—in the Bioinformatics field) proposed by DeepMind. Nevertheless, most civil engineering structures have a very complex mechanism, accordingly, the response data collected from them can have a very complex domain structure. Thus, the ability to estimate response data of different multiple civil structures even at nodes with no data collection depends on high-level multidisciplinary research activity between AI and SHM. This multidisciplinary collaboration can be similar to the case of “how the researchers had to understand the formation of protein structures throughout the development of the AlphaFold series”. Systematic collaboration between the field of SHM or structural engineering and the AI field might also lead to the emergence of a new discipline just as the discipline of Civionics (combination of Civil Engineering and Electronics Engineering) or Bioinformatics (combination of Biology and Computer Science).

Currently, Variational Autoencoders (VAE) and GAN are the most used generative DL models to learn the distribution of the data in to generate new data. Figure 20 summarizes the pros and cons of VAEs, GANs, and DMs. While VAEs are good at producing diverse samples at a faster rate considering their training and inference processes, the quality of generated outputs is not satisfactory (e.g., blurry) compared to the performance of GANs. On the other hand, GANs can produce high-quality outputs through adversarial training mechanisms but are mostly difficult and timely to train (e.g., mode collapse, vanishing gradients, failure of model convergence). DMs tackle the aforementioned challenges of VAEs and GANs and have demonstrated great success [(Dhariwal and Nichol, 2021; Ho et al., 2020)] in text-guided image generation such as Imagen [(Saharia et al., 2022; Ho et al., 2022)], DALL-E (Ramesh et al., 2021), GLIDE (Nichol et al., 2021), and Make-A-Video (Singer et al., 2022). Briefly, DMs learn the data distribution by destroying training data through the successive addition of Gaussian noise and then, recovering the input from the noise in a backward propagation (denoising), which is also called Markov-Chain since it is a sequence of stochastic events where each time step depends on the previous time step. One of the special properties of DMs is latent space has the same dimensionality as the input space. The generation of new data is implemented via a backward process where denoising is implemented. While there are multiple studies of VAEs and GANs in the SHM field and several studies use Transformers, there is no study that uses DM for SHM applications which could be very advantageous using for SHM applications considering the remarkable performance of recently released DM models [(Saharia et al., 2022); (Nichol et al., 2021; Ramesh et al., 2021; Singer et al., 2022)].

FIGURE 20

FIGURE 20. Generative learning trilemma (Vahdat and Kreis, 2022).

6 Summary and conclusion

6.1 Summary

Deteriorated and rapidly ageing civil engineering structures have been a concern for the safety of civil society. SHM utilizes continuous, periodic or intermittent data collection (via sensorial systems) from civil structures along with advanced analysis methods to improve conventional condition assessment methods (Figure 1 and Figure 2). On the other hand, ML and DL methods have been found to be advantageous in data-driven applications in the civil SHM field. Since its release in 2014, GANs have been used in a wide variety of applications in various disciplines, and it has been one of the most popular research topics in the AI-ML domain. Especially, GANs have been quite favored by researchers lately for civil SHM applications. Nonetheless, it is observed by the authors of this study that while there are several studies about GANs in civil SHM, no literature review study is available, which is very important for the advancement of GANs in the field. Thereby, this study aimed to fulfill this gap by presenting a comprehensive literature review of the studies conducted via GANs in the civil SHM field. It is of authors’ opinion that this paper contributes to the civil SHM field by providing essential information for researchers about GANs and their applications in the field.

7 Conclusion

This paper presents a literature review of the studies utilizing GANs in the civil SHM field with its initial release in 2014 to date. The conclusions of this paper can be listed in the following.

• Based on the literature review conducted in this paper, 45 studies using GANs in the civil SHM field are identified. Among the 45 studies, 19 of them worked on 2-D and 26 of them worked on 1-D data. The studies are also classified based on the use of GANs as follows: “Data Generation (only)”, “Lost Data Reconstruction”, “Data Augmentation”, “Data Domain Translation”, “Anomaly and Novelty Detection”, and “Others”: “Data denoising, deblurring, and repair”, “Displacement measurement and crack detection after increased resolution”, “Track irregularity estimation”, “Damage identification”, and “Annotation reduction via transfer learning”.

• It is observed in the studies that are reviewed that most of them addressed the data scarcity problem. Therefore, they used GAN or its variant (GANs) to overcome the data scarcity challenge. Some of the other studies, on the other hand, utilized GAN or its variant for only novelty or anomaly or track irregularity detection. Although the remaining studies did not directly address the data scarcity problem (such as one of the studies measure structural displacement on the image frames after increased resolution of the images), those studies indirectly point out the data scarcity issue by generating the desired data.

• The very first study using GAN in the civil SHM field is observed in the year 2018 as per an online publication (Wang et al., 2019). Since then, the number of studies per year tends to increase gradually. It is safe to assume that GANs are at the beginning stage of research and development in the civil SHM field. Thus, it is expected to observe more studies in the following years.

• The GAN-based models proposed in the studies reviewed must be extensively tested on real civil structures for further validation and to investigate their applicability in real-life cases.

• Having acceleration, displacement, or strain response data from one civil structure and estimating the response of other similar structures or even different structures could be very beneficial for the civil SHM field as it would minimize or eliminate the data collection effort. To achieve such a concept, domain adaptation techniques incorporating zero-shot, one-shot, or few-shot learning and/or adopting the mechanism of text-to-image synthesis such as using GAN in collaboration with Transformer and/or Diffusion Models could be very advantageous.

• Many remarkable studies have been presented in the AI field lately [(Dhariwal and Nichol, 2021; Ho et al., 2020; Devlin et al., 2018; Saharia et al., 2022; Ho et al., 2022; van den Oord et al., 2016; Rae et al., 2021; Brown et al., 2020; Ramesh et al., 2021; Nichol et al., 2021; Jumper et al., 2021; Cheng et al., 2022; Vahdat and Kreis, 2022)]. As some of them required interdisciplinary collaboration, some of them did not. Most civil engineering structures have a very complex mechanism, accordingly, the collected response data from them can have a very complex domain structure. Thus, the ability to estimate response data of different civil structures at locations with no data (new data generation) depends on a high-level systematic multidisciplinary research activity between SHM and AI fields. This multidisciplinary collaboration should be at least at the level of “how the researchers had to understand the formation of protein structures throughout the development of the AlphaFold series (Cheng et al., 2022)”.

Author contributions

All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

Acknowledgments

The authors would like to thank members of the CITRS (Civil Infrastructure Technologies for Resilience and Safety) Research Initiative at the University of Central Florida. The second author (FC) would like to acknowledge the support of the National Aeronautics and Space Administration (NASA) Award No. 80NSSC20K0326.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Akcay, S., Atapour-Abarghouei, A., and Breckon, T. P. (2018). GANomaly: Semi-Supervised anomaly detection via adversarial training.

Google Scholar

Aktan, A. E., Farhey, D. N., Brown, D. L., Dalal, V., Helmicki, A. J., Hunt, V. J., et al. (1996). Condition assessment for bridge management. J. Infrastruct. Syst. 2 (3), 108–117. doi:10.1061/(asce)1076-0342(1996)2:3(108)

CrossRef Full Text | Google Scholar

Ali, R., and Cha, Y. J. (2022). Attention-based generative adversarial network with internal damage segmentation using thermography. Autom. Constr. 141, 104412. doi:10.1016/j.autcon.2022.104412

CrossRef Full Text | Google Scholar

Arjovsky, M., Chintala, S., and Bottou, L. (2017). Wasserstein GAN.

Google Scholar

Avci, O., Abdeljaber, O., Kiranyaz, S., Hussein, M., Gabbouj, M., and Inman, D. J. (2021). A review of vibration-based damage detection in civil structures: From traditional methods to Machine Learning and Deep Learning applications. Mech. Syst. Signal Process. 147, 107077. doi:10.1016/j.ymssp.2020.107077

CrossRef Full Text | Google Scholar

Bianchi, E., and Hebdon, M. (2021). “Forecasting infrastructure deterioration with GAN-inversion,” in Applications of machine learning 2021. Editors M. E. Zelinski, T. M. Taha, and J. Howe (San Diego, CA: SPIE), 3.

CrossRef Full Text | Google Scholar

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., et al. (2020). Language models are few-shot learners.

Google Scholar

Catbas, F. N., and Khuc, T. (2018). “Computer vision-based displacement and vibration monitoring without using physical target on structures,” in Bridge design, assessment and monitoring. 1st ed. (Milton Park, Abingdon-on-Thames, Oxfordshire United Kingdom: Taylor & Francis).

CrossRef Full Text | Google Scholar

Chen, G., Teng, S., Lin, M., Yang, X., and Sun, X. (2022). Crack detection based on generative adversarial networks and deep learning. KSCE J. Civ. Eng. 26 (4), 1803–1816. doi:10.1007/s12205-022-0518-2

CrossRef Full Text | Google Scholar

Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. (2016). InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, 05 December 2016, IEEE, 2180–2188.

Google Scholar

Cheng, S., Wu, R., Yu, Z., Li, B., Zhang, X., Peng, J., et al. (2022). FastFold: Reducing AlphaFold training time from 11 Days to 67 hours.

Google Scholar

Colombera, G., Rosafalco, L., Torzoni, M., Gatti, F., Mariani, S., Manzoni, A., et al. (2021). “A generative adversarial network based autoencoder for structural health monitoring,” in The 1st Online Conference on Algorithms session Evolutionary Algorithms and Machine Learning IOCA 2021, Basel, Switzerland, 06 September 2021 (MDPI), 9.

CrossRef Full Text | Google Scholar

Devlin, J., Chang, M. W., Lee, K., and Toutanova, K. (2018). Bert: Pre-Training of deep bidirectional transformers for language understanding.

Google Scholar

Dhariwal, P., and Nichol, A. (2021). Diffusion models beat GANs on image synthesis.

Google Scholar

Dong, C. Z., and Catbas, F. N. (2021). A review of computer vision–based structural health monitoring at local and global levels. Struct. Health Monit. 20 (2), 692–743. doi:10.1177/1475921720935585

CrossRef Full Text | Google Scholar

Dunphy, K., Sadhu, A., and Wang, J. (2022). Multiclass damage detection in concrete structures using a transfer learning-based generative adversarial networks. Struct. Control Health Monit. 29 (11), e3079. doi:10.1002/stc.3079

CrossRef Full Text | Google Scholar

Fan, G., Li, J., Hao, H., and Xin, Y. (2021). Data driven structural dynamic response reconstruction using segment based generative adversarial networks. Eng. Struct. 234, 111970. doi:10.1016/j.engstruct.2021.111970

CrossRef Full Text | Google Scholar

Farrar, C. R., and Worden, K. (2007). An introduction to structural health monitoring. Phil. Trans. R. Soc. A 365, 303–315. doi:10.1098/rsta.2006.1928

PubMed Abstract | CrossRef Full Text | Google Scholar

Feng, D., and Feng, M. Q. (2018). Computer vision for SHM of civil infrastructure: From dynamic response measurement to damage detection – a review. Eng. Struct. 156, 105–117. doi:10.1016/j.engstruct.2017.11.018

CrossRef Full Text | Google Scholar

Gao, Y., Zhai, P., and Mosalam, K. M. (2021). Balanced semisupervised generative adversarial network for damage assessment from low-data imbalanced-class regime. Computer-Aided. Civ. Infrastructure Eng. 36 (9), 1094–1113. doi:10.1111/mice.12741

CrossRef Full Text | Google Scholar

Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., et al. (2014). Generative adversarial networks.

Google Scholar

Goodfellow, I. (2016). NIPS 2016 tutorial: Generative adversarial networks.

Google Scholar

Google (2022). Generative adversarial networks. Available at: https://developers.google.com/machine-learning/gan/gan_structure.

Google Scholar

Gucunski, N., Romero, F., Kruschwitz, S., Feldmann, R., Abu-Hawash, A., and Dunn, M. (2010). Multiple complementary nondestructive evaluation Technologies for condition assessment of concrete bridge decks. Transp. Res. Rec. 2201 (1), 34–44. doi:10.3141/2201-05

CrossRef Full Text | Google Scholar

Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. (2017). Improved training of Wasserstein GANs.

Google Scholar

Guo, Q., Li, Y., Song, Y., Wang, D., and Chen, W. (2020). Intelligent fault diagnosis method based on full 1-D convolutional generative adversarial network. IEEE Trans. Ind. Inf. 16 (3), 2044–2053. doi:10.1109/tii.2019.2934901

CrossRef Full Text | Google Scholar

Heesch, M., Mendrok, K., and Dworakowski, Z. (2021). Time-domain signal synthesis with style-based generative adversarial networks applied to guided waves. Berlin, Germany: Springer, 78–88.

CrossRef Full Text | Google Scholar

Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models.

Google Scholar

Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., et al. (2022). Imagen video: High definition video generation with diffusion models.

Google Scholar

Hou, J., Jiang, H., Wan, C., Yi, L., Gao, S., Ding, Y., et al. (2022). Deep learning and data augmentation based data imputation for structural health monitoring system in multi-sensor damaged state. Measurement 196, 111206. doi:10.1016/j.measurement.2022.111206

CrossRef Full Text | Google Scholar

Housner, G. W., Bergman, L. A., Caughey, T. K., Chassiakos, A. G., Claus, R. O., Masri, S. F., et al. (1997). Structural Control: Past, present, and future. J. Eng. Mech. 123 (9), 897–971. doi:10.1061/(asce)0733-9399(1997)123:9(897)

CrossRef Full Text | Google Scholar

Huang, Y., Zhang, H., Li, H., and Wu, S. (2020). Recovering compressed images for automatic crack segmentation using generative models. Mech. Syst. Signal Process. 146, 107061. doi:10.1016/j.ymssp.2020.107061

CrossRef Full Text | Google Scholar

Isola, P., Zhu, J. Y., Zhou, T., and Efros, A. A. (2016). Image-to-Image translation with conditional adversarial networks.

Google Scholar

Janiesch, C., Zschech, P., and Heinrich, K. (2021). Machine learning and deep learning. Electron. Mark. 31 (3), 685–695. doi:10.1007/s12525-021-00475-2

CrossRef Full Text | Google Scholar

Jiang, H., Wan, C., Yang, K., Ding, Y., and Xue, S. (2022). Continuous missing data imputation with incomplete dataset by generative adversarial networks–based unsupervised learning for long-term bridge health monitoring. Struct. Health Monit. 21 (3), 1093–1109. doi:10.1177/14759217211021942

CrossRef Full Text | Google Scholar

Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., et al. (2021). Highly accurate protein structure prediction with AlphaFold. Nature 596, 583–589. doi:10.1038/s41586-021-03819-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Kaartinen, E., Dunphy, K., and Sadhu, A. (2022). LiDAR-based structural health monitoring: Applications in civil infrastructure systems. Sensors 22 (12), 4610. doi:10.3390/s22124610

PubMed Abstract | CrossRef Full Text | Google Scholar

Kanghyeok, L., and do Hyoung, S. (2019). Generative model of acceleration data for deep learning-based damage detection for bridges using generative adversarial network. J. KIBIM 9 (1), 42–51. doi:10.13161/kibim.2019.9.1.042

CrossRef Full Text | Google Scholar

Karras, T., Aila, T., Laine, S., and Lehtinen, J. (2017). Progressive growing of GANs for improved quality, stability, and variation.

Google Scholar

Karras, T., Laine, S., and Aila, T. (2018). A style-based generator architecture for generative adversarial networks.

Google Scholar

Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. (2019). Analyzing and improving the image quality of StyleGAN.

Google Scholar

Kim, T., Cha, M., Kim, H., Lee, J. K., and Kim, J. (2017). Learning to discover cross-domain relations with generative adversarial networks.

Google Scholar

Kingma, D. P., and Ba, J. (2014). Adam: A method for stochastic optimization.

Google Scholar

Kuo, P. H., Lin, S. T., and Hu, J. (2020). DNAE-GAN: Noise-free acoustic signal generator by integrating autoencoder and generative adversarial network. Int. J. Distrib. Sens. Netw. 16 (5), 155014772092352. doi:10.1177/1550147720923529

CrossRef Full Text | Google Scholar

Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., et al. (2017). “Photo-realistic single image super-resolution using a generative adversarial network,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 25 May 2017 (Piscataway, New Jersey, United States: IEEE), 105–114.

CrossRef Full Text | Google Scholar

Lei, X., Sun, L., and Xia, Y. (2021). Lost data reconstruction for structural health monitoring using deep convolutional generative adversarial networks. Struct. Health Monit. 20 (4), 2069–2087. doi:10.1177/1475921720959226

CrossRef Full Text | Google Scholar

Li, H. N., Ren, L., Jia, Z. G., Yi, T. H., and Li, D. S. (2016). State-of-the-art in structural health monitoring of large and complex civil infrastructures. J. Civ. Struct. Health Monit. 6 (1), 3–16. doi:10.1007/s13349-015-0108-9

CrossRef Full Text | Google Scholar

Li, Y., Lin, C., Li, H., Hu, W., Dong, H., and Liu, Y. (2020). Unsupervised domain adaptation with self-attention for post-disaster building damage detection. Neurocomputing 415, 27–39. doi:10.1016/j.neucom.2020.07.005

CrossRef Full Text | Google Scholar

Liu, G., Niu, Y., Zhao, W., Duan, Y., and Shu, J. (2022). Data anomaly detection for structural health monitoring using a combination network of GANomaly and CNN. Smart Struct. Syst. 39 (1), 195–206. doi:10.12989/sss.2022.29.1.053

CrossRef Full Text | Google Scholar

Liu, Y., Yeoh, J. K. W., and Chua, D. K. H. (2020). Deep learning–based enhancement of motion blurred UAV concrete crack images. J. Comput. Civ. Eng. 34 (5), 907. doi:10.1061/(asce)cp.1943-5487.0000907

CrossRef Full Text | Google Scholar

Liu, Y., and Yeoh, J. K. W. (2020). Vision-based semi-supervised learning method for concrete crack detection.In Construction Research Congress 2020. Reston, VA, 26 November 2020. American Society of Civil Engineers, 527–536.

CrossRef Full Text | Google Scholar

Loshchilov, I., and Hutter, F. (2017). Decoupled weight decay regularization.

Google Scholar

Luleci, F., Catbas, F. N., and Avci, O. (2022a). CycleGAN for undamaged-to-damaged domain translation for structural health monitoring and damage detection. Available from: https://arxiv.org/abs/220207831.

Google Scholar

Luleci, F., Catbas, F. N., and Avci, O. (2022b). Generative adversarial networks for data generation in structural health monitoring. Front. Built Environ. 88, 816644. doi:10.3389/fbuil.2022.816644

CrossRef Full Text | Google Scholar

Luleci, F., Catbas, F. N., and Avci, O. (2022c). Generative adversarial networks for labeled acceleration data augmentation for structural damage detection. J. Civ. Struct. Health Monit. doi:10.1007/s13349-022-00627-8

CrossRef Full Text | Google Scholar

Luleci, F., Catbas, N., and Avci, O. (2022d). Improved undamaged-to-damaged acceleration response translation using CycleGAN for structural health monitoring. Engineering Applications of Artificial Intelligence.

Google Scholar

Luleci, F., Catbas, F. N., and Avci, O. (2023). “Generative adversarial networks for labelled vibration data generation,” in Special Topics in Structural Dynamics & Experimental Techniques, Volume 5. Conference Proceedings of the Society for Experimental Mechanics Series. Springer, Cham. doi:10.1007/978-3-031-05405-1_5

CrossRef Full Text | Google Scholar

Luo, T. J., Fan, Y., Chen, L., Guo, G., and Zhou, C. (2020). EEG signal reconstruction using a generative adversarial network with Wasserstein distance and temporal-spatial-frequency loss. Front. Neuroinform. 14, 15. doi:10.3389/fninf.2020.00015

PubMed Abstract | CrossRef Full Text | Google Scholar

Mao, J., Wang, H., and Spencer, B. F. (2021). Toward data anomaly detection for automated structural health monitoring: Exploiting generative adversarial nets and autoencoders. Struct. Health Monit. 20 (4), 1609–1626. doi:10.1177/1475921720924601

CrossRef Full Text | Google Scholar

Mao, X., Li, Q., Xie, H., Lau, R. Y. K., Wang, Z., and Smolley, S. P. (2016). Least squares generative adversarial networks.

Google Scholar

McLaughlin, E., Charron, N., and Narasimhan, S. (2020). Automated defect quantification in concrete bridges using robotics and deep learning. J. Comput. Civ. Eng. 34 (5), 04020029. doi:10.1061/(asce)cp.1943-5487.0000915

CrossRef Full Text | Google Scholar

Mirza, M., and Osindero, S. (2014). Conditional generative adversarial nets. Available at: https://arxiv.org/abs/14111784.

Google Scholar

Misra, D. (2019). Mish: A self regularized non-monotonic activation function.

Google Scholar

Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., et al. (2021). Glide: Towards photorealistic image generation and editing with text-guided diffusion models.

Google Scholar

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., et al. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. Syst. Rev. 10 (1), 89. doi:10.1186/s13643-021-01626-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Radford, A., Metz, L., and Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks.

Google Scholar

Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., et al. (2021). Scaling language models: Methods, analysis & insights from training gopher.

Google Scholar

Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., et al. (2021). Zero-shot text-to-image generation.

Google Scholar

Rastin, Z., Ghodrati Amiri, G., and Darvishan, E. (2021). Generative adversarial network for damage identification in civil structures. Shock Vib. 2021, 1–12. doi:10.1155/2021/3987835

CrossRef Full Text | Google Scholar

Reed, S., Akata, Z., Yan, X., Lajanugen, L., Schiele, B., and Lee, H. (2016). Generative adversarial text to image synthesis.

Google Scholar

Ribeiro, D., Santos, R., Cabral, R., Saramago, G., Montenegro, P., Carvalho, H., et al. (2021). Non-contact structural displacement measurement using Unmanned Aerial Vehicles and video-based systems. Mech. Syst. Signal Process. 160, 107869. doi:10.1016/j.ymssp.2021.107869

CrossRef Full Text | Google Scholar

Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., et al. (2022). Photorealistic text-to-image diffusion models with deep language understanding.

Google Scholar

Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. (2016). Improved techniques for training GANs.

Google Scholar

Sathya, K., Sangavi, D., Sridharshini, P., Manobharathi, M., and Jayapriya, G. (2020). Improved image based super resolution and concrete crack prediction using pre-trained deep learning models. J. Soft Comput. Civ. Eng. 4 (3), 40–51. doi:10.22115/SCCE.2020.229355.1219

CrossRef Full Text | Google Scholar

Scott, M., Rezaizadeh, A., Delahaza, A., Santos, C. G., Moore, M., Graybeal, B., et al. (2003). A comparison of nondestructive evaluation methods for bridge deck assessment. NDT E Int. 36 (4), 245–255. doi:10.1016/s0963-8695(02)00061-0

CrossRef Full Text | Google Scholar

Shin, H., Ahn, Y., Tae, S., Gil, H., Song, M., and Lee, S. (2021). Enhancement of multi-class structural defect recognition using generative adversarial network. Sustainability 13 (22), 12682. doi:10.3390/su132212682

CrossRef Full Text | Google Scholar

Singer, U., Adam, P., Hayes, T., Yin, X., An, J., Zhang, S., et al. (2022). Make-A-Video: Text-To-Video generation without text-video data. Available from: https://ai.facebook.com/blog/generative-ai-text-to-video/.

Google Scholar

Soleimani-Babakamali, M. H., Sepasdar, R., Nasrollahzadeh, K., Lourentzou, I., and Sarlo, R. (2022). Toward a general unsupervised novelty detection framework in structural health monitoring. Comput. aided. Civ. Eng. 37 (9), 1128–1145. doi:10.1111/mice.12812

CrossRef Full Text | Google Scholar

Soleimani-Babakamali, M. H., Sepasdar, R., Nasrollahzadeh, K., and Sarlo, R. (2022). A system reliability approach to real-time unsupervised structural health monitoring without prior information. Mech. Syst. Signal Process. 171, 108913. doi:10.1016/j.ymssp.2022.108913

CrossRef Full Text | Google Scholar

Soleimani-Babakamali, M. H., Soleimani-Babakamali, R., and Sarlo, R. (2022). A general framework for supervised structural health monitoring and sensor output validation mitigating data imbalance with generative adversarial networks-generated high-dimensional features. Struct. Health Monit. 21 (3), 1167–1182. doi:10.1177/14759217211025488

CrossRef Full Text | Google Scholar

Stanley, R. K., Moore, P. O., and McIntire, P. (1995). Nondestructive testing handbook: Special nondestructive testing methods (nondestructive testing handbook).

Google Scholar

Sun, C., Gu, D., Zhang, Y., and Lu, X. (2022). Vision-based displacement measurement enhanced by super-resolution using generative adversarial networks. Struct. Control Health Monit. 29 (10), e3048. doi:10.1002/stc.3048

CrossRef Full Text | Google Scholar

Tilon, S., Nex, F., Kerle, N., and Vosselman, G. (2020). Post-disaster building damage detection from earth observation imagery using unsupervised and transferable anomaly detecting generative adversarial networks. Remote Sens. (Basel). 12 (24), 4193. doi:10.3390/rs12244193

CrossRef Full Text | Google Scholar

Tilon, S. M., Nex, F., Duarte, D., Kerle, N., and Vosselman, G. (2020). Infrastructure degradation and post-disaster damage detection using anomaly detecting generative adversarial networks. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2020, 573–582. doi:10.5194/isprs-annals-v-2-2020-573-2020

CrossRef Full Text | Google Scholar

Tsialiamanis, G., Champneys, M. D., Dervilis, N., Wagg, D. J., and Worden, K. (2022). On the application of generative adversarial networks for nonlinear modal analysis. Mech. Syst. Signal Process. 166, 108473. doi:10.1016/j.ymssp.2021.108473

CrossRef Full Text | Google Scholar

Tsialiamanis, G., Chatzi, E., Dervilis, N., Wagg, D., and Worden, K. (2020). “An application of generative adversarial networks in structural health monitoring,” in XI International Conference on Structural Dynamics, Athens, Greece, 23-26 November 2020, 3816–3831.

CrossRef Full Text | Google Scholar

Tsialiamanis, G., Wagg, D. J., Dervilis, N., and Worden, K. (2022). On generating parametrised structural data using conditional generative adversarial networks. Berlin, Germany: Springer, 35–46.

CrossRef Full Text | Google Scholar

Vahdat, A., and Kreis, K. (2022). Improving diffusion models as an alternative to GANs. Available from: https://developer.nvidia.com/blog/improving-diffusion-models-as-an-alternative-to-gans-part-1/.

Google Scholar

van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., et al. (2016). WaveNet: A generative model for raw audio.

Google Scholar

van Nguyen, L., Gibb, S., Pham, H. X., and La, H. M. (2018). “A mobile robot for automated civil infrastructure inspection and evaluation,” in 2018 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), Philadelphia, PA, USA, 06-08 August 2018 (IEEE), 1–6.

CrossRef Full Text | Google Scholar

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., et al. (2017). Attention is all you need. Available from: https://arxiv.org/abs/1706.03762?context=cs.

Google Scholar

Wang, K., Zhang, X., Hao, Q., Wang, Y., and Shen, Y. (2019). Application of improved least-square generative adversarial networks for rail crack detection by AE technique. Neurocomputing 332, 236–248. doi:10.1016/j.neucom.2018.12.057

CrossRef Full Text | Google Scholar

Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., et al. (2018). Esrgan: Enhanced super-resolution generative adversarial networks.

Google Scholar

Xiong, J., and Chen, J. (2019). A generative adversarial network model for simulating various types of human-induced loads. Int. J. Str. Stab. Dyn. 19 (08), 1950092. doi:10.1142/s0219455419500925

CrossRef Full Text | Google Scholar

Xu, Y., Tian, Y., Zhang, Y., and Li, H. (2021). Deep-learning-based bridge condition assessment by probability density distribution reconstruction of girder vertical deflection and cable tension using unsupervised image transformation model. Berlin, Germany: Springer, 35–45.

CrossRef Full Text | Google Scholar

Yang, K., Ding, Y., Jiang, H., Zhao, H., and Luo, G. (2022). A two-stage data cleansing method for bridge global positioning system monitoring data based on bi-direction long and short term memory anomaly identification and conditional generative adversarial networks data repair. Struct. Control Health Monit. 29 (9), e2993. doi:10.1002/stc.2993

CrossRef Full Text | Google Scholar

Yao, D., Sun, Q., Yang, J., Liu, H., and Zhang, J. (2020). Railway fastener fault diagnosis based on generative adversarial network and residual network model. Shock Vib. 2020, 1–15. doi:10.1155/2020/8823050

CrossRef Full Text | Google Scholar

Yasuno, T., Ishii, A., Fujii, J., Amakata, M., and Takahashi, Y. (2020). Generative damage learning for concrete aging detection using auto-flight images.

Google Scholar

Yu, Y., Cai, C. S., and Liu, Y. (2021). Probabilistic vehicle weight estimation using physics-constrained generative adversarial network. Computer-Aided. Civ. Infrastructure Eng. 36 (6), 781–799. doi:10.1111/mice.12677

CrossRef Full Text | Google Scholar

Yuan, Z., Luo, J., Zhu, S., and Zhai, W. (2021). A Wasserstein generative adversarial network-based approach for real-time track irregularity estimation using vehicle dynamic responses. Veh. Syst. Dyn. 2021, 1–20. doi:10.1080/00423114.2021.1999480

CrossRef Full Text | Google Scholar

Zhang, H., Goodfellow, I., Metaxas, D., and Odena, A. (2018). Self-attention generative adversarial networks.

Google Scholar

Zhang, K., Zhang, Y., and Cheng, H. D. (2020). Self-supervised structure learning for crack detection based on cycle-consistent generative adversarial networks. J. Comput. Civ. Eng. 34 (3), 04020004. doi:10.1061/(asce)cp.1943-5487.0000883

CrossRef Full Text | Google Scholar

Zhang, T., and Wang, Y. (2019). “Deep learning algorithms for structural condition identification with limited monitoring data,” in International Conference on Smart Infrastructure and Construction 2019 (ICSIC), Cambridge, UK, 08 April 2019, ICE Publishing, 421–426.

CrossRef Full Text | Google Scholar

Zhang, X., Zhu, X., Zhang, X. Y., Zhang, N., Li, P., and Wang, L. (2018). “SegGAN: Semantic segmentation with generative adversarial network,” in 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), Xi'an, China, 13-16 September 2018 (IEEE), 1–5.

CrossRef Full Text | Google Scholar

Zhu, D., Guo, J., Cho, C., Wang, Y., and Lee, K. M. (2012). Wireless mobile sensor network for the system identification of a space frame bridge. Ieee. ASME. Trans. Mechatron. 17 (3), 499–507. doi:10.1109/tmech.2012.2187915

CrossRef Full Text | Google Scholar

Zhu, J. Y., Park, T., Isola, P., and Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks.

Google Scholar

Zhuang, Y., Qin, J., Chen, B., Dong, C., Xue, C., and Easa, S. M. (2022). Data loss reconstruction method for a bridge weigh-in-motion system using generative adversarial networks. Sensors 22 (3), 858. doi:10.3390/s22030858

PubMed Abstract | CrossRef Full Text | Google Scholar