Semi-supervised surface defect detection of wind turbine blades with YOLOv4

0 Introduction

Wind is the most widely used renewable energy source worldwide.Wind turbines are key in generating wind energy, ensuring efficiency and cost-effectiveness [1, 2].The increased installation of wind turbines has significantly increased the likelihood of blade defects [3].Blades are the most exposed components of wind turbines and have many surface defects in wind turbine blades.The most critical defects are cracks and erosion, which can significantly affect the wind turbine performance.Therefore, the timely detection of blade defects is necessary for the safety and reliability of wind turbine systems and for reducing the cost of wind power generation [4].

The surface damage to wind turbines includes leadingedge erosion, surface cracks, lightning receiver damage,and vortex generator damage [5].Traditionally, the identification of surface defects in wind turbines has been primarily based on ultrasonic detection [6] and visual inspection.Although ultrasonic detection can be performed in harsh environments, signal attenuation is problematic.Visual inspection is risky, consumes considerable workforce and time, and affects the accuracy of judgment based on human experience.More efficient machine- and deep-learning methods have been used in recent studies.Deng et al.proposed an LPSO algorithm and an adaptive filter with a log-Gabor filter to construct a classifier for identifying and classifying defect types [7].Wang et al.proposed a machine-learning algorithm for the UAV-based automatic detection of surface cracks on wind turbines[8].This automatic detection method is based on the Haarlike features of blade cracks [9] and creates a new cascade classifier from various machine-learning models, including logitBoost [10], and support vector machines [11], which are very effective in identifying the number of cracks and defect localization.However, this study was limited to detecting cracks and relied on traditional machine-learning methods.In recent years, deep-learning technology has led to breakthroughs in object detection.The two main methods are two-stage detection and single-stage models.In twostage models, the first stage predicts the bounding box while the second stage predicts and corrects the bounding box to improve accuracy.The disadvantages of two-stage models are cumbersome training steps, slow testing processes, and excessive physical space occupation.Single-stage detection models integrate the entire process of generating an alternative bounding box and predicting object position and category.Yu et al.used a DCNN trained on the ImageNet dataset to extract deep features of blade images for defect recognition, which performed better with small samples[12].Shihavuddin et al.used UAV-based high-definition images for surface defect detection of wind turbine blades using a two-stage object detection algorithm, Faster R-CNN[5].Zhu et al.proposed a residual network-based deeplearning method by fusing multiple features to detect blade defects, significantly improving the detection efficiency and accuracy compared to benchmarking models [13].

In this study, a wind turbine blade defect detection method was developed based on the improved YOLOv4[14].The single-stage object detection method no longer needs to generate a candidate bounding box, but it directly predicts the entire image, simplifying the process and reducing computation and time [15].However, collecting sufficiently labeled images of defective wind turbine blades to train deep-learning models is difficult.Owing to the lack of images of wind turbine blades with defects, the defect identification of blades is a few-shot learning problem.This study takes advantage of the large amount of available unlabeled data by proposing a semi-supervised learning(SSL) approach based on a GAN [16].The improved SSL module can acquire additional features from a large number of unlabeled images, which are later passed to the defect detection network.This process enables the network to obtain enhanced features in addition to the limited labeled data to improve the identification of defects in different blades.The scSE attention mechanism [17] is used to enhance critical features on feature maps passing through different levels of the network, which improves the ability of the network to detect smaller-sized targets in blade defects.A new loss function was used to address the imbalance in the defect categories of the blade images and expedite the network convergence [18, 19].

1 Proposed method

1.1 Localization and classification of surface defects on blades

Advancements in UAV technology have facilitated the collection of high-resolution images of various wind turbine blade components.Based on blade images captured by the UAV, this study established an object detection model for small-scale blade defects by constructing an SSL method based on a GAN using a small amount of labeled data.This method can identify surface defects in blade images, locate the location of the defects, and output the defect window for defect contour extraction.The framework of the proposed wind turbine blade defect detection method is illustrated in Fig.1.

Fig.1 Framework of the semi-supervised defect-detection method

Blade surface defects usually occupy fewer pixels and smaller areas in images captured by UAVs.Such blade defect characteristics can be handled by improving the YOLOv4 object-detection network, which enhances essential features in the feature map by restructuring the network and enhancing the small-scale object-detection capabilities.Training deep learning-based detection models often requires a large amount of labeled data, whereas manual labeling requires considerable labor.Moreover, a lack of experience in defect labeling may lead to mistakes,impairing the quality of the dataset.In this study, an SSL using a GAN was proposed to enable the model to acquire features from a small number of labeled images and additional features from many unlabeled images.By capturing features from unlabeled data, the model can acquire defect information that is unavailable in supervised learning.The enhancement of the generalization capacity of the model allows it to identify poorly characterized shape-diverse defects more easily.Therefore, only a small amount of labeled sample data is required to increase the accuracy of blade defect detection using the proposed semisupervised learning method, thereby effectively reducing the cost of manual labeling.

1.2 Semi-supervised learning methods using a small amount of labeled data

To train the blade defect-detection model, professionals must label the location and type of defects in the blade image, which is a large workload.Many original unlabeled images can be used as potential training samples.Based on this, this study employed a combination of labeled and unlabeled data to train a semi-supervised network and then used a small amount of labeled data to train an improved YOLOv4 network.This study investigated a semisupervised learning method using a GAN to effectively train a blade defect-detection model.This study innovates an object-detection algorithm and constructs a YOLOv4 model based on semi-supervised adversarial learning for blade defect localization and identification.An SSL model for blade-surface defect localization and identification was constructed, as shown in Fig.2.The model includes the reconstruction network R, the object detection model YL, and the T layer that shares information between them.R is a GAN that includes generator G, which consists of the backbone of YOLOv4 as an encoder, and a discriminator D, which is composed of inverse convolutional layers.G reconstructs the input image, whereas D discriminates the image reconstructed from G and the original input image.Generator G is a deep autoencoder neural network in which the encoder is the CSPDarknet53 network used as the backbone of the YOLOv4 model and the decoder uses an upsampling layer.The features pass through the decoder to produce a reconstructed image of the input image for use in the subsequent loss functions.Discriminator D is a deep convolutional neural network primarily used to distinguish the false image generated by G from the true image fed into the network.This study employs an improved YOLOv4 model as the object detection model, YL, which utilizes CSPDarknet53 as a backbone network for deep feature extraction and image meshing to identify and locate defects.Unlike previous transfer learning methods that share parameters, this study utilizes the learned network layer T to share feature maps from R. T consists of convolutional layers with batch normalization and ReLU activation functions.

Fig.2 Training method for the defect localization and classification model based on semi-supervised adversarial learning

The training process of the YOLOv4 model based on semi-supervised adversarial learning consists of three steps.

(a) Initialize the YL model using the CSPDarket53 parameters trained on the VOC dataset to transfer its image feature extraction capability.

(b) Train R using all the blade images. Because R does not require labeling information, the model can be trained based on all the images.For N images, the labeled or unlabeled data are Xn, n=1, 2, ..., N, and the loss of the entire R structure is given by (1).

where lbc is the cross-entropy of D to counteract the loss,and lmse is the mean square reconstruction loss of G.In this study, the Adam algorithm was used to optimize the loss function.A better training effect was achieved by setting the weight decay to zero.

(c) Use labeled data to post-tune the model.The model was post-optimized by combining the parameters obtained from the previous pre-training and the labeled data.Suppose that the features of the labeled data are represented by Xm and the corresponding labels are Ym, m = 1, 2, ..., M. Then,the loss function is defined by (2).

where lYL denotes the loss function of YOLOv4. Consistent with the pre-training model, the Adam algorithm was used to optimize the loss function when post-tuning the model,and the weight decay was the same as that for the pretraining step.

1.3 Object detection model for small-scale blade defects

Small-scale defect targets typically occupy fewer pixels in a blade image.The features extracted with multiple convolution layers were not apparent and were difficult to utilize in subsequent networks.In this study, the attention mechanism module in the detection network was designed to accurately detect small-scale targets.Simultaneously,considering the imbalance problem of different categories of defects, a suitable loss function will be designed to further improve the detection accuracy.Blade defects typically occupy fewer pixels in the captured images.For this small-scale object detection problem, this study improves the existing YOLOv4 model structure by utilizing the attention mechanism to enhance essential features.This study also improves the loss function of YOLOv4 to solve the imbalance problem of defect categories.The details of this method are discussed below.

1.3.1 Improved YOLOv4 model structure

As the hierarchy of the convolutional network increases,the features of the small-scale targets in the feature map gradually weaken.Therefore, the network may have missed or misdetected small-scale targets.In this study, defect detection was performed based on the YOLOv4 model(shown in Fig.3), which uses a path aggregation network(PAN) at the neck of the YOLOv4 model.This structure can convey strong semantic features from the top down and strong localization features from the bottom up.Thus,small-scale targets can be identified.To further improve the detection capability, this study aims to enhance the spatial and channel features in feature maps using the scSE attention module in image semantic segmentation.Hence, the proposed network can capture target features for focused learning.The scSE attention module incorporates attention sSE and attention cSE.The former integrates information about the spatial dimension of the feature map and attention cSE, whereas the latter integrates information about the channel dimension of the feature map, resulting in feature maps UˆscSE:

Fig.3 YOLOv4 Model Structure

In (3), UˆcSE is the output obtained from the intermediate features after the cSE module, UˆsSE is the output obtained from the intermediate features after the sSE module, and the feature map fusion method is the addition of various elements in the matrix.

We compared the attention module embedded in the backbone network, neck, and detection head to determine the optimal module embedding position, as shown in Fig.4.

1.3.2 Optimization of loss function

The model output of YOLOv4 consists of six components: the center coordinates of the defective bounding box, the corresponding true bounding box boundary dimensions, the representation of the defective bounding box confidence level, and the probability that the output belongs to each class of defects.The overall loss function is given by (4).

where the CIoU Loss function lCIoU is:

Fig.4 Position of Attention Module

In (5), IoU is the intersection and concurrency ratio between the predicted defect window and the real defect window of the image output by the network.d denotes the length of the center of the real bounding box from the center of its corresponding predicted bounding box, c is the diagonal distance of the smallest box that contains both the predicted box of the network output and the real box of the labeled image, and (wˆ, hˆ) is the size of the defect window of the network output.

The confidence loss lconf is calculated following (6):

Because the YOLOv4 model adopts the cross-entropy(CE) loss function to characterize the classification errors,it cannot solve the category imbalance problem.During the operation of wind turbines, different categories of bladesurface defects occur with different probabilities, resulting in an imbalance of data for various defect categories.The effect of the category imbalance problem on the model training is reduced in this study by integrating the focal loss[20] and CE loss functions to construct the focal-cross (FC)loss function in (8).

In (8), e is Euler’s number, and η and γ are adjustable parameters.Fig.5 compares the FC loss with the focal loss and CE loss functions.

The proposed loss function is close to the focal loss for well-classified defects and the CE loss function for poorly classified samples.Therefore, this loss function can make the model pay more attention to the defect category with fewer samples to balance the proportion of different categories, thus effectively improving the detection accuracy.

Fig.5 Loss Function Comparison

2 Experiments results

2.1 Dataset

Two datasets were used to evaluate the model performance.Dataset I is a public dataset used in [21] and consists primarily of wind turbine and solar photovoltaic panel images captured by UAVs.In Dataset I, multiple defect types have been classified into a single damage category.Dataset II, which contains drone inspection images of a wind turbine (DTU), is a public dataset with 606 labeled images.The DTU dataset contains 701 high-resolution images of wind turbine blades captured using a UAV [22].Many types of defects in this dataset are of interest for defect detection of wind turbine blades and satisfy the need for defect detection of wind turbine blades.Following [5], there are four categories of labels, including vortex generator panel(VG), VG panel with missing teeth (VGMT), leading-edge erosion (LE), and lightning receptor (LR).Table 1 shows the details of the dataset, with the number of defects indicated in parentheses.The unlabeled images used for semi-supervised training were obtained from approximately 940 wind turbine images collected by the authors.

Table 1 Details of the dataset

Dataset Number of defects in different categories Quantity Resolution Dataset I Damage (312) 391 100 × 100 -1844 × 1281 Dataset II VG (482), VGMT (103),LE (157), LR (127) 606 512 × 512

2.2 Training Strategies and Evaluation Metrics

In the numerical experiment, the Pytorch deep-learning API was used to conduct experiments on YOLOv4 objectdetection methods with different network structures.The input image size for training was 512 × 512 pixels, with a batch size of 4.The training, test, and validation sets were divided at a 3:1:1 ratio for 100 training rounds.Before inputting the image into the network, data augmentation,such as scaling, flipping, and color gamut transformation,was performed randomly.The evaluation metrics used in[17], including precision (%), recall (%), F1-Score (%),and mAP50 (%), were considered in this study.If the box predicted by the network overlapped with the true box by more than 0.5, it was retained.The mean average precision(mAP) was used as an indicator to evaluate the algorithm performance.

Table 2 Results of ablation experiments on scSE attention insert position

Algorithm Precision/% Recall/% F1-Score/% mAP50/%Faster R-CNN 60.34 88.61 71.79 83.02 DETR 58.68 89.87 71.00 85.96 YOLOv4 93.06 84.81 88.74 89.84 backbone 89.19 83.54 86.27 90.08 neck 93.15 86.08 89.48 91.25 detection header 92.65 79.75 85.72 89.80 all 90.41 83.54 86.84 91.78

2.3 Ablation Experiment and Comparative Experiment

The three improvement techniques for the YOLOv4 application were validated by ablation experiments on the scSE attention mechanism insert position conducted on Dataset I; the results are shown in Table 2.With only a single defect category, if the scSE attention is placed at the neck, the network obtains the largest mAP50 enhancement(+1.41%).However, if scSE attention is placed on the detection head, a slight decrease in the F1-Score and mAP50 is observed.This may be because Dataset I labels all defects of different types as damage categories, whereas different types of defects behave differently in terms of shape and texture.After feature extraction by the network, different defect features are obtained for the same defect category.This can mislead the network training, which leads to a decrease in the final Precision and Recall.

Table 3 Results of the ablation experiments

*The values on the left side of ‘/’ indicate the results of Dataset I, and those on the right side indicate the results of Dataset II.The first column indicates the combination of different algorithms, and ‘√’ and ‘-’ indicate whether the method was used or not, respectively

Algorithm Semi-supervised structure FC loss scSE attention Precision/% Recall/% F1-Score/% mAP50/%YOLOv4 - - - 93.06/97.27 84.81/75.31 88.74/84.89 89.84/90.75 1-√-94.29/95.78 83.54/80.70 88.59/87.60 90.43/92.70 2 90.41/98.49 83.54/70.54 86.84/82.20 91.78/91.86 3√--90.00/96.82 79.75/72.05 84.57/82.62 90.62/92.00--√4 92.75/96.69 81.01/77.75 86.48/86.19 92.55/93.25 5√-√94.12/89.68 81.01/86.04 87.07/87.82 92.27/93.30-√√6 95.24/92.82 75.95/83.66 84.51/88.00 91.59/93.44 7√√√96.88/94.09 78.48/90.03 86.71/92.02 92.89/95.58√√-

The added attention at the detection head blurred the discriminative ability of the classifier, resulting in a significant decrease in the recall rate.When scSE attention was added at all sites, mAP50 improved significantly(+1.94%).The corresponding small decreases in precision and recall were acceptable.Therefore, the following experiments used improvements in attention at all three sites.In addition, the improved algorithm shows a significant improvement in precision (+38.2%), F1-Score(+15.71%), and mAP50 (+6.93%) compared to DETR [23];recall is significantly reduced, and the overall detection ability of the improved algorithm is better than that of the classical algorithm.

Table 3 lists the ablation experiments conducted by combining different improvement strategies with YOLOv4 on Datasets I and II.Because Dataset I has a single defect category and the defect morphology is quite different, the enhancement using the FC loss function is not apparent,and the recall decreases slightly.However, adding a semisupervised structure and attention can effectively enhance the network’s generalization and feature extraction capability, significantly improving both precision and mAP50.For multicategory defect detection on Dataset II,the increase in detection difficulty due to multicategorization results in a small decrease in precision.The FC loss demonstrated promising results in multicategory defect detection, with significant improvements in recall (+14.72%)and mAP50 (+4.83%).Table 4 shows the AP values for each defect for all the algorithms in Table 3 on Dataset II; the improvement in the network is significant for each defect category, especially for VGMT (+11.55%).The above analyses show that using the FC loss function can achieve a good balance between positive and negative samples, particularly for multicategory defect detection.

The addition of scSE attention can make the network focus on information that can be easily ignored and enhance its capability to identify small targets.Introducing a semisupervised structure can also enable the network to obtain the features of unlabeled images, significantly improving the performance of the network.

Table 4 AP values of each algorithm on Dataset II

*The algorithmic implications are the same as in Table 3

Algorithm LE/% LR/% VG/% VGMT/% mAP50/%YOLOv4 89.05 100 96.15 77.78 90.75 1 90.48 100 97.64 82.69 92.70 2 90.59 100 94.86 81.98 91.86 3 87.38 99.77 97.13 83.73 92.00 4 91.59 100 94.43 86.99 93.25 5 91.24 100 98.03 83.91 93.30 6 91.58 100 97.49 84.71 93.44 7 95.60 100 98.47 89.33 95.58

3 Conclusion

In this study, an improved semi-supervised object detection method based on the YOLOv4 network was proposed.This method can detect surface defects in wind turbines more accurately than previous detection methods.Considering difficulties in practice (e.g., the difficulty of obtaining high-quality images and correctly labeling images), this study developed a semi-supervised structure based on a GAN, which enables the network to take advantage of the features of a large number of unlabeled images.The proposed method not only facilitates the model in differentiating defective features and improves its generalization capability but also handles the imbalance problem of samples and enhances important spatial and channel features in the feature map.The loss function was improved, and the scSE attention mechanism enabled the network to capture the target features and perform focused learning.These improvements significantly improve the feature extraction capacity of the network and the identification of targets of different defect categories.Experiments showed that the improved model can be applied to the surface defect detection of wind turbines with multiple defect categories.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under grants 62202044 and 62372039, Scientific and Technological Innovation Foundation of Foshan under grant BK22BF009, Excellent Youth Team Project for the Central Universities under grant FRF-EYIT-23-01, Fundamental Research Funds for the Central Universities under grants 06500103 and 06500078,Guangdong Basic and Applied Basic Research Foundation under grant 2022A1515240044, and Beijing Natural Science Foundation under grant 4232040.

Declaration of Competing Interest

We declare that we have no conflict of interest.