A bright spot detection and analysis method for infrared photovoltaic panels based on image processing

The energy crisis and environmental problems have attracted global attention, thus the photovoltaic (PV) power generation technology comes to people’s mind. The application of unmanned aerial vehicle (UAV) inspection technology can overcome the disadvantages of large scale and high risk of this project. The application of unmanned aerial vehicle (UAV) infrared detection technology in PV power generation can not only improve work efficiency, but also have high economic benefits. This paper based on U-Net network and HSV space, proposes a method of PV infrared image segmentation and location detection of hot spots, which is used to detect and analyze the shielding of PV panels. Firstly, the main PV modules are automatically split from the different infrared image background based on U-Net. In order to quickly locate the defection location, the mask image is multiplied by the original image and then converted to HSV. The discriminant of bright spot features is introduced, and the discriminant mechanism is summarized according to the experiment, and the formation reason is analyzed. The experiment result shows that the method is not affected by the infrared image under the different background, provides data for the maintenance of power station and improves the detection accuracy. The accuracy rate of analyzing the causes of defects is 92.5%.

1 Introduction

With the rapid development of photovoltaic PV power generation industry, safe and reliable maintenance of power equipment is particularly important. Traditional detection methods are not suitable for large-scale PVPS because of low efficiency and high cost. However, the use of infrared imaging technology for fault detection is currently an effective method (Zhang et al., 2017). Cubukcu and Akanalci (2020) used infrared thermal imaging equipment to detect different PVPS and classify defects according to fault types. For the hotspots shown by the defect module in infrared image (Kamran et al., 2019), cluster segmentation was used to detect and provide quantitative data such as temperature and area (Salazar and Macabebe, 2016). Due to the resolution of infrared images and blurry edges, image segmentation is also a concern. In order to improve the effect of infrared image segmentation, a method based on an improved genetic algorithm to optimize the OTSU segmentation function was proposed (Wang, 2018). Kumar, and Ansari (Kumar et al., 2018) used watershed transform and color space methods for infrared image segmentation and fault detection of electrical equipment. In fact, plants show too many bright spots in infrared image, which makes some methods ineffective and image processing methods based on deep learning has been widely applied.Nie et al. (2020) eliminated noise according to traditional image processing and crop the infrared Image, then proposed an automatic hot spot detection model based on CNN. Zhang et al. (2021) proposed an industrial smoke image segmentation method based on FCN-LSTM, during the basic feature extraction, the time information of image sequence was extracted by using the short and long time memory network, and the moving smoke and background were distinguished by the dynamic characteristics of smoke and dust. Tang et al. (2020) Combined Seg-Net network with low-altitude UAV image, established a ground feature information extraction model in Huixian dissolved rock wetland. Ronneberger et al. (2015) proposed a new network structure, U-Net, which is widely used in the segmentation of organs in medical images (Lei et al., 2019), prediction of different crop types in agriculture observations (Wei et al., 2019), forest ecological management (Wagner et al., 2019). Its effect is better than traditional pattern recognition methods.

In this paper, the U-Net network is used for infrared image segmentation, bright spots are detected in the HSV space and located in the original image. The classification standard of bright spots is proposed as the maintenance basis, which has accuracy and effectiveness. U-Net is a neural network composed of contraction path and symmetric expansion path, which can use fewer images for end-to-end training, with high segmentation accuracy and fast speed (Ronneberger et al., 2015). It is suitable for the case of small data sets in this paper. In order to remove more interference factors, this paper makes an improvement on the traditional U-Net, and increases the convolutional layer in the coding stage to improve the segmentation accuracy of infrared PV panels. Meanwhile, the Dropout layer is added behind the low-level network to improve the generalization ability of the data, reduce the over fitting probability. Then, in order to realize the quick location of the spot on the UAV infrared PV panel, according to the characteristics of the infrared PV panel and spot in HSV space, the infrared PV image and mask image can be multiplied to quickly detect the bright spot and locate it to the original image. At the same time, a classification standard for the bright spot of infrared PV panels is put forward, which is taken as the maintenance basis. Finally, the advanced and reliability of the method is verified by a detailed experiment. Although the U-net network can train small data, in order to improve the training effect, image enhancement is carried out on the data set in advance. Through cutting and stitching images collected from PVPS, the data set eventually increases from 300 to 450.

1) This paper analyzes the applicability of the U-Net model in the field of infrared PV panel detection, and aim at providing a more robust and accurate solution to the specific problem of hot spots caused by green plants and dust accumulation.

2) Bright spots contained in the data set have been accurately tagged and located in advance, and our method is less affected by light. It also provides a benchmark for the rapid detection of bright spots on PV panels by using novel image processing methods.

3) In order to analyze the cause of defect formation, the bright spot measurement mechanisms N_d and N_g were introduced, and the analysis accuracy was proved to be 92.5% through experiments.

2 Data acquisition

Using unmanned aerial vehicle (UAV) to collect images can improve the efficiency and reduce monitoring costs (Grimaccia and Sonia LevaNiccolai, 2017). As shown in Figure 1A, the flight equipment used has strong wind resistance and stability. The image acquisition equipment of FLIR infrared thermal imager is chosen as shown in Figure 1B. Figure 1C shows the scenario where the UAV is at work.

FIGURE 1

FIGURE 1. Data acquisition equipment: (A) UAV equipment. (B) Infrared equipment. (C) UAV is capturing images.

A total of 450 images of the dataset set were used in the experiment, it contains 350 training sets and 100 test sets. We resized it to 512 × 512 (pixel) for the following image processing. Figures 2A, B shows the actual infrared image, and we use the LabelMe tool to mark the complete PV module. Figure 2C shows the label sample.

FIGURE 2

FIGURE 2. The acquired PV infrared images: (A) Image 1 (B) Image 2. (C) Masking sample.

3 U-Net network model

U-Net network is a distributed structure on both sides of encoding and decoding, and it is a very robust model for edge extraction using a small amount of data (Ronneberger et al., 2015). The left encoding is used to extract image features, and the right decoding is used to restore image feature edge information. Figure 3 shows the U-Net network structure in this paper. The network adopts the 3 × 3 convolutional layer structure for 23 times (stride = 1, padding), the activation function of ReLU, and the maximum pooling layer structure of 2 × 2 (stride = 2). The characteristic graph output by each encoder unit is connected to the corresponding decoder reconstruction unit according to the jump connection mechanism. Add Dropout layer after the low-level network to improve the generalization ability of data and reduce the probability of overfitting. In order to improve the robustness of the model, reduce the semantic gap and ensure the operation speed, this paper adds one more 3 × 3 convolution layer for each subsampling process. Figure 3 shows our U-Net network structure in this paper.

FIGURE 3

FIGURE 3. The structure of our U-Net network.

In this paper, the experimental environment was Windows system (CPU Intel i7, NVIDIA GeForce RTX 2070, 16 GB memory), python 3.7.0. The deep learning network framework PyTorch was used for model training, with a learning rate of 0.0001, 300 iterations and batch size of 5 times. Figure 4 shows the loss value of PV infrared images training in TensorBoard. The training loss value is 0.01041. It shows that the U-Net model is feasible to segment the infrared PVP.

FIGURE 4

FIGURE 4. Training loss curves.

As shown in Figure 5, the PV modules segmented by U-Net in this paper are relatively complete. The phenomenon of over-segmentation or under-segmentation is relatively less.

FIGURE 5

FIGURE 5. The segmentation results: (A) Original U-NET. (B) Our U-NET.

Table 1 proves the superiority of the U-Net with three layers compared with FPS, Seg-Net and U-Net through experiments. We can see our algorithm FPS get 19.62 f/s and detection times arrive 12.53 ms for one picture, its mAP compared to FCN just below 3.68. However, the overall evaluation is relatively better.

TABLE 1

TABLE 1. The performance of different training model.

4 PV infrared image analysis process

4.1 Bright spots segmentation

RGB color mode (RGB) is more susceptible to lighting than HSV color model (Wang et al., 2021). So, Maiti and Chakraborty, (2012) converted RGB image into HSV space, processed and combined the images of each channel to obtain brain tumor segmentation image. Malik et al. (2018) proposed an improved tomato detection algorithm based on HSV color space and watershed segmentation, with high accuracy. So, in this paper, the infrared image was first converted into HSV space, the difference between the two channel images as shown in Eq. 1.

$\mathit{s}\mathit{u}\mathit{b}=\mathit{d}\mathit{o}\mathit{u}\mathit{b}\mathit{l}\mathit{e}\left(V-H\right).(1)$

The sub image is processed by adaptive threshold segmentation and multiplied by the original infrared image to obtain the bright spots image, as shown in Eq. 2.

$\mathit{r}\mathit{e}\mathit{s}\mathit{u}\mathit{l}\mathit{t}=\frac{C\left(i,j\right)}{255}\cdot I\left(i,j\right),(2)$

where C (i, j) is a three-channel image synthesized from result of threshold segmentation, I (i, j) is an original infrared image.

The segmentation results are fused with the original infrared image and converted into HSV space, as shown in Figure 6.

FIGURE 6

FIGURE 6. PV modules and bright spots results: (A) PV modules segmentation. (B) Bright spots segmentation.

The size of the image changed during U-Net network training, we use morphological opening operations to obtain the centroid coordinates of the connected regions and the marker points as shown in Table 2. In this paper, the original pixel size of the infrared image in the experiment is 704 × 576. Set the original bright spot centroid coordinate is $\left(x_0,y_0\right)$, it can be calculated by

$\frac{X}{x_0}=\frac{512}{704},\frac{Y}{y_0}=\frac{512}{576}.(3)$

TABLE 2

TABLE 2. Centroid coordinate point.

On the basis of the bright spots image, we use the human checking, the Nie’s method (Nie et al., 2020) as benchmarks to demonstrate ours advantages. Nie’s method eliminated noise according to traditional image processing and crop the infrared image. Figure 7 shows the detection accuracy of the three methods in different numbers of test images. As the number of test images grow, the accuracy of manual detection has huge decreased. The accuracy of Nie’s method and our method still keep at 90%, and our method is more stable.

FIGURE 7

FIGURE 7. Comparison of detection accuracy.

4.2 Infrared image surface features

The bright spots on the surface of the infrared image include hot spots and more dust covering the orange-red area. In order to analyse the image features, the image is first introduced into overview interface, the imtool command to display the image processing interface, select the target area to analyse the starting point, the block-by-block generated grey value and then the Pixel Region is used to move and select the target area (Peng and Liu, 2018; Liu et al., 2020).

In order to judge the feature of bright spots, it is convenient to calculate the average grey value and the number of pixels points the measurement standard as shown in Eq. 4.

$\begin{array}{l}{A}_g=\frac{{\sum }_{i,j\in \left[m,n\right]}\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{y}\left(i,j\right)}{N\left(\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{y}\left(i,j\right)\ne 0\right)},\\ \left\{\begin{array}{cc}100<\mathrm{g}\mathrm{r}\mathrm{e}\mathrm{y}\left(i,j\right)<200,& N_d+1,\\ \mathrm{g}\mathrm{r}\mathrm{e}\mathrm{y}\left(i,j\right)>200,& N_g+1,\end{array}\right.\end{array}(4)$

where A_g is the average grey value of image, grey (i, j) is the grey value of the pixel with coordinate (i, j), N is number, N_d is the number of pixels with grey values between 100 and 200, N_g is greater than 200, we can get them from MATLAB.

In this paper, when A_g is less than 150 and N_d is greater than N_g, it can be considered that bright spots are mainly caused by dust, otherwise, the hotspots are mostly caused by green plants. Figure 8A are test areas including normal area, dust covered area and plants shelter area. Figures 8B, C shows the normal area, the G value in the three channels of RGB is the smallest and the grey value is less than 100. Figures 8D, E shows dust covered area, the B values is the smallest and the grey value is between 100 and 200. Figures 8F, G shows plants shelter area, the RGB channel values and gray values are both greater than (b) and (d). Figure 9 shows the grayscale images of different image segmentation results. Table 3 shows the measured values and the main types of defects on the PV infrared image.

FIGURE 8

FIGURE 8. RGB and grey values of target areas: (A) Test areas from left to right: normal area, dust covered area, plants shelter area. (B) RGB values of normal area. (C) Grey values of normal area. (D) RGB values of dust covered area. (E) Grey values of dust covered area. (F) RGB values of the area shaded by plants. (G) Grey values of the area shaded by plants.

FIGURE 9

FIGURE 9. (A-F) shows the grayscale images of different image segmentation results.

TABLE 3

TABLE 3. Different image detection results.

5 Conclusion

In this study, for the inspection and evaluation of large-scale PVPS, a method based on U-Net network and HSV space to segment and detect PV infrared images captured by UAV was proposed. The proposed method uses the U-Net network training model to segment PV modules from the different background, then locates the bright spot pixel coordinates in the HSV space, and classifies defects according to the image features. The experimental results show that the complex background can be effectively eliminated, and can provide qualitative and quantitative evaluation results. To prove its advantages, the other two methods are used for comparison. In future works, the combination of UAV, infrared thermal imaging and image processing technology can quickly patrol large-scale PVPS, find out the bright spots and analyze the defect in time which is conducive to the later maintenance of the power station.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

Author contributions

JL completed the main work, and NJ assisted in making some modifications of the paper, they approved it for publication.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The handling editor shares affiliation with author JL, HW.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Cubukcu, M., and Akanalci, A. (2020). Real-time inspection and determination methods of faults on photovoltaic power systems by thermal imaging in Turkey. Renew. Energy 147 (1), 1231–1238. doi:10.1016/j.renene.2019.09.075

CrossRef Full Text | Google Scholar

Grimaccia, Francesco, Leva, Sonia, and Niccolai, Alessandro (2017). PV plant digital mapping for modules’ defects detection by unmanned aerial vehicles. IET Renew. Power Gener. 11 (10), 1221–1228. doi:10.1049/iet-rpg.2016.1041

CrossRef Full Text | Google Scholar

Kamran, N., Wajahat, A., Abbas, K. H., Yang, Y., and Athar, S. (2019). Hotspot diagnosis for solar photovoltaic modules using a Naive Bayes classifier. Sol. Energy 190, 34–43. doi:10.1016/j.solener.2019.07.063

CrossRef Full Text | Google Scholar

Kumar, Deepak, Amit, Kumar, and Ansari, M. A. (2018). “A flexible scheme to fault detection for electrical assets using infrared thermography,” in Advances in electronics, communication and computing (Singapore: Springer). doi:10.1007/978-981-10-4765-7_57

CrossRef Full Text | Google Scholar

Lei, Yang, Liu, Yingzi, Xue, Dong, Tian, Sibo, Wang, Tonghe, Jiang, Xiaojun, et al. (2019). Automatic multi-organ segmentation in thorax CT images using U-Net-GAN. Computer-Aided Diagn. 46 (5), 2157–2168. doi:10.1002/mp.13458

PubMed Abstract | CrossRef Full Text | Google Scholar

Liu, Jun, Liu, Yang, and Yichen, Ke (2020). Detection and analysis of a quay crane surface based on the images captured by a UAV. Remote Sens. Lett. 11 (1), 76–85. doi:10.1080/2150704x.2019.1686779

CrossRef Full Text | Google Scholar

Maiti, Ishita, and Chakraborty, Monisha “A new method for brain tumor segmentation based on watershed and edge detection algorithms in HSV colour model,” in Proceedings of the Paper presented at the 2012 National conference on computing and communication systems, Durgapur, India, November 2012, 192–196. doi:10.1109/NCCCS.2012.6413020

CrossRef Full Text | Google Scholar

Malik, Muhammad Hammad, Zhang, Ting, Han, Li, Zhang, Man, Shabbir, Sana, and Ahmed, Saeed (2018). Mature tomato fruit detection algorithm based on improved HSV and watershed algorithm. IFAC-PapersOnLine 51 (17), 431–436. doi:10.1016/j.ifacol.2018.08.183

CrossRef Full Text | Google Scholar

Nie, J. F., Luo, T., and Li, H. (2020). Automatic hotspots detection based on UAV infrared images for large-scale PV plant. Electron. Lett. 10 (18), 193–194. doi:10.1049/el.2020.1542

CrossRef Full Text | Google Scholar

Peng, Lin, and Liu, Jun (2018). Detection and analysis of large-scale WT blade surface cracks based on UAV-taken images. IET Image Process. 12 (11), 2059–2064. doi:10.1049/iet-ipr.2018.5542

CrossRef Full Text | Google Scholar

Ronneberger, Olaf, Fischer, Philipp, and Brox, Thomas (2015). “U-Net: Convolutional networks for biomedical image segmentation,” in Paper presented at the International Conference on Medical image computing and computer-assisted intervention (Cham, Switzerland: Springer). doi:10.1007/978-3-319-24574-4_28

CrossRef Full Text | Google Scholar

Salazar, April M., and Macabebe, Erees Queen B. (2016). Hotspots detection in photovoltaic modules using infrared thermography. MATEC Web Conf. 70, 10015. doi:10.1051/matecconf/20167010015

CrossRef Full Text | Google Scholar

Tang, Tingyuan, Fu, Bolin, and Li, Ying (2020). Application of Seg-Net in extracting ground feature information of Huixian Karst Wetland from low-altitude UAV images. Wetl. Sci. 18 (04), 413–423. doi:10.13248/j.cnki.wetlandsci.2020.04.005

CrossRef Full Text | Google Scholar

Wagner, Fabien H., Sanchez, Alber, Tarabalka, Yuliya, Lotte, Rodolfo G., Ferreira, Matheus P., Aidar, Marcos P. M., et al. (2019). Using the U-net convolutional network to map forest types and disturbance in the Atlantic rainforest with very high resolution images. Remote Sens. Ecol. Conserv. 5 (4), 360–375. doi:10.1002/rse2.111

CrossRef Full Text | Google Scholar

Wang, Di, Dong, Sufen, and Cheng, Fang (2021). Improved intutitionistic fuzzy C-means clustering pork image detection in HSV space. Acta Metrol. Sin. 42 (08), 986–992.

Google Scholar

Wang, Ya. “Improved OTSU and adaptive genetic algorithm for infrared image segmentation,” in Proceedings of the Paper presented at the 2018 Chinese Control and Decision Conference (CCDC), Shenyang, China, June 2018. doi:10.1109/CCDC.2018.8408116

CrossRef Full Text | Google Scholar

Wei, Sisi, Zhang, Hong, Wang, Chao, Wang, Yuanyuan, and Xu, Lu (2019). Multi-temporal SAR data large-scale crop mapping based on U-Net model. Remote Sens. 11 (1), 68. doi:10.3390/rs11010068

CrossRef Full Text | Google Scholar

Zhang, Junpeng, Liu, Hui, and Li, Qingrong (2021). Industrial smoke image segmentation based on FCN-LSTM. Comput. Eng. Sci. 43 (05), 907–916.

Google Scholar

Zhang, Peng, Zhang, Lifu, Wu, Taixia, Zhang, Hongming, and Sun, Xuejian (2017). Detection and location of fouling on photovoltaic panels using a drone-mounted infrared thermography system. J. Appl. Remote Sens. 11 (1), 016026–16111. doi:10.1117/1.jrs.11.016026

CrossRef Full Text | Google Scholar