Solar Irradiance Nowcasting for Virtual Power Plants Using Multimodal Long Short-Term Memory Networks

The rapid penetration of photovoltaic generation reduces power grid inertia and increases the need for intelligent energy resources that can cope in real time with the imbalance between power generation and consumption. Virtual power plants are a technology for coordinating such resources and monetizing them, for example on electricity markets with real-time pricing or on frequency reserves markets. Accurate short-term photovoltaic generation forecasts are essential for such virtual power plants. Although significant research has been done on medium- and long-term photovoltaic generation forecasting, the short-term forecasting problem requires special attention to sudden fluctuations due to the high variability of cloud cover and related weather events. Solar irradiance nowcasting aims to resolve this variability by providing reliable short-term forecasts of the expected power generation capacity. Sky images captured in proximity to the photovoltaic panels are used to determine cloud behavior and solar intensity. This is a computationally challenging task for conventional computer vision techniques and only a handful of Artificial Intelligence (AI) methods have been proposed. In this paper, a novel multimodal approach is proposed based on two Long Short-Term Memory Networks (LSTM) that receives a temporal image modality of a stream of sky images, a temporal numerical modality of a time-series of past solar irradiance readings and cloud cover readings as inputs for irradiance nowcasting. The proposed nowcasting pipeline consists of a preprocessing module and an irradiance augmentation module that implements methods for cloud detection, Sun localization and mask generation. The complete approach was empirically evaluated on a real-world solar irradiance case study across the four seasons of the northern hemisphere, resulting in a mean improvement of 39% for multimodality.

Introduction

VPP (Virtual Power Plants) are emerging as a key solution for power grid decarbonization and the integration of photovoltaic and wind power generation. Yet the inclusion of these renewable generation resources to VPPs remains limited, partly due to the uncertainties involved in their production (Liu et al., 2021). Mahmud et al. (2020) emphasize the need for VPPs to have a capability to cope with sudden weather changes impacting PV production as well as other energy consuming assets under the same VPP. Yu et al. (2019) review the major categories of uncertainty that a VPP needs to manage. They identify market prices and consumption demand as uncertainties that are interdependent with renewable generation uncertainty. A common way to manage the uncertainty of photovoltaic generation is to invest in significant battery storage capacity Liu et al. (2019), but such resource intensive investments could be reduced by better predictions of the photovoltaic generation. A VPP must have a capacity to operate in real-time and to adjust plans that were made earlier based on forecasts of PV generation and other assets (Fan et al., 2020). The adjustments can be made either to ensure compliance with bids that were made ahead of time, or to allow the VPP to perform a second round of trading on real-time markets (Qiu et al., 2017). In either case, an accurate short-term photovoltaic generation forecast is needed to support the real-time operations of a VPP.

Generation and consumption in a power grid must be balanced at all times in order to maintain the grid frequency close to a rated frequency. Excess generation causes over-frequency and excess consumption causes under-frequency. Frequency reserves are services purchased by the grid operator to react to such situations. Providers of reserve resources commit to increasing production or decreasing consumption in underfrequency situations, and to decrease production or increase consumption in over-frequency situations. An example of photovoltaic participation in frequency reserves is curtailing the production in over-frequency situations. Subramanya et al. (2021) report of such a commercially operational installation, managed by a VPP and participating in Finnish frequency reserve markets. Thus, frequency reserves are another category of applications benefitting from photovoltaic generation forecasts. Subramanya et al. address the problem with day-ahead forecasts for the purpose of bidding on the day-ahead frequency reserve markets. However, there is a need for further work on more accurate real-time forecasts that allow the VPP to address actual frequency deviations as they occur. Such forecasts would be applicable to work on dynamic planning of a VPP participating on frequency reserves (Javadi et al., 2019).

This paper is motivated by the need for development of real-time capabilities to VPPs, for example to participate on real-time electricity markets, to react to frequency deviations when participating on frequency reserves markets, or simply to cope with commitments that were made ahead of time based on uncertain photovoltaic generation predictions. A capability for real-time forecasting of photovoltaic generation, “nowcasting,” is developed for these purposes. Artificial Intelligence (AI) techniques such as classic neural networks Al-Dahidi et al. (2019) and more recently Long Short-Term Memory Networks (LSTM) Li et al. (2020), Stüber et al. (2021) have performed well for medium-term solar forecasting. These applications of LSTM are based on time-series data, but the short-term forecasting context could significantly benefit from the additional consideration sky image data. The processing of diverse kinds of inputs by a machine learning model is known as the multimodal approach Baltrusaitis et al. (2019), which is applied in this paper to consider time-series data alongside sky image data. In this paper, following research contributions are reported to advance the state-of-the-art on photovoltaic nowcasting:

1 An AI pipeline for nowcasting that receives a temporal image modality of a stream of sky images, a temporal numerical modality of a time-series of past solar irradiance readings and cloud cover readings as inputs, followed by a data preprocessing module and an irradiance augmentation module leading up to the irradiance nowcasting module.

2 A novel method for cloud detection, Sun localization and mask generation in the irradiance augmentation module that computes the impact of cloud cover on irradiance intensity.

3 A multimodal approach based on two LSTMs for the image and time-series modalities in the irradiance nowcasting module that can deliver real-time capabilities to VPP.

4 Finally, the complete pipeline study is evaluated across the four seasons of the northern hemisphere, using a real-world solar irradiance case study conducted by the NREL SRRL labs.

Related Work

Nowcasting for VPP With Photovoltaic Generation

A resource intensive way to manage photovoltaic integration is to invest in large-scale battery resources to complement the photovoltaic generation capacity, and one line of research is to investigate the optimal sizing of battery storages for this purpose e.g., (Kong et al., 2019; Kosmadakis et al., 2021; Muñoz-Rodríguez et al., 2021). A more intelligent and less resource intensive approach is to employ a VPP Thavlov and Bindner (2015) to coordinate diverse energy resources such as photovoltaic generation, smart loads and electric vehicles. A typical planning horizon for a VPP is day-ahead, in which case the VPP requires forecasts of its photovoltaic generation as well as all the resources under its control for this time horizon (Xing et al., 2021). In this context, real-time pricing usually means hourly changing prices that are known day ahead Nge et al. (2019), but other shorter-term real-time pricing approaches exist and shall be introduced below. A minority of research considers a shorter timeframe in which the photovoltaic generation nowcasting could be exploited. Recognizing the inevitable errors in predicting wind power generation day ahead, Hu et al. (2021) propose a 2-stage dispatch, day-ahead and real-time, to exploit the more accurate short-term forecast; a similar approach is applicable to photovoltaic dispatch if short-term solar irradiation forecasts are available. Das et al. (2021) propose to optimize photovoltaic generation and battery storage use in the context of hourly changing electricity prices. Elkazaz et al. (2020) consider a household with local photovoltaic generation and battery storage, and propose a scheme consisting of day-ahead optimization as well as real-time control to manage the errors caused to the optimization by uncertain forecasts. Elkazaz et al. (2020), Das et al. (2021), Hu et al. (2021) are all examples of methods that use real-time measurements as the input to an algorithm that handles the error of medium-term renewable generation forecasts, so an obvious line of further research would be to exploit nowcasting. Nefedov et al. (2018) exploit electric vehicle batteries for short-term storage of surplus photovoltaic energy, updating plans as vehicles arrive or if they leave unexpectedly. Wu et al. (2020) and Seyyedeh Barhagh et al. (2020), present similar approaches with the additional capability to consider dynamic electricity prices. Kolodziejczyk et al. (2021) operate a system consisting of photovoltaic generation and battery storage on a real-time electricity market with hourly changing prices. An example of a close to real-time electricity market is Singapore’s half-hourly spot market (Zhang et al., 2021); at this time-scale, photovoltaic generation nowcasting exploiting sky image analysis begins to be advantageous. A few works address a very short-term scenario, in which the photovoltaic generation nowcasting approach is most beneficial. In particular, Larsen et al. (2017) consider a virtual power plant with smart loads participating on a real-time market with prices changing at 5 min intervals.

Photovoltaic generation is at the same time a problem and a potential solution for power grid frequency reserves. In fact, increased photovoltaic penetration without VPPs is a part of the problem, as it decreases the grid inertia Su et al. (2020), Zhang et al. (2020) and increases the demand for frequency reserves (Karbouj et al., 2019). This increased demand is specifically due to the short-term variability of photovoltaic generation (Sinsel et al., 2020). A straightforward approach is to invest in battery storage that is dedicated for frequency reserves (Koller et al., 2015). However, a VPP can be more economical for managing uncertainties and optimizing a set of diverse resources with different characteristics. Forecasting the grid frequency involves considerable uncertainty which is compounded by other uncertainties such as photovoltaic generation Mashlakov et al. (Mashlakov et al., 2021). As the future grid frequency deviations are not known even in the short term, it is the frequency reserve resource provider’s duty to ensure that the reserve resource capacity is available and ready to be activated in the event of a frequency disturbance (Subramanya et al., 2021). Several innovative solutions have been proposed for this purpose, exploiting the specific characteristics of the intelligent energy resources that are being exploited as frequency reserve resources. Angenendt et al. (Angenendt et al., 2020) identify an alternative way to exploit batteries for supporting local photovoltaic generation. Such batteries could be traded on frequency reserves in the low photovoltaic generation season. Other authors simply rely on smart loads to cover the demand for frequency reserves caused by large photovoltaic penetration (Conte et al., 2021). Bagheri-Sanjareh and Nazari (2020) present a comprehensive approach for coordinating photovoltaics with other smart energy resources for the purpose of providing frequency reserves. In the case of these and other systems for providing frequency reserves by coordinating intelligent energy resources, whenever photovoltaic generation is part of the mix, the nowcasting of such generation can support the planning of the coordination on a short time horizon, e.g., 5, 10 or 15 min. This is especially beneficial for reserve resources that need several minutes to be ramped up. Finally, it is noted that frequency reserve markets are traditionally operated by transmission system operators, but the emergence of photovoltaic microgrids is leading to the emergency of real-time frequency response being managed at the level of the microgrid Shi et al. (2015) or cluster of collaborative microgrids (Qazi et al., 2021).

Related Work in Solar Irradiance Nowcasting

In recent work, several approaches have been proposed for solar irradiance nowcasting, using sky cameras, satellite imagery, ground sensor networks, and time-series of historical PV generation.

Chen et al. (2019) proposed a spatio-temporal PV nowcasting method using a sensor network with predictor preselection. It enabled preselection in different cloud movement scenarios; stationary, and ramp, and interpolate cloud information to provide consistent PV nowcasts. Zheng et al. (2020) proposed a method to predict the output of solar power plants based on time series forecasting, primarily focusing on multiple regions. They used a Long Short-Term Memory Network (LSTM) model for the forecasting and multiple LSTM structures were compared to determine the final prediction model with sensitivity analysis. Additionally, particle swarm optimization was employed to improve the prediction by optimizing the parameters used for the LSTM models. Kong et al. (2020) proposed a series of novel deep learning approaches for short-term PV forecasting based on whole-sky-images. They developed both static and dynamic architectures composed of Convolutional Neural Networks (CNN), LSTM, ConvLSTM and PredNet deep learning models. Caldas and Alonso-Suárez (2019) have reported a hybrid forecasting method to forecast 1-min averaged solar irradiance using all-sky images and past irradiance measurements. Detection of clouds and cloud motion estimation were done using pre-existing methods with the addition of a post-processing procedure. After that, the predicted cloud information was used to infer the solar irradiance by forecasting the clear-sky index using the proposed method that uses a simple deterministic model. Zhang et al. (2018) proposes a deep photovoltaic nowcasting approach using sky images and historical PV power values. Several CNN based models were trained to learn the relationship between sky appearance and future PV power output. Feng and Zhang (2020) proposes a CNN based approach for intra-hour solar forecasting exclusively using sky images. This contains a set of models that generate fixed-step global horizontal irradiance (GHI) in parallel.

Infrared (IR) technology has also been applied in related work on sky imaging based solar irradiance forecasting. Mammoli et al. (2019) proposed short-term solar forecasting by merging far-field images from multiple longwave infrared sensors. Terrén-Serrano and Martínez-Ramón (2021) presented a short-term solar forecasting approach based on processing of global solar irradiance and ground-based infrared sky images. Both approaches focus on the cloud behavior and the latter also categorized the sky condition of an infrared image into four main cloud formations, clear-sky, cumulus, stratus, and nimbus. Apart from forecasting, infrared is also used in solar position identification which is an important aspect that even helps with forecasting. Nespoli and Niccolai (2020) proposed an approach for solar position identification using infrared thermal all-sky images of which the sunspot is defined clearer than color images. However, to capture infrared images a special camera system is specified.

As delineated above, several statistical, machine learning, deep learning, infrared-based and hybrid methods have been proposed in recent literature to address the technical challenges of solar irradiance nowcasting. However, none of these approaches consider the multimodality of the problem domain in terms of the need for development of real-time capabilities for VPP. The significance of cloud detection, Sun localization and mask generation in computing the impact of cloud cover on irradiance intensity as a necessary augmentation phase prior to the consideration of its multimodality is a further limitation of existing work. Drawing on this context, a new approach that delivers an AI pipeline consisting of preprocessing, augmentation, and a multimodal deep learning model, is presented in the following section. Table 1 lists the nomenclature.

TABLE 1

TABLE 1. Nomenclature.

Proposed Methodology

The proposed approach for solar irradiance nowcasting is illustrated in Figure 1. Given the multimodal nature of this approach, it receives two input modalities, a temporal image modality of a stream of sky images, a temporal numerical modality that consists of a time-series of past solar irradiance readings and cloud cover readings. Streams from both modalities are input to a data preprocessing module, which conducts missing data handling and redundancy reduction for images and temporal sequence alignment, normalization, and range correction for the solar irradiance readings. The preprocessed sky image stream is funneled through an irradiance augmentation module, which further refines the irradiance information using cloud detection, Sun localization and mask generation techniques. The irradiance augmentation module receives a further time-series of solar angle readings. This augmented sky image stream, the preprocessed irradiance readings and cloud cover data converge in the final phase for irradiance nowcasting. The irradiance nowcasting module is composed of two LSTM Networks, irradiance readings and cloud cover data LSTM (irLSTM) and sky image LSTM (siLSTM). The irLSTM was designed with four 2-dimensional layers of 128, 128, 64, 64 nodes and three fully connected layers of 64, 32, 16 nodes. The siLSTM was designed with five 2-dimensional Conv-LSTM layers of 256, 256, 128, 64, 32 nodes that generates a flattened 1-dimensional output which is then integrated with a repeat vector of 32 nodes, followed by a time distributed dense layer of 32 nodes that is again flattened prior to final fully connected layers of 64, 32 and 16 nodes. This deep learning architecture was formulated using a grid optimization process, which is considered as the state of the art in global optimization (Bergstra et al., 2012). Each module is further elaborated in the following subsections.

FIGURE 1

FIGURE 1. The proposed approach for solar irradiance nowcasting.

Data Preprocessing Module

Scaling and normalization are the initial preprocessing steps conducted to ensure that variations in magnitude, range and units of measurement do not have a negative impact on the model architecture or the learning process of the proposed multimodal approach based on deep learning in LSTM networks. For example, input features of high magnitude are allocated with large weights resulting in unstable models during the learning and testing phases. Temporal sequence alignment, normalization and range correction techniques were used to address these variations in numerical time-series data streams of solar irradiance and cloud cover readings. In the nowcasting prediction module, the final output was reverted to the original scale and magnitude for evaluation and application.

The sky image data stream was initially checked for missing data due to maintenance and technical faults. The missing days are removed entirely to maintain the accuracy of the model. Then, the raw image data are saved in JPEG file format with the size of 288 × 352. A raw image consists of the sky region and other segments of the surroundings or the machinery involved. In redundancy reduction, all images were cropped to 270 × 270 (as shown in Figure 2) to focus on the area of interest, which is the sky region.

FIGURE 2

FIGURE 2. Redundancy reduction of the sky image for irradiance (A) original and (B) after reduction.

A correlation analysis was performed using the Pearson correlation coefficient to determine features of importance from the time series of irradiance readings. Hour of the day, total cloud cover, and opaque cloud cover were identified as the most correlated to the actual solar irradiance reading. Figure 3A illustrates this correlation, here it can be observed that the presence of clouds leads to a decrease in the level of irradiance.

FIGURE 3

FIGURE 3. (A) Solar irradiance behavior with cloud cover (B) Mean solar irradiance for each month.

Figure 3B depicts solar irradiance in the Northern hemisphere across the 12 months of a calendar year. During winter months, irradiance is low and during summer, irradiance reaches the peak. This seasonal fluctuation impacts the predictive accuracy of an irradiance nowcasting approach. To overcome this, the dataset was divided into four seasons and trained a separate model for each season.

Irradiance Augmentation Module

Following the preprocessing phase, the irradiance augmentation phase further refines the irradiance information using cloud detection, Sun localization and mask generation techniques. Each technique is described below.

Cloud Detection

Red Blue Ratio (RBR) method was utilized for cloud detection (Li et al., 2019). First, the images were separated into the red, blue, and green channels. Then, the pixel-wise ratio of the red and the blue channels were calculated which reduces the 3-channel image to a 2-D array with ratio values. After this, commonly a single threshold is used for cloud detection. However, in this approach three thresholds were used where each threshold detects clouds with different intensities, to increase the cloud details that can be extracted from the image. As shown in Figure 3, when a lower threshold is used, all the clouds with lower intensity are detected and when a higher threshold is used, only the clouds with higher intensity are detected. In the final image, all the outputs which were acquired using different thresholds were concatenated giving each layer a value depending on the threshold where the layer acquired from the highest threshold gets 255. Figure 4 illustrates the input and the outputs of each stage of the cloud detection technique.

FIGURE 4

FIGURE 4. Cloud detection using the Red Blue Ratio method.

Sun Localization and Mask Generation

A new algorithmic method is proposed for Sun localization using the zenith angle, the azimuth angle and their corresponding astronomical details. Here the zenith angle is considered as the angle between the direction of the Sun and the zenith (directly above). Also, the azimuth angle is defined as the angle between the horizontal direction of the Sun and the reference (usually north but sometimes taken as south). Figure 5 illustrates the zenith angle and the azimuth angle.

FIGURE 5

FIGURE 5. Zenith angle and Azimuth angle.

The coordinates of the Sun on the image (x, y) are calculated using Eq1, 2 and 3,

$\begin{array}{c}l=\mathit{S}\ast \sin \left(\alpha \right)\end{array}(1)$

$\begin{array}{c}\mathit{y}=\mathit{R}+\mathit{l}\ast \cos \left(\beta -\raisebox{1ex}{$\pi $}\!\left/ \!\raisebox{-1ex}{$2$}\right.\right)\end{array}(2)$

$\begin{array}{c}\mathit{x}=\mathit{R}+\mathit{l}\ast \sin \left(\beta - \raisebox{1ex}{$\pi $}\!\left/ \!\raisebox{-1ex}{$2$}\right.\right)\end{array}(3)$

Here, the R is the radius of the hemisphere that the image has covered. Next, the Sun localization is used for solar mask generation, which is used to make the region closer to the Sun more effective since the clouds close to the Sun has the most impact when it comes to nowcasting. The solar mask is a gradient from 1 to 0, with 1 being directly where the Sun is located and then gradually decreasing to zero across the image. Finally, element-wise multiplication is used to apply the mask to the processed image. Figure 6 illustrates the estimation of the solar location (B) on samples representing different times of the day and the year (A) and the corresponding solar mask (C). Figure 7 illustrates the process in terms of outputs at each stage, and how the solar mask is merged with the preprocessed image.

FIGURE 6

FIGURE 6. Solar location estimation and solar mask generation.

FIGURE 7

FIGURE 7. Pictorial representation of the process for merging the sun mask with preprocessed image output.

Irradiance Nowcasting Module

As described earlier, the irradiance nowcasting module is based on LSTM networks. LSTM is a type of deep Recurrent Neural Network (RNN). RNNs are designed for sequential data with internal memory states and feedback connections. An LSTM unit is illustrated in Figure 8. It mainly consists of four parts, namely, cell, input gate, output gate and forget gate. The cell stores the feedback values while the gates control the information flow to the cell. Each of these cells can be considered as neurons of a standard feedforward neural network where the activation of a weighted sum is calculated. In that sense, if the activations for the input, output and forget gates are considered as i_t, o_t, and f_t, respectively, there are calculated using Eq 4, 5 and 6.

$\begin{array}{c}{\mathit{f}}_t=\sigma \left({\mathit{W}}_{\mathit{f}}{\mathit{x}}_{\mathit{t}}+{\mathit{U}}_{\mathit{f}}{\mathit{h}}_{\mathit{t}-1}+{\mathit{b}}_{\mathit{f}}\right)\end{array}(4)$

$\begin{array}{c}{\mathit{i}}_{\mathit{t}}=\sigma \left({\mathit{W}}_{\mathit{i}}{\mathit{x}}_{\mathit{t}}+{\mathit{U}}_{\mathit{i}}{\mathit{h}}_{\mathit{t}-1}+{\mathit{b}}_{\mathit{i}}\right)\end{array}(5)$

$\begin{array}{c}{\mathit{o}}_{\mathit{t}}=\sigma \left({\mathit{W}}_{\mathit{o}}{\mathit{x}}_{\mathit{t}}+{\mathit{U}}_{\mathit{o}}{\mathit{h}}_{\mathit{t}-1}+{\mathit{b}}_o\right)\end{array}(6)$

FIGURE 8

FIGURE 8. LSTM unit.

W represents the input weights and U represents the recurrent connection weights of the cell or the three gates. The cell output c_t and the upward output h_t is calculated using Eq. 7, 8 and 9,

$\begin{array}{c}{\tilde{c}}_{\mathit{t}}=\mathit{tanh}\left({\mathit{W}}_{\mathit{c}}{\mathit{x}}_{\mathit{t}}+{\mathit{U}}_{\mathit{c}}{\mathit{h}}_{\mathit{t}-1}+{\mathit{b}}_{\mathit{c}}\right)\end{array}(7)$

$\begin{array}{c}{\mathit{c}}_{\mathit{t}}={\mathit{f}}_{\mathit{t}} \otimes {\mathit{c}}_{\mathit{t}-1}+{\mathit{i}}_{\mathit{t}} \otimes {\tilde{c}}_{\mathit{t}}\end{array}(8)$

$\begin{array}{c}{\mathit{h}}_{\mathit{t}}={\mathit{o}}_{\mathit{t}} \otimes \mathit{tanh}\left(c_t\right)\end{array}(9)$

Here, σ represents the sigmoid function and in both Figure 8 and the equations, $\otimes$ denotes the matrix multiplication. Furthermore, h_t goes through an activation function, before it is considered as the output of the unit.

The proposed irradiance nowcasting module is composed of multimodal LSTM networks Hochreiter and Schmidhuber (1997), irradiance readings LSTM (irLSTM) and sky image LSTM (siLSTM). Although LSTM has been used in solar forecasting Zhang et al. (2018), Kong et al. (2020), Zheng et al. (2020), the multimodal LSTM approach and the learning architecture that is proposed in this paper are novel contributions. Conv-LSTM is a variant of the LSTM network which is specifically designed to learn the spatial information of data. Conv-LSTM is designed for multi-dimensional data in which the all-sky solar image dataset can be learned. The matrix multiplications at each gate in the LSTM are replaced by the convolutional operations in the Conv-LSTM to capture the underlying spatial information in data.

The model architecture of the multimodal LSTM networks used for nowcasting is illustrated in Figure 9. For time series data, four LSTM layers with 128, 128, 64, and 64 units were used and three dense layers with 64, 32, and 16 units were integrated at the end. For image data, five 2D Conv-LSTM layers with 256, 256, 128, 64, and 32 units were used. Then the output was flattened, and a repeat vector was integrated before the LSTM layer with 32 units. Following a time distributed dense layer with 32 units the output gets flattened again before the final dense layers with 64, 32 and 16 units. This deep learning architecture was formulated using a grid optimization process (Bergstra et al., 2012). Finally, the output is irradiance nowcasting with a prediction horizon of 10 min.

FIGURE 9

FIGURE 9. Model architecture of the multimodal LSTM networks.

Experiments and Results

The proposed approach was experimented on a solar irradiance case study conducted by the NREL SRRL labs (Stoffel and Andreas, 1981). The solar irradiance data, considered is the global horizontal hemispheric shortwave irradiance, measured by a Kipp and Zonen CMP22 pyranometer with calibration factor traceable to the World Radiometric Reference (WRR). The values are measured in Watt per square meter (Wm-2). Solar angles data are also acquired, namely, zenith angle and azimuth angle. The ground-based All-sky images used are captured using a Yankee Total Sky Imager 880 (TSI-880) which stores image data every 10 min. It also analyzes and calculates the cloud cover percentage data of both total and opaque clouds, which are also used in the research. Although the time-series data are available in 1 min, 10 min interval was selected since the image data are only available for 10 min. Four separate models were designed and trained for the four seasons. During the training, a call-back function was used to prevent overfitting by stopping the training if no improvement of the validation loss was observed for five continuous epochs. For the LSTM layers, the sequence size of six steps (1 h) was used. Hence, the rolling window method was used for training as well as the evaluations. The hardware specifications of the experimental setup are composed of eighth Gen. Intel i7 processor, 16 GB random access memory and NVIDIA GeForce GTX 1050 Ti graphics card.

Three evaluation metrics were used to validate all models. They are, Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and normalized RMSE (nRMSE). MAE measures an average model accuracy giving equal weights to the errors that are calculated at each observation. Considering Ai and Fi as the actual and the forecasted values respectively, for n observations, Eq 4 calculates the MAE.

$\begin{array}{c}\mathit{MAE}= \frac{{{\displaystyle \sum }}_{\mathit{i}=1}^{\mathit{n}}\left|{\mathit{A}}_{\mathit{i}}-{\mathit{F}}_{\mathit{i}}\right|}{\mathit{n}}\end{array}(10)$

The RMSE takes the residuals between the forecasted values and the actual value to compare prediction errors, giving higher weights to large errors. Considering A_i and F_i as the actual and the forecasted values respectively, for n observations, Eq 5 calculates the RMSE.

$\begin{array}{c}\mathit{RMSE}= \sqrt{{\displaystyle \sum }_{\mathit{i}=1}^{\mathit{n}}\frac{{\left({\mathit{A}}_{\mathit{i}}-{\mathit{F}}_{\mathit{i}}\right)}^2}{\mathit{n}}}\end{array}(11)$

The nRMSE represents the RMSE to the actual range of the predicted variable. It is used to facilitate the comparison between models developed for data with different scales. Also, it can be interpreted as a fraction of the overall range that is typically resolved by the model. Considering the $\overline{X}$ bar as the mean of the actual variable, nRMSE is calculated using Eq. 6.

$\begin{array}{c}\mathit{nRMSE}=\frac{\mathit{RMSE}}{\overline{X}}\end{array}$

Data from 3 years were used for the evaluation, January 2018 to December 2020, following 2-fold cross validation procedure. To evaluate and demonstrate the advantages of using both time series and image data, the irLSTM and siLSTM models were tested separately. The architectures are illustrated in Figure 9 and additionally for each model a dense layer with 1 unit was integrated following the dense layer with 16 units. Also, a univariate LSTM model with the same architecture as of irLSTM was tested as the benchmark model. Table 2 shows the experimental results of the models, separately for “BM,” “irLSTM” and “siLSTM,” as well as the “multimodal” model. Here “BM” represents the benchmark model.

TABLE 2

TABLE 2. Comparison of nowcasting results for benchmark (BM), irLSTM, siLSTM and multimodal models.

Based on Table 1, the proposed multimodal model surpasses the benchmark model and the irLSTM model also outperforms the benchmark in terms of the error metrics demonstrating the positive effect of the added time series features. Then, it can be observed that a considerable performance gain could be achieved by using both irLSTM and siLSTM models as multimodal model, with some fluctuations only during the spring season. Especially during the winter and autumn, 14.28 and 10.8% improvements in nRMSE were recorded. The multimodal model for the spring and the summer produced nRMSE of 28.74 and 32.16%, respectively. The model performed better during the winter and the autumn scoring nRMSE values of 21.38 and 21.05%, respectively.

Table 3 presents the model performance of the multimodal model with image data before and after the Sun masking. For each season, the performance of the model was improved with the Sun masking. Especially during spring and summer, the nRMSE of the models were improved by 8.41 and 6.32%, respectively.

TABLE 3

TABLE 3. Comparison of nowcasting results of the multimodal model with before and after the Sun masking.

Figure 10 illustrates the actual solar irradiance vs the prediction of the benchmark and the multimodal model, for 4 days during winter. As depicted by the “total cloud cover,” Day 1 is all-day cloudy with a percentage close to 100% and Day 4 is sunny throughout the day with percentages close to zero. During Day 1, the predictions from the multimodal model manage to get closer to the actual value. During day 4, multimodal model predicts closer to the actual value but a bit lower during the peak. Then, during both day 2 and day 3, cloud cover is recorded a lower value and increases during the latter part of the day, where the irradiance reading is significantly affected. It is observed that model produces lower values at the peaks during this period, while maintaining the prediction closer to the actual value.

FIGURE 10

FIGURE 10. Actual irradiance, irradiance nowcasting, cloud cover and uncertainty measurement across 4 days during winter.

Also, the uncertainty of the prediction of the multimodal model is also explored. The method proposed in Gal and Ghahramani (2015) which utilizes dropout for representing the model uncertainty was used. The measurement of uncertainty was defined as the variance of the values that the model predicts with dropout for 50 iterations. A dropout ratio of 0.16 was selected in a way that the percentages of the actual values in each confidence interval are closer to the confidence interval itself. This was carried out to make sure that the model confidence interval is neither overestimated nor underestimated. Nonetheless, the ‘variance’ in Figure 10 shows that the proposed model has a lower uncertainty during cloudy conditions than during clear sky conditions.

Conclusion

In this paper, the challenge of solar irradiance nowcasting in virtual power plants is addressed using a novel approach that leverages the multimodality of irradiance data streams. This approach receives a temporal image modality of a stream of sky images, a temporal numerical modality of a time-series of past solar irradiance readings and cloud cover readings as inputs into an AI pipeline composed of preprocessing and irradiance augmentation modules followed by a multimodal LSTM network architecture that nowcasts solar irradiance within a prediction horizon of 10 minutes. These short burst predictions inform the upstream decision-making of real-time pricing and frequency reserves optimization in VPP. The complete approach was empirically evaluated across the four seasons of the northern hemisphere using a real-world solar irradiance case study. The results confirm that our approach is more effective than single modalities and outperforms current benchmark models. As future work, we intend to formulate and integrate further modalities such as weather, climate and mobility that improve the predictive accuracy of solar irradiance nowcasting. Also, automated machine learning methods offer a promising approach for optimizing the deep neural network architecture, and their application is a topic for further work. One relevant starting point for such work is the extensive discussion of Neural Architecture Search (NAS) methods in the survey (He et al., 2021).

Data Availability Statement

Publicly available datasets were analyzed in this study. This data can be found here: https://www.nrel.gov/esif/solar-radiation-research-laboratory.html.

Author Contributions

DH, DS, SS, DA, AJ and VV contributed to conception, design and development of the study. DH and RN completed the implementation and conducted experiments. DH, DS, SS and DA wrote the first draft of the manuscript. RN, AJ and VV wrote sections of the manuscript. All authors contributed to manuscript revision, read, and approved the submitted version.

Funding

This research was partially supported by Business Finland grant 7439/31/2018.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Al-Dahidi, S., Ayadi, O., Adeeb, J., and Louzazni, M. (2019). Assessment of Artificial Neural Networks Learning Algorithms and Training Datasets for Solar Photovoltaic Power Production Prediction. Front. Energ. Res. 7, 130. doi:10.3389/fenrg.2019.00130

CrossRef Full Text | Google Scholar

Angenendt, G., Zurmühlen, S., Figgener, J., Kairies, K.-P., and Sauer, D. U. (2020). Providing Frequency Control reserve with Photovoltaic Battery Energy Storage Systems and Power-To-Heat Coupling. Energy 194, 116923. doi:10.1016/J.ENERGY.2020.116923

CrossRef Full Text | Google Scholar

Bagheri-Sanjareh, M., and Nazari, M. H. (2020). Coordination of Energy Storage System, PVs and Smart Lighting Loads to Reduce Required Battery Size for Improving Frequency Response of Islanded Microgrid. Sustainable Energ. Grids Networks 22, 100357. doi:10.1016/J.SEGAN.2020.100357

CrossRef Full Text | Google Scholar

Baltrusaitis, T., Ahuja, C., and Morency, L.-P. (2019). Multimodal Machine Learning: A Survey and Taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41, 423–443. doi:10.1109/TPAMI.2018.2798607

PubMed Abstract | CrossRef Full Text | Google Scholar

Bergstra, J., Ca, J. B., and Ca, Y. B. (2012). Random Search for Hyper-Parameter Optimization Yoshua Bengio. Available at: http://scikit-learn.sourceforge.net (Accessed June 2, 2021).

Google Scholar

Caldas, M., and Alonso-Suárez, R. (2019). Very Short-Term Solar Irradiance Forecast Using All-Sky Imaging and Real-Time Irradiance Measurements. Renew. Energ. 143, 1643–1658. doi:10.1016/j.renene.2019.05.069

CrossRef Full Text | Google Scholar

Chen, X., Du, Y., Lim, E., Wen, H., and Jiang, L. (2019). Sensor Network Based PV Power Nowcasting with Spatio-Temporal Preselection for Grid-Friendly Control. Appl. Energ. 255, 113760. doi:10.1016/j.apenergy.2019.113760

CrossRef Full Text | Google Scholar

Conte, F., Crosa di Vergagni, M., Massucco, S., Silvestro, F., Ciapessoni, E., and Cirio, D. (2021). Performance Analysis of Frequency Regulation Services provided by Aggregates of Domestic Thermostatically Controlled Loads. Int. J. Electr. Power Energ. Syst. 131, 107050. doi:10.1016/j.ijepes.2021.107050

CrossRef Full Text | Google Scholar

Das, R., Wang, Y., Busawon, K., Putrus, G., and Neaimeh, M. (2021). Real-time Multi-Objective Optimisation for Electric Vehicle Charging Management. J. Clean. Prod. 292, 126066. doi:10.1016/j.jclepro.2021.126066

CrossRef Full Text | Google Scholar

Elkazaz, M., Sumner, M., Naghiyev, E., Pholboon, S., Davies, R., and Thomas, D. (2020). A Hierarchical Two-Stage Energy Management for a home Microgrid Using Model Predictive and Real-Time Controllers. Appl. Energ. 269, 115118. doi:10.1016/j.apenergy.2020.115118

CrossRef Full Text | Google Scholar

Fan, S., Liu, J., Wu, Q., Cui, M., Zhou, H., and He, G. (2020). Optimal Coordination of Virtual Power Plant with Photovoltaics and Electric Vehicles: A Temporally Coupled Distributed Online Algorithm. Appl. Energ. 277, 115583. doi:10.1016/j.apenergy.2020.115583

CrossRef Full Text | Google Scholar

Feng, C., and Zhang, J. (2020). SolarNet: A Sky Image-Based Deep Convolutional Neural Network for Intra-hour Solar Forecasting. Solar Energy 204, 71–78. doi:10.1016/j.solener.2020.03.083

CrossRef Full Text | Google Scholar

Gal, Y., and Ghahramani, Z. (2015). Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In 33rd Int. Conf. Mach. Learn. ICML 2016 3, 1651–1660, New York, United States, Available at: https://arxiv.org/abs/1506.02142v6 (Accessed July 16, 2021).

Google Scholar

He, X., Zhao, K., and Chu, X. (2021). AutoML: A Survey of the State-Of-The-Art. Knowledge-Based Syst. 212, 106622. doi:10.1016/J.KNOSYS.2020.106622

CrossRef Full Text | Google Scholar

Hochreiter, S., and Schmidhuber, J. (1997). Long Short-Term Memory. Neural Comput. 9, 1735–1780. doi:10.1162/neco.1997.9.8.1735

PubMed Abstract | CrossRef Full Text | Google Scholar

Hu, S., Xiang, Y., Liu, J., Li, J., and Liu, C. (2021). A Two-Stage Dispatching Method for Wind-Hydropower-Pumped Storage Integrated Power Systems. Front. Energ. Res. 9, 65. doi:10.3389/fenrg.2021.646975

CrossRef Full Text | Google Scholar

Javadi, M., Amraee, T., and Capitanescu, F. (2019). Look Ahead Dynamic Security-Constrained Economic Dispatch Considering Frequency Stability and Smart Loads. Int. J. Electr. Power Energ. Syst. 108, 240–251. doi:10.1016/j.ijepes.2019.01.013

CrossRef Full Text | Google Scholar

Karbouj, H., Rather, Z. H., Flynn, D., and Qazi, H. W. (2019). Non-synchronous Fast Frequency Reserves in Renewable Energy Integrated Power Systems: A Critical Review. Int. J. Electr. Power Energ. Syst. 106, 488–501. doi:10.1016/j.ijepes.2018.09.046

CrossRef Full Text | Google Scholar

Koller, M., Borsche, T., Ulbig, A., and Andersson, G. (2015). Review of Grid Applications with the Zurich 1MW Battery Energy Storage System. Electric Power Syst. Res. 120, 128–135. doi:10.1016/j.epsr.2014.06.023

CrossRef Full Text | Google Scholar

Kolodziejczyk, W., Zoltowska, I., and Cichosz, P. (2021). Real-time Energy purchase Optimization for a Storage-Integrated Photovoltaic System by Deep Reinforcement Learning. Control. Eng. Pract. 106, 104598. doi:10.1016/j.conengprac.2020.104598

CrossRef Full Text | Google Scholar

Kong, J., Kim, S. T., Kang, B. O., and Jung, J. (2019). Determining the Size of Energy Storage System to Maximize the Economic Profit for Photovoltaic and Wind Turbine Generators in South Korea. Renew. Sustain. Energ. Rev. 116, 109467. doi:10.1016/j.rser.2019.109467

CrossRef Full Text | Google Scholar

Kong, W., Jia, Y., Dong, Z. Y., Meng, K., and Chai, S. (2020). Hybrid Approaches Based on Deep Whole-Sky-Image Learning to Photovoltaic Generation Forecasting. Appl. Energ. 280, 115875. doi:10.1016/j.apenergy.2020.115875

CrossRef Full Text | Google Scholar

Kosmadakis, I. E., Elmasides, C., Koulinas, G., and Tsagarakis, K. P. (2021). Energy Unit Cost Assessment of Six Photovoltaic-Battery Configurations. Renew. Energ. 173, 24–41. doi:10.1016/j.renene.2021.03.010

CrossRef Full Text | Google Scholar

Larsen, E. M., Pinson, P., Leimgruber, F., and Judex, F. (2017). Demand Response Evaluation and Forecasting - Methods and Results from the EcoGrid EU experiment. Sustain. Energ. Grids Networks 10, 75–83. doi:10.1016/j.segan.2017.03.001

CrossRef Full Text | Google Scholar

Li, H., Li, H., Lu, W., Wang, Z., and Bian, J. (2020). Optimal Power Flow Calculation Considering Large-Scale Photovoltaic Generation Correlation. Front. Energ. Res. 8, 338. doi:10.3389/fenrg.2020.590418

CrossRef Full Text | Google Scholar

Li, X., Lu, Z., Zhou, Q., and Xu, Z. (2019). A Cloud Detection Algorithm with Reduction of Sunlight Interference in Ground-Based Sky Images. Atmosphere 10, 640. doi:10.3390/atmos10110640

CrossRef Full Text | Google Scholar

Liu, C., Yang, R. J., Yu, X., Sun, C., Wong, P. S. P., and Zhao, H. (2021). Virtual Power Plants for a Sustainable Urban Future. Sustain. Cities Soc. 65, 102640. doi:10.1016/j.scs.2020.102640

CrossRef Full Text | Google Scholar

Liu, J., Chen, X., Cao, S., and Yang, H. (2019). Overview on Hybrid Solar Photovoltaic-Electrical Energy Storage Technologies for Power Supply to Buildings. Energ. Convers. Manage. 187, 103–121. doi:10.1016/j.enconman.2019.02.080

CrossRef Full Text | Google Scholar

Mahmud, K., Khan, B., Ravishankar, J., Ahmadi, A., and Siano, P. (2020). An Internet of Energy Framework with Distributed Energy Resources, Prosumers and Small-Scale Virtual Power Plants: An Overview. Renew. Sustain. Energ. Rev. 127, 109840. doi:10.1016/j.rser.2020.109840

CrossRef Full Text | Google Scholar

Mammoli, A., Terren-Serrano, G., Menicucci, A., Caudell, T. P., and Martínez-Ramón, M. (2019). An Experimental Method to Merge Far-Field Images from Multiple Longwave Infrared Sensors for Short-Term Solar Forecasting. Solar Energy 187, 254–260. doi:10.1016/J.SOLENER.2019.05.052

CrossRef Full Text | Google Scholar

Mashlakov, A., Pournaras, E., Nardelli, P. H. J., and Honkapuro, S. (2021). Decentralized Cooperative Scheduling of Prosumer Flexibility under Forecast Uncertainties. Appl. Energ. 290, 116706. doi:10.1016/j.apenergy.2021.116706

CrossRef Full Text | Google Scholar

Muñoz-Rodríguez, F. J., Jiménez-Castillo, G., de la Casa Hernández, J., and Aguilar Peña, J. D. (2021). A New Tool to Analysing Photovoltaic Self-Consumption Systems with Batteries. Renew. Energ. 168, 1327–1343. doi:10.1016/j.renene.2020.12.060

CrossRef Full Text | Google Scholar

Nefedov, E., Sierla, S., and Vyatkin, V. (2018). Internet of Energy Approach for Sustainable Use of Electric Vehicles as Energy Storage of Prosumer Buildings. Energies 11, 2165. doi:10.3390/en11082165

CrossRef Full Text | Google Scholar

Nespoli, A., and Niccolai, A. (2020). Solar Position Identification on Sky Images for Photovoltaic Nowcasting Applications. In Proc. - 2020 IEEE Int. Conf. Environ. Electr. Eng. 2020 IEEE Ind. Commer. Power Syst. Eur. EEEIC/I CPS Eur. 2020, June 2020, Spain. doi:10.1109/EEEIC/ICPSEUROPE49358.2020.9160490

CrossRef Full Text | Google Scholar

Nge, C. L., Ranaweera, I. U., Midtgård, O.-M., and Norum, L. (2019). A Real-Time Energy Management System for Smart Grid Integrated Photovoltaic Generation with Battery Storage. Renew. Energ. 130, 774–785. doi:10.1016/j.renene.2018.06.073

CrossRef Full Text | Google Scholar

Qazi, H. S., Zhao, T., Liu, N., Wang, T., and Ullah, Z. (2021). Optimal Operation of Isolated Micro-grids-cluster via Coalitional Energy Scheduling and Reserve Sharing. Front. Energ. Res. 9, 25. doi:10.3389/fenrg.2021.629131

CrossRef Full Text | Google Scholar

Qiu, J., Meng, K., Zheng, Y., and Dong, Z. Y. (2017). Optimal Scheduling of Distributed Energy Resources as a Virtual Power Plant in a Transactive Energy Framework. IET Generation, Transm. Distribution 11, 3417–3427. doi:10.1049/iet-gtd.2017.0268

CrossRef Full Text | Google Scholar

Seyyedeh Barhagh, S., Abapour, M., and Mohammadi-Ivatloo, B. (2020). Optimal Scheduling of Electric Vehicles and Photovoltaic Systems in Residential Complexes under Real-Time Pricing Mechanism. J. Clean. Prod. 246, 119041. doi:10.1016/j.jclepro.2019.119041

CrossRef Full Text | Google Scholar

Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W., and Woo, W. (2015). Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting, Adv. Neural Inf. Process. Syst., 802–810. Available at: http://arxiv.org/abs/1506.04214 (Accessed January 14, 2021).

Google Scholar

Sinsel, S. R., Riemke, R. L., and Hoffmann, V. H. (2020). Challenges and Solution Technologies for the Integration of Variable Renewable Energy Sources-A Review. Renew. Energ. 145, 2271–2285. doi:10.1016/j.renene.2019.06.147

CrossRef Full Text | Google Scholar

Stoffel, T., and Andreas, A. (1981). NREL Solar Radiation Research Laboratory (SRRL): Baseline Measurement System (BMS). Golden: Colorado (Data). doi:10.7799/1052221

CrossRef Full Text

Stüber, M., Scherhag, F., Deru, M., Ndiaye, A., Sakha, M. M., Brandherm, B., et al. (2021). Forecast Quality of Physics-Based and Data-Driven PV Performance Models for a Small-Scale PV System. Front. Energ. Res. 9, 639346. doi:10.3389/fenrg.2021.639346

CrossRef Full Text | Google Scholar

Su, L., Qin, X., Zhang, S., Zhang, Y., Jiang, Y., and Han, Y. (2020). Fast Frequency Response of Inverter-Based Resources and its Impact on System Frequency Characteristics. Glob. Energ. Interconnection 3, 475–485. doi:10.1016/j.gloei.2020.11.007

CrossRef Full Text | Google Scholar

Subramanya, R., Yli-Ojanperä, M., Sierla, S., Hölttä, T., Valtakari, J., and Vyatkin, V. (2021). A Virtual Power Plant Solution for Aggregating Photovoltaic Systems and Other Distributed Energy Resources for Northern European Primary Frequency Reserves. Energies 14, 1242. doi:10.3390/en14051242

CrossRef Full Text | Google Scholar

Terrén-Serrano, G., and Martínez-Ramón, M. (2021). Processing of Global Solar Irradiance and Ground-Based Infrared Sky Images for Very Short-Term Solar Forecasting. Available at: https://arxiv.org/abs/2101.08694v3 (Accessed July 16, 2021).

Google Scholar

Thavlov, A., and Bindner, H. W. (2015). Utilization of Flexible Demand in a Virtual Power Plant Set-Up. IEEE Trans. Smart Grid 6, 640–647. doi:10.1109/TSG.2014.2363498

CrossRef Full Text | Google Scholar

Wu, Y., Zhang, J., Ravey, A., Chrenko, D., and Miraoui, A. (2020). Real-time Energy Management of Photovoltaic-Assisted Electric Vehicle Charging Station by Markov Decision Process. J. Power Sourc. 476, 228504. doi:10.1016/j.jpowsour.2020.228504

CrossRef Full Text | Google Scholar

Xing, Q., Cheng, M., Liu, S., Xiang, Q., Xie, H., and Chen, T. (2021). Multi-Objective Optimization and Dispatch of Distributed Energy Resources for Renewable Power Utilization Considering Time-Of-Use Tariff. Front. Energ. Res. 9, 68. doi:10.3389/fenrg.2021.647199

CrossRef Full Text | Google Scholar

Yu, S., Fang, F., Liu, Y., and Liu, J. (2019). Uncertainties of Virtual Power Plant: Problems and Countermeasures. Appl. Energ. 239, 454–470. doi:10.1016/j.apenergy.2019.01.224

CrossRef Full Text | Google Scholar

Zhang, J., Verschae, R., Nobuhara, S., and Lalonde, J.-F. (2018). Deep Photovoltaic Nowcasting. Solar Energy 176, 267–276. doi:10.1016/j.solener.2018.10.024

CrossRef Full Text | Google Scholar

Zhang, K., Troitzsch, S., Zhang, S.-Y., Teh, E. S. P., Subramanian, L., and Massier, T. (2021). Transactive Energy in an Urban Environment: A Case Study of Local Generation and Flexibility Potentials in a Singaporean Urban District. Front. Energ. Res. 9, 22. doi:10.3389/fenrg.2021.632992

CrossRef Full Text | Google Scholar

Zhang, Z., Du, E., Zhu, G., Zhang, N., Kang, C., Qian, M., et al. (2020). Modeling Frequency Response Dynamics in Power System Scheduling. Electric Power Syst. Res. 189, 106549. doi:10.1016/j.epsr.2020.106549

CrossRef Full Text | Google Scholar

Zheng, J., Zhang, H., Dai, Y., Wang, B., Zheng, T., Liao, Q., et al. (2020). Time Series Prediction for Output of Multi-Region Solar Power Plants. Appl. Energ. 257, 114001. doi:10.1016/j.apenergy.2019.114001

CrossRef Full Text | Google Scholar