TempNet – temporal super-resolution of radar rainfall products with residual CNNs

From disaster preparedness to response and recovery needs, the availability and quality of environmental datasets have gained significant importance in recent years with the increased impact of natural disasters. Rainfall datasets have been an important component in many modeling applications such as flood forecasting (Seo et al. 2018, 2021; Sit et al. 2021a; Xiang & Demir 2022a), water quality modeling (Jha et al. 2007; Demir et al. 2009), wastewater management (Cahoon & Hanke 2017), along with other data products based on sensor networks and instrumentation (Muste et al. 2017). Since spatial and temporal distributions of rain exert importance in such modeling efforts, the quality and availability of precipitation maps hold the utmost importance in advancing research on disaster mitigation (Teague et al. 2021; Alabbad et al. 2022), risk assessment (Yildirim & Demir 2021), and decision support (Ewing & Demir 2021).

Rainfall products from quantitative precipitation estimation (QPE) systems are three-dimensional, including the time component. The first two dimensions reflect spatial coordinates on earth, including latitude and longitude. The third dimension here is temporal resolution. Weather radars have been a primary instrument for QPE and allow us to capture space-time features of rainfall, which is required for environmental (i.e., hydrologic) predictions at relevant space-time scales. Due to its sensing nature, the accuracy of QPE broadly depends on many factors (Villarini & Krajewski 2010). Since a composite of multiple radars is often used to produce large-scale rain maps, computational capabilities and methodology play an essential role in the process of combining multiple radar data. Consequently, it should be noted that synchronization of different observation times among multiple radars (interpolation or extrapolation over time) is a major challenge to generate a composite rain map. Another QPE issue regarding space-time resolutions of radar product is the misrepresentation of rainfall accumulation arising from radar's intermittent sampling strategy (e.g., Fabry et al. 1994; Liu & Krajewski 1996; Seo & Krajewski 2015).

Mitigating the uncertainty caused by radars in rainfall products has been an essential task in radar hydrology (Krajewski & Smith 2002). Most efforts in radar hydrology for better rainfall datasets rely on understanding uncertainty factors and developing their technical solution in the processing algorithms. Regarding the technology used, efforts usually focus on getting more accurate precipitation data for a better sense of weather in terms of space or time. Once the data have been acquired, however, the effort of generating better datasets becomes a different task.

The temporal resolution of rainfall products is one of the key elements that determine the prediction accuracy of modeling efforts (e.g., Atencia et al. 2011). Interpolating 2D rain maps is not a unique problem for rainfall estimation. The same approach has been a topic of interest in the field of computer vision and, with advancements in computational capacities, in the deep learning applications field, precisely for video frame interpolation, or in other words, video temporal interpolation. Recording every video at high frame rates is impractical or expensive for many settings. Thus, developing models or methods to increase video frame rates has been extensively studied in recent years. In computer vision, temporal interpolation is done with both conventional statistical methods and data-driven machine-learning models. Even though proposed solutions could be realized on 1-channel versions of 2D images that form videos, performing temporal interpolations over 3-channel images is more explored, making the task at hand a problem of estimating the tensor between two 3D tensors. Regardless, the methodology that could be employed and challenges remain vastly similar among video temporal interpolation and temporal interpolation of 2D rain maps.

With inspiration from video frame interpolation studies, this paper proposes a convolutional neural networks (CNNs) based deep learning architecture to account for radar's intermittent sampling and thus improve its product temporal resolution. The proposed CNN model, temporal-resolution network, which will be referred to as TempNet throughout this paper, is compared to three baseline methods: the nearest frame, optical flow, and a relatively simple but deep CNN that will be referred to as CNN-baseline. As with baseline methods to compare, the proposed CNN, TempNet produces an intermediate 2D rain map between two temporally consecutive rain maps. In other words, the methodology presented here aims to perform temporal interpolation of 2D rain maps.

The rest of this paper is structured as follows: in section 2, a brief review of the literature is provided. Subsequently, in section 3, the overview of the methodology is presented, starting with the description of the dataset used, followed by the proposed deep neural networks and baseline methods to compare them. In section 4, the results from the nearest frame, optical flow, CNN-baseline, and the TempNet are presented and discussed. Finally, in section 5, the conclusions are summarized with final remarks regarding the study.

Research studies for better rainfall products can be broadly classified as either quality- or resolution-related. Dataset improvements using deep neural networks include data cleaning to eliminate noise (Lepetit et al. 2021), increasing the resolution or accuracy of datasets by various statistical or data-driven methods (Li et al. 2019; Demiray et al. 2021a, 2021b), synthetic data generation (Gautam et al. 2020), and bias correction (Hu et al. 2021). Resolution-related improvements, on the other hand, either focus on increasing the resolution of two spatial dimensions or the temporal dimension.

Unlike deep learning literature, in meteorology and climatology, whether it is temporal or spatial, super-resolution is referred to as downscaling. Downscaling of precipitation products has been a topic of interest for domain scientists. For instance, Hou et al. (2017) built an ensemble of a Markov chain and a Support Vector Machine to increase the spatial resolution of daily precipitation. Likewise, Vandal et al. (2019) explored five statistical downscaling methods for global climate models (GCMs) including machine learning approaches such as autoencoders and support vector machines. Beyond traditional machine learning models, ANNs have been employed for spatial downscaling as well. Alizamir et al. (2017) integrated particle swarm optimization (PSO) into ANNs for arid regions. Similarly, Salimi et al. (2019) utilized PSO, ANNs, as well as genetic algorithm for downscaling of precipitation events in Austria.

Utilizing adversarial training, Chaudhuri & Robertson (2020) proposed CliGAN, a Generative Adversarial Network (GAN) that uses an encoder–decoder structure for the generator, trained with a combination of adversarial loss, content loss with the Nash–Sutcliff Model Efficiency (NSE), and structural loss with the multi-scale structural similarity index (MSSIM). The authors showed that CliGAN can downscale an ensemble of large-scale annual global maximum precipitation products to regional precipitation products. CNNs were used by Tu et al. (2021). They developed their method using ERA-Interim datasets (European Centre for Medium-Range Weather Forecasts 2006) and then applied it to super-resolution precipitation in Japan's Kuma River Watershed.

Temporal downscaling in the literature is not as profound as spatial downscaling. Lee & Jeong (2014) employed a stochastic take with the genetic algorithm in order to increase the temporal resolution of daily precipitation into hourly. Downscaling daily rain gauge measured precipitation into sub-daily precipitation data with 6 h of temporal resolution, Ryo et al. (2014) used satellite rainfall products. Seo & Krajewski (2015) increased the temporal resolution of rain maps produced by a QPE system from 5 to 1 min using advection correction and validated the results with measurements of ground rain gauges. We are not aware of any other studies that increase the temporal resolution of 2D rainfall maps.

Aside from super-resolution, or downscaling, on the other hand, there are studies that impute missing values in rainfall sequences. In such a study, Norazizi & Deni (2019) utilized ANNs to complete rainfall time series for the data from the Malaysian Meteorological Department. Similarly, for a spatiotemporal imputation of monthly rainfall data in 45 gauge stations located in southwestern Colombia, Canchala-Nastar et al. (2019) trained an ANN. In a similar manner to this study, Gao et al. (2021) tested two deep ANNs for temporal imputation of 2D rainfall maps. They comparatively employed a 3DCNN and a Convolutional Bidirectional LSTM and showed that the LSTM-based method they proposed outperforms the 3DCNN as well as some baselines, namely, optical flow, linear interpolation, and nearest frame.

Since the approach this paper utilizes is similar to video frame interpolation, computer vision literature provides more insight regarding the problem and plausible ways to address it. Besides the widely established method of calculating optical flow with statistical methods in order to generate videos with better frame rates, there are many studies exploring deep neural networks. Niklaus et al. (2017a, 2017b) presented two incremental methods for converting low-temporal-resolution videos to high-temporal-resolution videos that used adaptive convolutions.

In another study, Liu et al. (2017) combined traditional optical flow calculation methods with deep neural networks in a method called deep voxel flow, to both temporally interpolate and extrapolate videos. Beyond that, Jiang et al. (2018) used a neural network structure (Unet) that was built for image segmentation purposes and trained it to learn optical flow to generate slow-motion videos out of regular low frame-rate videos. In a similar but extensive effort, Xiang et al. (2020) proposed a convolutional LSTM-based model for both spatial and temporal interpolation of video frames. Cheng & Chen (2021), using separable convolutions, proposed an approach, namely enhanced deformable separable convolution, that aimed to provide a way to increase the frame rates of videos by estimating more than a frame between two available frames.

Even though the literature over 2D radar rainfall data is quite comprehensive for forecast studies using deep learning, to the best of our knowledge, deep learning has not been explored for the temporal interpolation of 2D radar rainfall prior to this study. Furthermore, temporal interpolation of 2D radar rainfall is not well studied using relatively conventional statistical models. The only method from the literature that had a similar aim as ours employed advection correction, which will be one of the baselines utilized in this study to validate the approach that will be presented in the next section.

The IowaRain dataset chronologically provides 64, 67, 76, and 81 events for each year between 2016 and 2019. In order to do the dataset separation in a fashion closer to a 70/30 split (%70 training, %30 testing), we decided to use the rainfall events for 2019 as the test set and all the rainfall events before 2019 as the training set, making our set lengths 207 and 81 for the training and test sets, respectively. The final dataset entries sum up to 35,258 and 6,725 2D rainfall maps for the training and testing sets, respectively. It should be noted that aside from the test/train split and augmenting the dataset by reversing frame sequences, there was no form of preprocessing. Since the min–max normalization was already applied on the dataset, we did not make any further modifications to the measurements.

As one of the baselines, we employed the nearest frame as it was previously used in radar rainfall imputation comparisons (Gao et al. 2021). The nearest frame simply refers to assuming the interpolated frame is the same as the temporally closest frame to the forecasted frame. In other words, when interpolating t₅ from t₀ and t₁₅, t₅ is assumed to be the same as t₀. In our case of interpolating t₅ from t₀ and t₁₀, on the other hand, we will use the predecessor frame, t₀, even though both input frames have the same distance to the output frame.

In order to form a baseline method that is comparable to the model we propose, we chose to use an optical flow-based temporal interpolation method. This method depends on the optical flow calculation between two 2D rain maps and creating an intermediate frame between those rain maps that is at the same distance temporally. Optical flow has been widely used in the radar rainfall literature (Gao et al. 2021) as a method to build deep learning forecasts upon (Nie et al. 2021; Yan et al. 2021), as a baseline method to compare in the temporal extrapolation of radar rainfall products (Ayzel et al. 2020; Kumar et al. 2020; Xie et al. 2020; Xu et al. 2021) as well as in temporal imputation of 2D rainfall maps (Gao et al. 2021) since it was first proposed by Bowler et al. (2004).

Optical flow is the summary of the spatial changes of objects between two frames of a scene. In other words, it shows the motion of the objects in order to estimate the mobility of velocity fields between two frames. Consequently, an optical flow calculation algorithm is a way to determine how objects move from one scene to another. What an optical flow calculation algorithm actually does is calculate the change of pixel intensities. In computer vision, pixel intensity is defined as the properties a pixel carries in a scene. For a 2D tensor, each value's pixel intensity is determined by its position in the 2D tensor and its value. The same phenomenon is also valid for the case we tackle in this study since rainfall maps are, in fact, 2D tensors.

There are many optical flow calculation algorithms in the computer vision literature. In this paper, we employed the Gunnar–Farneback optical flow (Farnebäck 2003), or sometimes referred to as dense optical flow. Gunnar–Farneback optical flow calculates pixel intensities for each pixel in the scene, as opposed to feature-level intensity calculations, hence the name ‘dense’, as opposed to sparse. For the rain map case, that would mean calculating the changes for each of the measurements for the 0.5 km × 0.5 km areas that make up the 2D rain map. The mathematical formulation of the algorithm is beyond the scope of this paper; we suggest the reader refer to the paper for further details. Once the optical flow is calculated, the next step in the approach is to transfer every measurement depending on their motion vectors in the calculated optical flow and their location in the first and second frames.

For comparison purposes, the optical flow between input snapshots was calculated for each of the entries in the test dataset. Then, color propagation was done using that calculation in order to estimate the intermediate frame. Once all the estimations were done for the testing dataset, the performance metrics reported in the next section were calculated using estimated and actual snapshots. The entirety of the baseline method was implemented using the Gunnar–Farneback optical flow implementation using the OpenCV (Bradski & Kaehler 2000) and NumPy (Harris et al. 2020) numeric computing libraries.

Convolutional Neural Networks, ConvNets, or simply CNNs work relying on convolutional layers and are primarily used for computer vision tasks. A convolutional layer in a neural network uses weight tensors called filters to explore patterns in inputs. In contrast to weights in a fully connected neural network layer, a filter in a convolutional layer typically works two-dimensionally in order to grasp multi-dimensional (both vertical and horizontal in a 2D black and white image, or in a 2D rain map, similarly) spatial relations over the given input tensor. Building upon this core idea, a CNN with multiple convolutional layers can map complex spatial relations in a given tensor that only one convolutional layer fails to achieve (Goodfellow et al. 2016). For further technical details regarding CNNs, readers could refer to LeCun (1989) and Goodfellow et al. (2016). Considering CNN's applicability and success in multi-dimensional space, where the data comprise spatial representations, and the comprehensive usage of CNNs for 2D rainfall forecasting, CNNs make a sound candidate over soft computing methods for temporal interpolation of radar rainfall, as in radar rainfall datasets, it is crucial to take spatial correlations and patterns into account.

Unlike typical artificial neural networks, CNNs work two-dimensionally over a tensor. Since 2D rainfall maps fall into that description, excluding their temporal dimension, one can assume CNNs are the go-to neural network structures for the interpolation of 2D rainfall maps. Since CNNs are typically used with images containing three color channels, a CNN that processes an input image typically has three filters. The dataset we employed, on the other hand, by its nature, has only one channel, a rain rate (mm/h) map.

To form a CNN-baseline that can be compared to TempNet, we built a four-layer CNN where Rectified Linear Unit (ReLU) was used as the activation function after each layer, namely CNN-baseline. The input frames for the CNN were concatenated into a 3D tensor, as if they were separate channels. In other words, when interpolating t₅, t₀ of shape 1,088 × 1,760 and t₁₀ of shape 1,088 × 1,760 were concatenated to form the input tensor of shape 2 × 1,088 × 1,760. Details regarding the architecture can be found in Table 1.

In order to explore how the temporal resolution of 2D rain maps could be improved, we propose a CNN-based neural network architecture, TempNet. The TempNet provides a fast but effective alternative to the optical flow-based color transfer method and builds upon the CNN architecture we have described in previous subsections.

Both CNN-baseline and the TempNet described here were trained using L1 Loss (also known as mean absolute error – MAE) as the cost function and Adam optimizer (Kingma & Ba 2014) with the help of a Reduce-on-Plateau learning rate scheduler over the training loss. The network was trained on the training dataset described in the Dataset subsection on NVIDIA Titan V GPUs using the PyTorch numeric computing library (Paszke et al. 2019). As for hyperparameters, a batch size of 32, an initial learning rate of 0.001, and a number of epochs of 50 without early stop were used.

We reported six metrics for each of the methods described earlier, namely MAE, root mean square error (RMSE), coefficient of determination (CoD), probability of detection (POD), false alarm ratio (FAR), and critical success index (CSI). MAE reports the mean of the absolute values of the differences between estimated and actual 2D rain maps in the test dataset. RMSE is a similar error metric, but it is calculated by taking the root of the sum of the squared differences between estimated and actual rain maps. On the other hand, CoD is the proportion of dependent variable variation predicted by the independent variable(s), where the best score is 1. We used variance-weighted CoD from the scikit-learn library (Pedregosa et al. 2011) in Python. POD, FAR, and CSI are calculated by (1–3), respectively, using the number of hits (H), false alarms (F), and mises (M). In this case, H, F, and M are calculated in a binary fashion (e.g., threshold is zero) (Table 2); H is defined as the number of correctly estimated rainfall cells, or in other words, the number of elements in the 2D tensor that were correctly estimated as nonzero values. F is defined as the number of wrongly estimated rainfall cells: while the cells were estimated to have rain, the ground truth for the same indices in the 2D map had zero. M is defined as the number of rainfall cells that were estimated as zero, while they have a nonzero value in the ground truth. It should be noted that while the best value for POD and CSI is 1.0 (the greater the better), it is 0.0 for FAR (the lower the better).

Beyond accuracy, one upside of using neural network models is the runtime. Since calculating optical flow and building the new frame out of two sequential frames needs work over individual pixels or measurements, parallelization is challenging and, consequently, time-consuming. Conversely, the same task takes a trivial amount of time on both GPUs and CPUs compared to the baseline method's runtime using TempNet or CNN-baseline once the training is done. As for POD, the optical flow approach seems to be a better methodology but coupled with the FAR, one can easily suggest that optical flow fills radically more cells with values than other approaches. In other words, optical flow focuses on true positives while estimating significantly more false positives than either of the neural network model estimates. Another take here for neural networks is that optical flow being better at POD is most likely due to the fact that the neural networks we presented were trained by optimizing them over MAE. One can argue that optimizing networks' weights by other metrics would make them better than optical flow as well.

Applying a single-frame temporal interpolation method recursively to generate several intermediate frames is an interesting idea, but there are some drawbacks to this approach. First, recursive temporal interpolation cannot be fully parallelized because certain frames cannot be computed until others are generated. Second, only 2i − 1 intermediate frames can be generated (e.g., 3, 5, 7). Third, during recursive interpolation, errors also build up. Subsequently, in order to understand how recursion affects the performance of the approaches presented, we have run a series of tests. Our first test is to understand how the baseline methods and TempNet perform in interpolating t₁₀ from t₀ and t₂₀, namely estimating t′₁₀. Table 4 shows each method's performance on this task using the aforementioned metrics. According to the presented scores, the performances of the models change drastically in recursion, and CNN-baseline and TempNet lose first place to optical flow. Furthermore, the change in performance suggests that any of the presented methodologies do not scale well enough. Nevertheless, considering optical flow calculates the difference between two frames and fills in the middle frame by moving the pixels by half of the calculated motion vectors, and TempNet and CNN-baseline performed similarly to optical flow in terms of MAE while providing better FAR and CSI scores, we can infer that neural networks are promising in interpolating the temporal distances they were not trained for. It also deserves mentioning that optical flow seems to spatially over-estimate rainfall again since FAR is significantly greater than that of other approaches.

Finally, for comparison purposes, we did some tests to understand the visual performance of the CNN-baseline and the TempNet by increasing the temporal resolution of some events in the test dataset by three iterations over the resolutions they were not trained for. In other words, rainfall maps' temporal resolution was increased from 5 to 0.7 min. After visualizing the actual and generated rain maps with the Iowa Flood Information System's rainfall map color scheme, even though we were able to see that the TempNet and the CNN-baseline were not able to carry all the information regarding the rainfall in specific areas, they were still able to represent most of the accumulated rainfall clearly. Alternatively, the optical flow method would visually average two input frames that had lots of shadows scattered over the maps.

In summary, it was demonstrated that neural network-based approaches outperform optical flow-based advection correction in the literature for the metric that they were trained for. This is largely due to the fact that the neural networks presented here are designed to explore nonlinear patterns as well as linear ones, while optical flow calculations work purely on linear connections. The difference between TempNet and CNN-baseline, on the other hand, is thanks to the residual connection, where the learned patterns and the original data are combined for better estimations. In order for TempNet or CNN-baseline to be more competitive over RMSE and CoD, one could easily incorporate them into the cost function for better scores. As for rainfall metrics such as CSI and POD, since they do not comprise formulas that the derivatives could easily be taken from, they cannot be part of the cost function right off the bat.

In this study, we presented a CNN-based temporal-resolution improvement model, TempNet, and compared it to the nearest frame approach, an optical flow-based baseline method, and a CNN-based baseline method, namely CNN-baseline. Results show that TempNet broadly outperforms baseline models according to MAE as well as a combination of other metrics such as FAR, POD, and CSI, while doing that with a significantly better computation time than optical flow. We consider this work as a significant step towards creating better rainfall maps for hydrological modeling needs such as flood forecasting (Sit et al. 2021a; Xiang et al. 2021), climate change modeling (Rolnick et al. 2019; Sit et al. 2020), and missing rainfall map imputation (Gao et al. 2021). Nevertheless, it comes with shortcomings. One shortcoming of TempNet that strikes first is the fact that it does not perform as well as optical flow in terms of some metrics such as RMSE and CoD. Even though, as previously stated, this is due to the cost function selection, a better neural network that performs better across those metrics is theoretically possible. However, since TempNet is a relatively small neural network with fast training and estimation times, this tendency is somewhat expected. Making TempNet more complex at the expense of computation time could be the solution.

This study and the approaches provided here were only tested over a dataset that covered the state of Iowa; thus, one of the shortcomings here is that the generalization performance of the approaches is not known. As a future direction, we aim to develop new iterations of TempNet and CNN-baseline and test them over datasets that cover wider areas, potentially the whole earth. Also, to support operational flood forecasting needs (Xiang & Demir 2022b), we aim to extend the neural networks presented in this study in two different directions: (1) deeper neural networks for better interpolation accuracy; and (2) neural networks with similar performance but doing so with faster computation times by building more efficient neural networks. In order to train better models, various loss functions could be utilized, such as a loss function where the metrics we used in this are combined by differentiable weights.

One challenge with the process was that weather radar-based rainfall maps contain significant artifacts. Although the IowaRain dataset provides rain maps that are vastly cleansed of those artifacts, some remaining noise (e.g., ground and wind turbine clutter; Seo et al. 2015) may affect both accuracy and visual representations. In order to test the dataset better and to develop more accurate models for both imputation and temporal super-resolution of radar rainfall products, datasets with fewer artifacts would become useful. Thus, in order to create improved datasets, as a future perspective, we aim to focus on denoising radar rainfall maps to decrease the number of artifacts in the original dataset with neural networks and train the TempNet over a cleaner dataset. This highlights the need for better benchmark datasets for research communities (Demir et al. 2022). Another future aspect of this problem would be to rectify the issues we mentioned regarding recursive temporal interpolation by introducing a neural network that is capable of producing intermediate frames for any given temporal distance to the input frames, as in Jiang et al. (2018).

All relevant data are included in the paper or its Supplementary Information.

The authors declare there is no conflict.