Predicting time-series for water demand in the big data environment using statistical methods, machine learning and the novel analog methodology dynamic time scan forecasting

The growing demand for water and the imminent risk of shortages are associated with the increase in urban population, per capita consumption and irregular distribution of rainfall. In the last 100 years, the world population has increased threefold, while water consumption has increased sixfold (Cirilo 2015). Considering population growth, between 2009 and 2050, the world population is projected to increase by around 2.3 billion inhabitants, from 6.8 to 9.1 billion, at the same time that the urban population will increase from 3.4 billion in 2009 to 6.3 billion in 2050, increasing the stress on water availability (WWAP 2012).

Despite all the existing urban water infrastructures, many cities are currently facing water stress. In fact, a large proportion of the world population has been affected by water stress (Vörösmarty et al. 2000; Maddocks et al. 2015). Studies indicate that one quarter (25% ± 4%) of the population in large cities, or 381 ± 55 million people, has water supplies that are stressed (McDonald et al. 2014). This scenario represents a major challenge for providing a sufficient amount of water of adequate quality.

To minimize the effects of water stress, water supply system (WSS) management techniques have been used to maintain a balance between water supply and demand. This balance is achieved by applying operational actions, many of which require the application of statistical tools for water demand forecasting.

The importance of water demand forecasting has been justified by the increase in interest among researchers and professionals. Over the past 20 years, the number of articles published on this topic has increased exponentially (Groppo et al. 2019), reflecting the need to develop efficient systems to manage water demand. The literature presents numerous models for urban water demand forecasting using statistical techniques such as linear regression (Campisi-Pinto et al. 2012; Santos & Pereira Filho 2014), nonlinear regression (Adamowski et al. 2012), time-series analysis (Caiado 2010; Huang et al. 2014; Al-Zahrani & Abo-Monasar 2015; Arandia et al. 2016; Rajballie et al. 2022), similarity-based approaches (Alvisi et al. 2007; Bakker et al. 2013; Tian et al. 2016; Gagliardi et al. 2017a; Pacchin et al. 2019), Markov chains (Gagliardi et al. 2017b; Pacchin et al. 2019) and techniques based on Soft Computing.

For water demand forecasting, several Soft Computing methods are presented and applied to historical water demand series. It is known that these series have stochastic and nonlinear components, making water demand forecasting a complex issue. In this context, Soft Computing methods such as Fuzzy Logic (Firat et al. 2009a; Ambrosio et al. 2019), Neural Computing (Firat et al. 2009b, 2010; Santos & Pereira Filho 2014; Al-Zahrani & Abo-Monasar 2015; Pacchin et al. 2019), Evolutionary Computation (Bai et al. 2014; Romano & Kapelan 2014; Leon et al. 2020; Shirkoohi et al. 2021), Support Vector Machines (Herrera et al. 2010; Brentan et al. 2016; Ambrosio et al. 2019), Random Forests (Chen et al. 2017; Ambrosio et al. 2019), Long Short-Term Memory (Boudhaouia & Wira 2021), Dual-Scale Deep Belief Network (Xu et al. 2018), Continuous Deep Belief Echo State Network (Xu et al. 2019b) and hybrid models (e.g., Nasseri et al. 2011; Adamowski et al. 2012; Campisi-Pinto et al. 2012; Odan & Reis 2012; Huang et al. 2014, 2021, 2022; Tiwari & Adamowski 2015; Guo et al. 2022; Rajballie et al. 2022) have provided more accurate results for urban water demand forecasting. In general, hybrid models are more robust for water demand forecasting compared with Feed Forward Neural Network (FFNN), Multiple Linear Regression (RLM), Multiple Nonlinear Regression (MNLR) and ARIMA models.

In general, the main factors that impact urban water demand are often difficult to identify using traditional algorithms. For instance, Xu et al. (2019a) applied the energy spectrum (Oshima & Kosuda 1998) and the highest Lyapunov coefficient (Tsonis 1992) to examine the main characteristics of the water demand time-series. Results indicate that the water demand time-series can be represented as chaotic time-series. Soft Computing methods, without proper pre-processing, can become unstable and produce erroneous results if applied to water demand forecasting (Zhang & Qi 2005).

Simultaneously, with the profusion of methods developed by the scientific community, the digitization process has been advancing. Consequently, data acquisition has been increasing, creating a ‘big data’ problem. De Mauro et al. (2014) claim that big data represents information assets characterized by high volume, speed and variety, which require specific technology and analytical methods to turn it into value. Data values can be divided into three groups. Values associated with the characteristics of the data set; values associated with the specific technology and analytical methods for manipulating the data; and, values associated with the insights, i.e., knowledge extracted from the data. Therefore, the goal of big data analytics is knowledge discovery from massive data sets (Chen & Han 2016).

Due to the rapid increase in data volume, a large amount of storage is necessary, requiring greater bandwidth and high latency in data processing (Caiza et al. 2020), making the Industrial Internet of Things (IIoT) a challenge for the current infrastructure (Sabireen & Neelanarayanan 2021). A solution to mitigate this problem is ‘cloud computing’, which, according to Sabireen & Neelanarayanan (2021), constitutes a new component of Industry 4.0. Cloud computing solves the big data problem, reducing energy consumption in network industrial sensors, improving security, processing and real-time data storage. In addition to solving major limitations in processing large databases, algorithms with low computational complexity (CC) have been developed (Arnaiz-González et al. 2016; Baldán & Benítez 2019; Sarma et al. 2019; Baldán et al. 2021).

In this scenario, the present work evaluates the feasibility of using a novel, analog-based methodology named dynamic time scan forecasting (DTSF) (Costa et al. 2021) to perform multi-step forecasting in univariate time-series in order to predict short-term (hourly) demand in WSS. This is done by comparing several known univariate alternatives in terms of computational efficiency and cost. Statistical, machine learning and hybrid approaches were used. As in Costa et al. (2021), this study employed the Naive Bayes (naive 1 method), the Pattern Sequence-Based Forecasting (PSF) analog approach, the time-series-based approach Box–Jenkins (SARIMA), Trigonometric Box–Cox transformation model, ARMA errors, Trend and Seasonal components (TBATS), the hybrid model using Seasonal and Trend decomposition using Loess filter (STL) and the exponential smoothing method (ETS) STL + ETS, the Hybrid.2 approach, and the NNET.2 approach – an automatic method that employs extreme learning machines (ELM). Additionally, we used the hybrid autoregressive neural network approach with bootstrap (BNNAR), THETA model and the ETS, which, according to Makidrakis et al. (2018), comparing numerous statistical and machine learning methods to predict a step forward, obtained the best accuracy among all evaluated methods.

The water demand time-series data were monitored using the ‘Data Logger’ device, equipment equipped with nonvolatile memory that performs data acquisition with the most varied types of existing sensors. The equipment was configured to register the flow rate (L/s) of the water exiting the water treatment plant (WTP-A) at 5 min intervals, and on the macrometer that connects the WTP-B to the study area at 15 min intervals. In the analyzed period, missing data comprise days with no data collection due to failures in the ‘Data Logger’ device. Missing data were replaced using the Naive Bayes imputation method. The choice of this method is due to its simplicity.

Prior to the forecasting analysis, the data were aggregated into 1 h intervals, using the mean function. The final data set showed a mean water flow rate of 533.63 (L/s). The forecast horizon was set at 24 h; i.e., a 24-step ahead.

Table 1 presents the descriptive measures of the historical demand series used in this study. The median flow rate for the period analyzed was 522.81 L/s, standard deviation of 123.16 L/s and coefficient of variation of 23.08%.

The DTSF method is based on scan statistics (Glaz et al. 2009), which comprise a class of statistical methods aiming to estimate anomalous behavior in databases with temporal, spatial or spatio-temporal components. This method was originally presented by Joseph Naus (Naus 1965) and adapted for epidemiological surveillance systems using spatial (Kulldorff 1997), temporal and spatio-temporal data (Kulldorff et al. 1998; Kulldorff 2001). Briefly, a scanning window with a pre-fixed geometry scans the data, alternating its position and dimension. A test statistic is calculated for each window configuration. The configuration yielding the highest value for the test statistic represents a potential candidate for anomalous behavior. Statistical inference is obtained using Monte Carlo simulations (Mooney 1997), with the null hypothesis that the data set does not present anomalous data. Detailed information about scan statistics is found in Glaz et al. (2009).

The DTSF method scans a time-series using a fixed window. The objective is to find historical windows in which the data patterns are similar to the most recent values in the time-series. A test statistic, or similarity statistic, is calculated for each window. In addition, a similarity function is estimated for each window. The similarity function aims to define an equation for mapping historical data in the scanning window to the most recent data in the time-series. Once the most similar historical windows are detected, the respective similarity functions are applied to the subsequent values of the selected windows, thus generating future forecasts of the time-series.

The final predictions are generated using an aggregation function, such as average or median functions. The DTSF method requires three parameters: the length of the scanning window, w, the parametric specification of the similarity function (the degree of polynomial, j, chosen to set the similarity function) and the number of the best matches, m (the number of the most similar historical windows). To guarantee the high computational speed of the method, linear similarity functions such as linear, quadratic or cubic equations are preferable. The number of best matches can be selected dynamically using, for example, those windows with similarity statistics above a previously defined threshold. In the present study, the number of the most similar historical windows was previously chosen. Briefly, different values for the length of the scanning window and the number of best matches were evaluated. The length of the scanning window and the number of best matches with minimum forecasting error were selected.

An important requirement for the application of the DTSF method is the availability of a large time-series. It is worth mentioning that DTSF is a data-oriented method.

The main Soft Computing tools used in this work are implemented in the R software. The Nonlinear Autoregressive Neural Network methods with Bootstrap, Autoregressive Integrated Moving Average, Trigonometric seasonality, Box–Cox transformation, ARMA errors, Trend and Seasonal components, Exponential Smoothing and Theta were applied using the Forecast package (Hyndman & Khandakar 2008; Hyndman et al. 2017). It is worth mentioning that the Theta model, developed by Assimakopoulos & Nikolopoulos (2000), became the winner of the M3 Competition and was used as one of the benchmark methods in the M4 Competition (Makidrakis et al. 2020). The hybrid model, using the STL filter with exponential smoothing (STL + ETS), and Hybrid.2 (consisting of an ensemble of seven different time-series forecasts – arima, tbats, ets, theta, stlm, snaive and nnetar – which are individually fitted and combined to create an ensemble mean forecast), were applied using the hybrid Model function from the forecast Hybrid package (Shaub & Ellis 2019). This package makes it possible to combine several models with equal weights, or with weights based on in-sample errors, and is based on the study by Bates & Granger (1969). The extreme learning machines ELM model was implemented using the nnfor package (Kourentzes 2017), which is based on studies by Crone & Kourentzes (2010) and Kourentzes et al. (2014). According to Lendasse et al. (2010), this method was ranked as the second-best method in ESTSP'08 – 2nd European Symposium of Time-Series Prediction. The PSF algorithm (Martinez Alvarez et al. 2011) was implemented using the PSF package (Bokde et al. 2017). The DTSF method was implemented using the DTScanF package (Costa & Mineti 2019).

According to Tashman (2000), rolling window cross-validation is the preferred procedure to evaluate forecasting performances.

The metrics used to evaluate the accuracy of the candidate models are: mean square error (MSE), the root of the mean square error (RMSE), the mean absolute error (MAE) and the mean absolute percentage error (MAPE). Model Fitting (MF) was also used as a measure of the fitness of historical data to the prediction model; it represents the MSE of the model predictions, normalized by the average of the time-series under study.

Computational time was estimated using a computer having a CPU with 7th Generation Intel^® Core ™ i7-7500 U @ 3.5 GHz processor, 16 GB DDR4 memory and 64× based processor.

Table 2 shows the results of predictions for the evaluated models. From January to May, and then in December, no model was more accurate than the others for forecasting. From June to November, the ELM method showed the best results. Regarding CC, the DTSF algorithm was extremely fast, as compared with the other methods.

The DTSF model was superior to SARIMA and ETS in all metrics evaluated (RMSE, MAE and MAPE) for all months of 2016. The STL-ETS hybrid model was more accurate based on the RMSE metric for May, July, August and September. In relation to the MAPE metric, the STL-ETS hybrid model was slightly better for the months of July and September, with values of 19.3% and 17.1%, respectively. For the same period, the DTSF model reached values of 19.8 and 17.2%, respectively. When compared with the PSF analogue, regarding the MAPE metric, the DTSF model obtained the best results for all months of 2016. Regarding the RMSE metric, TBATS was more accurate than DTSF in January, February, March, May, July, August, November and December. However, in relation to the MAPE metric, DTSF was more accurate in the periods from March to June and from August to October. Compared with the THETA method, DTSF obtained the best results for all months of the year using the RMSE and MAPE metrics. Regarding the MAE metric, THETA was more accurate for the months of July to September and in the month of November.

Next, a comparative analysis of the DTSF, ELM and BNNAR methods is presented. ELM proved to be more accurate, not only in relation to DTSF but also to BNNAR, for the entire second semester of 2016 for the RMSE and MAPE metrics. BNNAR presented superior results in relation to ELM and DTSF in April and May for the MAPE metric and superior results in April, May and December for the MAE metric. With respect to this metric, ELM was superior to DTSF and BNNAR in the first quarter of the year.

Figure 11 shows the accuracy of the models in the study period using the MF metric. The worst performance for all methods occurred in the summer (January, February and December) and spring (September to November) months, while the best performance occurred in the autumn (March to May) and winter (June to August) months, when the water demand is lower. The two best models, ELM and BNNAR, are similar. DTSF and TBATS alternated best performance accuracy throughout the studied period.

Table 4 presents the evaluation of the forecasting models for the different seasons in 2016 using the root mean square error, average absolute error, average percentage error and MF. During the summer, which is a period of higher water consumption, no model achieved better forecasting performance for any metrics. Nonetheless, the ELM achieved the best results during the winter months.

The results, shown in Table 5, show the computational overhead (log 10) of the methods used in the present study.

DTSF presented computational overhead ranging between −0.029 and 0.477. THETA was the method with the second-lowest computational overhead, with values ranging from 0.278 to 0.660. This was followed by ETS with values ranging from 1.714 to 2.583. Both the THETA and ETS methods are used as benchmarks of the M4 Competition. It is worth mentioning that the THETA method relies on a linear decomposition of the time-series into two components. In sequence, a linear regression model is applied to the first component and an exponential smoothing model is applied to the second component. The linear decomposition of the time-series, the fit of the linear regression model and the fit of the exponential smoothing model are extremely fast procedures. On the other hand, ELM, BNNAR, HYBRID, TBATS and PSF showed marked increases in computational overhead as new data were incorporated into the forecasts (recalibration of the rolling time window) (Tashman 2000). The computational overhead of these five methods ranged from 2.592 to 4.188, 3.345 to 3.981, 3.165 to 3.542, 3.096 to 3.523 and 2.868 to 3.265, respectively. In December 2016, BNNAR, ELM, HYBRID and PSF showed computational overheads of 3.981, 3.640, 3.506 and 3.255, respectively, compared with the DTSF computational overhead of 0.364.

After applying the seven-day moving average to the median of forecast errors, the results were obtained as shown in Table 6. The results indicate that in a one-week horizon with hourly forecasts, the forecast error for the different models varies greatly. Therefore, there is no better model for predicting water demand for every day of the year. The best water demand forecasting model in 2016 was NB, with 21.45% of the best forecasts, after applying a weekly moving average. This was followed by the hybrid neuronal model BNNAR with 20.06%; the DTSF method, with 17.27%; ELM, with 16.43%; THETA, with 5.85%; and PSF, with 5.01%; this was followed by hybrid model ETS-STL, with 3.62%. The statistical methods TBATS, SARIMA and ETS presented 4.74%, 3.06% and 1.39% of the best predictions, respectively. The HYBRID model showed the worst performance, with 1.11%. It is worth noting that a more complex method was not as accurate as a simpler method (Makidrakis et al. 2018). Dalessandro (2013) shows that the use of computationally less expensive algorithms, which are also less accurate in the intermediate stages, define a model that performs equally well in big data forecasting. These algorithms may have more iterations compared with those that are computationally more expensive. However, the iterations are much faster. Consequently, less expensive algorithms tend to converge much faster, while giving the same precision.

Table 7 shows the daily frequency of methods with the best forecast performance for each month during 2016. As previously mentioned, NB, DTSF, ELM and BNNAR achieved improved forecast performance, most frequently. A daily analysis of the best forecast methods is found in the Supplementary Material.

Based on the literature and the forecast results using different univariate methodologies, it is possible to conclude that algorithms with high computational overhead often are not applicable in large-scale, real-world situations. In this context, DTSF is extremely fast as compared with the remaining methods and provides reasonably lower prediction errors. The great advantage of this data-oriented method is that, given a large amount of data, in general, the performance improves. Thus, the use and evaluation of DTSF for water demand forecasting is an important and novel contribution of this study.

Furthermore, scanning and scaling previous observations using similarity functions compensate changes in the mean and in the scale throughout the time-series. Thus, observations under the influence of pipe failures and leakages can be used for forecasting purposes, as long as the water demand time-series pattern under these circumstances is strongly correlated with the last observations.

Results show that, in summer and autumn, no model surpassed the others in terms of forecasting accuracy. However, from June to November (winter and spring), ELM presented the best results, i.e., ELM presented the best forecasting performance. However, its computational overhead is a major disadvantage. The other methods based on neural networks presented the best results for the metrics RMSE, MAE, MAPE and MF. Similar to ELM, computational overhead increases significantly if cross-validation with sliding windows and recalibration is applied.

In general, the main findings show that there is no uniform method to predict water demand with high accuracy. Nonetheless, there is a set of methods with optimal performances. Each method can be further investigated to improve its own performance. Alternatively, future work will aim at investigating the switching of methods, since each method achieved better forecast accuracy at different parts of the year (see Figures A.1–A.4 in the Supplementary Material).

Finally, forecasting results using data from 2016 indicate that, within a week-long horizon with an hourly forecast resolution, errors using the different models vary. Therefore, there is no best model for predicting daily water demand. Results indicate the use of different models at different times of the year.

The authors thank CNPq and Fapemig for financial support, and Sanitation Company of Minas Gerais (Copasa MG) for providing urban water demand data.

Data cannot be made publicly available; readers should contact the corresponding author for details.

The authors declare there is no conflict.