ABSTRACT
Missing streamflow data is a common issue in Peninsular Malaysia, as the technologies used in hydrological studies often fail to collect data accurately. Additionally, conventional methods are still widely used in the region, which are less accurate compared to artificial intelligence (AI) methods in estimating missing streamflow data. Therefore, this study aims to estimate the missing streamflow data from 11 stations in Peninsular Malaysia by using different AI methods and determine the most appropriate method. Four homogeneity tests were applied to check the quality of data, and the results of the tests indicated that the streamflow data in most stations were homogenous. Two AI methods were applied in this study, which were artificial neural network and artificial neuro-fuzzy inference systems (ANFIS). The proposed AI methods were compared with five different conventional methods. All streamflow missing data, constituting 30% of data from each year were estimated on a daily time scale, and evaluated using root mean square error, mean absolute error and correlation coefficient values. The results indicated that ANFIS was the best due to its learning abilities and the fuzzy inference systems, which enable it to handle complicated input–output patterns and provide highly accurate estimation results.
HIGHLIGHTS
Various estimation methods were compared and evaluated.
Homogeneity tests were applied to assess the quality of historical streamflow data from 11 stations.
The estimation results were rigorously evaluated using three evaluation methods: root mean square error, mean absolute error and correlation coefficient.
Adaptive neuro-fuzzy inference system emerged as the best method for estimating missing streamflow data.
INTRODUCTION
The completeness of streamflow data is vital in the aspect of the hydrological system, including weather forecast, water resources management and flood and drought prediction. However, the missing streamflow data are frequently found during the data collection process because the gauging devices used in hydrological studies heavily rely on physical-based systems and sensors used to collect the data (Li et al. 2020). The data collected from the physical sensor system is often partial and inaccurate over an extended period of data collection due to exposure to various hazards, including battery depletion, physical damage and extreme environmental conditions (Hamzah et al. 2020). Other than that, the monsoon period in Malaysia can significantly impact streamflow discharge, leading to potential inaccuracies in streamflow measurement. High velocity and turbulence of the streamflow during this period can make measuring streamflow more challenging (Higgins et al. 2022). Besides, conventional methods such as the inverse distance weighting (IDW) method, arithmetic average (AM) method and normal ratio (NR) method are still applied in Peninsular Malaysia. These conventional methods possess a lower accuracy in the estimation of missing streamflow data compared to artificial intelligence (AI) methods (Ismail et al. 2017). The conventional methods are suitable for stable environments without floods and heavy rainfall. While commonly used for estimating missing data, it is recommended to use advanced methods for cases where missing data are significant or patterns are complex (Gao et al. 2018). The incompleteness of streamflow data will heavily affect the analysis, prediction and forecasting activities conducted by hydrologists and engineers.
There are several studies that applied conventional methods, including AM, NR, IDW and coefficient of correlation (CR), to estimate missing streamflow data. Ismail et al. (2017) applied AM, CR, IDW and NR to estimate the missing streamflow data and rainfall data in Terengganu by assuming 5, 10, 15 and 20% of the datasets are missing. The result showed that the NR was the best compared to all the other methods for missing streamflow data estimation in most of the stations. Meanwhile, Yilmaz & Bihrat (2019) evaluated the performances for regression analysis (REG), standardization with mean (SM), single donor station-based drainage area (DAR), standardization with mean and standard deviation (SMS), multiple donor stations-based drainage area ratio (MDAR) and IDW for missing streamflow estimation from stations located at the Porsuk River Basin. The results showed that the IDW was the best method, while DAR was the most below-average method in most of the stations. Kamwaga et al. (2018) applied linear regression, multi-donor linear regression, the rainfall–runoff relationship using the double mass curve method, flow duration matching, drainage area ratio and rainfall–runoff modelling using hydrologiska byrans vattenbalansavdelning (HBV)-light to estimate missing streamflow data on the catchment of the Little Ruaha at Tanzania. The results showed the multiple linear regression methods were the best in missing data estimation among all the other methods, while the runoff modelling using HBV-light possessed the poorest performance. Al-Taiee (2008) applied trend analysis of the time series of the river stages data at the Mosul station and concluded that Winter's multiplicative model is suitable to forecast missing streamflow data for the Tigris River.
Studies on evaluating the performance of AI methods for estimating missing streamflow data were also conducted in Malaysia. Gao et al. (2018) employed the auto regressive integrated moving average (ARIMA) and autoregressive conditional heteroscedasticity (ARCH) models to fill the gap of missing streamflow data and compare their performance with conventional methods including listwise and pairwise deletion, single imputation, arithmetic means and median imputation, regression-based imputation and principal component analysis, and multiple imputation. The results showed that ARIMA and ARCH are the more suitable methods for filling the missing data. Norazizi & Deni (2019) applied the multivariate imputation by chained equations method, bootstrapping and the expectation maximization algorithm method and artificial neural network (ANN) to estimate the missing data at eight meteorological stations in Kuantan. The results showed that the performance of ANN is the best among the other methods.
Many studies have been conducted to compare the performances between conventional and AI methods. Mesta et al. (2020) applied the Takagi–Sugeno fuzzy rule-based (FRB) model to estimate the missing daily streamflow data of the Meric–Ergene River located in Turkey. The performance of FRB was evaluated by comparing it with the physical-based Hydrologic Engineering Centre-Hydrologic Modelling System (HEC-HMS). The results showed that the data-based FRB model possesses a better performance compared to HEC-HMS. Saplioglu & Kucukerdem (2018) evaluated the performance of artificial neuro-fuzzy inference systems (ANFIS), multiple regression and NR in estimating missing streamflow data at the Yeşilırmak River, Turkey. The results indicated that the ANFIS was the best method with the lowest mean squared error in most of the data sets. Hamzah et al. (2020) reviewed the performance, advantages and disadvantages of different conventional and AI methods. This research indicated that the conventional method such as the deletion method brings different drawbacks and that the removal of variables that possess missing values using this method will cause the loss of valued data, reducing the size of the data sample and inaccuracy in the results. Besides, the research also showed that the AI methods based on machine learning techniques delivered a better performance in missing streamflow data estimation compared to conventional methods. Belotti et al. (2021) investigated the effectiveness of linear autoregressive (AR), periodic autoregressive (PAR) models and extreme learning machines (ELM) in streamflow forecasting. The accuracy of each model was analysed using mean square error (MSE), mean absolute error (MAE), correlation coefficient (CC) and Nash–Sutcliffe efficiency. The Friedman test was applied to verify the difference in the significance of errors in the forecast models and showed that ELM had the best performance compared to AR and PAR.
In general, missing streamflow data is an issue which needs to be treated seriously, as streamflow plays an important role for humans, different hydrological applications and fields (Heddam & Kisi 2021). In the aspects of engineering, complete streamflow data provide engineers with sufficient information to design the infrastructure, including reservoirs, drainage systems, bridges and roads. It ensures that the engineer can provide a suitable design, avoiding over- or under-design issues, which can result in excessive costs and the risk of structural collapse. Besides, a complete set of data enables hydrologists to conduct the hydrological analysis including streamflow prediction, flood and drought prediction more accurately. These analyses with higher accuracy enable related authorities to manage and provide wise decisions on water resources management. This will result in a stable water supply and better planning for natural disasters such as floods and drought (Fentaw et al. 2019), which eventually benefit people and industries strongly relying on water supply such as agriculture industries and meat production. Although previous studies have been conducted on comparing various missing data estimating methods, it is important to note that the performance of estimating methods may be different in different regions due to their distinct geographical and meteorological characteristics. Hence, this study aims to investigate the performance of different AI methods in estimating missing streamflow and how it performed compared to conventional methods in Peninsular Malaysia. At the end of this study, the most appropriate method for estimating missing streamflow data will be determined.
METHODOLOGY
Data collection and study area
Information on each streamflow station
Station code . | Station name . | Study period . | Duration . | Latitude . | Longitude . |
---|---|---|---|---|---|
1,737,451 | Sungai Johor at Rantau Panjang | 1972 ∼ 1992 | 20 years | 01° 46′ 50″E | 103° 44′ 45″N |
5,606,410 | Sungai Muda at Jambatan Syed Omar | 1974 ∼ 1994 | 20 years | 05° 36' 35″E | 100° 37' 35″N |
5,721,442 | Sungai Kelantan at Jambatan Guillemard | 1973 ∼ 1993 | 20 years | 05° 45' 45″E | 102° 09' 00″N |
2,224,432 | Sungai Kesang at Chin Chin | 1960 ∼ 1980 | 20 years | 02° 17' 25″E | 102° 29' 35″N |
2,723,401 | Sungai Kepis at Jambatan Kayu Lama | 1979 ∼ 1999 | 20 years | 02° 42' 20″E | 102° 21' 20″N |
3,519,426 | Sungai Bentong at Kuala Marong | 1970 ∼ 1990 | 20 years | 03° 30' 45″E | 101° 54' 55″N |
6,503,401 | Sungai Arau at Ladang Tebu Felda | 1984 ∼ 2004 | 20 years | 06° 30' 10″E | 100° 21' 05″N |
3,116,430 | Sungai Klang at Jambatan Sulaiman | 1995 ∼ 2015 | 20 years | 03° 08' 20″E | 101° 41' 50″N |
4,930,401 | Sungai Berang at Menerong | 1998 ∼ 2018 | 20 years | 04° 56' 20″E | 103° 03' 45″N |
3,813,411 | Sungai Bernam at Jambatan Skc | 1984 ∼ 2004 | 20 years | 03° 48' 27″E | 101° 21' 70″N |
4,907,422 | Sungai Kurau at Bt. 14 Jalan Taiping | 1975 ∼ 1995 | 20 years | 04 ° 58' 40″E | 100 ° 46' 50″N |
Station code . | Station name . | Study period . | Duration . | Latitude . | Longitude . |
---|---|---|---|---|---|
1,737,451 | Sungai Johor at Rantau Panjang | 1972 ∼ 1992 | 20 years | 01° 46′ 50″E | 103° 44′ 45″N |
5,606,410 | Sungai Muda at Jambatan Syed Omar | 1974 ∼ 1994 | 20 years | 05° 36' 35″E | 100° 37' 35″N |
5,721,442 | Sungai Kelantan at Jambatan Guillemard | 1973 ∼ 1993 | 20 years | 05° 45' 45″E | 102° 09' 00″N |
2,224,432 | Sungai Kesang at Chin Chin | 1960 ∼ 1980 | 20 years | 02° 17' 25″E | 102° 29' 35″N |
2,723,401 | Sungai Kepis at Jambatan Kayu Lama | 1979 ∼ 1999 | 20 years | 02° 42' 20″E | 102° 21' 20″N |
3,519,426 | Sungai Bentong at Kuala Marong | 1970 ∼ 1990 | 20 years | 03° 30' 45″E | 101° 54' 55″N |
6,503,401 | Sungai Arau at Ladang Tebu Felda | 1984 ∼ 2004 | 20 years | 06° 30' 10″E | 100° 21' 05″N |
3,116,430 | Sungai Klang at Jambatan Sulaiman | 1995 ∼ 2015 | 20 years | 03° 08' 20″E | 101° 41' 50″N |
4,930,401 | Sungai Berang at Menerong | 1998 ∼ 2018 | 20 years | 04° 56' 20″E | 103° 03' 45″N |
3,813,411 | Sungai Bernam at Jambatan Skc | 1984 ∼ 2004 | 20 years | 03° 48' 27″E | 101° 21' 70″N |
4,907,422 | Sungai Kurau at Bt. 14 Jalan Taiping | 1975 ∼ 1995 | 20 years | 04 ° 58' 40″E | 100 ° 46' 50″N |
Conventional methods in missing streamflow data estimation
Arithmetic average method
AM is the easiest method which is normally applied to estimate and fill in the missing gap of data in hydrological studies. AM is the method suitable to be applied when the data from the individual gauge is not too varied from the average data value and the gauges are homogenously allocated at the area. By using AM, the missing data is determined based on the data obtained from the nearest stations.
Inverse distance weighting method
Normal ratio method
The NR is one of the frequently applied methods for missing data estimation. It is applied when one or more of the streamflow stations data exceed the targeted streamflow data by 10% or more. The average ratio of the available data between the targeted station and the ith surrounding station is used to weight in the NR method. The ratios of entire streamflow data at the targeted and surrounding stations are applied to determine the concurrent streamflow data at neighbouring stations.
Coefficient of correlation
Linear interpolation
Auto regressive integrated moving average
Artificial intelligence methods in missing streamflow data estimation
Artificial neural network
Artificial neuro-fuzzy inference systems
Layer 1

Layer 2
Layer 3
Layer 4
In Equation (23), is the output obtained from layer 3, and pi, qi and ri are the parameter set.
Layer 5
Performance evaluation of different estimation methods
To effectively evaluate the performance of all estimation methods, the daily streamflow value computed from each method is compared to the actual daily streamflow data. Using estimated values, which constitute 30% of the data from each year, a comparison was made with the observed values. The longest periods without records are different across different streamflow stations, with some stations experiencing gaps of up to 7 days. Three performance evaluation methods were applied to the comparative analysis, including MAE, root mean square error (RMSE) and CC statistics, in order to evaluate each missing data estimation method. These evaluation methods provided us with the error of each estimation method, based on comparing the estimation result and the relevant observed value (Ismail et al. 2017).
Root mean square error

Mean absolute error
Correlation coefficient


Homogeneity test
Buishand range test



Higher Q statistic values imply that the data in the time series is not homogeneous. To assess the significance of the Q statistic, a p-value can be calculated by comparing the observed Q value to the distribution of Q values under the null hypothesis. The p-value represents the probability of observing a Q statistic as extreme as or more extreme than the observed value if the null hypothesis of homogeneity is true. If the p-value is below a predetermined significance level, the null hypothesis of homogeneity is rejected, indicating non-homogeneity.
Von Neumann ratio test
Standard normal homogeneity test



To assess the significance of the T0 statistic, a p-value can be calculated by comparing it to the distribution of T0 values under the null hypothesis. If the p-value is below the predetermined significance level, the null hypothesis of homogeneity is rejected.
Pettitt Test
To assess the significance of the Xk statistic, critical values can be determined based on the null distribution of the test statistic. If the observed Xk statistic exceeds the critical values, the null hypothesis of homogeneity is rejected.
RESULTS AND DISCUSSION
The estimation results were generated using the data from stations in different states of Peninsular Malaysia. Five proposed conventional methods and three AI methods were applied to estimate the missing data from 11 proposed targeted stations. The performances of the estimation methods were evaluated using RMSE, MAE and CC methods.
Root mean square error
Performance of each estimation method based on RMSE results
Stations . | Conventional methods . | AI methods . | ||||||
---|---|---|---|---|---|---|---|---|
AM . | IDW . | NR . | CR . | LI . | ARIMA . | ANN . | ANFIS . | |
1,737,451 | 9.96 | 7.65 | 6.14 | 15.11 | 8.92 | 1.37 | 2.14 | 1.74 |
2,224,432 | 5.43 | 0.63 | 0.20 | 7.37 | 5.59 | 0.08 | 0.05 | 0.26 |
2,723,401 | 3.50 | 2.79 | 2.24 | 4.08 | 3.47 | 0.23 | 0.26 | 0.14 |
3,116,430 | 14.83 | 9.78 | 2.53 | 23.69 | 17.37 | 1.59 | 2.62 | 1.46 |
3,813,411 | 8.18 | 9.43 | 6.76 | 9.70 | 5.43 | 2.58 | 3.25 | 1.58 |
4,907,422 | 5.92 | 1.16 | 0.34 | 2.71 | 6.57 | 0.43 | 0.06 | 0.21 |
5,606,410 | 20.42 | 26.76 | 7.76 | 22.10 | 10.98 | 0.37 | 0.46 | 0.44 |
6,503,401 | 7.23 | 0.10 | 0.13 | 21.90 | 7.29 | 0.07 | 0.0997 | 0.1031 |
5,721,442 | 143.55 | 147.22 | 13.27 | 145.65 | 75.86 | 9.97 | 9.96 | 9.85 |
4,930,401 | 64.75 | 11.02 | 5.11 | 68.70 | 62.99 | 9.91 | 3.18 | 4.04 |
3,519,426 | 42.04 | 54.71 | 1.95 | 29.84 | 43.35 | 1.06 | 0.7313 | 0.7309 |
Average | 29.62 | 24.66 | 4.22 | 31.90 | 22.53 | 2.51 | 2.07 | 1.87 |
Stations . | Conventional methods . | AI methods . | ||||||
---|---|---|---|---|---|---|---|---|
AM . | IDW . | NR . | CR . | LI . | ARIMA . | ANN . | ANFIS . | |
1,737,451 | 9.96 | 7.65 | 6.14 | 15.11 | 8.92 | 1.37 | 2.14 | 1.74 |
2,224,432 | 5.43 | 0.63 | 0.20 | 7.37 | 5.59 | 0.08 | 0.05 | 0.26 |
2,723,401 | 3.50 | 2.79 | 2.24 | 4.08 | 3.47 | 0.23 | 0.26 | 0.14 |
3,116,430 | 14.83 | 9.78 | 2.53 | 23.69 | 17.37 | 1.59 | 2.62 | 1.46 |
3,813,411 | 8.18 | 9.43 | 6.76 | 9.70 | 5.43 | 2.58 | 3.25 | 1.58 |
4,907,422 | 5.92 | 1.16 | 0.34 | 2.71 | 6.57 | 0.43 | 0.06 | 0.21 |
5,606,410 | 20.42 | 26.76 | 7.76 | 22.10 | 10.98 | 0.37 | 0.46 | 0.44 |
6,503,401 | 7.23 | 0.10 | 0.13 | 21.90 | 7.29 | 0.07 | 0.0997 | 0.1031 |
5,721,442 | 143.55 | 147.22 | 13.27 | 145.65 | 75.86 | 9.97 | 9.96 | 9.85 |
4,930,401 | 64.75 | 11.02 | 5.11 | 68.70 | 62.99 | 9.91 | 3.18 | 4.04 |
3,519,426 | 42.04 | 54.71 | 1.95 | 29.84 | 43.35 | 1.06 | 0.7313 | 0.7309 |
Average | 29.62 | 24.66 | 4.22 | 31.90 | 22.53 | 2.51 | 2.07 | 1.87 |
Note: Bolded value indicates the best estimation method.
Among all the conventional methods ARIMA was found to be the best method with the lowest average RMSE value of 2.51, as shown in Figure 3. NR, LI, IDW and AM followed with average RMSE values of 4.22, 22.53, 24.66 and 29.62, respectively. The CR was the worst as it possessed the highest average RMSE value of 31.90. The average RMSE values of conventional methods for each station were significantly higher compared to AI methods. Hence, it can be concluded that AI methods performed better than conventional methods at every station. These findings agree with the study by Hamzah et al. (2020), which showed that the performance of conventional methods was poorer than that of AI methods. This is because conventional methods suffer from different drawbacks, including the removal of correlation between variables, leading to the loss of important information and inaccuracy in the results (Faizin et al. 2019; Ratolojanahary et al. 2019).
Mean absolute error
Performance of each estimation method based on MAE results
Stations . | Conventional methods . | AI methods . | ||||||
---|---|---|---|---|---|---|---|---|
AM . | IDW . | NR . | CR . | LI . | ARIMA . | ANN . | ANFIS . | |
1,737,451 | 5.75 | 4.41 | 3.55 | 8.73 | 5.15 | 0.79 | 1.23 | 1.00 |
2,224,432 | 3.14 | 0.36 | 0.12 | 4.25 | 3.23 | 0.05 | 0.03 | 0.15 |
2,723,401 | 2.02 | 1.61 | 1.29 | 2.36 | 2.01 | 0.13 | 0.15 | 0.08 |
3,116,430 | 8.56 | 5.64 | 1.46 | 13.67 | 10.03 | 0.92 | 1.51 | 0.84 |
3,813,411 | 4.72 | 5.44 | 3.91 | 5.60 | 3.14 | 1.49 | 1.87 | 0.91 |
4,907,422 | 3.42 | 0.67 | 0.20 | 1.57 | 3.79 | 0.25 | 0.04 | 0.12 |
5,606,410 | 11.79 | 15.45 | 4.48 | 12.76 | 6.34 | 0.21 | 0.26 | 0.25 |
6,503,401 | 4.17 | 0.06 | 0.07 | 12.64 | 4.21 | 0.04 | 0.057 | 0.059 |
5,721,442 | 82.88 | 85.00 | 7.66 | 84.09 | 43.80 | 5.76 | 5.75 | 5.68 |
4,930,401 | 37.39 | 6.36 | 2.95 | 39.67 | 36.37 | 5.72 | 1.84 | 2.33 |
3,519,426 | 24.27 | 31.59 | 1.13 | 17.23 | 25.03 | 0.61 | 0.4222 | 0.4220 |
Average | 17.10 | 14.24 | 2.44 | 18.42 | 13.01 | 1.45 | 1.20 | 1.08 |
Stations . | Conventional methods . | AI methods . | ||||||
---|---|---|---|---|---|---|---|---|
AM . | IDW . | NR . | CR . | LI . | ARIMA . | ANN . | ANFIS . | |
1,737,451 | 5.75 | 4.41 | 3.55 | 8.73 | 5.15 | 0.79 | 1.23 | 1.00 |
2,224,432 | 3.14 | 0.36 | 0.12 | 4.25 | 3.23 | 0.05 | 0.03 | 0.15 |
2,723,401 | 2.02 | 1.61 | 1.29 | 2.36 | 2.01 | 0.13 | 0.15 | 0.08 |
3,116,430 | 8.56 | 5.64 | 1.46 | 13.67 | 10.03 | 0.92 | 1.51 | 0.84 |
3,813,411 | 4.72 | 5.44 | 3.91 | 5.60 | 3.14 | 1.49 | 1.87 | 0.91 |
4,907,422 | 3.42 | 0.67 | 0.20 | 1.57 | 3.79 | 0.25 | 0.04 | 0.12 |
5,606,410 | 11.79 | 15.45 | 4.48 | 12.76 | 6.34 | 0.21 | 0.26 | 0.25 |
6,503,401 | 4.17 | 0.06 | 0.07 | 12.64 | 4.21 | 0.04 | 0.057 | 0.059 |
5,721,442 | 82.88 | 85.00 | 7.66 | 84.09 | 43.80 | 5.76 | 5.75 | 5.68 |
4,930,401 | 37.39 | 6.36 | 2.95 | 39.67 | 36.37 | 5.72 | 1.84 | 2.33 |
3,519,426 | 24.27 | 31.59 | 1.13 | 17.23 | 25.03 | 0.61 | 0.4222 | 0.4220 |
Average | 17.10 | 14.24 | 2.44 | 18.42 | 13.01 | 1.45 | 1.20 | 1.08 |
Note: Bolded value indicates the best estimation method.
Correlation coefficient
Performance of each estimation method based on CC results
Location . | Conventional methods . | AI methods . | ||||||
---|---|---|---|---|---|---|---|---|
AM . | IDW . | NR . | CR . | LI . | ARIMA . | ANN . | ANFIS . | |
1,737,451 | 16.0 | 6.7 | 34.2 | 16.6 | 25.4 | 103.4 | 267.3 | 162.5 |
2,224,432 | 1.5 | 1.5 | 2.3 | 1.5 | 1.5 | 5.2 | 19.8 | 153.6 |
2,723,401 | 8.8 | 2.8 | 8.1 | 8.5 | 13.8 | 47.0 | 44.8 | 648.3 |
3,116,430 | 27.3 | 37.1 | 43.1 | 25.8 | 27.9 | 90.3 | 43.8 | 1,611.8 |
3,813,411 | 82.0 | 104.4 | 103.2 | 120.0 | 122.1 | 244.5 | 215.2 | 2,436.3 |
4,907,422 | 4.3 | 1.5 | 10.4 | 4.6 | 4.4 | 50.9 | 3,633.4 | 362.6 |
5,606,410 | 20.1 | 3.2 | 108.5 | 15.3 | 56.6 | 3,275.7 | 998.0 | 7,517.5 |
6,503,401 | 0.26 | 0.00 | 0.01 | 0.25 | 0.27 | 0.24 | 2.82 | 0.00 |
5,721,442 | 99.0 | 89.4 | 1,473.9 | 93.5 | 285.4 | 2,846.5 | 3,662.6 | 39,186.9 |
4,930,401 | 36.5 | 50.2 | 54.4 | 36.5 | 39.0 | 77.0 | 1,502.7 | 5,018.3 |
3,519,426 | 10.8 | 10.6 | 17.5 | 11.0 | 10.9 | 34.2 | 56.3 | 3,106.1 |
Average | 27.9 | 28.0 | 168.7 | 30.3 | 53.4 | 615.9 | 949.7 | 5,473.1 |
Location . | Conventional methods . | AI methods . | ||||||
---|---|---|---|---|---|---|---|---|
AM . | IDW . | NR . | CR . | LI . | ARIMA . | ANN . | ANFIS . | |
1,737,451 | 16.0 | 6.7 | 34.2 | 16.6 | 25.4 | 103.4 | 267.3 | 162.5 |
2,224,432 | 1.5 | 1.5 | 2.3 | 1.5 | 1.5 | 5.2 | 19.8 | 153.6 |
2,723,401 | 8.8 | 2.8 | 8.1 | 8.5 | 13.8 | 47.0 | 44.8 | 648.3 |
3,116,430 | 27.3 | 37.1 | 43.1 | 25.8 | 27.9 | 90.3 | 43.8 | 1,611.8 |
3,813,411 | 82.0 | 104.4 | 103.2 | 120.0 | 122.1 | 244.5 | 215.2 | 2,436.3 |
4,907,422 | 4.3 | 1.5 | 10.4 | 4.6 | 4.4 | 50.9 | 3,633.4 | 362.6 |
5,606,410 | 20.1 | 3.2 | 108.5 | 15.3 | 56.6 | 3,275.7 | 998.0 | 7,517.5 |
6,503,401 | 0.26 | 0.00 | 0.01 | 0.25 | 0.27 | 0.24 | 2.82 | 0.00 |
5,721,442 | 99.0 | 89.4 | 1,473.9 | 93.5 | 285.4 | 2,846.5 | 3,662.6 | 39,186.9 |
4,930,401 | 36.5 | 50.2 | 54.4 | 36.5 | 39.0 | 77.0 | 1,502.7 | 5,018.3 |
3,519,426 | 10.8 | 10.6 | 17.5 | 11.0 | 10.9 | 34.2 | 56.3 | 3,106.1 |
Average | 27.9 | 28.0 | 168.7 | 30.3 | 53.4 | 615.9 | 949.7 | 5,473.1 |
Note: Bolded value indicates the best estimation method.
Ranking of each method
Based on the obtained RMSE, MAE and CC values, a ranking of each estimation method can be acquired. Table 5 indicates the ranking of each method, where the lower total points indicate the higher performance of the methods. According to Table 5, ANFIS was selected as the best among all the methods, while ANN was the second best. These findings are similar to the study of Jimeno-Sáez et al. (2017), which indicated that ANFIS provided superior performance in peninsular Spain. ANFIS is armed with the learning abilities of neural networks and fuzzy inference systems to generate a series of fuzzy IF–THEN criteria by analysing the patterns in different data sets. By merging the neural network's learning ability with fuzzy inference systems, ANFIS can estimate the missing streamflow data better than the other methods (Zakaria et al. 2021). Besides, as shown in Table 5, ARIMA was ranked as number three, which was also the best among all the conventional methods. ARIMA took into account of the autocorrelation present in the streamflow data series, which means the model made estimation by capturing the trend and pattern of the data.
Ranking of each estimation method
Estimation method . | Total points . | Ranking . |
---|---|---|
ANFIS | 60 | 1 |
ANN | 73 | 2 |
ARIMA | 81 | 3 |
NR | 136 | 4 |
LI | 193 | 5 |
IDW | 206 | 6 |
AM | 210 | 7 |
CC | 229 | 8 |
Estimation method . | Total points . | Ranking . |
---|---|---|
ANFIS | 60 | 1 |
ANN | 73 | 2 |
ARIMA | 81 | 3 |
NR | 136 | 4 |
LI | 193 | 5 |
IDW | 206 | 6 |
AM | 210 | 7 |
CC | 229 | 8 |
Note: Bolded value indicates the best estimation method.
Homogeneity test
The homogeneity tests including PT, SNHT, BRT and VNRT were applied to the daily streamflow data collected from DID. If the null hypothesis is accepted with the p-value larger than 0.05, the streamflow data are considered homogeneous. The results produced by all four tests were categorized into three categories, which are ‘useful’, ‘doubtful’ and ‘suspect’. The result was categorized as ‘useful’ when all or three of the tests accepted the null hypotheses; ‘doubtful’ when two tests accepted the null hypotheses; ‘suspect’ when one or none of the tests accepted the null hypothesis. As shown in Table 6, the results of the PT indicated that the time series from all the stations were accepted. The rejections of SNHT occurred at three stations, which were stations 2,224,432, 3,813,411 and 5,721,442. Besides, the rejection of BRT was found at station 2,723,401. Furthermore, the rejections of VNRT occurred at most of the stations, except station 2,723,401. Because VNRT evaluates the time series as randomly distributed, hence the rejections indicated that the streamflow was normally distributed. This could be due to the influence of atmospheric pressure, temperature, humidity and wind speed. Previous research suggested that rainfall data in Peninsular Malaysia often exhibits a bell-shaped curve, which is one of the characteristics if a normal distribution (Tan et al. 2020; Ghani et al. 2022).
Summary of homogeneity test for daily time series
Stations . | Methods . | Results . | |||
---|---|---|---|---|---|
PT . | SNHT . | BRT . | VNRT . | ||
1,737,451 | 0.459 | 0.215 | 0.635 | < 0.0001 | Useful |
2,224,432 | 0.956 | 0.0003 | 1.000 | < 0.0001 | Doubtful |
2,723,401 | 0.125 | 0.083 | 0.041 | 0.351 | Useful |
3,116,430 | 0.714 | 0.138 | 0.947 | 0.038 | Useful |
3,813,411 | 0.592 | 0.001 | 0.589 | < 0.0001 | Doubtful |
4,907,422 | 0.258 | 0.072 | 0.376 | 0.003 | Useful |
5,606,410 | 0.959 | 0.061 | 0.951 | < 0.0001 | Useful |
6,503,401 | 0.822 | 0.150 | 0.857 | 0.001 | Useful |
5,721,442 | 0.672 | < 0.0001 | 0.789 | < 0.0001 | Doubtful |
4,930,401 | 0.966 | 0.119 | 0.973 | 0.044 | Useful |
3,519,426 | 0.244 | 0.330 | 0.599 | 0.004 | Useful |
Stations . | Methods . | Results . | |||
---|---|---|---|---|---|
PT . | SNHT . | BRT . | VNRT . | ||
1,737,451 | 0.459 | 0.215 | 0.635 | < 0.0001 | Useful |
2,224,432 | 0.956 | 0.0003 | 1.000 | < 0.0001 | Doubtful |
2,723,401 | 0.125 | 0.083 | 0.041 | 0.351 | Useful |
3,116,430 | 0.714 | 0.138 | 0.947 | 0.038 | Useful |
3,813,411 | 0.592 | 0.001 | 0.589 | < 0.0001 | Doubtful |
4,907,422 | 0.258 | 0.072 | 0.376 | 0.003 | Useful |
5,606,410 | 0.959 | 0.061 | 0.951 | < 0.0001 | Useful |
6,503,401 | 0.822 | 0.150 | 0.857 | 0.001 | Useful |
5,721,442 | 0.672 | < 0.0001 | 0.789 | < 0.0001 | Doubtful |
4,930,401 | 0.966 | 0.119 | 0.973 | 0.044 | Useful |
3,519,426 | 0.244 | 0.330 | 0.599 | 0.004 | Useful |
Note: Bolded value indicates that the null hypothesis of homogeneity is rejected.
It can be concluded that there were total of eight stations categorized as ‘useful’, three stations categorized as ‘doubtful’ and none of the station was categorized as ‘suspect’. These results indicated that the streamflow data at most of the stations are homogenous. The daily time series categorized as ‘useful’ can be applied for future studies related to missing streamflow data estimation. Those daily time series categorized as ‘doubtful’ should be further alerted, as non-homogeneity was detected in the series. Even though there were three streamflow stations categorized as ‘doubtful’, the streamflow data can still be applied if they are treated carefully. However, it is important to note that while homogeneity tests can be useful for detecting changes in the streamflow data, they do not account for the uncertainty associated with missing streamflow values. Additionally, these tests can sometimes be sensitive to the magnitude of streamflow values, making it difficult to capture extreme events or rare hydrological phenomena.
CONCLUSION
Research on missing data estimation has received much attention from researchers worldwide, as accurate and complete datasets are important for reliable analysis and better decision-making.
In this study, the estimation of missing streamflow data was carried out in 11 stations from Peninsular Malaysia. The missing streamflow data were estimated by six different conventional methods including AM, IDW, NR, CR, LI and ARIMA, and two AI methods including ANN and ANFIS. Homogeneity tests, including PT, SNHT, BRT and VNRT were applied to access the quality of streamflow data sets. Based on the performance evaluation results, ANFIS was ranked as the best method, while ANN, ARIMA, NR, LI, IDW, AM and CR were ranked as the second to eighth-best, respectively. This can be explained by the fact that ANFIS combined the learning abilities and fuzzy inference systems which allows it to model the non-linear patterns of input–output data. This study demonstrates that AI methods significantly outperform conventional methods due to their better capabilities in dealing with complicated relationships in data and adapting to new data patterns. The conventional methods fall short in situations requiring complicated handling of the missing data. The findings of the estimation of missing streamflow data are important for the accuracy of hydrological analysis, such as flood and drought prediction, and are useful for other regions with similar tropical climatic conditions. The hydrological analysis with higher accuracy can enable related authorities to make the proper decisions on water resources management.
It is recommended that the longer time scales of streamflow data should be applied in future studies, such as utilization of datasets spanning more than 30 years. Longer time scales can provide a comprehensive view of streamflow characteristics and improve the reliability of the results. Additionally, it is recommended to apply hybrid methodologies that integrate multiple AI models, such as the multilayer perceptron neural networks model (Narimani et al. 2023) and hybrid deep neural network approach (Sharma et al. 2022) to improve the precision of missing data estimation. They have the potential to integrate the advantages of diverse stand-alone AI models by offering a powerful tool to handle ambiguous and incomplete data.
ACKNOWLEDGEMENTS
The authors would like to express their gratitude towards the Department of Irrigation and Drainage (DID) of Malaysia for providing the streamflow data for this study.
AUTHOR CONTRIBUTIONS
All authors equally contributed to the preparation of this manuscript. All authors read and approved the final manuscript.
FUNDING
This research received no external funding.
DATA AVAILABILITY STATEMENT
All relevant data are included in the paper or its Supplementary Information.
CONFLICT OF INTEREST
The authors declare there is no conflict.