Evaluation of rural water supply sustainable operation and management based on cyclic correction framework – a case study of Chongqing, China Open Access

Water is an essential substance for human survival and development (Du 2017; Ostad-Ali-Askar et al. 2018). The proposal of the ‘three rural’ policy of China has made rural life the center of attention (Dai et al. 2020). The rural water supply project is a water supply facility that provides clean water for rural residents and guarantees their normal life. Implementation of rural water supply projects is an important work fundamental to building a new socialist countryside and harmonious society (Li 2013; Song et al. 2020). During the 13th Five-Year Plan period, China has attached great importance to the construction and development of rural areas and implemented drinking water projects in a vigorous manner (Li 2020). In August 2020, the Ministry of Water Resources of China declared that all the rural drinking water safety problems of the population in poverty have been solved. As the rural water supply projects progress, how to make them sustainably operated and managed has become the focus and difficulty of the current and future work (Montgomery et al. 2009; Behailu et al. 2016).

The operation of rural water supply projects involves many aspects, such as management organization, operation system, and operation funds (Oyebode & Muzammil 2019; Cook & Newell 2020). Evaluating the effectiveness of operation and management of rural water supply projects has caught many researchers' attention. Most researchers focused on the establishment of an indicator system, considering different aspects of project operation. Zhang (2018) established an indicator system to evaluate the performance of sustainable operation and management of rural drinking water safety projects from the aspects of management, safety, organization, economy, sustainability, and satisfaction. Feng & Qi (2012) established an indicator system for performance evaluation from the aspects of organizational effectiveness, technical effectiveness, economic effectiveness, and safety effectiveness. Deng et al. (2017) established an evaluation indicator system of operation management of rural centralized water supply projects from the aspects of water production, water supply service, operation finance, and organization management. Hoko & Hertle (2006) selected sustainable indicators including the reliability of the system, human capacity development, institutional arrangements, and the impact of the project on rural livelihoods, to evaluate the sustainability of a rural water project. Different indicator systems focus on different aspects of the operation and management of rural water supply projects. It would be preferable to establish a more comprehensive indicator system, including all aspects of the operation and management of rural water supply projects so that the situation of project operation can be better evaluated.

The existing studies mainly use an individual evaluation method to evaluate the operation of rural water supply projects. Such individual evaluation methods include the grey cluster evaluation model (Zhang & Fan 2013), the analytic hierarchy process (AHP) (Singh & Sarkar 2019; Lin 2020), Technique for order preference by similarity to an ideal solution (TOPSIS) (Komasi & Sharghi 2017; Meshram et al. 2020), the structural equation model (SEM) (Masduqi et al. 2010) and so on. However, the results of different evaluation methods are often inconsistent, due to the applicability and stability of the individual evaluation methods themselves.

To solve the problem of inconsistent evaluation results of different individual evaluation methods and fully integrate the advantages of each individual evaluation method, a cyclic correction framework for the evaluation of sustainable operation and management of rural water supply projects is proposed. Several typical counties in Chongqing, China, are selected to evaluate the operational effectiveness of rural water supply projects, as well as the effectiveness of the proposed cyclic correction framework we constructed.

Chongqing is located in the east of Sichuan Basin and the Three Gorges Reservoir area in the upper reaches of the Yangtze River. The land spans between 105°17′–110°11′N and 28°10′–32°13′ N (Pu et al. 2013). Chongqing started the construction of rural water supply projects in the mid-1990s, which underwent three stages: rural drinking and poverty relief, rural drinking water safety, and consolidation and upgrading of rural drinking water safety. By the end of 2020, the total number of rural water supply projects in Chongqing had reached 29,2200, the rural centralized water supply rate had reached 87%, the tap water popularization rate had reached 80%, the water supply guarantee rate had reached 92.3%, and the rural water supply guarantee level continued to improve.

By integrating the characteristics of geographical location, topography climate condition, water resources condition, social and economic developments, and the layout of rural water supply projects, Chongqing City is divided into three parts, the main urban metropolitan part, the Three Gorges reservoir part in northeastern Chongqing, and the Wuling Mountain part in southeastern Chongqing (Figure 1). Districts in the main urban metropolitan part have relatively flat terrain, and their economic levels are higher. Districts in the Three Gorges reservoir part in northeastern Chongqing are all within the Three Gorges Reservoir Area, the topography of this part is mainly mountains with large fluctuation. The economic level of this part is lower than the main urban metropolitan part. Districts in the Wuling Mountain part in southeastern Chongqing are poverty-stricken parts with large areas and mountainous topography.

Limited by early low investment standards and technology level, and due to the special construction conditions, such as topography, unevenly distributed water sources and population, the rural water supply in Chongqing still needs improvement in ensuring water sources, supporting facility constructions, and dealing with engineering water shortage and seasonal water shortage. At the same time, it is also faced with a poor financial situation of water supply project operation, the non-standard management of small centralized water supply projects, and outdated mechanisms of operation and management of rural water supply projects.

To enhance the construction of fundamental rural water supply facilities, explore the long-term mechanisms and operation modes of project operation and management, and improve the standard of rural water safety, Chongqing selected 13 districts (Yubei, Banan, Rongchang, Tongliang, Liangping, Yunyang, Fengjie, Chengkou, Wuxi, Qianjiang, Youyang, Pengshui, and Shishu) (Figure 2) as pilot districts, to conduct pioneer work on rural drinking water safety to significantly improve the rural drinking water facilities, the investment and financing mechanisms, and the operation and management mechanism. Since the 13 pilot districts have conducted pioneer work on rural water supply projects, they obtained more experience and detailed statistics about projects' sustainable operation and management. So in this study, the 13 pilot districts are chosen as the study area, and the operation and management situations of rural water supply projects in these pilot districts are evaluated. The basic information of rural water supply projects in these 13 districts is shown in Table 1.

A proper evaluation indicator system that can capture the major factors affecting the operation and management of rural water supply projects is an important basis for any evaluation method. The selection of evaluation indicators should follow the principles of being scientific, practical, systematic, and comprehensive. The indexes should be selected one by one based on avoiding cross-meaning or correlation among indicators. It is important that all the indexes in the indicator system should reflect all aspects of rural water supply projects' operation and management. After the index selection is determined, the indicators are divided into internal and external indicators according to their characteristics. The internal indicators are used to reflect the operation of rural water supply projects themselves, and the external indicators are used to measure the external role. In this paper, an indicator system including seven categories is established. The seven categories are organizational management, engineering management, operation management, safety management, economic management, satisfaction, and sustainable development. Different indicators are then chosen according to the criteria for each category. The 36 overall indicators are shown in Table 2.

When using the indicator system to evaluate the sustainable operation of the rural water supply projects, the basic material is the numerical data for each indicator of each district. From Table 2 it can be seen that some indicators are objective, which means they can be calculated by the collected data, while the others are subjective, and they should be assigned a score by subjective judgment. To better collect the data on rural water supply project operation, and eliminate individual subjective judgment as much as possible, we set up five teams to conduct site surveys in the 13 pilot districts in Chongqing from July 16th to August 20th in 2019. After the Covid-19 pandemic, we again set up three teams to conduct supplementary surveys in the 13 pilot districts from December 14th to December 26th in 2020. Each team had three or four team members, all of whom were engineers with rich experience in rural water supply projects. For the site surveys, each team went to several typical rural water supply projects to investigate the management, operation, and benefits of each project. Then each team visited beneficiary households to investigate the satisfaction and participation levels. In addition, each team held a seminar at each district with the local government and Water Resources Bureau, to collect the basic data for the objective indicators. As for the subjective indicators, each team member gave scores for the district based on the site survey observation, and then average scores were calculated as the input data of such indicators.

Different evaluation methods offer distinct advantages and disadvantages, resulting in different outcomes for the same evaluation object. Three frequently used evaluation methods are chosen in this work: the fuzzy matter-element method, the fuzzy identification method, and the weighted sum method for multi-objective optimization.

This method is based on fuzzy matter-element analysis and the concept of Euclid approach degree. The sustainable operation and management degree of each district is regarded as a matter element. Based on each evaluation indicator and its fuzzy value, the fuzzy matter element is established. The Euclid approach degree is calculated with the established fuzzy matter element and the standard fuzzy matter element. Finally, based on the weights obtained before, the water scarcity rankings among all districts can be calculated. See Liu & Zou (2012) for the detailed calculation formula.

Rural water supply projects sustainable operation and management evaluation is a multi-objective and multi-layered process, which can be regarded as a fuzzy identification problem. Based on the indicator system, the measured indicator matrix and the standard matrix are established. The degree of membership is obtained according to fuzzy change. The objective function is to obtain the minimum for the quadratic sum of generally weighted distances, and the parameters are the Hamming distance and the Euclidean distance. The evaluation model of Rural water supply projects sustainable operation and management based on the fuzzy identification method is established, and then, the scores of Rural water supply projects sustainable operation and management among all districts are obtained. See Li (2021) for the detailed calculation formula.

To use this method, all indicators are normalized. The weighted average method is used to calculate the score of each district and all the scores are then converted to rankings. See Marler & Arora (2010) for the detailed calculation formula.

While all of the above-discussed evaluation methods measure the operation of rural water supply projects, their results are often inconsistent. To take advantage of each evaluation method and provide consistent information for decision-makers, we apply a cyclic correction framework, which can effectively optimize the combination of different evaluation methods. The cyclic correction framework is a progressive optimization algorithm with the intent to optimize different combinational estimation results (Wang et al. 2018). In this framework, the Kendall test is used to verify the consistency of the results from different individual methods, and the Spearman test is employed to verify the consistency of the results from combinational evaluation methods. Combinational evaluation is executed only when previous individual evaluations pass the Kendall correlation test. Three combinational evaluation methods, namely the Average method (Wang et al. 2003), Borda method (Du & Gao 2021), and the Copeland method (Favardin et al. 2002), are applied. This framework involves the following five steps, as shown in Figure 3.

Step 0 Establish an indicator system that captures all the important aspects of sustainable operation and management of rural water supply projects.

Step 1 Use the AHP method to calculate the weight of each indicator.

Step 2 Assess rural water supply projects sustainable operation and management by three individual evaluation methods, namely the fuzzy matter-element method, the fuzzy identification method, and the multi-objective linear weighting function method, and obtain the evaluation results from each method in terms of ranking.

Step 3 Verify the consistency of the evaluation from the individual method using the Kendall test. If the evaluation results from all methods are consistent, a further combinational evaluation will be performed. Otherwise, return to Step 2 to select a new individual evaluation method or return to Step 1 to recalculate the weight of each index.

Step 4 Three combinational evaluation methods, the Average method, Borda method, and Copeland method, are used to optimize the results from Step 3. The overall water scarcity level is also presented in the form of ranks.

Step 5 Verify the consistency of the three combining methods by the Spearman test. If the test result is positively correlated, these methods are consistent and then the combined evaluation results are accepted. Otherwise, return to Step 4 and select a new evaluation method or adjust the weight of each individual method.

The necessary data for this framework is a set of numerical values for all indicators in the indicator system established in Step 1 of all districts.

The underlying theory of the Kendall rank correlation coefficient Test is that for the same evaluated object, the results from the three individual methods should not vary greatly. Suppose there are m individual methods used to evaluate water scarcity degree in n districts, and q_ij represents the rank of the i-th district using the j-th method. The Kendall test is described as follows with coefficient W representing the correlation of different ranks.

Null hypothesis: All individual methods are uncorrelated.

Alternative hypothesis: All individual methods are correlated.

The critical value can be obtained for a given significance level α. When is less than the critical value at a given significance level α, the null hypothesis is rejected, indicating that all individual methods are positively correlated and the evaluation results are consistent.

The Spearman test is used to verify the consistency of the results from three combinational evaluation methods.

Null hypothesis: The methods p and q are uncorrelated.

Alternative hypothesis: The methods p and q are correlated.

For a given significance level α, the null hypothesis is rejected if ⁠, indicating methods p and q are positively correlated. If all methods are positively correlated, the results are consistent.

In this study, five experts in the field of rural water supply were invited to provide a judging matrix in order to conduct the AHP method. The basic information of the experts is shown in Table 3. With each judging matrix, a set of weights were calculated for the indicator system. The final weights of the indicator system were then obtained by taking the average of those of the five experts, and are shown in Table 4. As for the criterion layer, the ‘economic management’ layer had the highest weight of 0.2932, while the ‘mass satisfaction layer’ had the lowest weight of 0.0436. As for the indicator layer, since there were 30 indexes in the indicator system, so the weight of each index was no more than 0.15. The ‘establishment of management agency’ index under the organizational management layer had the highest weight of 0.1436. The ‘assurance level of operating funds’ under the economic management layer had the second-highest weight of 0.1382. Apart from these two indexes, the weight of other indexes was less than 0.1. The ‘publicity and training’ index under sustainable development had the lowest weight of 0.0022.

The evaluation results in terms of ranking from the fuzzy matter-element method, the fuzzy identification method, and the multi-objective linear weighting function method were shown in Table 5. The ranking results from Table 5 were used to draw Figure 4. The three coordinate planes in Figure 4 represented the ranks from three different methods. It can be seen that the ranking results for the same district were not consistent from the projection to the three coordinate planes. Comparing the results from the fuzzy matter-element method and the fuzzy identification method, 7 districts had different ranks. Comparing the results from the fuzzy matter-element method and the multi-objective linear weighting function method, 10 districts had different ranks. Comparing the results from the fuzzy identification method and the multi-objective linear weighting function method, 12 districts had different ranks. The inconsistency of evaluations revealed the shortcoming of using individual evaluation methods.

By calculating the Kendall rank correlation coefficient, we obtained the statistical magnitude χ²(13 − 1) = 32.88 at 5% of significance, larger than the critical value χ²_0.05/2(12) = 21.03. This indicated that all three individual methods were positively correlated and passed the Kendall rank correlation test without returning to Step 2. Therefore, in order to obtain more consistent and optimized results, a combinational evaluation was subsequently performed to synthesize the results from individual methods.

The three combinational evaluation methods showed the same ranking for each district, which means that the three methods were positively correlated and pass the Spearmen test. The evaluation rankings, therefore, were accepted as the final evaluation results for sustainable operation and management of rural water supply projects in each district (Table 6). To analyze the performance of each aspect of sustainable operation and management, this framework was then applied to each criterion layer of the indicator system (Figure 5).

A comparison of Tables 5 and 6 showed the cyclic correction framework outperforms the individual methods. Table 5 showed that the results of the three individual methods were inconsistent. Naturally, the three individual methods generate different results due to their different assumptions and methodologies. It is different to determine the best suitable method, as each method places emphasis on a different aspect. The cyclic correction framework can resolve this issue by combining different individual methods and finally providing a consistent result. From Table 6, it can be seen that the cyclic correction framework produces consistent results after combining the different evaluation results from the three individual methods using three combinational evaluation methods. Therefore, the cyclic correction framework is an effective way to integrate different methods and provide more reliable evaluation results.

In addition, not only can the cyclic correction framework be used in the rural water supply projects' operation and management evaluation, but it can also be applied in other situations, such as water scarcity evaluation and water resources capacity evaluation. The successful implementation of the method depends on a meaningful indicator system for which data are available.

According to the final rankings obtained from the cyclic correction framework, the rural water supply projects of the 13 pilot districts, from the highest-ranking to the lowest based on sustainable operation and management, are Rongchang > Yubei > Banan > Liangping = Tongliang > Qianjiang > Pengshui > Fengjie > Youyang > Yunyang > Wuxi > Shizhu > Chengkou. In order to analyze the spatial distribution of the rank, the rank of each district in the map was drawn, as shown in Figure 6. To further analyze the ranks and the underlying reasons, the three divided parts can be considered as one of the factors. It can be observed that districts in the main urban metropolitan area are ranked higher, which means that rural water supply projects in these districts are operated better, while the districts in the other two parts are ranked relatively low. This can be explained by two factors, natural environment, and economic development level. On one hand, the topographic features of the three parts are different. The main urban part has relatively flat terrain, so it is possible and suitable for these districts to build large-scale water supply projects (supporting more than 10,000 people) and urban-rural integrated pipe network extension projects. However, the other two parts are mountainous with large fluctuation, making them not suitable to build large-scale projects. So the main types of projects of these districts are small-scale centralized projects (supporting 20–10,000 people), and decentralized projects (supporting less than 20 people). On the other hand, the economic level of the main urban part is higher than the other two parts. A higher economic level means sufficient funding and better equipment. As a result, the operation and management of the projects in the main urban area are relatively more intelligent and specific, while for the other two areas, with less funding and more small-scale projects, it is more difficult to manage all the projects well, and the sustainable operation and management need to be improved in the future.

The weights of the indicator system can also show the important factors that affect the sustainable operation and management of rural water supply projects. As for the criterion layer, the criteria in the order of highest influence to the lowest on the sustainable operation and management of rural water supply projects are as follows: economic management > organizational management > operation management > engineering management > safety management > sustainable development > mass satisfaction. As for the indicator layer, the two indexes with the highest weights are ‘establishment of management agency’ and ‘assurance level of operating funds, which shows that the influences of organizational management and economic management on the sustainable operation of rural water supply projects are larger than those of the other criterion layer. The observation makes sense since the establishment of a management agency is a necessary condition to ensure the sustainable operation of the projects. If no management agency exists for a rural water supply project, no one has the obligation to manage the project, and then the project would end up unmanaged and unattended, failing to be sustainably operated. The establishment of a management agency alone is not enough: operating funds are also very important. For rural water supply projects in the mountainous area, the coverage area of the pipe network is relatively large, with a higher difference in elevation. Correspondingly, the risk of pipe network damage is higher, the operation and management of such projects are more challenging than those in the plain area. In addition, it is difficult to collect water fees for rural water supply projects. There are occasions when the collected water fee cannot cover the operating costs, so the government has to fill the gap. If the operating funds cannot be guaranteed, then the projects cannot be operated sustainably.

According to the final rankings of the criterion layer, for each criterion layer, there are more than one districts that share the same ranking, especially for the index of mass satisfaction, seven districts are ranked the same. This indicates that different districts may have the same performance in every aspect, and the gap between districts in all seven aspects is not large. As a result, we should concentrate on the districts that are ranked higher, and extend their valuable and useful experience and practice to the whole city. The results indicate that the top five districts are Rongchang, Yubei, Banan, Liangping, and Tongliang. Apart from Liangping, the other four districts all belong to the main urban part of Chongqing. The economic level of these four districts is more developed, and the scale of rural water supply projects and the degree of urban-rural integration is higher, which facilitates unified management of the project. In addition, the governments of these districts provided additional funds for the operation and maintenance of the projects. As for Liangping, although its economic level is not as high as the other districts, in order to better manage the rural water supply projects, the water resources bureau of Liangping founded a water supply company, with a water supply station in every town. All the rural water supply projects are uniformly managed by this company. Furthermore, the staff of the company take training lessons every two or three months, to learn the basic skills of operating the disinfection and purification equipment. This is a good experience that can be learned by other districts, especially those in the northeastern and southeastern parts of Chongqing.

This paper proposes a cyclic correction framework to evaluate the sustainable operation and management of rural water supply projects. The cyclic correction framework combined three individual methods (fuzzy matter-element method, fuzzy identification method, and multi-objective linear weighting function method) with three combinational methods (Average method, Borda method, and Copeland method) by using the Kendall test and Spearman test. This framework can effectively solve the inconsistency problem of different estimation methods. With two site surveys in 2019 and 2020, 13 pilot districts of Chongqing were chosen as the study area, and the sustainable operation and management of rural water supply projects were evaluated. A comprehensive indicator system including seven categories (organizational management, engineering management, operation management, safety management, economic management, satisfaction, and sustainable development) were established. The APH method was used to calculate the weight of each index. Two indexes with the highest weights are ‘establishment of management agency (0.1436)’ and ‘assurance level of operating funds (0.1382)’. Through the Kendall test and Spearman test, the cyclic correction framework greatly enhances the accuracy of the evaluation results. Compared with the traditional comprehensive evaluation model, systematic deviation and randomness error in the evaluation process can be effectively reduced, and the research conclusion is more reliable. According to the final ranks of the 13 pilot districts, the districts in the main urban metropolitan part, such as Rongchang, Yubei, Banan, Liangping, and Tongliang, are ranked higher; in other words, the rural water supply projects in these districts meet the requirements of sustainable operation management. On the contrary, the districts in the southeast and northeast of Chongqing are ranked relatively low, indicating that there is still a gap between the rural water supply projects and the requirements of sustainable operation and management in these areas. It is necessary to improve the level of sustainability of rural water supply projects. In the future, the government should enhance the ability of management agencies of rural water supply projects and increase funding for project operation and management.

This paper was supported by the Fundamental Research Funds for Central Public Welfare Research Institutes (grant No. CKSF 2017036/NS), and the Funds for the technology demonstration project of the Ministry of Water Resources (grant No. SF-2020004).

All relevant data are included in the paper or its Supplementary Information.