Performance evaluation for urban water supply services in China Free

Since the 1990s, urban water supply performance evaluation has gradually become one of the hot issues in the field of water supply management. In the past 30 years, the International Water Association (IWA) (Alegre et al. 2006; Kun et al. 2007), Office of Water Services (OFWAT) (Li & Han 2012) and Portugal's ERSAR have successively established corresponding water supply performance evaluation systems to analyze the management performance of water companies and achieved good results (Su & He 2014; Cabrera and Cabrera 2016).

In 2002, China began to promote the market-oriented reform of urban water supply, and has made significant progress in water supply services and operation management since then. However, a unified performance evaluation system for water supply has not been formed yet. Most systems of performance management are limited to internal water companies, in order to meet their own management needs. Due to the inconsistency of indicator definitions and standards, it is difficult to promote the application on a large scale at the water industry level.

In recent years, the China National Water Pollution Control and Treatment Major Project has successively established research subjects on performance evaluation of water supply during the ‘Eleventh Five-Year Plan’ and ‘Twelfth Five-Year Plan’ period, to strengthen industry supervision and improve industry efficiency. These research subjects incorporated international advanced experience and carried out in-depth research on indicator systems, evaluation methods and the establishment of a management mechanism based on China's national conditions.

This paper constructed a set of performance indicators system of water supply, combining quantitative indicator evaluation with qualitative evaluation of good practices to form a comprehensive and systematic performance evaluation method system, which was applied in a typical region. The study will lay a foundation for the promotion and application of performance evaluation in China's water supply industry.

Water companies all over the country have various differences, such as water source conditions, pipe network configuration, local economic development level and so on. Therefore, it is difficult to carry out benchmarking for the water industry. How to set reasonable performance indicators to make the horizontal performance comparison of water companies under different backgrounds is the primary problem to be considered in performance evaluation work. In addition, data is the basis of performance evaluation. Without high-quality data input, the results of performance evaluation will lose their significance. How to define the quality of data collection and evaluate the reliability of data collection are always difficult for performance evaluation.

Performance evaluation is only a tool itself and its main purpose is to promote management performance and improve the service level of the water industry. Therefore, it should be considered whether it can promote the performance management level of water companies participating in the evaluation, when the whole performance evaluation system is designed.

Referring to the performance evaluation indicator system of the IWA and based on China's actual national conditions, the water supply performance evaluation indicator system is divided into six categories: service, operation, resources, assets, finance and personnel (Han et al. 2016). It evaluates the management levels of water companies in these six different dimensions systematically and comprehensively. It can change the common situation of ‘emphasizing service and operation while neglecting finance and manpower’ in the present conditions of water industry evaluation. It is more instructive for the management of water companies, and also helpful for grasping some common problems faced by the whole water supply industry.

The selection of quantitative performance indicators follows the principles of reliability, sufficiency, availability and minimization. After repeated screening and in-depth discussion with domestic and foreign experts, 23 performance indicators are set under the six performance evaluation categories.

• (1)

  The service category includes four indicators, which are the call-center connection rate, the timely rate of complaint handling, the timely rate of pipe network repair and the comprehensive satisfaction rate of user service. It covers the four aspects of customer service management, complaint handling efficiency, leak repair efficiency and comprehensive service evaluation.

• (2)

  The operation category includes five indicators, which are the qualified rate of water quality, qualified rate of pipe network service pressure, unit power consumption of water distribution, water loss rate, and recovery rate of water fee. It covers the five aspects of water quality management, water pressure management, energy consumption management, production and water loss control and water fee recovery.

• (3)

  The asset category includes three indicators, which are the utilization rate of water plant capacity, the renewal rate of large-diameter pipelines and the water storage ratio of the water distribution system. Its aim is to evaluate the asset management capacity of the water plant and the pipe network and the anti-risk of the water supply system.

• (4)

  The resource category includes three indicators, which are the utilization rate of water resource, self-use water rate of water plants and leakage rate. Its aim is to evaluate the water resource utilization level of water companies from the perspective of water conservation.

• (5)

  The finance category includes five indicators, which are the profit margin of the water selling business, the profit margin of the main business, the asset–liability ratio, return on assets and the operating cost of water selling. Its aim is to evaluate the financial status of water companies.

• (6)

  The personnel category includes three indicators, which are per capita daily water volume, the ratio of personnel with a college degree or above, and the ratio of professional and technical personnel. Its aim is to evaluate the human resource status of water companies.

Most of the selected quantitative indicators are common indicators used widely in the water industry, which can quantify the management level of water companies to some extent.

For each type of performance evaluation, referring to international experience (Krause et al. 2015), a series of good practices of water companies with excellent operation and management level are selected as qualitative evaluation criteria. The score of water companies in this part depends on the number of good practices completed in this category. More than 100 questions are included in the qualitative indicator system. The water company being evaluated should answer ‘yes’ or ‘no’ to each question and provide corresponding supporting documents.

The categories and related sub-items are shown in Table 1.

Take ‘customer complaints’ as an example, shown in Table 2.

The selection of qualitative indicators integrates the good management practices of many excellent water companies. The qualitative evaluation is the process of benchmarking management with excellent water companies. Through this horizontal comparison, the reason for high or low quantitative scores can be revealed to a certain extent. Water companies can plan their own improvement goals and strategies against the gap between good practices and their own operating practices, which can be used as a guide to lead the progress of companies.

In addition, the results of qualitative evaluation reflect the internal management logic of water companies; if a certain water company gets a high quantitative score and a low qualitative score, this indicates that it is difficult to maintain good performance management in the future. Therefore, the evaluation of qualitative indicators can also reflect the sustainability of water companies’ development. Quantitative indicators are more concerned with the current performance of the water company, but a company that performs well but lacks good practices may face the problem of how to achieve sustainable development, while another company that performs poorly but has good management must have great improvement in the future.

The total score of the water supply performance evaluation is 100 points. The weights of different categories are set through the improved AHP (Analytic Hierarchy Process) method, which includes: (1) determining the weights from different experts based on group decision theory; (2) establishing a hierarchical structure model for water supply performance evaluation; (3) inviting experts to fill out questionnaires; (4) determining the weights of every indicator, combining the group judgment matrix results and expert weight coefficients.

There are different weights for quantitative assessment and qualitative assessment of good practices in each category, as shown in Table 3. Different indicators in specific categories also have corresponding weight settings.

For quantitative indicators, the design of the standardized equation should be carried out according to the setting of the reference value, so as to convert the indicator value into a score value. Figure 1 shows the standardizing process of the ‘complaint feedback timely rate’ indicator. A 90% indicator value yields a normalized value of 0, whereas 100% compliance yields the maximum score of 100.

The importance of data quality to water supply performance evaluation is self-evident. In order to evaluate effectively the quality of the data reported by water companies, confidence evaluation units were set up in the data collection forms of quantitative data and qualitative data collection forms. First, the water companies should fill in the data by themselves. During the field investigation, the expert group verified the sources of quantitative and qualitative data carefully to determine the confidence level of all kinds of data eventually.

This study selected the regional demonstration of water supply performance evaluation in Jiangsu Province, China. Jiangsu Province is an economically developed province in the east of China, and the urbanization process is obviously faster than for the whole country. By the end of 2016, 161 public water supply plants had been built in Jiangsu Province, with a total water supply capacity of 28 million m³/d. To carry out regional water supply performance evaluation research in Jiangsu Province is to improve the level of water supply performance management in Jiangsu Province, and lay a foundation for nationwide promotion and application.

In the first stage of water supply performance evaluation, four demonstration water companies in Jiangsu Province were selected according to the differences of type of ownership, water supply capacity, company size, treatment process and so on. As shown in Table 4, they included the two categories of state-owned water companies and market-oriented investment water companies, respectively. The water supply capacity ranged from 600,000 to 3 million m³/d, and the company size ranged from 600 to 3,000 employees.

The evaluation work was carried out in 2017. It included investigating the current performance conditions of demonstration water companies, conducting performance evaluation and optimizing the whole performance evaluation methodology. The complete evaluation reports were given to the demonstration water companies finally, including some professional performance improvement suggestions.

The evaluation work was supported strongly by the demonstration water companies. The comparison of various evaluations of the four water companies is shown in Figure 2. On the whole, all demonstration water companies have good management foundations, and all kinds of performance management work are standardized and in place, which reflects the overall good level of water supply management in Jiangsu Province. For example, each demonstration water company has made careful management measures for water loss control and the water loss rate is lower than the national water industry benchmark.

As can be seen from Figure 2, each demonstration water company pays special attention to the management of service and operation, and pays insufficient attention to the management of resources and finance. This may be due to the lack of multidimensional performance evaluation of the water industry.

This study provided a detailed performance evaluation report for Jiangsu Provincial Department of Housing and Urban-Rural Development, which summed up the common problems in the water industry, and made targeted recommendations (Han et al. 2019). The study suggests that all demonstration water companies in Jiangsu Province should continue to promote relevant work in the following aspects to promote continual improvement of water supply service:

• (1)

  Promote the third-party evaluation of the survey on customer service satisfaction of water supply service, so as to strengthen the customer service awareness of water enterprises.

• (2)

  From the perspective of water resources protection, it is suggested to strengthen the measurement of raw water, including installing flowmeters for raw water.

• (3)

  It is suggested to optimize the water tariff mechanism, pay attention to the current situation of general losses in the main business of each water company, and give reasonable guidance.

• (4)

  It is suggested to promote the top-level design of a water supply information platform to lay a foundation for future smart water.

• (1)

  Based on China's national conditions, this paper combined quantitative indicators with good practices and the introduction and establishment of a relatively complete water supply performance evaluation method system, which will change the common situation of ‘emphasizing service and operation, while neglecting finance and manpower’ in the previous industry evaluation, and is more instructive for the water industry.

• (2)

  Regional demonstration application has been carried out in four water companies in Jiangsu Province, and good results have been achieved. In the future, the water supply performance evaluation will be carried out continually in Jiangsu Province, which will lay a foundation for comprehensive application in the whole country.

All relevant data are included in the paper or its Supplementary Information.