Building an indicator-based assessment framework for adaptive capacity of drainage systems in Beijing

Climate change has become a serious reality over recent decades with increasing attention directed toward its detrimental side effects. The IPCC's Fifth Assessment Report claims that the mean temperature of the land and ocean has increased 0.85 °C in the last 130 years, even the average temperature could be 1.5 °C higher than 100 years ago with the minimal carbon emission (Rockström et al. 2014). For the global water cycle, precipitation and evaporation patterns are also greatly modified by climate change, so either frequency or volume of extreme precipitation events will be inevitably augmented (Arnbjerg-Nielsen et al. 2013). In urban areas, changing precipitation will require amendments to the drainage networks highly relied upon by cities, which were designed based on past climate conditions and will not work properly in future (Kirshen et al. 2014). Hence, it is urgent to implement adaptive actions in response to climate change trends in urban drainage fields.

Although adaptation is a relatively new topic among climate change research compared to mitigation strategies, more organizations are realizing the necessity of studying adaptive capacity (Kutlu 2015). In terms of urban drainage systems, adaptation can be considered as the capabilities of socio-economic systems and ecosystems to exploit beneficial opportunities and moderate or avoid the harm of (extreme) precipitation. For example, several new planning patterns have focused on the environmental and social problems for traditional drainage systems under climate change (Che et al. 2009). First, Best Management Practices (BMPs) determined two category adaptations of rain-flood management, including engineering and non-engineering measures, to solve the problems faced by the urban water environment. To control the rainwater runoff from sources, Low Impact Development (LID) aims to keep the hydrology conditions of urban areas consistent with pre-development periods by using decentralized small-scale measures. Next, Sustainable Urban Drainage System (SUDS) take a holistic view of urban water systems to maintain a virtuous circle of rainwater. Similarly, the popular governance strategy of Integrated Urban Water Management (IUWM) emphasizes the interactions of components in urban water systems in order to maintain effective and safe water services. Overall, from rainfall discharging to water cycle management, combined physical and natural infrastructure, these concepts all belong to modern rain-flood management with consideration of adaptation to climate change.

Smit and Wandel state that adaptation to external stresses is the manifestation of adaptive capacity of a system (Smit & Wandel 2006). Thus, an important premise to carry out adaptation to heavy precipitation is to form an accurate understanding about the adaptive capacity of drainage systems. Over past decades, the vulnerability assessment of drainage systems has been paid enough attention compared to adaptive capacity (Shiau et al. 2012; Del et al. 2016). Fortunately, vulnerability assessment is a valuable reference for evaluating adaptive capacity because common points exist between these two concepts, where the latter is widely used as a criterion of the former. For example, the Fourth IPCC Assessment Report divided adaptive capacity into two categories in view of vulnerability assessments (Eriksen & Kelly 2007), including generic and specific capabilities. Moreover, considering the ambiguous constitution of adaptive capacity (Engle 2011), some researchers applied a framework of indicators to assess the adaptive capacity of water systems based on systematic thinking. For instance, Pandey et al. constructed a conceptual framework with an index, four parameters and seven indicators to assess the adaptive capacity of a water resource system in Nepal (Pandey et al. 2011). As a framework on climate governance, the Adaptive Capacity Wheel (ACW) was widely used to assess the inherent characteristics of institutions to enable the adaptive capacity of water safety and society (Gupta et al. 2010; van den Brink et al. 2011).

However, it is worth noting that few frameworks are specialized for assessing the adaptive capacity of drainage systems. Besides, each water system is unique and a framework developed at a specific location may not be suitable for other regions owing to different characteristics of society, economics, ecology and culture. Thus, the indicators that assess the adaptive capacity of a certain system should be area-specific (Carter et al. 2015). From the perspective of Beijing, an appropriate framework to assess adaptive capacity of drainage systems is yet to be developed and this study is the first step in that direction.

Considering the deficiencies in current systematic methods for constructing such frameworks, this study aims to develop an appropriate technique for building a representative indicator-based framework of adaptive capacity in Beijing's drainage systems. First, a framework with three levels is determined, extensively reviewing current assessments of adaptive capacity. Then, this paper discusses the specific process of the Delphi method used to select adaptive capacity indicators of drainage systems in Beijing. Finally, an indicator-based framework, with three levels was determined and an inherent linkage was found from the indicators of five dimensions that contributes to the adaptive capacity of drainage systems.

Since adaptive capacity may originate from several fields of socio-economic systems and ecosystems, an authoritative framework should contain various sources of adaptive capacity as much as possible. To illustrate adaptive capacity clearly, a three-level structure of assessment framework, including index, parameter, and indicator of adaptive capacity, was determined following the study of Nepal's water resources, where the interpretation of adaptive capacity is orderly and reasonable (Pandey et al. 2011).

First, the Adaptive Capacity Index (ACI) was identified based on the study objective to represent an integrated adaptive capacity of drainage systems, which assists researchers in determining the priority of policy innovations among various districts of Beijing. A larger ACI reflects a stronger adaptive capacity of drainage systems in response to climate change.

Next, the level of parameter aims to reflect the adaptive capacity of different dimensions in urban drainage systems. As an authoritative publication on adaptation assessment, the Third IPCC Assessment Report summarized the common characteristics of adaptive capacity based on research into sustainable development and disaster management, where the elements of adaptive capacity are divided into six parameters: economic resource, information and skills, technology, infrastructure, institutions, and equality (IPCC 2001). However, the dimension of equality in urban drainage system is usually related with other aspects to a great extent. For example, the distribution of various types of resources can be considered as a gauge of equality. In this study, five parameters are determined to assist researchers in comprehensively analyzing the distribution of adaptive capacity in a certain district.

This framework is shown in Figure 1, where the ACI is weighted using these five parameters to determine the integrated adaptive capacity and some indicators are affiliated to one of the five parameters.

In building the indicator-based framework to assess the adaptive capacity of drainage systems, the main assignment was to select valid indicators for qualifying adaptive capacity of different dimensions. Representative literature on water management for climate change was reviewed in the process of identifying primary indicators of adaptive capacity, which offers important references to justify this framework building in terms of the same or similar institutional background (Table 1). Meanwhile, some basic principles were adopted: that the indicators were easy to understand, available for data, sufficient to represent the characteristics of drainage systems in Beijing and applicable to numerous other areas. Therefore, 25 indicators (in the section of ‘Results and discussion’) were initially determined for further optimization in the Delphi process.

To determine adaptive capacity indicators, a Delphi technique was chosen for its significant merits and wide application. Unlike a conventional screening process, where a panel discusses and scores each indicator face to face, the Delphi process allows experts to assess the indicators anonymously to avoid the disturbance of group pressure (Clayton 1997). And many studies have successfully applied the Delphi method to establish indicator-based frameworks in various fields (Hyun et al. 2008; Moglia et al. 2009; Al-Jawhar & Rezouki 2012).

In this research, the Delphi process was executed to refine the primary 25 indicators until an expected consensus (shown as ‘Statistical analysis’) was reached. Gradually, the iterative nature of the procedure generates new information for experts during each round, enabling them to modify the judgement beyond their own subjective opinions. According to the Delphi idea, the focus should be on the opinion of group rather than individuals while analyzing the data. Thus, we hold that a consensus can be reached after all the Delphi rounds terminate as opposed to obtaining an agreement after only a certain number of rounds. For reaching the informed consensus among experts, all feedback of opinions is returned to the researchers, and an authoritative statistical tool is utilized to process the abundant data. Each step of the Delphi process is described in detail in the following sections.

Research that uses the Delphi process does not depend on a statistical sample that attempts to be representative of any population. The mechanism of having a group decision requires qualified experts who have deep understanding of the issues at hand (Veltri 1985). From this perspective, focusing on qualified experts rather than the size of the panel is a key to the process of expert selection. Based on this principle, three comprehensive criteria were considered by combining the professional knowledge and distribution of occupations of individual experts to guarantee eligible participants for the Delphi surveys:

• The experts participating in the Delphi process must have extensive working or research experience of drainage systems to offer authoritative responses.

• The scope of selected experts should be considered to obtain a full-scale evaluation of the study area. Geographically, participants should distribute evenly in study area so as to cognize the climate conditions of different districts.

• Functionally, experts ought to be adept in various fields of drainage systems, which include academic research, administrative agency, design, and construction.

With the consideration of the absence of experts during survey process, 14 experts were nominated, who represented a wide distribution in various districts of Beijing from three main fields of occupations: administration, service institutions, and scientific research, which guarantees that the research has representation from various sides.

The basic information of experts in the first and second rounds is summarized in Table 2.

Based on the selected 25 primary indicators and considering the expression of the experts' opinions, the Delphi survey was designed, seeking to obtain the maximum consensus on the opinions of the panel through a series of structured questionnaires. As the top priority of the Delphi technique, the initial structured questionnaire was divided into three parts, sections A, B and C.

Section A introduced the background of climate change, as well as the purpose of establishing an indicator-based assessment framework for adaptive capacity of drainage systems in response to climate change in Beijing. Section B included the participants' job descriptions, job titles, and working years that would be collected, as well as the requirements for participants. Section C presented a rating scale to quantify the assessment indicators.

When scoring on the questionnaire, experts determined the score of each indicator by their own methods, including model calculation, empirical analysis, etc. Thus, experts also needed to explain their degree of confidence in their judgement, where the authority of the panel (criteria of the judgement and degree of familiarity) was also judged and evaluated via self-assessment in order to confirm credibility of the results (Table 3). Simultaneously, the suggestions of experts were added as an efficient feedback to modify the indicators system at the end of the questionnaire.

Note that the later questionnaire needs to present the feedback of previous rounds for experts, where added, deleted, and modified indicators were presented to offer the latest information for experts' analyses. The questionnaires were delivered to the respondents via e-mail.

As the iterative feature of the Delphi method, all feedback will be presented to the researchers, where a rigorous mathematical statistics analysis will be applied to present the opinions on each indicator, such as mean value, satisfactory rate, coefficient of variation, and overall consensus of experts under circumstances of ‘no group pressure’. Table 4 shows the coefficients and calculation equations in this process of statistical analysis.

• Degree of participation. This aspect of statistical analysis is reflected by Response rate (P), which indicates the concerned extent of experts in reality.

• Degree of concentration. As a common but important feedback of the Delphi technique, the mean value (M_j) and satisfactory rate (K_j) of each indicator indicate the level of concentration of experts' opinions and the significance of indicator. A larger value of Mj indicates a more important indicator. Kj’ varies from 0 to 1 as a supplementary variable of Mj.

• Degree of concordance. This statistical analysis of the degree of concordance is reflected by two variables, coefficient of variation (Vj) and coefficient of concordance (W).

It is noted that the coefficient of variation (Vj) indicates fluctuation extent of the significance of indicator j, where a smaller value means a higher concordance for indicator j. However, because the data collected for nearly every study is unique, guidance that describes the level of variance that represents consensus is not available (Hallowell & Gambatese 2009). Thus, the degree of concordance could be only qualitatively evaluated by the variance, and the variance was developed into one of the screening conditions to optimize the framework.

For measuring the consensus among experts, additionally, the coefficient of concordance (W) was computed (the Kendall coefficient of concordance), which reflects the consistency of experts' scores and the reliability of the assessment results. A concordance coefficient of 1 indicates that all participants scored the assessment indicators identically, and a concordance coefficient of 0 means that all participants scored assessment indicators differently. Thus, the coefficient W was used to judge whether the consensus satisfied the requirements. During two or three rounds of the Delphi process, W usually fluctuates around 0.5, which explains a fine control of deviation (Wang & Si 2011). Given the number of indicators, this paper set a target consensus where W reached 0.4 or higher.

If the score of the indicator was higher than the cut-off value, the indicator was considered reasonable.

If the score of the indicator was lower than the cut-off value, the indicator was considered reasonable.

More importantly, the modified suggestions that were proposed by the panel beyond group pressure also could be a determining factor for selecting indicators to demonstrate the superiority of the Delphi technique.

In terms of different contributions of indicators to adaptive capacity, the method of proportion distribution was utilized to ensure the weights of each indicator after the final indicators were determined. Based on the final scores of each indicator, their weights were calculated. Then the parameters would be weighted by the affiliated indicators.

The statistical analysis work was accomplished using the software SPSS 20.0.

Two rounds of the Delphi process were executed to complete consensus building. Table 5 presents the degree of attention of the panel on this study for building indicator-based assessment framework of drainage systems in Beijing.

There are two main reasons for the absence of participants. Firstly, due to a long interval between rounds, some experts explained that their memory and concentration for the questionnaires had slackened, leading to unconvincing responses. Secondly, some experts were occupied by heavy workload, which decreased their concern and care for the questionnaire.

Nevertheless, according to relevant literature, the number of experts participating in the Delphi process is usually between 8 and 20 (Xu 2012). Thus, for 10 experts to participate in the group process is acceptable for statistical analysis in the second Delphi round. Meanwhile, even though the number of experts in the second Delphi round decreased, there were still respondents from all three domains, presenting an even distribution in terms of job title, which provided a representative group of experts. Overall, the mean coefficient of participation was 86%, which presents a satisfying result compared to other studies that have applied Delphi rounds (Al-Jawhar & Rezouki 2012). It can be concluded that the experts presented a relatively high concern on this research, and it is significant to study the indicator selection method to form a representative framework of adaptive capacity.

By analyzing the authority of the panel on Delphi rounds, a promising result with high credibility was obtained, as shown in Table 6.

From the result of round 1, the mean comprehensive value of 0.609 reveals that the degree of authority was upper-middle, which indicates that most of the selected experts had good knowledge of the questions on the questionnaire. The most authoritative parameter was Information and skills, which indicates that this field might have already been noticed for a time due to its importance to adaptive capacity.

In the second round, although the mean degree of familiarity somewhat decreased, both the mean criteria of judgment and mean comprehensive value show an upward trend compared to the first round. The overall trend indicates that the panel of experts may desire to strengthen their knowledge about the contribution factors of adaptive capacity of drainage systems during the Delphi process. From the improvements of these parameters, it can be concluded that the modifications based on experts' suggestions were positive, and judgments of the experts were reliable.

Overall, the authority of the panel performed well. And it proved that the Delphi process can absorb the consensus of experts to the greatest extent and tackle the divergence among them.

The score results, shown in Table 7, indicate the importance and divergence of each indicator that experts determined from theory and practical experiences in the first Delphi round. Based on the screening criteria (Table 8) and suggestions of experts (Table 9), nine indicators were deleted, where three were directly removed for not satisfying the three threshold values and six were deleted with consideration of both the threshold values and suggestions of experts. Meanwhile, to optimize the expression of indicators, several indicators that conveyed similar meaning were combined or rephrased. In the dimension of Information and skills, for instance, ‘The availability of information from government when heavy rainfall occurs’ aims to describe the capacity of government to manage precipitation information in detail, which was rephrased as ‘The capacity of government that guarantees timely and accurate notification before rainfall’. Similarly ‘The degree of participation by the public during heavy rainfall disasters’ and ‘The popularity of knowledge about heavy rainfall by the public’ represent the aspect of the public to exploit climatic information, which are combined with ‘The self-consciousness of the public about heavy rainfall disasters’. For the dimension of Infrastructure, ‘The number of shelter to heavy rainfall’ was rephrased and combined with ‘the number of retention facilities to rainfall’. ‘The rate of reaching the standard of the capacity of drainage in pipes’ was combined with the urban rain-flood management standards. Eight new indicators were added in this framework based on experts' suggestions.

Table 10 indicates the score results for each indicator in the second round. The main selection criteria are four principal categories for application combined with screening criteria (Table 8), so the indicators within each parameter are determined in such a way that they are easy to quantify for the availability of data, combined for simplification, and reflect the differentiation of regions and years. Thus, ‘The development level of urban’ was directly eliminated for dissatisfaction of the three threshold values, and the other 11 indicators were deleted based on discussions from the principles. For the final eight indicators (in the section of ‘Indicator-based framework’), seven of them had high mean values greater than 4.0 point, which proves that the selected indicators are efficient enough for the adaptive capacity of drainage system. Although the K of 1.1.2 indicator was zero, its coefficient of variation was 0.111 with a relatively low level. Based on the experts' perspectives, this indicator should be a representative indicator because of the uncertainty in climate change that requires various investments. The mean value of the 1.1.2 indicator was 3.80, which was higher than the threshold value and represented an upper-middle significance. Thus, the indicator ‘investment types of drainage systems to climate change’ can be considered reasonable.

After analyzing the data in Tables 8, a satisfying trend was found that the consensus improved during the two rounds of Delphi process. As a factor to reflect the degree of consensus of experts' decisions, the average coefficient of variation decreased from 0.244 (SD = 0.123) in the first round to 0.153 (SD = 0.079) in the second round. More importantly, the mean score of importance increased from 4.04 (SD = 0.6) to 4.28 (SD = 0.5) and the mean rate of satisfaction ascended from 44.57% (SD = 0.28) to 46.00% (SD = 0.29). From the improved data, we can conclude that the opinions of experts tend to be more consistent through anonymous negotiations of the Delphi method. Meanwhile, the level of significance of consensus between rounds 1 and 2 was examined using the T test for significance. The results indicate that the coefficient of variation between these two rounds was significantly different below a confidence level of 95% (P = 0.007 < 0.05), in other words, the consensus among the panel and applicability of framework are improved significantly through the Delphi process.

In order to obtain the consistency of experts' opinions, the Kendall coefficient of concordance (W) was calculated to obtain the consistency of experts' opinions using SPSS. In the first Delphi process, W was 0.368, and the value, which was higher than the threshold value of 36.42, was 123.54. Although the level of significance (P) was less than 0.01, which represents a relatively fine consistency from a statistical standpoint, the coefficient W did not reach the target consensus (0.4). Thus, another Delphi round was executed, where the coefficient of concordance was determined to be 0.410, and the value, which was higher than the threshold value (30.14), was 77.81 with P < 0.05.

From Table 11, it can be seen that the coefficient of concordance increased from 0.368 to 0.410, which explains that the consensus of our study improved over successive rounds. Moreover, the coefficient of concordance, W, reached the target consensus, resulting in the termination of the Delphi process with a good control of deviation. Further, P values below 0.01 in both of the rounds indicate that there was good consistency of responses from experts and that our results have a high confidence level. It is concluded that it is an effective method to establishing consensus using the Delphi technique to process opinions among various experts.

During the indicator selection process for building the framework, the initial 25 indicators were modified into 20 indicators, which were further screened into eight refined indicators to reflect the attention degree of each dimension of adaptive capacity (Table 12). It should also be noted that there was high consistency among the final 8 indicators, where the coefficient of concordance (W) was 0.480 with P < 0.01 below a confidence level of 95%.

According to the indicator-based assessment framework, the composition of the adaptive capacity of drainage system to climate change can be explained from three perspectives, including ACI, parameter value and indicator value.

Three relatively important domains of the drainage system were determined based on the weight they accounted for, including Economic resource (0.2290), Infrastructure (0.2346), and Institutions (0.2738). Because the parameter institutions accounted for the most weight, some effective management measures in this dimension can be considered as key factors to respond to climate change, which implies the necessity for an organizational form to face various climate change impacts.

Eight indicators of adaptive capacity have been determined from the five dimensions of a synergistic state, as mentioned above.

• Proportion of investment of urban drainage systems. This indicator reflects the economic strength of a district. An abundant investment in drainage systems is the basis for guaranteeing the normal functioning of all aspects of the drainage system in response to climate change.

• Investment types of drainage systems to climate change. This indicator indicates that climate change involves various aspects, so comprehensive investments should be completed to make it easier to ensure the adaptive capacity is equal to development and recovery after disasters.

• The capacity of government to guarantee timely and accurate notification when heavy rainfall is going to occur. The timely popularity of information can effectively reduce the risk that residents are exposed to the risk of flood disasters. And the public sector plays a pivotal role in managing relevant information on precipitation.

• The knowledge and skill level of employees in drainage systems. As an important technical reserve, professional staff who serve drainage systems play a key role in the construction of drainage facilities and emergencies.

• The number of waterlogged places on the road after rainfall. This increases the security risks for public traffic when heavy rain occurs, and reflects the completeness of drainage infrastructure (drainage pipe network, pumping stations, etc.). It should be noted that it is an indicator of the negative aspects that reduce the adaptive capacity of drainage systems.

• The rate of hardening of urban ground. If ‘high rate of hardening of urban ground’ exists, rain cannot easily permeate the ground and combine with groundwater, which obviously imposes greater pressure on draining rainfall via drainage systems. It is an indicator of the negative influences that reduce the adaptive capacity of drainage systems.

• Form contingency plan to heavy rainfall. When a city encounters extreme events like a rainstorm, one of the tasks of the water department is to coordinate the relevant stakeholders to carry out post-disaster response work in an orderly manner. Thus, formulating an effective contingency plan could dramatically reduce the loss of property and life.

• The starting time of contingency plan to heavy rainfall. This indicator reflects the executive force of institutions, which is a key factor in guaranteeing timely and targeted rescue procedures. Owing to the possibility of urban flooding causing serious risk to life and property within a short time, the shorter the start-up time of the plan, the more adaptive capacity an institution can apply. This is also an indicator of the negative aspects.

This work proposes a representative assessment framework composed of three levels to reflect the adaptive capacity of drainage systems in Beijing. In this framework, the ACI is an aggregate of five parameters, where each is an aggregate of one or more adaptive capacity indicators that were screened by the Delphi method to maximize consensus among various decisions of experts.

To take full advantage of the merits of anonymous discussion in the Delphi technique, both the scores of adaptive capacity indicators and added suggestions from the panel without any group pressure were analyzed. The former indicates the significance of each indicator from three aspects, which are importance score, rate of satisfaction, and coefficient of variation, while the latter can be the imperative criteria to judge the controversial indicators. In the Delphi process, the mean coefficient of variation decreased from 0.244 in the first round to 0.153 in the second round. Meanwhile, the mean score of importance increased from 4.04 to 4.28, and the mean rate of satisfaction varied from 44.57% to 46.00%. Eventually, eight indicators were chosen to assess the adaptive capacity of drainage in response to climate change in Beijing. From the improved data, it is concluded that the opinions of experts tended to be uniform through anonymous negotiations of the Delphi process and the Delphi method is a suitable technique for forming objective opinions and consensus in relatively subjective environments.

The final eight indicators for assessing the adaptive capacity of drainage systems can be found in Table 12. In total, these eight indicators are representative of the five fields of adaptive capacity in drainage systems, which indicates that the construction of adaptive capacity requires a systematic engineering. According to the composition of this proposed framework, an inherent linkage can be found from the weights of five parameters that contributes to the adaptive capacity of drainage systems. The Economic resource parameter can be considered as a basic and crucial guarantee for adaptive capacity in view of the necessity of financial support. Based on abundant funds, local government can be equipped with a more complete infrastructure to address the threats of climate change. However, Infrastructure is vulnerable to extreme climatic events and effective guideline is necessary for aiding higher adaptive capacity. For example, administrative departments must implement timely response to any climatic events. Meanwhile, the executive force of Institutions is necessary to guarantee that a project could be accurately implemented in a timely manner. Furthermore, the parameter Technology and information should be a significant complement to Institutions. Technology, as a reserve strength, can ensure that present and future extreme events are coped with reasonably. Meanwhile, Information and skills could enhance the modernization of monitoring, managing, and replying, which undoubtedly saves considerable manpower and resources when responding to climate change.

In summary, the indicator-based assessment framework represents adaptive capacity from a full-dimension perspective, which offers a fundamental framework to comprehend the adaptive capacity of drainage systems in Beijing. More importantly, the methodology applied in this study may serve as a guideline for constructing relevant assessment frameworks of adaptive capacity in urban water systems.

Some deficiencies were found while the Delphi method was carried out. One concern is that the long time interval for responding may affect the concentration and memory of experts. A suitable time to perform the Delphi method deserves to be studied to maintain the enthusiasm for participants. As for the second deficiency, the abundant data, which originates from the experts' suggestions during the Delphi rounds, brings about complicated analysis processes for researchers. Thus, future research should explore a reasonable number of participants to be used in a Delphi process in terms of a certain study area.

It is worth noting that this framework will facilitate the assessment practices of adaptive capacity. Further work can be done to apply the framework to assess the adaptive capacity, with the case of the Beijing drainage system at district level.

The author would like to acknowledge Beijing Water Authority and the funding by Clean Development Mechanism Grant in China [grant number 2013027].