Abstract

In urban areas, the increasing frequency of heavy precipitation caused by climate change has generated huge pressure on drainage systems, which could result in severe environmental or social issues if not adapted to. A critical step to enhance the adaptive capacity of drainage systems is to build an authoritative indicator-based framework for assessing the adaptive capacity. An assessment framework with three levels, including an index (Adaptive Capacity Index), five parameters (Economic resource, Information and skills, Technology, Infrastructure, and Institutions) and several indicators, was established after literature review and expert consultation. The Delphi method was applied to optimize indicators in order to form the applicable indicator-based assessment framework. Through two rounds of Delphi process, a consensus among experts was reached and was checked for consistency by statistical analysis. Eventually, eight indicators of adaptive capacity were determined with high consistency and an inherent linkage was revealed from the five dimensions (parameters) that contribute to the adaptive capacity of drainage systems. Overall, the construction of adaptive capacity of drainage systems is a systematic engineering, and the Delphi method proved to be an appropriate technique for framework building.

INTRODUCTION

Climate change has become a serious reality over recent decades with increasing attention directed toward its detrimental side effects. The IPCC's Fifth Assessment Report claims that the mean temperature of the land and ocean has increased 0.85 °C in the last 130 years, even the average temperature could be 1.5 °C higher than 100 years ago with the minimal carbon emission (Rockström et al. 2014). For the global water cycle, precipitation and evaporation patterns are also greatly modified by climate change, so either frequency or volume of extreme precipitation events will be inevitably augmented (Arnbjerg-Nielsen et al. 2013). In urban areas, changing precipitation will require amendments to the drainage networks highly relied upon by cities, which were designed based on past climate conditions and will not work properly in future (Kirshen et al. 2014). Hence, it is urgent to implement adaptive actions in response to climate change trends in urban drainage fields.

Although adaptation is a relatively new topic among climate change research compared to mitigation strategies, more organizations are realizing the necessity of studying adaptive capacity (Kutlu 2015). In terms of urban drainage systems, adaptation can be considered as the capabilities of socio-economic systems and ecosystems to exploit beneficial opportunities and moderate or avoid the harm of (extreme) precipitation. For example, several new planning patterns have focused on the environmental and social problems for traditional drainage systems under climate change (Che et al. 2009). First, Best Management Practices (BMPs) determined two category adaptations of rain-flood management, including engineering and non-engineering measures, to solve the problems faced by the urban water environment. To control the rainwater runoff from sources, Low Impact Development (LID) aims to keep the hydrology conditions of urban areas consistent with pre-development periods by using decentralized small-scale measures. Next, Sustainable Urban Drainage System (SUDS) take a holistic view of urban water systems to maintain a virtuous circle of rainwater. Similarly, the popular governance strategy of Integrated Urban Water Management (IUWM) emphasizes the interactions of components in urban water systems in order to maintain effective and safe water services. Overall, from rainfall discharging to water cycle management, combined physical and natural infrastructure, these concepts all belong to modern rain-flood management with consideration of adaptation to climate change.

Smit and Wandel state that adaptation to external stresses is the manifestation of adaptive capacity of a system (Smit & Wandel 2006). Thus, an important premise to carry out adaptation to heavy precipitation is to form an accurate understanding about the adaptive capacity of drainage systems. Over past decades, the vulnerability assessment of drainage systems has been paid enough attention compared to adaptive capacity (Shiau et al. 2012; Del et al. 2016). Fortunately, vulnerability assessment is a valuable reference for evaluating adaptive capacity because common points exist between these two concepts, where the latter is widely used as a criterion of the former. For example, the Fourth IPCC Assessment Report divided adaptive capacity into two categories in view of vulnerability assessments (Eriksen & Kelly 2007), including generic and specific capabilities. Moreover, considering the ambiguous constitution of adaptive capacity (Engle 2011), some researchers applied a framework of indicators to assess the adaptive capacity of water systems based on systematic thinking. For instance, Pandey et al. constructed a conceptual framework with an index, four parameters and seven indicators to assess the adaptive capacity of a water resource system in Nepal (Pandey et al. 2011). As a framework on climate governance, the Adaptive Capacity Wheel (ACW) was widely used to assess the inherent characteristics of institutions to enable the adaptive capacity of water safety and society (Gupta et al. 2010; van den Brink et al. 2011).

However, it is worth noting that few frameworks are specialized for assessing the adaptive capacity of drainage systems. Besides, each water system is unique and a framework developed at a specific location may not be suitable for other regions owing to different characteristics of society, economics, ecology and culture. Thus, the indicators that assess the adaptive capacity of a certain system should be area-specific (Carter et al. 2015). From the perspective of Beijing, an appropriate framework to assess adaptive capacity of drainage systems is yet to be developed and this study is the first step in that direction.

Considering the deficiencies in current systematic methods for constructing such frameworks, this study aims to develop an appropriate technique for building a representative indicator-based framework of adaptive capacity in Beijing's drainage systems. First, a framework with three levels is determined, extensively reviewing current assessments of adaptive capacity. Then, this paper discusses the specific process of the Delphi method used to select adaptive capacity indicators of drainage systems in Beijing. Finally, an indicator-based framework, with three levels was determined and an inherent linkage was found from the indicators of five dimensions that contributes to the adaptive capacity of drainage systems.

CONCEPTUAL FRAMEWORK AND METHODOLOGY

Conceptual framework

Since adaptive capacity may originate from several fields of socio-economic systems and ecosystems, an authoritative framework should contain various sources of adaptive capacity as much as possible. To illustrate adaptive capacity clearly, a three-level structure of assessment framework, including index, parameter, and indicator of adaptive capacity, was determined following the study of Nepal's water resources, where the interpretation of adaptive capacity is orderly and reasonable (Pandey et al. 2011).

First, the Adaptive Capacity Index (ACI) was identified based on the study objective to represent an integrated adaptive capacity of drainage systems, which assists researchers in determining the priority of policy innovations among various districts of Beijing. A larger ACI reflects a stronger adaptive capacity of drainage systems in response to climate change.

Next, the level of parameter aims to reflect the adaptive capacity of different dimensions in urban drainage systems. As an authoritative publication on adaptation assessment, the Third IPCC Assessment Report summarized the common characteristics of adaptive capacity based on research into sustainable development and disaster management, where the elements of adaptive capacity are divided into six parameters: economic resource, information and skills, technology, infrastructure, institutions, and equality (IPCC 2001). However, the dimension of equality in urban drainage system is usually related with other aspects to a great extent. For example, the distribution of various types of resources can be considered as a gauge of equality. In this study, five parameters are determined to assist researchers in comprehensively analyzing the distribution of adaptive capacity in a certain district.

This framework is shown in Figure 1, where the ACI is weighted using these five parameters to determine the integrated adaptive capacity and some indicators are affiliated to one of the five parameters.

Figure 1

Framework of indicators to assess adaptive capacity of drainage systems.

Figure 1

Framework of indicators to assess adaptive capacity of drainage systems.

In building the indicator-based framework to assess the adaptive capacity of drainage systems, the main assignment was to select valid indicators for qualifying adaptive capacity of different dimensions. Representative literature on water management for climate change was reviewed in the process of identifying primary indicators of adaptive capacity, which offers important references to justify this framework building in terms of the same or similar institutional background (Table 1). Meanwhile, some basic principles were adopted: that the indicators were easy to understand, available for data, sufficient to represent the characteristics of drainage systems in Beijing and applicable to numerous other areas. Therefore, 25 indicators (in the section of ‘Results and discussion’) were initially determined for further optimization in the Delphi process.

Table 1

Determinants of adaptive capacity in water systems

  Study scope Elements of adaptive capacity Source 
Capacity for flooding prevention in the Rhine Delta Availability of technological options and resources;
structure of critical institutions;
stock of human capital and social capital;
access to and manage information;
awareness of the implications of climate change 
Yohe (2002)  
Factors influencing adaptive capacity of drainage pipe network Technology; infrastructure; management, etc. Semadeni-Davis (2004)  
Adaptive ability for communities in north-western Victoria Economic; socio–cultural; infrastructure–institutional Sietchiping (2006)  
Adaptive capacity of water management to climate change Education and wealth resources; knowledge and information; participation; equality of decision; experience, etc. Engle & Lemos (2010)  
Capacity of water safety institutions to flood risk Adaptive Capacity Wheel; variety; learning; room for autonomous change; leadership; resources; fair governance van den Brink et al. (2011)  
Resilience of urban drainage system under various rainfall events Response indicators; population density, traffic density, land use recovery indicators: relief availability, budget allocation, inundation subsidence Birgani & Yazdandoost (2016)  
  Study scope Elements of adaptive capacity Source 
Capacity for flooding prevention in the Rhine Delta Availability of technological options and resources;
structure of critical institutions;
stock of human capital and social capital;
access to and manage information;
awareness of the implications of climate change 
Yohe (2002)  
Factors influencing adaptive capacity of drainage pipe network Technology; infrastructure; management, etc. Semadeni-Davis (2004)  
Adaptive ability for communities in north-western Victoria Economic; socio–cultural; infrastructure–institutional Sietchiping (2006)  
Adaptive capacity of water management to climate change Education and wealth resources; knowledge and information; participation; equality of decision; experience, etc. Engle & Lemos (2010)  
Capacity of water safety institutions to flood risk Adaptive Capacity Wheel; variety; learning; room for autonomous change; leadership; resources; fair governance van den Brink et al. (2011)  
Resilience of urban drainage system under various rainfall events Response indicators; population density, traffic density, land use recovery indicators: relief availability, budget allocation, inundation subsidence Birgani & Yazdandoost (2016)  

METHODOLOGY: THE DELPHI METHOD

To determine adaptive capacity indicators, a Delphi technique was chosen for its significant merits and wide application. Unlike a conventional screening process, where a panel discusses and scores each indicator face to face, the Delphi process allows experts to assess the indicators anonymously to avoid the disturbance of group pressure (Clayton 1997). And many studies have successfully applied the Delphi method to establish indicator-based frameworks in various fields (Hyun et al. 2008; Moglia et al. 2009; Al-Jawhar & Rezouki 2012).

In this research, the Delphi process was executed to refine the primary 25 indicators until an expected consensus (shown as ‘Statistical analysis’) was reached. Gradually, the iterative nature of the procedure generates new information for experts during each round, enabling them to modify the judgement beyond their own subjective opinions. According to the Delphi idea, the focus should be on the opinion of group rather than individuals while analyzing the data. Thus, we hold that a consensus can be reached after all the Delphi rounds terminate as opposed to obtaining an agreement after only a certain number of rounds. For reaching the informed consensus among experts, all feedback of opinions is returned to the researchers, and an authoritative statistical tool is utilized to process the abundant data. Each step of the Delphi process is described in detail in the following sections.

Selection of the expert panel

Research that uses the Delphi process does not depend on a statistical sample that attempts to be representative of any population. The mechanism of having a group decision requires qualified experts who have deep understanding of the issues at hand (Veltri 1985). From this perspective, focusing on qualified experts rather than the size of the panel is a key to the process of expert selection. Based on this principle, three comprehensive criteria were considered by combining the professional knowledge and distribution of occupations of individual experts to guarantee eligible participants for the Delphi surveys:

  • The experts participating in the Delphi process must have extensive working or research experience of drainage systems to offer authoritative responses.

  • The scope of selected experts should be considered to obtain a full-scale evaluation of the study area. Geographically, participants should distribute evenly in study area so as to cognize the climate conditions of different districts.

  • Functionally, experts ought to be adept in various fields of drainage systems, which include academic research, administrative agency, design, and construction.

With the consideration of the absence of experts during survey process, 14 experts were nominated, who represented a wide distribution in various districts of Beijing from three main fields of occupations: administration, service institutions, and scientific research, which guarantees that the research has representation from various sides.

The basic information of experts in the first and second rounds is summarized in Table 2.

Table 2

Basic information of experts in Delphi round

Name of job
 
Job title
 
Working years
 
Ages
 
Groups Round 1 Round 2 Groups Round 1 Round 2 Groups Round 1 Round 2 Groups Round 1 Round 2 
Administration agency of urban drainage systems Senior 5–10 30–39 
Operating and management departments Associate 11–20 40–49 
Designing and research departments Middle 21–30 50–59 
Education and research departments          
Other fields of drainage systems          
Total 14 10  14 10  14 10  14 10 
Name of job
 
Job title
 
Working years
 
Ages
 
Groups Round 1 Round 2 Groups Round 1 Round 2 Groups Round 1 Round 2 Groups Round 1 Round 2 
Administration agency of urban drainage systems Senior 5–10 30–39 
Operating and management departments Associate 11–20 40–49 
Designing and research departments Middle 21–30 50–59 
Education and research departments          
Other fields of drainage systems          
Total 14 10  14 10  14 10  14 10 

Formulation of Delphi questionnaire

Based on the selected 25 primary indicators and considering the expression of the experts' opinions, the Delphi survey was designed, seeking to obtain the maximum consensus on the opinions of the panel through a series of structured questionnaires. As the top priority of the Delphi technique, the initial structured questionnaire was divided into three parts, sections A, B and C.

Section A introduced the background of climate change, as well as the purpose of establishing an indicator-based assessment framework for adaptive capacity of drainage systems in response to climate change in Beijing. Section B included the participants' job descriptions, job titles, and working years that would be collected, as well as the requirements for participants. Section C presented a rating scale to quantify the assessment indicators.

When scoring on the questionnaire, experts determined the score of each indicator by their own methods, including model calculation, empirical analysis, etc. Thus, experts also needed to explain their degree of confidence in their judgement, where the authority of the panel (criteria of the judgement and degree of familiarity) was also judged and evaluated via self-assessment in order to confirm credibility of the results (Table 3). Simultaneously, the suggestions of experts were added as an efficient feedback to modify the indicators system at the end of the questionnaire.

Table 3

Score table of questionnaire

Evaluation grade Quantitative values 
Extremely important 
Very important 
Important 
Somewhat important 
A little important 
Not important 
The criteria of judgment,  
Practical experience 0.8 
Theory analysis 0.6 
Realization through peer 0.4 
Intuition 0.2 
The degree of familiarity,  
Extremely familiar 
Very familiar 0.8 
Familiar 0.6 
Somewhat familiar 0.4 
A little familiar 0.2 
Unfamiliar 
Evaluation grade Quantitative values 
Extremely important 
Very important 
Important 
Somewhat important 
A little important 
Not important 
The criteria of judgment,  
Practical experience 0.8 
Theory analysis 0.6 
Realization through peer 0.4 
Intuition 0.2 
The degree of familiarity,  
Extremely familiar 
Very familiar 0.8 
Familiar 0.6 
Somewhat familiar 0.4 
A little familiar 0.2 
Unfamiliar 

Note: The criteria of judgment is divided into practical experience, theory analysis, realization through peers, intuition, which varies from 0 to 1. means the criteria of judgment makes an impact on panel and the result of judgment is considered more reliable; presents the criteria makes little impact. represents the degree of familiarity of panel on indicators. When the index is close to 1 it indicates that experts are more familiar with and understanding for the research content.

Note that the later questionnaire needs to present the feedback of previous rounds for experts, where added, deleted, and modified indicators were presented to offer the latest information for experts' analyses. The questionnaires were delivered to the respondents via e-mail.

Statistical analysis

As the iterative feature of the Delphi method, all feedback will be presented to the researchers, where a rigorous mathematical statistics analysis will be applied to present the opinions on each indicator, such as mean value, satisfactory rate, coefficient of variation, and overall consensus of experts under circumstances of ‘no group pressure’. Table 4 shows the coefficients and calculation equations in this process of statistical analysis.

  • Degree of participation. This aspect of statistical analysis is reflected by Response rate (P), which indicates the concerned extent of experts in reality.

  • Degree of concentration. As a common but important feedback of the Delphi technique, the mean value (Mj) and satisfactory rate (Kj) of each indicator indicate the level of concentration of experts' opinions and the significance of indicator. A larger value of Mj indicates a more important indicator. Kj’ varies from 0 to 1 as a supplementary variable of Mj.

  • Degree of concordance. This statistical analysis of the degree of concordance is reflected by two variables, coefficient of variation (Vj) and coefficient of concordance (W).

Table 4

Coefficients and calculation equations on statistical analysis

Statistical analysis aspect Degree of Degree of concentration
 
Degree of concordance
 
participation Response rate Mean value Rate of satisfaction Coefficient of variation Coefficient of concordance 
Variable      
Equation  
formula
(1)
 
 
formula
(2)
 
 
formula
(3)
 
 
formula
(4)
 
 
formula
(5)

 
formula
(6)

 
formula
(7)
 
Statistical analysis aspect Degree of Degree of concentration
 
Degree of concordance
 
participation Response rate Mean value Rate of satisfaction Coefficient of variation Coefficient of concordance 
Variable      
Equation  
formula
(1)
 
 
formula
(2)
 
 
formula
(3)
 
 
formula
(4)
 
 
formula
(5)

 
formula
(6)

 
formula
(7)
 

Here, NR is the number of returned questionnaires and Np is the number of issued questionnaires; mj is the number of experts marking indicator j and mj is the number of experts who scored 5 points for the indicator j; Cij is the score of indicator j that the ith expert marked; δj is the standard deviation of indicator j (this index indicates fluctuations in the significance of the indicator, and a smaller value means a higher concordance for indicator j); Msj represents the mean value of sum of all indicators; and n is the sum of indicators. Sj represents the sum of the score of indicator j that different experts marked (a larger value of Sj means higher significance of indicator j) and m is the number of experts, while n is the number of indicators.

It is noted that the coefficient of variation (Vj) indicates fluctuation extent of the significance of indicator j, where a smaller value means a higher concordance for indicator j. However, because the data collected for nearly every study is unique, guidance that describes the level of variance that represents consensus is not available (Hallowell & Gambatese 2009). Thus, the degree of concordance could be only qualitatively evaluated by the variance, and the variance was developed into one of the screening conditions to optimize the framework.

For measuring the consensus among experts, additionally, the coefficient of concordance (W) was computed (the Kendall coefficient of concordance), which reflects the consistency of experts' scores and the reliability of the assessment results. A concordance coefficient of 1 indicates that all participants scored the assessment indicators identically, and a concordance coefficient of 0 means that all participants scored assessment indicators differently. Thus, the coefficient W was used to judge whether the consensus satisfied the requirements. During two or three rounds of the Delphi process, W usually fluctuates around 0.5, which explains a fine control of deviation (Wang & Si 2011). Given the number of indicators, this paper set a target consensus where W reached 0.4 or higher.

More importantly, the significance of W was analyzed to guarantee the reliability of consensus, so the test was performed:  
formula
(8)
The threshold value () would be looked up on the standard table of values based on the degree of freedom and level of significance (see http://www.docin.com/p-1207903338.html). If > , W indicates a satisfying significance after testing, and the result can be accepted. On the contrary, below a confidence level of 95%, if P > 0.05, the results can be disregarded since they are considered to be obtained by chance and are not convincing enough to be used.

Screening of indicators

The threshold value was utilized to screen indicators. The threshold of mean value and the rate of satisfaction was calculated by:  
formula

If the score of the indicator was higher than the cut-off value, the indicator was considered reasonable.

The threshold value of the coefficient of variation was calculated by:  
formula

If the score of the indicator was lower than the cut-off value, the indicator was considered reasonable.

More importantly, the modified suggestions that were proposed by the panel beyond group pressure also could be a determining factor for selecting indicators to demonstrate the superiority of the Delphi technique.

Weighting of indicators

In terms of different contributions of indicators to adaptive capacity, the method of proportion distribution was utilized to ensure the weights of each indicator after the final indicators were determined. Based on the final scores of each indicator, their weights were calculated. Then the parameters would be weighted by the affiliated indicators.

Statistical software

The statistical analysis work was accomplished using the software SPSS 20.0.

RESULTS AND DISCUSSION

Coefficient of participation

Two rounds of the Delphi process were executed to complete consensus building. Table 5 presents the degree of attention of the panel on this study for building indicator-based assessment framework of drainage systems in Beijing.

Table 5

Coefficient of participation in Delphi process

  Round 1 Round 2 
Distributed questionnaire numbers 14 14 
Returned questionnaire numbers 14 10 
Response rate 100% 72% 
  Round 1 Round 2 
Distributed questionnaire numbers 14 14 
Returned questionnaire numbers 14 10 
Response rate 100% 72% 

There are two main reasons for the absence of participants. Firstly, due to a long interval between rounds, some experts explained that their memory and concentration for the questionnaires had slackened, leading to unconvincing responses. Secondly, some experts were occupied by heavy workload, which decreased their concern and care for the questionnaire.

Nevertheless, according to relevant literature, the number of experts participating in the Delphi process is usually between 8 and 20 (Xu 2012). Thus, for 10 experts to participate in the group process is acceptable for statistical analysis in the second Delphi round. Meanwhile, even though the number of experts in the second Delphi round decreased, there were still respondents from all three domains, presenting an even distribution in terms of job title, which provided a representative group of experts. Overall, the mean coefficient of participation was 86%, which presents a satisfying result compared to other studies that have applied Delphi rounds (Al-Jawhar & Rezouki 2012). It can be concluded that the experts presented a relatively high concern on this research, and it is significant to study the indicator selection method to form a representative framework of adaptive capacity.

Authority of panel of experts

By analyzing the authority of the panel on Delphi rounds, a promising result with high credibility was obtained, as shown in Table 6.

Table 6

The degree of authority in Delphi process

Parameters Criteria of judgment
 
Degree of familiarity
 
Comprehensive value
 
Round 1 Round 2 Round 1 Round 2 Round 1 Round 2 
Economic Resource 0.464 0.667 0.518 0.547 0.491 0.607 
Information and Skills 0.626 0.613 0.743 0.640 0.684 0.626 
Technology 0.506 0.650 0.617 0.660 0.562 0.655 
Infrastructure 0.582 0.580 0.694 0.632 0.638 0.606 
Institutions 0.639 0.611 0.704 0.534 0.672 0.572 
Mean value 0.563 0.624 0.655 0.603 0.609 0.614 
Parameters Criteria of judgment
 
Degree of familiarity
 
Comprehensive value
 
Round 1 Round 2 Round 1 Round 2 Round 1 Round 2 
Economic Resource 0.464 0.667 0.518 0.547 0.491 0.607 
Information and Skills 0.626 0.613 0.743 0.640 0.684 0.626 
Technology 0.506 0.650 0.617 0.660 0.562 0.655 
Infrastructure 0.582 0.580 0.694 0.632 0.638 0.606 
Institutions 0.639 0.611 0.704 0.534 0.672 0.572 
Mean value 0.563 0.624 0.655 0.603 0.609 0.614 

From the result of round 1, the mean comprehensive value of 0.609 reveals that the degree of authority was upper-middle, which indicates that most of the selected experts had good knowledge of the questions on the questionnaire. The most authoritative parameter was Information and skills, which indicates that this field might have already been noticed for a time due to its importance to adaptive capacity.

In the second round, although the mean degree of familiarity somewhat decreased, both the mean criteria of judgment and mean comprehensive value show an upward trend compared to the first round. The overall trend indicates that the panel of experts may desire to strengthen their knowledge about the contribution factors of adaptive capacity of drainage systems during the Delphi process. From the improvements of these parameters, it can be concluded that the modifications based on experts' suggestions were positive, and judgments of the experts were reliable.

Overall, the authority of the panel performed well. And it proved that the Delphi process can absorb the consensus of experts to the greatest extent and tackle the divergence among them.

Selection of indicators

The score results, shown in Table 7, indicate the importance and divergence of each indicator that experts determined from theory and practical experiences in the first Delphi round. Based on the screening criteria (Table 8) and suggestions of experts (Table 9), nine indicators were deleted, where three were directly removed for not satisfying the three threshold values and six were deleted with consideration of both the threshold values and suggestions of experts. Meanwhile, to optimize the expression of indicators, several indicators that conveyed similar meaning were combined or rephrased. In the dimension of Information and skills, for instance, ‘The availability of information from government when heavy rainfall occurs’ aims to describe the capacity of government to manage precipitation information in detail, which was rephrased as ‘The capacity of government that guarantees timely and accurate notification before rainfall’. Similarly ‘The degree of participation by the public during heavy rainfall disasters’ and ‘The popularity of knowledge about heavy rainfall by the public’ represent the aspect of the public to exploit climatic information, which are combined with ‘The self-consciousness of the public about heavy rainfall disasters’. For the dimension of Infrastructure, ‘The number of shelter to heavy rainfall’ was rephrased and combined with ‘the number of retention facilities to rainfall’. ‘The rate of reaching the standard of the capacity of drainage in pipes’ was combined with the urban rain-flood management standards. Eight new indicators were added in this framework based on experts' suggestions.

Table 7

Indicators of the first Delphi round

Index Parameter Indicator Mean score Rate of satisfaction (%) Variation 
Drainage systems Economic resource 1. Gross regional production 3.62 30.77 0.431 
2. Per capita gross regional production 3.38 15.38 0.392 
3. Proportion of investment of urban drainage systems 4.00 46.15 0.270 
4. Investment types of drainage systems in response to climate change 4.15 38.46 0.193 
Information and skills 5. The availability of information from government when heavy rainfall occurs 4.92 92.31 0.056 
6. The degree of participation by the public during heavy rainfall disasters 4.15 53.85 0.238 
7. The general literacy of residents 3.31 7.69 0.286 
8. The popularity of knowledge about heavy rainfall by the public 4.46 61.54 0.174 
9. Propaganda work of reactive measures on heavy rainfall by media 4.54 69.23 0.171 
Technology 10. The number of universities that awarding point of master degrees in municipal engineering or relevant speciality 3.62 15.38 0.349 
11. The highest ranking of the drainage design level of municipal design institutes in domestic 3.31 15.38 0.416 
12. The proportion of the employees of drainage domains in total residents 2.77 15.38 0.492 
13. The number of excellent experts in the drainage domains 3.38 15.38 0.392 
14. Project funding undertaken by company, for studying and constructing the drainage systems 3.92 38.46 0.265 
Infrastructure 15. Urban rain-flood management standards 4.92 92.31 0.056 
16. The number of shelters for protection during heavy precipitation 3.77 23.08 0.246 
17. The rate of reaching the standard for the drainage capacity of pipes 4.85 84.62 0.077 
18. The rate of hardening of urban ground 4.00  53.85 0.382 
19. The number of retention facilities for rainfall 3.85 15.38 0.179 
20. The number of waterlogged places on the road after rainfall 4.23 61.54 0.336 
21. The proportion of the population who live in dangerous buildings 4.08 38.46 0.211 
Institutions 22. Establish special institutions and administration for climate change 4.62 76.92 0.188 
23. Form a contingency plan to respond to heavy rainfall 4.92 92.31 0.056 
24. The starting time of a contingency plan when heavy rainfall occurs 4.77 76.92 0.092 
25. The stability of policies for heavy rainfall 4.00 30.77 0.204 
Index Parameter Indicator Mean score Rate of satisfaction (%) Variation 
Drainage systems Economic resource 1. Gross regional production 3.62 30.77 0.431 
2. Per capita gross regional production 3.38 15.38 0.392 
3. Proportion of investment of urban drainage systems 4.00 46.15 0.270 
4. Investment types of drainage systems in response to climate change 4.15 38.46 0.193 
Information and skills 5. The availability of information from government when heavy rainfall occurs 4.92 92.31 0.056 
6. The degree of participation by the public during heavy rainfall disasters 4.15 53.85 0.238 
7. The general literacy of residents 3.31 7.69 0.286 
8. The popularity of knowledge about heavy rainfall by the public 4.46 61.54 0.174 
9. Propaganda work of reactive measures on heavy rainfall by media 4.54 69.23 0.171 
Technology 10. The number of universities that awarding point of master degrees in municipal engineering or relevant speciality 3.62 15.38 0.349 
11. The highest ranking of the drainage design level of municipal design institutes in domestic 3.31 15.38 0.416 
12. The proportion of the employees of drainage domains in total residents 2.77 15.38 0.492 
13. The number of excellent experts in the drainage domains 3.38 15.38 0.392 
14. Project funding undertaken by company, for studying and constructing the drainage systems 3.92 38.46 0.265 
Infrastructure 15. Urban rain-flood management standards 4.92 92.31 0.056 
16. The number of shelters for protection during heavy precipitation 3.77 23.08 0.246 
17. The rate of reaching the standard for the drainage capacity of pipes 4.85 84.62 0.077 
18. The rate of hardening of urban ground 4.00  53.85 0.382 
19. The number of retention facilities for rainfall 3.85 15.38 0.179 
20. The number of waterlogged places on the road after rainfall 4.23 61.54 0.336 
21. The proportion of the population who live in dangerous buildings 4.08 38.46 0.211 
Institutions 22. Establish special institutions and administration for climate change 4.62 76.92 0.188 
23. Form a contingency plan to respond to heavy rainfall 4.92 92.31 0.056 
24. The starting time of a contingency plan when heavy rainfall occurs 4.77 76.92 0.092 
25. The stability of policies for heavy rainfall 4.00 30.77 0.204 
Table 8

Criteria of screening in Delphi round

  Mean value
 
Standard deviation
 
Cut-off value
 
Round 1 Round 2 Round 1 Round 2 Round 1 Round 2 
Score 4.04 4.28 0.60 0.50 3.44 3.78 
Rate of satisfaction (%) 44.57 46.00 28.07 29.81 16.50 16.19 
Coefficient of variation 0.244 0.153 0.123 0.079 0.367 0.232 
  Mean value
 
Standard deviation
 
Cut-off value
 
Round 1 Round 2 Round 1 Round 2 Round 1 Round 2 
Score 4.04 4.28 0.60 0.50 3.44 3.78 
Rate of satisfaction (%) 44.57 46.00 28.07 29.81 16.50 16.19 
Coefficient of variation 0.244 0.153 0.123 0.079 0.367 0.232 

Only when all three cut-off values do not meet requirements can an indicator be directly deleted. Otherwise, as an important reference, the suggestions of experts should be closely heeded in determining whether indicators were eliminated.

Table 9

Suggestions of experts and their roles for indicators selection after the first Delphi round

Parameters Suggestions Discussions 
Economic
Resource 
1. Delete indicator 1.
2. Add ‘the development level of urban district’. 
As a support to delete indicator 1. 
Information and Skills 1. Delete indicators 6, 7.
2. Pay attention to the education of the public on daily protection of drainage systems and the emergency treatment for flood danger;
3. Reflect the application and advance of information management system. 
As a support to delete indicators 7, 8.
The role of information and knowledge should consider both the professionals in drainage field and the public who suffer from disaster. 
Technology 1. Delete indicators 10, 13.
2. Integrate indicator 14 into Economy Resource.
3. Focus on the capacity of the employees in urban drainage systems;
4. Focus on the degree of concordance between the competent departments of drainage system and water affairs in cities. 
As a support to delete indicators 10, 13.
Technical indicators should focus on technical reserves and practical experience 
Infrastructure 1. Delete indicator 16.
2. Modify indicator 19 into ‘the volume of retention facilities per capita in each urban city’.
3. Focus on the quality of the old drainage pipe network in the city;
4. Represents the resilience of urban housing, roads, energy and other infrastructures, such as the proportion of vegetation area and drainage facilities; 
As a support to delete indicator 16.
Infrastructure indicators should reflect the interaction of natural and artificial infrastructures. 
Institutions 1. Focus on contingency plan to heavy rainfall and its implementation efficiency
2. Reflect the effects of urban and municipal special planning. 
Contingency plan is crucial for the management and disposal for heavy rainfall events 
Parameters Suggestions Discussions 
Economic
Resource 
1. Delete indicator 1.
2. Add ‘the development level of urban district’. 
As a support to delete indicator 1. 
Information and Skills 1. Delete indicators 6, 7.
2. Pay attention to the education of the public on daily protection of drainage systems and the emergency treatment for flood danger;
3. Reflect the application and advance of information management system. 
As a support to delete indicators 7, 8.
The role of information and knowledge should consider both the professionals in drainage field and the public who suffer from disaster. 
Technology 1. Delete indicators 10, 13.
2. Integrate indicator 14 into Economy Resource.
3. Focus on the capacity of the employees in urban drainage systems;
4. Focus on the degree of concordance between the competent departments of drainage system and water affairs in cities. 
As a support to delete indicators 10, 13.
Technical indicators should focus on technical reserves and practical experience 
Infrastructure 1. Delete indicator 16.
2. Modify indicator 19 into ‘the volume of retention facilities per capita in each urban city’.
3. Focus on the quality of the old drainage pipe network in the city;
4. Represents the resilience of urban housing, roads, energy and other infrastructures, such as the proportion of vegetation area and drainage facilities; 
As a support to delete indicator 16.
Infrastructure indicators should reflect the interaction of natural and artificial infrastructures. 
Institutions 1. Focus on contingency plan to heavy rainfall and its implementation efficiency
2. Reflect the effects of urban and municipal special planning. 
Contingency plan is crucial for the management and disposal for heavy rainfall events 

Table 10 indicates the score results for each indicator in the second round. The main selection criteria are four principal categories for application combined with screening criteria (Table 8), so the indicators within each parameter are determined in such a way that they are easy to quantify for the availability of data, combined for simplification, and reflect the differentiation of regions and years. Thus, ‘The development level of urban’ was directly eliminated for dissatisfaction of the three threshold values, and the other 11 indicators were deleted based on discussions from the principles. For the final eight indicators (in the section of ‘Indicator-based framework’), seven of them had high mean values greater than 4.0 point, which proves that the selected indicators are efficient enough for the adaptive capacity of drainage system. Although the K of 1.1.2 indicator was zero, its coefficient of variation was 0.111 with a relatively low level. Based on the experts' perspectives, this indicator should be a representative indicator because of the uncertainty in climate change that requires various investments. The mean value of the 1.1.2 indicator was 3.80, which was higher than the threshold value and represented an upper-middle significance. Thus, the indicator ‘investment types of drainage systems to climate change’ can be considered reasonable.

Table 10

Indicators of the second Delphi round

Index Parameter Indicator Mean score Rate of satisfaction(%) Variation 
Drainage systems Economic Resource 1. Proportion of investment of urban drainage systems 4.40 50.00 0.159 
2. Investment types of drainage systems in response to climate change 3.80 0.00 0.111 
3. The development level of urban district 3.10 0.00 0.283 
Information and Skills 4. Propaganda work of reactive measures on heavy rainfall by media 4.30 40.00 0.157 
5. The capacity of government that guarantees timely and accurate notification before rainfall. 5.00 100.00 0.000 
6. The self-consciousness of the public about heavy rainfall disasters 4.40 50.00 0.159 
Technology 7. The degree of concordance between the competent departments of drainage system and water affairs 4.50 60.00 0.157 
8. The knowledge and skill level of employees in drainage systems 4.40 50.00 0.159 
Infrastructure 9. The number of waterlogged places on the road after rainfall 4.30 30.00 0.112 
10. The rate of hardening of urban ground 4.10 30.00 0.214 
11. The period of updating old pipe networks in urban drainage systems 4.50 60.00 0.157 
12. The volume capture ratio of annual runoff 4.00 10.00 0.167 
13. The volume of retention facilities per capita in each district 3.30 20.00 0.351 
Institutions 14. The stability of policies for heavy rainfall 4.00 10.00 0.118 
15. Establishing special institutions and administration for climate change 4.60 60.00 0.112 
16. Forming a contingency plan to respond to heavy rainfall 4.90 90.00 0.065 
17. The starting time of a contingency plan when heavy rainfall occurs 4.90 90.00 0.065 
18. The effects of urban and municipal special planning 4.80 80.00 0.088 
19. The policy requirements proposed by management institutions for measures to heavy rainfall of relevant departments 3.90 20.00 0.189 
20. The professional level of the management institutions 4.30 60.00 0.246 
Index Parameter Indicator Mean score Rate of satisfaction(%) Variation 
Drainage systems Economic Resource 1. Proportion of investment of urban drainage systems 4.40 50.00 0.159 
2. Investment types of drainage systems in response to climate change 3.80 0.00 0.111 
3. The development level of urban district 3.10 0.00 0.283 
Information and Skills 4. Propaganda work of reactive measures on heavy rainfall by media 4.30 40.00 0.157 
5. The capacity of government that guarantees timely and accurate notification before rainfall. 5.00 100.00 0.000 
6. The self-consciousness of the public about heavy rainfall disasters 4.40 50.00 0.159 
Technology 7. The degree of concordance between the competent departments of drainage system and water affairs 4.50 60.00 0.157 
8. The knowledge and skill level of employees in drainage systems 4.40 50.00 0.159 
Infrastructure 9. The number of waterlogged places on the road after rainfall 4.30 30.00 0.112 
10. The rate of hardening of urban ground 4.10 30.00 0.214 
11. The period of updating old pipe networks in urban drainage systems 4.50 60.00 0.157 
12. The volume capture ratio of annual runoff 4.00 10.00 0.167 
13. The volume of retention facilities per capita in each district 3.30 20.00 0.351 
Institutions 14. The stability of policies for heavy rainfall 4.00 10.00 0.118 
15. Establishing special institutions and administration for climate change 4.60 60.00 0.112 
16. Forming a contingency plan to respond to heavy rainfall 4.90 90.00 0.065 
17. The starting time of a contingency plan when heavy rainfall occurs 4.90 90.00 0.065 
18. The effects of urban and municipal special planning 4.80 80.00 0.088 
19. The policy requirements proposed by management institutions for measures to heavy rainfall of relevant departments 3.90 20.00 0.189 
20. The professional level of the management institutions 4.30 60.00 0.246 

After analyzing the data in Tables 8, a satisfying trend was found that the consensus improved during the two rounds of Delphi process. As a factor to reflect the degree of consensus of experts' decisions, the average coefficient of variation decreased from 0.244 (SD = 0.123) in the first round to 0.153 (SD = 0.079) in the second round. More importantly, the mean score of importance increased from 4.04 (SD = 0.6) to 4.28 (SD = 0.5) and the mean rate of satisfaction ascended from 44.57% (SD = 0.28) to 46.00% (SD = 0.29). From the improved data, we can conclude that the opinions of experts tend to be more consistent through anonymous negotiations of the Delphi method. Meanwhile, the level of significance of consensus between rounds 1 and 2 was examined using the T test for significance. The results indicate that the coefficient of variation between these two rounds was significantly different below a confidence level of 95% (P = 0.007 < 0.05), in other words, the consensus among the panel and applicability of framework are improved significantly through the Delphi process.

Degree of concordance

In order to obtain the consistency of experts' opinions, the Kendall coefficient of concordance (W) was calculated to obtain the consistency of experts' opinions using SPSS. In the first Delphi process, W was 0.368, and the value, which was higher than the threshold value of 36.42, was 123.54. Although the level of significance (P) was less than 0.01, which represents a relatively fine consistency from a statistical standpoint, the coefficient W did not reach the target consensus (0.4). Thus, another Delphi round was executed, where the coefficient of concordance was determined to be 0.410, and the value, which was higher than the threshold value (30.14), was 77.81 with P < 0.05.

From Table 11, it can be seen that the coefficient of concordance increased from 0.368 to 0.410, which explains that the consensus of our study improved over successive rounds. Moreover, the coefficient of concordance, W, reached the target consensus, resulting in the termination of the Delphi process with a good control of deviation. Further, P values below 0.01 in both of the rounds indicate that there was good consistency of responses from experts and that our results have a high confidence level. It is concluded that it is an effective method to establishing consensus using the Delphi technique to process opinions among various experts.

Table 11

Degree of concordance in Delphi process

Variables Round 1 Round 2 
14 10 
Kendall W 0.368 0.410 
 123.54 77.81 
df 24 19 
Sig (P) 0.000 0.000 
Variables Round 1 Round 2 
14 10 
Kendall W 0.368 0.410 
 123.54 77.81 
df 24 19 
Sig (P) 0.000 0.000 

Indicator-based framework

During the indicator selection process for building the framework, the initial 25 indicators were modified into 20 indicators, which were further screened into eight refined indicators to reflect the attention degree of each dimension of adaptive capacity (Table 12). It should also be noted that there was high consistency among the final 8 indicators, where the coefficient of concordance (W) was 0.480 with P < 0.01 below a confidence level of 95%.

Table 12

Indicators and weights of assessment framework of the adaptive capacity of drainage systems in Beijing.

Index (Weight) Parameters (Weight) Indicators (Weight) 
Drainage systems (1.0) 1.1. Economic resource (0.2290) 1.1.1 Proportion of investment of urban drainage system (0.1229);
1.1.2 Investment types of drainage systems to climate change (0.1061) 
1.2. Information and skills (0.1397) 1.2.1 The capacity of government that guarantees timely and accurate notification when heavy rainfall is going to occur (0.1397) 
1.3. Technology (0.1229) 1.3.1 The knowledge and skill level of employees in drainage systems (0.1229) 
1.4. Infrastructure (0.2346) 1.4.1 The number of waterlogged places on the road after rainfall (0.1201);
1.4.2 The rate of hardening of urban ground (0.1145) 
1.5. Institutions (0.2738) 1.5.1 Form contingency plan to heavy rainfall (0.1369);
1.5.2 The starting time of contingency plan to heavy rainfall (0.1369) 
Index (Weight) Parameters (Weight) Indicators (Weight) 
Drainage systems (1.0) 1.1. Economic resource (0.2290) 1.1.1 Proportion of investment of urban drainage system (0.1229);
1.1.2 Investment types of drainage systems to climate change (0.1061) 
1.2. Information and skills (0.1397) 1.2.1 The capacity of government that guarantees timely and accurate notification when heavy rainfall is going to occur (0.1397) 
1.3. Technology (0.1229) 1.3.1 The knowledge and skill level of employees in drainage systems (0.1229) 
1.4. Infrastructure (0.2346) 1.4.1 The number of waterlogged places on the road after rainfall (0.1201);
1.4.2 The rate of hardening of urban ground (0.1145) 
1.5. Institutions (0.2738) 1.5.1 Form contingency plan to heavy rainfall (0.1369);
1.5.2 The starting time of contingency plan to heavy rainfall (0.1369) 

According to the indicator-based assessment framework, the composition of the adaptive capacity of drainage system to climate change can be explained from three perspectives, including ACI, parameter value and indicator value.

Three relatively important domains of the drainage system were determined based on the weight they accounted for, including Economic resource (0.2290), Infrastructure (0.2346), and Institutions (0.2738). Because the parameter institutions accounted for the most weight, some effective management measures in this dimension can be considered as key factors to respond to climate change, which implies the necessity for an organizational form to face various climate change impacts.

Eight indicators of adaptive capacity have been determined from the five dimensions of a synergistic state, as mentioned above.

  • Proportion of investment of urban drainage systems. This indicator reflects the economic strength of a district. An abundant investment in drainage systems is the basis for guaranteeing the normal functioning of all aspects of the drainage system in response to climate change.

  • Investment types of drainage systems to climate change. This indicator indicates that climate change involves various aspects, so comprehensive investments should be completed to make it easier to ensure the adaptive capacity is equal to development and recovery after disasters.

  • The capacity of government to guarantee timely and accurate notification when heavy rainfall is going to occur. The timely popularity of information can effectively reduce the risk that residents are exposed to the risk of flood disasters. And the public sector plays a pivotal role in managing relevant information on precipitation.

  • The knowledge and skill level of employees in drainage systems. As an important technical reserve, professional staff who serve drainage systems play a key role in the construction of drainage facilities and emergencies.

  • The number of waterlogged places on the road after rainfall. This increases the security risks for public traffic when heavy rain occurs, and reflects the completeness of drainage infrastructure (drainage pipe network, pumping stations, etc.). It should be noted that it is an indicator of the negative aspects that reduce the adaptive capacity of drainage systems.

  • The rate of hardening of urban ground. If ‘high rate of hardening of urban ground’ exists, rain cannot easily permeate the ground and combine with groundwater, which obviously imposes greater pressure on draining rainfall via drainage systems. It is an indicator of the negative influences that reduce the adaptive capacity of drainage systems.

  • Form contingency plan to heavy rainfall. When a city encounters extreme events like a rainstorm, one of the tasks of the water department is to coordinate the relevant stakeholders to carry out post-disaster response work in an orderly manner. Thus, formulating an effective contingency plan could dramatically reduce the loss of property and life.

  • The starting time of contingency plan to heavy rainfall. This indicator reflects the executive force of institutions, which is a key factor in guaranteeing timely and targeted rescue procedures. Owing to the possibility of urban flooding causing serious risk to life and property within a short time, the shorter the start-up time of the plan, the more adaptive capacity an institution can apply. This is also an indicator of the negative aspects.

CONCLUSION AND SUGGESTION

Conclusions

This work proposes a representative assessment framework composed of three levels to reflect the adaptive capacity of drainage systems in Beijing. In this framework, the ACI is an aggregate of five parameters, where each is an aggregate of one or more adaptive capacity indicators that were screened by the Delphi method to maximize consensus among various decisions of experts.

To take full advantage of the merits of anonymous discussion in the Delphi technique, both the scores of adaptive capacity indicators and added suggestions from the panel without any group pressure were analyzed. The former indicates the significance of each indicator from three aspects, which are importance score, rate of satisfaction, and coefficient of variation, while the latter can be the imperative criteria to judge the controversial indicators. In the Delphi process, the mean coefficient of variation decreased from 0.244 in the first round to 0.153 in the second round. Meanwhile, the mean score of importance increased from 4.04 to 4.28, and the mean rate of satisfaction varied from 44.57% to 46.00%. Eventually, eight indicators were chosen to assess the adaptive capacity of drainage in response to climate change in Beijing. From the improved data, it is concluded that the opinions of experts tended to be uniform through anonymous negotiations of the Delphi process and the Delphi method is a suitable technique for forming objective opinions and consensus in relatively subjective environments.

The final eight indicators for assessing the adaptive capacity of drainage systems can be found in Table 12. In total, these eight indicators are representative of the five fields of adaptive capacity in drainage systems, which indicates that the construction of adaptive capacity requires a systematic engineering. According to the composition of this proposed framework, an inherent linkage can be found from the weights of five parameters that contributes to the adaptive capacity of drainage systems. The Economic resource parameter can be considered as a basic and crucial guarantee for adaptive capacity in view of the necessity of financial support. Based on abundant funds, local government can be equipped with a more complete infrastructure to address the threats of climate change. However, Infrastructure is vulnerable to extreme climatic events and effective guideline is necessary for aiding higher adaptive capacity. For example, administrative departments must implement timely response to any climatic events. Meanwhile, the executive force of Institutions is necessary to guarantee that a project could be accurately implemented in a timely manner. Furthermore, the parameter Technology and information should be a significant complement to Institutions. Technology, as a reserve strength, can ensure that present and future extreme events are coped with reasonably. Meanwhile, Information and skills could enhance the modernization of monitoring, managing, and replying, which undoubtedly saves considerable manpower and resources when responding to climate change.

In summary, the indicator-based assessment framework represents adaptive capacity from a full-dimension perspective, which offers a fundamental framework to comprehend the adaptive capacity of drainage systems in Beijing. More importantly, the methodology applied in this study may serve as a guideline for constructing relevant assessment frameworks of adaptive capacity in urban water systems.

Suggestions

Some deficiencies were found while the Delphi method was carried out. One concern is that the long time interval for responding may affect the concentration and memory of experts. A suitable time to perform the Delphi method deserves to be studied to maintain the enthusiasm for participants. As for the second deficiency, the abundant data, which originates from the experts' suggestions during the Delphi rounds, brings about complicated analysis processes for researchers. Thus, future research should explore a reasonable number of participants to be used in a Delphi process in terms of a certain study area.

It is worth noting that this framework will facilitate the assessment practices of adaptive capacity. Further work can be done to apply the framework to assess the adaptive capacity, with the case of the Beijing drainage system at district level.

ACKNOWLEDGEMENTS

The author would like to acknowledge Beijing Water Authority and the funding by Clean Development Mechanism Grant in China [grant number 2013027].

REFERENCES

REFERENCES
Al-Jawhar
H. D.
&
Rezouki
S. E.
2012
Identification of procurement system selection criteria in the construction industry in Iraq by using Delphi method
.
International Proceedings of Economics Development & Research
45
,
142
147
.
Arnbjerg-Nielsen
K.
,
Willems
P.
,
Olsson
J.
,
Beecham
S.
,
Pathirana
A.
,
Bülow Gregersen
I.
,
Madsen
H.
&
Nguyen
V.-T.-V.
2013
Impacts of climate change on rainfall extremes and urban drainage systems: a review
.
Water Science & Technology
68
,
16
28
.
Birgani
Y. T.
&
Yazdandoost
F.
2016
Resilience in urban drainage risk management systems
.
Proceedings of the Institution of Civil Engineers-Water Management
169
,
3
16
.
Carter
J. G.
,
Cavan
G.
,
Connelly
A.
,
Guy
S.
&
Handley
J.
2015
Climate change and the city: building capacity for urban adaptation
.
Progress in Planning
95
,
1
66
.
Che
W.
,
Lv
F. F.
,
Li
J. Q.
,
Li
H. Y.
&
Wang
J. L.
2009
Typical stormwater and flood management systems in developed countries and their inspiration
.
China Water & Wastewater
25
(
20
),
12
17
.
Del
G. G.
,
Padulano
R.
&
Siciliano
D.
2016
Multivariate probability distribution for sewer system vulnerability assessment under data-limited conditions
.
Water Science & Technology
73
,
751
760
.
Engle
N. L.
2011
Adaptive capacity and its assessment
.
Global Environmental Change
21
(
2
),
647
656
.
Eriksen
S. H.
&
Kelly
P. M.
2007
Developing credible vulnerability indicators for climate adaptation policy assessment
.
Mitigation and Adaptation Strategies for Global Change
12
,
495
524
.
Gupta
J.
,
Termeer
C.
,
Klostermann
J.
,
Meijerink
S.
,
Van den Brink
M.
&
Jong
P.
2010
The adaptive capacity wheel: a method to assess the inherent characteristics of institutions to enable the adaptive capacity of society
.
Environmental Science & Policy
13
,
459
471
.
Hallowell
M. R.
&
Gambatese
J. A.
2009
Qualitative research: application of the Delphi method to CEM research
.
Journal of Construction Engineering & Management
136
,
99
107
.
Hyun
C. T.
,
Cho
K. M.
,
Koo
K. J.
,
Hong
T. H.
&
Moon
H. S.
2008
Effect of delivery methods on design performance in multifamily housing projects
.
Journal of Construction Engineering & Management
134
,
468
482
.
IPCC
2001
Climate Change 2001: Impacts, Adaptation, and Vulnerability
.
Cambridge University Press
,
Cambridge
,
UK
.
Kirshen
P.
,
Caputo
L.
,
Vogel
R. M.
,
Mathisen
P.
,
Rosner
A.
&
Renaud
T.
2014
Adapting urban infrastructure to climate change: a drainage case study
.
Journal of Water Resources Planning & Management
141
(
4
),
04014064
.
Kutlu
G.
2015
Cancun Adaptation Framework and the Adaptive City Copenhagen – A Critical Urban Theory Perspective
.
Roskilde University
,
Roskilde
,
Denmark
.
Moglia
M.
,
Burn
S.
&
Tjandraatmadja
G.
2009
Vulnerability of water services in Pacific Island countries: combining methodologies and judgment
.
Water Science & Technology
60
,
1621
1631
.
Pandey
V. P.
,
Babel
M. S.
,
Shrestha
S.
&
Kazama
F.
2011
A framework to assess adaptive capacity of the water resources system in Nepalese river basins
.
Ecological Indicators
11
(
2
),
480
488
.
Shiau
J. T.
,
Chen
C. N.
&
Tsai
C. T.
2012
Physiographic drainage-inundation model based flooding vulnerability assessment
.
Water Resources Management
26
,
1307
1323
.
Sietchiping
R.
2006
Applying an index of adaptive capacity to climate change in north-western Victoria
.
Australia. Applied GIS
2
(
3
),
16.1
16.28
.
Smit
B.
&
Wandel
J.
2006
Adaptation, adaptive capacity and vulnerability
.
Global Environmental Change
16
,
282
292
.
van den Brink
M.
,
Termeer
C.
&
Meijerink
S.
2011
Are Dutch water safety institutions prepared for climate change?
Journal of Water and Climate Change
2
,
272
287
.
Veltri
A. T.
1985
Expected use of Management Principles for Safety Function Management
.
PhD dissertation
,
University of West Virginia
,
Morgantown, WV
,
USA
.
Wang
C. Q.
&
Si
Q.
2011
Process methods of mathematical statistics and application study of Delphi method
.
Journal of Inner Mongolia University of Finance and Economics
4
,
92
96
.
Xu
G. X.
2012
Qualitative prediction method
. In:
Statistical Forecasting and Decision-making
.
Publication of Shanghai University of Finance and Economics
,
Shanghai
,
China
, pp. 8–9.