Approach to develop a climate change resilience assessment framework

Urban areas are complex, vulnerable and continuously evolving, with interacting strategic services, such as water supply, wastewater, storm water and solid waste management, energy supply or mobility. Interacting services, interdependencies between systems and the involvement of multiple stakeholders add complexity to the management of resilience in these areas. Climate dynamics effects on urban areas, including intense precipitation, increased coastal water level or droughts, can significantly affect strategic urban services, people, natural and built environment and economy. Furthermore, climate change (CC) trends aggravate current conditions and lead potentially to emergence of new hazards. These challenges require an integrated and forward-looking approach to resilient and sustainable urban development, incorporating the interdependencies between systems and a continuously improvement process.

Assessing resilience constitutes the basis for cities to know where they stand, to support decision on strategies and measures to adopt, adding robustness to planning in the long, medium and short terms, while facilitating the assessment of progress. As part of the United Nations Agenda 2030 for Sustainable Development, several agendas have been adopted such as the Sendai Framework for Disaster Risk Reduction 2015–2030, the Sustainable Development Goals, the New Urban Agenda and the Paris Agreement (Panda 2018), all considering assessment steps to track implementation (UN-GA 2016). The World Health Organization (WHO 2009), in its five main conclusions from the vision 2030 study, states that systematic assessments of climate change resilience are required by all utilities and rural water and sanitation programmes, together with simple tools that can be used in diverse areas, such as rapid assessment of water utilities' vulnerability to climate change. Assessment tools provide a decision support to city and urban services' managers, planners and decision-makers by identifying aspects that need to be enhanced, to establish priorities for intervention, to identify resilience strategies, to develop resilience action plans and to predict and monitor effectiveness and efficiency of their implementation (Sharifi 2016; Cardoso et al. 2018).

The recognition of the relevance to assess resilience resulted in the development of several tools and frameworks for assessing resilience presented hereafter. However, and considering the relevance of continuous improvement (ISO 9001:2015), the need was identified for a framework to be usable by cities and urban services managers, considering a structured and freely available assessment, objective driven and with the purpose of supporting the development and monitoring of cities' and urban services' resilience action plans. This, particularly regarding the scope of climate change with focus on the urban water cycle, addressing a multisectorial approach that integrates the city, services and infrastructure resilience.

In this context, a resilience assessment framework (RAF) was developed within the EU H2020 RESCCUE project (Resilience to cope with climate change in urban areas) and applied to three cities' research sites, Lisbon, Bristol and Barcelona (Velasco et al. 2018). The scope is urban resilience to climate change-related hazards (meaning that, e.g., earthquakes or economic crises are not considered), with focus on the urban water cycle, with emphasis on city, services and infrastructure resilience. The paper presents an outline of the structure of the framework and details the approach used in its development, including engagement tools and actions undertaken to assure stakeholder involvement in its development, validation and testing.

The approach for the RAF development follows a step-by-step process, considering four main steps:

1. Analysis of existing assessment frameworks

2. Definition of a RAF preliminary proposal

3. Validation of the RAF preliminary version

4. Proposal of a RAF final version

Step 1 is based on a literature review of existing assessment frameworks for assessing resilience. Several tools and frameworks for assessing resilience have been developed in diverse fields of study by a wide variety of organisations such as ICLEI (2010), UN-Habitat CRPT (2018), Rockefeller Foundation and Arup Group (2014), World Bank (2015), UNISDR (2017), EPA (2017), among others (Patel & Nosal 2016; EPA 2017; Summers et al. 2017; UNIDSR 2017). They present substantial variation in their structure, content and complexity, therefore a review is carried out taking into consideration the mentioned scope (CC) and focus (urban water cycle and water, wastewater, storm water, solid waste, energy and mobility sectors) of the present work.

Step 2 is grounded on the previous step results and considers the same scope and focus in the development of a preliminary RAF proposal, keeping alignment with international benchmarks for resilience assessment (ARUP 2015; UNIDSR 2017), following the principle of continuous improvement (ISO 9001:2015) and adopting the recommendations for objective-driven assessment and management urban water services and assets (ISO 24510:2007; ISO 24511:2007; ISO 24512:2007; ISO 55000:2014; ISO 9001:2015). Regarding international resilience assessment frameworks, the analysis of the scope, structure, content as well as addressed concerns and sectors was carried out, in order to identify those aspects that are common to the scope and fit the purpose of the RAF. Considering the adoption of recommendations for objective-driven assessment and continuous improvement, it is fundamental to define resilience objectives, translating the ambitions to be achieved in the medium–long term by the city and services. For each objective, key criteria are specified, which express the different points of view through which the objectives are to be assessed. For each criterion, metrics are identified corresponding to parameters or functions that allow to quantify or classify the criteria. By comparison of the result of the metrics with reference values, it is possible to assign a judgement to the responses, reflecting the resilience maturity of the city or of the service under assessment.

Step 3 considers the need to ensure a broader engagement of stakeholders in the development and implementation of assessment tools (Larkin et al. 2015). The RAF preliminary version was subject to a two-step validation. Internal validation (mainly involving RESCCUE partners) and external validation (involving mainly external stakeholders to the project). Engagement tools were applied to promote interaction with stakeholders, namely, receiving and returning contributions and to ensure stakeholder involvement in specific activities. In this context, stakeholders include not only people from the city and services under assessment (i.e., individuals that may be RESCCUE partners or not, involved with resilience in the city management and in each service under the RAF scope), and other players relevant to the project's aims.

The implemented validation activities consider the following:

• Working group discussions (for internal validation): with the purpose to create awareness of RAF specificities, to align development with project expectations and to fine-tune some details.

• Workshops (for both internal and external validation): held in each city – Barcelona, Bristol and Lisbon – providing an opportunity to engage every stakeholder, by raising awareness of their individual contribution to city resilience, and integrating their contributions in the framework. Workshops intend to get stakeholders' opinions on RAF relevance and applicability and to incorporate stakeholders' concerns into its contents.

• RAF testing (for internal validation): includes the application of the RAF to the cities, providing a preliminary assessment of the cities' resilience. The purpose of the testing was to ensure the coherence and feasibility of the approach.

This process allows stakeholders to contribute to the final version of the RAF by incorporating their concerns as well as their own context and reality, and validating the framework applicability. For this, several methods and tools are considered:

• Surveys: questionnaires addressed to each participant followed by a broader debate; they are used in working group discussions and workshops.

• Sectoral brainstorming (brainstorming with urban services' teams): brainstorming within groups organised by sectors, focused on problems proposed to the group, which is composed of participants from or related to a specific urban service, followed by a broader debate; this tool is used in workshops.

• Combined brainstorming (mixed-teams' brainstorming): brainstorming within groups organised by mixing diverse sectors, to discuss problems with participants from different stakeholders, followed by broader debate; this tool is used in workshops.

• RAF app: web-based application tool reproducing the RAF structure facilitating the application of the RAF (Brito et al. 2019); this tool is applied in the RAF testing.

• RAF sharing: presentations of the RAF proposal and of its results, followed by a broader debate; tool used in working group discussions and in RAF testing.

Step 4 is based on the analysis of the validation results, considering the identification of opportunities for the RAF improvement, regarding its applicability to different cities and services having distinct contexts, as well as its fitness for the assessment purpose of identifying city and services' resilience improvement opportunities.

Extensive literature reviews characterising existing frameworks for assessing resilience are presented in Lavelle et al. (2015), Schipper & Langston (2015), Patel & Nosal (2016), Sharifi (2016), Summers et al. (2016), Tafidis et al. (2017) and UN-Habitat (2018). From these, for assessing resilience, it is important to take into account that cities are multi-dimension entities and, therefore, urban resilience needs to consider multidisciplinary insights. Additionally, resilience of a city is determined by diverse interacting systems and their relationships. For this reason, resilience also depends on the overall performance and capacity of its systems, not solely on its ability to cope with specific natural hazards or to adapt targeted areas to the impacts of climate change (Brugmann 2012). Thus, it is essential to address interdependencies and cascading effects (Vallejo & Mullan 2017). Another relevant aspect is that it needs to include both sudden crises as well as interacting long-term stressors. Noteworthy work developed assumes that resilience is structured in terms of dimensions, while different dimensions are considered depending on the themes under assessment. Summers et al. (2017) identified the five most common dimensions as environmental, social, economic, built environment and infrastructure, and institutional. Each dimension was then divided into sub-dimensions, with several resilience criteria distributed among them.

Grounded in the analysis of these frameworks and considering the RESCCUE scope (CC and water) and focus (city, services and infrastructure), gaps and needs were identified, particularly with regard to a framework with the following characteristics: freely available to be used by cities and urban services managers for assessing strategic urban sectors, their interactions with both other sectors and the wider urban system; considering an objective-driven structured assessment with the purpose of supporting the development of cities' resilience action plans and monitoring of their implementation, considering a multisectorial approach integrating the city, services and infrastructure resilience.

The preliminary RAF proposal considers alignment with international benchmarks for resilience assessment (ISO 24510:2007; ISO 55000:2014; ISO 9001:2015; ARUP 2015; UNIDSR 2017) and provides significant developments with regard to its focus on urban services.

It considers the four UN-Habitat resilience dimensions (Pagani et al. 2018): organisational, spatial, functional and physical. The organisational dimension integrates governance relations and urban population involvement, at the city level. The spatial dimension, also at the city level, refers to urban space and environment. The resilience of strategic services is assessed in the functional dimension, while the physical dimension focuses on the resilience of their infrastructure. The last two dimensions also allow knowing the contribution of each service to the city's resilience (Cardoso et al. 2019). Other dimensions, such as social or political, are only taken into account whenever important for city, services and infrastructure resilience. The services within the RAF scope are water supply, wastewater, storm water and solid waste management, electrical energy supply and mobility.

The RAF purpose (Cardoso et al. 2018) is to contribute to the following:

• Direct and facilitate an objective-driven resilience diagnosis of urban cities and services, using common criteria and identifying data gaps, strengths, weaknesses, opportunities and threats, considering performance, risk and cost.

• Support decisions on selection of resilience measures and on the development of strategies to enhance resilience to climate change-related hazards.

• Outline a path to co-build resilience action plans tailored to each city.

• Track the progress of resilience in the city or service over time.

The RAF has a tree structure that follows the approach described in step 2 of the methods. For each dimension, resilience objectives are defined. The functional and physical dimensions first unfold into sub-dimensions (one for each service under assessment), which are then assigned specific objectives. For each objective (Obj), key criteria (Crit) are specified and, for each criterion, metrics (Met) are identified, including reference values. This structure is illustrated in Figure 1.

The framework is complemented with the characterisation of the city and the services context, essential to the interpretation of the assessment results. The main characterisation themes are geography, climate, population, economy and governance, built environment and infrastructure, type of customers, components and dimension of services and infrastructures.

For all the RAF metrics, a set of answers are defined and associated with resilience development levels. The resilience development levels are classified as incipient (for results that are still non-existent or at an early stage of development), progressing (for situations where significant steps have already been taken and the city or the service are still developing the specific aspect addressed by the metric) or advanced (for already consolidated results). The tree structure allows identification of the development level for each criterion, taking into account the various metrics that contribute to it. Likewise, it is possible to assess the development level of a certain objective, service or resilience dimension, allowing a more aggregated result.

The RAF provides the assignment of a degree of relevance: essential, complementary, and comprehensive to each metric. Based on this feature, the deeper insight assessment may firstly be carried out for the essential metrics, if a city is still initiating its path on resilience, then for the complementary metrics and further on for the comprehensive metrics. Therefore, the proposed RAF enables a tailored assessment of any city, regardless of their resilience maturity, and supports the identification of a resilience development level for each dimension and for each service.

The RAF validation through working group discussions used survey tools addressing the following themes:

• Adequacy of the RAF dimensions

• Assessment frameworks available in the cities

• Feasibility of usage and flexibility of the RAF preliminary version

• Selection of assessment scenarios

• Disaggregation of the mobility service

• Details of the city and service characterisation profiles

From Figure 2, it is evident that the main purposes for the RAF identified by the participants (Figure 2(a)) are to support the resilience action plans in the cities, to raise awareness of the services' contributions to the cities' resilience, to identify the importance of the infrastructures to the cities' resilience as well as the measures to improve resilience. Stakeholders strongly agree to assess predicted consequences of climate change scenarios and historical extreme events and agree with less intensity to consider everyday life events (Figure 2(b)). Also evident is the agreement with inclusion in the assessment of the most probable and most severe climate change scenarios, for the different hazards to be studied (Figure 2(c)).

Workshops were held with a 1-day duration in each city, Lisbon, Bristol and Barcelona. In Lisbon, 38 stakeholders' representatives attended the session, from 14 different organisations, including Lisbon Municipality that was represented by the civil protection, urban hygiene, urban planning and green infrastructure sectors; wastewater, mobility and electrical energy distribution sectors; national cultural heritage, territorial governance, mobility, transports and meteorological services institutions; engineering consultants and the Lisbon underground railway operator. In Bristol, 24 stakeholders' representatives attended the sessions, from 13 different organisations. Involved sectors included those such as water supply, electricity distribution and waste management sectors; engineering consultants; Bristol City Council represented by sustainability, transport, open data and civil protection sectors; railway infrastructure; government environment agency. In Barcelona, 28 stakeholders' representatives attended the session, from 18 different organisations. Here the involved organisations included those such as the Barcelona metropolitan area management; Barcelona City Council represented by infrastructures, mobility, urban planning and waste sectors; water cycle management; water technologies and consultants; metropolitan transport and energy management authorities; energy research institute of Catalonia and electrical energy distribution. All the involved organisations contributed with a multidisciplinary knowledge for the purpose of the workshops.

Organisational and spatial resilience dimensions were addressed by mixed-teams' brainstorming and functional and physical dimensions were addressed by sector teams' brainstorming, regarding each RESCCUE service. The results of these sessions allowed improvement of the RAF, namely, regarding the metrics definitions, the decision on whether or not to keep them in the RAF and sustaining the assignment of the relevance of each metric (Figure 3). As an overall summary:

• Most of the proposed criteria were considered essential.

• 15–30% of the criteria were considered either complementary or comprehensive.

• Differences between the cities were taken into account considering cities' context and stakeholder diversity.

• For all the dimensions, depending on the criteria, between up to two or to 14 stakeholders expressed their availability to contribute to quantify all the metrics in the criteria (Figure 4).

In these workshops, stakeholders' validated all the RAF criteria, namely, regarding their relevance for each objective (essential, complementary or comprehensive) and applicability to their respective city, including identification of the possible contribution of each entity to the determination of metrics and providing opportunities for RAF improvement. It was also possible to collect stakeholders' opinion on the relevance of a city's properties for resilience – such as redundancy, flexibility, adaptation capacity, self-reliance or autonomy, reliability, recovery capacity, capacity for continuous improvement and emergency response capacity – to ensure that the RAF addresses all of the most relevant concerns.

The purpose of the RAF testing was to ensure coherence, feasibility and effectiveness of the approach, and of its purpose. The RAF testing on the cities provided results regarding the following:

• Identification of answered or unanswered metrics (Figure 5).

• Clarification of the reason why metrics were not answered (i.e., whether they were not applicable to the city, there were no data available for calculation or data would still be available during the project's timeline).

• Preliminary city and service assessments (Figure 6).

In Figure 5, the overall three cities testing results are presented, regarding the percentage of metrics that were answered (Answered), not answered (NAnswered) and those that were not answered because they are related to the assessment of climate change scenarios (NAns_Sc), still being used by the cities and services. For each objective, in the organisational (O.Org.) and spatial (O.S.) dimensions, between 90% and 100% of the metrics were answered. For the functional (O.F.) and physical (O.P.) dimensions, above 70% and above 58% were answered, respectively. In these two last dimensions, a higher percentage of metrics was reported as still being under assessment for the climate change scenarios.

From the results, it is possible to conclude that all the objectives in the RAF can be evaluated by the cities. For the organisational and spatial dimensions, no relevant difficulties were identified. For the physical dimension, greater difficulty is evident even compared to the functional dimension. This can be associated with less information available. Based on the cities' testing, it was possible to identify the RAF components that benefited from additional improvements and those that less fitted the cities' available information, thus supporting revisions.

In Figure 6, to illustrate the results related to the RAF purpose, overall three cities testing results for the organisational and spatial dimensions are presented, regarding the preliminary assessment. Figure 6(a) presents the main strengths for the organisational dimension (inner circle), identifying the resilience objectives (middle circle) and criteria (outer circle) where the cities present metrics with advanced level of development. The cities are well developed regarding leadership and management (O.Org.2) and city preparedness (O.Org.3), and present some relevant developments on collective engagement and awareness (O.Org.1). Figure 6(b) presents the main weaknesses of the spatial dimension, identifying the resilience objectives and criteria where the cities have metrics with incipient level of development. The main opportunities for development are those related to the provision of protective infrastructure and ecosystems (O.S.2) as well as the spatial risk management (O.S.1) from the points of view of the resilient urban development and hazard and exposure mapping, although there are already significant developments in the cities in this dimension of resilience.

The analysis of results from the RAF validation provided a final version summarised in Table 1 regarding organisational and spatial dimensions, and Table 2 regarding functional and physical dimensions.

A resilience assessment framework was developed, within the scope of CC and focused on the urban water cycle, in order to direct and facilitate an objective-driven resilience diagnosis of urban cities and services; to support decisions on selection of resilience measures and development of strategies to enhance resilience to climate change; to outline a path to co-build resilience action plans; and to track the progress of resilience in the city or in the service over time. The main purpose of the RAF application is to identify the real needs of the cities and services to enhance urban resilience, since it directs and facilitates a structured resilience diagnosis of the cities and strategic urban sectors.

The development of the RAF followed a step-by-step approach that considered the main recommendations indicated in the literature. Particularly, keeping the alignment with international frameworks for resilience assessment, in order to facilitate its use by the cities that are already applying them, as is the case of Lisbon, Barcelona and Bristol; adopting an objective-driven assessment that showed to be flexible during its testing application by these cities and services, and facilitates a structured diagnosis; ensuring a broader engagement of stakeholders in the development and implementation of this assessment tool, namely, by organisation of workshops with the relevant stakeholders in these cities. Involving stakeholders in the validation of the RAF preliminary version proved to be a relevant process, by integrating their contributions in the framework and raising awareness of their role in the path to city resilience.

Even though it was developed within RESCCUE and to support RESCCUE cities and services, replication was its foundation. Given its different assessment levels, it may be used by any city, service or organisation that intends to undertake a resilience assessment or to develop a RAP with this scope and focus, regardless of their resilience maturity. The RAF allows alignment with the resilience path and integrate the work already in place in the cities and services, as well as to consider the information provided by diverse analysis approaches and tools, already in use or to be used by the city and services managers.

The approach for the RAF development proved to be successful since it provided a flexible and structured framework allowing further inclusion of additional dimensions, such as social or economic, and of other objectives, criteria and metrics, for the services already addressed. Moreover, it may be strengthened with the incorporation of other services, such as telecommunication, education or health.

The main purpose of the RAF application is that the investment to be realised in the knowledge of the cities' resilience may be translated by concrete benefits, resulting from the implementation of measures and strategies, ensuring a sustained path to enhance the resilience of cities.

The RESCCUE project has received funding from the European Commission by means of Horizon 2020, the EU Framework Programme for Research and Innovation, under Grant Agreement no. 700174. Acknowledgements are due to the Research Site Leaders of Barcelona, Bristol and Lisbon and to Cristina Lucas Pereira, LNEC Research Fellow.