Facing global transitions in water management: Advances in knowledge and capacity development and towards adaptive approaches

Water management is carried out by stakeholders and institutions – the organisations and the rules that govern and administer society's activities. They function by virtue of their capacity to make decisions and get these implemented, supported by a combination of their knowledge and (political) commitment (North, 1990; Sveiby, 2001). Knowledge is necessary to articulate the subject matter – ‘what we need to do or achieve’ – as well as the implementation process – ‘how we will do it’. Knowledge is central to capacity which is the capability to decide on goals, act to attain these, and to learn from this experience. This paper aims to review the advances made in the past three decades in practising knowledge and capacity development (KCD) by assessing the complementary roles of institutions, knowledge, and policy design and implementation. It will argue that new systemic challenges are arising in the next three decades that will emphasise the need for more effective policy implementation for adaptive water management. This forward-looking review thus sits at the mid-point of a period spanning about six decades bounded by the first identification in 1991 of weak institutional capacity as one main reason for the partial failure of the 1981–1990 UN Drinking Water Supply and Sanitation Decade (IDWSSD) (Alaerts et al., 1991; O'Rourke, 1992) and by about the year 2050 that according to many water and climate forecasts risks becoming a tipping point and by which time better resilience should be achieved (see further). While the paper concerns primarily countries in development, the knowledge and capacity challenges are common to all countries irrespective of their wealth.

Epistemology has defined knowledge as the capacity to act (Polanyi, 1958). Knowledge is acquired by an individual through reasoning, discovery, and learning. Whereas in industrialised economies and in firms the literature is concerned with knowledge management to enhance performance, in public management in developing economies these processes are generally considered capacity development (Alaerts, 1999). In economies with a robust education system, strong governance and high levels of communication and transparency, new knowledge can dissipate relatively fast across institutions and society and be applied through decisions in commercial transactions, regulations and procedures decided at national and local government levels. Citizens and their representatives participate in these processes. In developing economies, on the other hand, new knowledge travels more slowly through a nation. Policy innovation and reform that imply redistribution of power and delegated decision-making can only be implemented stage-wise as the prospective new actors need time to acquire the capacity to understand their new role, agree to it, and start acting accordingly. Policy reform thus rests on simultaneity of political decision and capacity. For capacity to be acquired, political will must exist, and to inform sound political decision-making, capacity is a prerequisite. The effective impact of a policy is predicated on implementation capacity.

Knowledge is central to institutional capacity yet it rarely receives explicit strategic attention. Typically, it is identified as an issue when its absence is considered the cause of current poor performance, but is rarely subject of strategic policy and sector-wide management. Alaerts & Kaspersma (2009) propose an analytical framework that emphasises that capacity and knowledge must exist at the following four nested levels: (1) individuals need to possess generic and functional knowledge in the meanings of factual knowledge (in the sense of the scientific consensus on discoverable and refutable physical, economic and behavioural phenomena), understanding, skills, and attitudes; (2) organisations, i.e. institutions in which the individuals work or interact must dispose of specific problem-solving heuristics (‘operational practices’) by aggregating individual staff knowledge but also by rendering this knowledge effective and acquire and embed new knowledge in these practices – the organisations becoming ‘learning organisations’; (3) the institutional enabling environment with the institutions of government, policies, regulations, and agreements in which these organisations operate; and (4) civil society, i.e. civic groups or society, as they are seeking to understand their challenges, negotiate shared values, and acquire new knowledge.

For conceptual and operational reasons, KCD concerns itself with learning and knowledge generation and sharing. Many insights are drawn from traditional educational and pedagogical principles but others were formulated only in the second half of the 20th century, often in the context of management or behavioural sciences. Epistemology highlights the difference between explicit knowledge that is formalised, e.g. in text books and can be easily transferred, and implicit (tacit) knowledge that cannot be codified and needs to be discovered through example and trial and error, like riding a bike (Polanyi, 1958). Generally, knowledge acquisition presupposes a willingness to discover new insight, or learn, and takes five main routes. Two routes occur within an individual's own cognitive capability through deduction or induction. Three other routes are exogenous (or a hybrid): experiential learning (e.g. as learning-by-doing [e.g. Kolb, 1984] or structured experiential learning [Pritchett et al., 2013]), absorption of external knowledge (e.g. in classroom teaching and most training) and mimicking (e.g. in formal or informal mentoring and peer-learning in organisations, communities-of-practice or networks [e.g. Schein, 1978; Guilmette, 2009; Mangin & Dunsmore, 2014]). Learning processes in which we are emotionally engaged or discover by experiment generate knowledge that sticks better to our memory (e.g. Niedenthal et al., 2006). Finally, pedagogy and behavioural sciences offer insight into methodologies for effective teaching and knowledge transfer in both physical teaching and internet-based education.

The insight that knowledge is also an institutional attribute emerged in the mid-20th century when knowledge as a productive economic input became valued in macro-economic models (e.g. OECD, 1996). Knowledge operates by inducing changes in behaviour and decision-making and adds value to society and firms, creating a competitive advantage by taking the form of an educated workforce and of know-how and innovation capability in the organisation (as in intellectual capital and patents). When new policies are to be implemented in complex environments where many institutions are actors – typical for water management – the capacity of all institutions often needs to be developed. Institutional capacity can be developed through, e.g. formal education (for the longer term); training for imparting competencies, skills, and attitudes; human resources management; research and innovation; development of policy and administrative arrangements based on, and reinforcing knowledge; peer-learning inside and among institutions; coaching and mentoring; and learning-by-doing. A central element in institutional learning is structured monitoring and evaluation (M&E) of results against planned outcomes and feeding back lessons learned during this implementation. Public institutions are still slow in adopting learning strategies though the corporate sector already in the 1980s and 1990s started to treat knowledge as a firm's asset that needs to be managed for the organisations to be effective. Firms like Toyota, General Electric, and Asea Brown Boveri pioneered new management approaches. Senge (1990), Argyris (1992), Nonaka & Takeuchi (1995), Sveiby (1997) and Easterby-Smith & Lyles (2003), among others, developed the discipline of organisational knowledge management. It applies concepts of human resources management and competences management; the articulation and retaining of experience, such as in best-practices and knowledge repositories; and internal learning protocols as well as managerial procedures that create space for experimentation including occasional failure¹. The recognition that knowledge is a fundamental attribute of public sector policy and administration is gaining traction only slowly. Here, it leads to more effective policies and improved problem-solving heuristics in administration and policy implementation. Some countries, such as The Netherlands, have explicit knowledge-management policies for ‘top’ sectors including that of water (Ministry of Economic Affairs, Agriculture & Innovation, 2011; CBS, 2012).

Capturing KCD in definitions has proved challenging; they tend to reflect two orientations. Morgan (1993), whose work inspired many development agencies, casts it as a fundamental governance endeavour: ‘Capacity development is the ability of individuals, groups, institutions and organisations to identify and solve development problems over time’. This broad perspective at the same time also is a weakness as it suggests capacity development is almost synonymous with development itself, without providing insight in how to operationalise ability. Other authors stress both the role in self-determination and the instrumental nature of KCD, e.g. ‘KCD is the capability of a society or community to identify and understand their development issues, to act to address these, and to learn from experience and accumulate knowledge for the future’ (Alaerts, 2009).

Over the past three decades, KCD has tended to take place in approaches that can be pragmatically grouped into six arenas characterised by purpose, context, processes, and instruments. Some of these arenas are distinct, but others overlap or intersect, or are nested in others. They are as follows: (1) development of skills for well-defined executive tasks; (2) development of complex behavioural skills; (3) education; (4) research, innovation, and R&D; (5) hardware and software for communication and decision-making; and (6) policy and administrative reform.

Arena 1 activities concern short-term development of vocational, cognitive, and sometimes interpersonal skills and competencies needed to carry out specific straightforward tasks of generally technical or mono-disciplinary nature, such as pipe-laying, water-meter reading and repair, accounting, community organising, mathematical modelling, etc. These activities have well-defined objectives, draw upon established training approaches (notably classroom training, workshops and coaching) and yield easily measurable outcomes with a clear causal training–outcome relation.² These activities are intuitively worthwhile and uncontroversial.

Arena 2 activities encompass development of more complex competencies, often of managerial or behavioural nature. This more open-ended KCD may concern professionals in managerial or other leading positions, but also diverse groups in civil society which need to be coached to identify and address their local challenges. It typically necessitates a broader more specialist toolset of teaching, structured and experiential learning, and personal discovery in a coaching frame; much of the knowledge imparting occurs through self-assessment and simulations. It draws on approaches and instruments of Arena 1 but also builds on networks, communities-of-practice, and peer-learning and peer-assessments. In local communities, processes are applied such as Participatory Rural Assessment, actor analysis and behaviour models, and Motivation and Ability frameworks. Aims and scope vary. Minimally they include developing personal management and leadership skills of individuals (on technical tasks, organisation, leadership, results-orientation, etc.) to enhance staff and institutional performance. In a broader scope, they form part of Arena 6 in transitions of (part of) organisations by enhancing the skills sets of individual staff to support the redesigned organisation.

Arena 3 concerns education and knowledge-generation. Commonly these aim to prepare young generations for their future and help develop their basic knowledge, cognitive, vocational and interpersonal skills, and attitudes, as well as their capability for continuous further learning. A vast educational infrastructure exists to achieve these aims and simultaneously set up the platform for research (Arena 4). The KCD instruments range from classroom teaching to experiential learning, coaching, peer-learning, mimicking and mid-career education; pedagogy offers menus of tested instruments and approaches for different contexts and purposes. The outcome of education is generally well discernible at aggregate level in the form of the individual's and economic development, although the extent of the contribution of schooling to personal development is a matter of debate. The 70–20–10 rule suggests that – at least in a professional context – executives acquire 70% of their knowledge from hands-on experience, 20% through social learning, mentoring, collaborative learning and other interaction with peers, and only 10% from formal traditional courseware instruction (Lombardo & Eichinger, 1996). In a world of progressing complexity, advanced specialised knowledge has premium value yet many problems are of a wicked nature, i.e. are difficult to describe precisely in advance, subject to changing circumstances and of transdisciplinary nature. This calls for generalist knowledge. Some professionals and institutions may be able to accommodate both specialist and generalist knowledge; Uhlenbrook & de Jong (2012) argue that professionals are best equipped with T-shaped competencies. In most cases, however, effective heuristics will increasingly depend on the capability of individuals and institutions to collaborate in teams or networks able to combine different specialised competencies.

Arena 4 focuses on the discovery and generation of knowledge, such as in scientific research, R&D, innovation, and citizen-based science. The processes involved often are partially integrated with those in the other arenas. Arena 5 concerns the development and use of hardware and software upon which much of the communication, information sharing, training, simulations and decision-making is resting nowadays. The deeper penetration of hand-held phones and internet-based tools keep contributing to faster and wider knowledge dissemination and decision-making.

Finally, in Arena 6 the KCD ambition extends to the support to a change, reform programme or transition in organisations, policies or even assemblies of sectoral actors and institutions such as ministries, (water) agencies as well as political entities. In many instances – as water is affected by the behaviour and decisions of households and other users – this may also encompass civic stakeholders; e.g. farmers play key roles in irrigation, and industry in water pollution control. Focus lies on both policy articulation and its implementation. Like in Arena 2, KCD is rarely of simple technical nature and generally implies articulation of preferences and political choices for which consensus needs to be sought and sustained over prolonged periods, often 10–20 years. Such organisational change programmes are common in the corporate world but less so in the public sector or academic institutions, and when they exist, are often not well documented. Examples for the water sector refer to, e.g. the Uganda National Water and Sewerage Corp. and other African utilities (Mugisha et al., 2007; Mugisha & Brown, 2010), the Philippines and Malaysia (Jensen, 2019) and the Netherlands Ministry of Transport and Water Management (Metze, 2008). Recent studies propose ‘social learning’ to enhance management, e.g. for irrigation in Australia (Nikkels et al., 2019). Depending on scope, ambition and readiness, the duration of a KCD intervention likely takes many years if not decades. KCD outcomes are not well quantifiable or predictable and impact may have to be assessed through proxies that measure the institution's effectiveness on Key Performance Indicators or through perception surveys. A direct causality between interventions and indicators is often tenuous. Critically, in Arena 6, KCD intervenes in the management purpose, style and internal workings of the organisation, and can succeed only if political commitment exists to adopt and sustain the changes that re-arrange power. Though many sectoral assessments identify organisational weaknesses and the ‘need to change’ to achieve more effectiveness, political, institutional and often personal reluctance create roadblocks. Beside the complexity of the problem analysis and the KCD interventions at such large scale, often across several institutional jurisdictions, and the political contentiousness, the ultimate goals, pathways, benefits and risks of the reform process are generally not well known up-front and need to be discovered during step-wise adaptive implementation. Activities typically include approaches and instruments of Arenas 1 and 2 aimed at specific groups of stakeholders and decision-makers, beside more process-oriented tools such as Results Frameworks to monitor and evaluate interim achievements, identify and address constraints, reveal unanticipated changes in the environment or internal objectives, and adapt to these new circumstances (see The next challenge: KCD to support adaptive policy implementation).

It remains challenging for notably developing nations to design and implement policies that are effective and help meet the targets of, e.g. the Sustainable Development Goals (SDGs). The High-Level Forums on Aid Effectiveness (Paris, 2005; Accra, 2008) identified weak institutional capacities for policymaking and implementation as the key impediment (OECD, 2019a). Reviewing international development performance, Andrews et al. (2017) conclude that implementation of policy is the main challenge, more than its design. Implementation capacity is at risk also in developed economies; notwithstanding tall ambitions to boost infrastructure and adopt green pathways, many initiatives are slow to materialise and generate realistic investment opportunities (OECD, 2021; Alaerts, 2022). The 2009 American Recovery and Reinvestment Act to boost infrastructure spending after the 2008 financial crisis partly failed due to a shortage of ‘shovel-ready’ proposals. In the Netherlands, concern is growing that government is increasingly beholden to short-term scoring with policy and legislation, at the expense of implementation realism (NRC, 2020). The articulation and implementation of realistic policies and investment programmes concerning water are especially challenging compared to, e.g. energy and transport, because of their complexity (as discussed below).

The process of articulating and implementing effective policies has been studied extensively. Kuhn (1962) posits paradigm shifts as the basis for scientific (and by extension, political) revolutions driven by growing consensus among experts regarding progressing dysfunction of the prevailing paradigm. Van der Brugge et al. (2005) show that institutions and policy domains tend to stability (with no or minimal fundamental change), i.e. internal optimisation rather than system innovation, to minimise transaction costs (North, 1990) or because policies are captured by groups of actors seeking the status quo (Meijerink & Huitema, 2009). According to the Multiple Streams Theory, policy changes occur when three streams (of problems, policies, and politics) converge (Kingdon, 1995). To achieve effective and efficient policy setting, impact monitoring and policy implementation must be approached as knowledge-intensive. The Advocacy Coalition Approach (Sabatier & Jenkins-Smith, 1993) stresses the crucial role of knowledge and longer-term learning processes in the shaping of the coalitions. The Contextual Interaction Theory (Bressers, 2007) identifies motivation, information, and power as decisive factors for change. To these three change factors and to the three ‘streams’, before, one should add the role of personalities (personal ambitions of decision-makers) as well as institutional inertia that may work against change. Gidron & Bar-Almog (2010) recommend to observe policy change over a period of a decade or longer to find out how policy analysis shapes the agenda and how learning takes place. Also, Rotmans et al. (2001) emphasise learning as central to policy transitions. Most literature on policy development is geared to industrialised economies which possess a good governance architecture and educated citizenry. The Contextual Interaction Theory therefore highlights the limitations of replication of institutional models and their intrinsic knowledge from one cultural and political context to another (Bressers, 2007; Bressers & de Boer, 2013). Policy change is particularly problematic where the government is at the same time a main actor in its implementation, because public sector agencies are more subject to rigidities and inefficiencies than the private sector. This challenge notably affects developing countries where bureaucracies tend to be more prone to inertia and elite capture, and where knowledge on implementation is less easily available.

Also, health services and environmental management now consider weak implementation capacity a systemic challenge. However, the focus here is on more rapid and systematic uptake of research results into practice (for the health sector ‘from lab bench to bedside’, down from the average 17 years), taking into account the institutional context and applying a structured monitoring framework. They appear less concerned with the management of the environmental or health management system itself (Bauer et al., 2015; Hering, 2018)³.

While the above suggests that a deeper understanding of policy and institutional reform and the role of KCD has evolved over the past decades, a discrepancy is often observed between the diagnostics – the analysis of why a system is functioning poorly – and what improvements can be achieved by KCD on the ground. Examples of diagnostic frameworks are those of EuropeAid (2005), UNDP (2007), ECDPM (Keijzer et al., 2011), ADB (2011) and GiZ (2015). In line with Morgan's definition (see higher), these diagnostic frameworks generally have ambitious analytical scopes covering a sector, or a set of policies and organisations regarding their policy and governance (Arena 6) but are less clear on what actual KCD programme would be able to develop this capacity. They tend to fall back on common intuitive instruments, such as education and training of individual staff, that can be executed in a project's timeframe of 3–4 years. Possible causes for the discrepancy are as follows: (i) over-reliance on exogenous inputs instead of building on local initiatives, (ii) underestimation of the need of political negotiation (e.g. Meijerink & Huitema, 2009) to secure commitment, even on part of the goals, (iii) the assumption that a change process can be planned rather than built up iteratively, and (iv) the high cost and duration of the engagement.

Current global policy narratives on water are shaped by the SDG6 ‘to achieve universal access to safe drinking water and sanitation for all by 2030’, halve pollution and apply Integrated Water Resources Management. It continues the 1981–1990 IDWSSD and the Millennium Development Goal 7 (2000–2015). These goals and their implementation strategies were based on scaling up past experiences, extrapolating historical data without addressing the growing implicit competition for water caused by the simultaneous goals to increase food and energy production and job creation in a world where water is already scarce. More recent policy discourses, on the other hand, are founded increasingly on forecasts and scenarios derived from mathematical simulation models based on deeper insight in these interconnected systems. The suites of climate change models for instance describe relatively reliable future scenarios linking changes in temperature, sea level and precipitation and evapotranspiration with locations. With expanding knowledge of these systems and more powerful simulation capability, planning will become more able to forecast non-linearity, competition and discontinuity in the water system.

Water sits at the nexus of several natural and social systems that require water to thrive (health, food production, ecosystems, etc.) and, in turn, decide on policies that impact heavily on water. Demographic and economic growth are the main stressors on water. Water over-abstraction and land use changes, including urbanisation, increasingly determine local water excess or shortfall, and quality. Most rivers and lakes are already heavily over-used. In many places the abstraction rate from aquifers exceeds recharge; concern is growing across the globe – from the Ogallala aquifer in the mid-west US, to south-western Europe, India's Rajasthan and China's north-eastern Hebei-Shandong region – that many local stocks will run dry in the next 2–3 decades unless drastic measures are taken to return to sustainable abstraction rates (Famiglietti, 2014; Turner et al., 2019; Gleeson et al., 2020). Over-pumping in urbanised areas, or for irrigation and land reclamation, is now causing land subsidence, by 10–30 cm per year, over vast areas in Indonesia, the Beijing-Shandong basin, Mexico City, Central Valley (US), the Ganges and Indus plains, the Teheran plain, to name a few. Much of this subsidence started only in the 1970–1980s. Herrera-García et al. (2021) estimate that from 2010 to 2040 this will expose a population of 480–660 million people and 12–16% of global GDP to intensified flooding. This starkly illustrates how compartmentalised water exploitation and poor land management are exhausting water resources while at the same time dramatically increasing flood risk. In the mid-term, the main drivers for aridification and increased flooding are conversions in land use that channel rain run-off faster to the river, such as by removal of forest canopies and marshland, and urban ‘hard surface’ such as roofs, roads and parking lots. At continental scale, vegetative cover evaporates moisture into the atmosphere that feeds precipitation elsewhere. In South America, the Río de la Plata basin depends on evaporation from the Amazon forest for 70% of its water resources. Thus, intensive conversion of forest to urban and agricultural land will have a major negative impact on water availability (Van der Ent et al., 2010).

Water, ecosystems, and biodiversity exist in critical mutual dependence. On an estimated 23% of the global terrestrial area, current biomass productivity is estimated to have been depressed by land and water use changes and other interventions, compared to the undisturbed situation; this is affecting 36% of all cropland, pasture and forestry systems and 15% of natural areas. Biodiversity and ecosystems are by themselves very productive in a narrow economic sense (fish, timber, and produce) but the combination of landscape and vegetation (‘green infrastructure’) in turn ensures a healthy regenerative hydrologic system. OECD (2017, 2019b) estimates the current global productive value of ecosystems and biodiversity at about US$140 trillion, or 1.5 times global GDP, yet these systems are being degraded at an increasing rate. Biodiversity loss was estimated at 34% by 2010 and is projected to continue with about 10% of additional loss up to 2050, a juncture that may turn critical with respect to reversibility in land and soil degradation jeopardising water security (Van der Esch et al., 2017; WWF, 2020). Studies have assessed the overall implications of combined water (over-)use and land use change. Drought appears as the deadliest physical hazard with at present an estimated 3.6 billion people living in areas that are water-scarce for at least one month per year, to increase to 4.8–5.7 billion by 2050 (i.e. 55–65% of the world population) with over 40% of the world population living in water-scarce river basins under severe water stress, up from about 16% in 2010 (OECD, 2012; Sadoff et al., 2015; WWAP, 2018). Thus, the world started transitioning from a place that is predominantly wet to one that is predominantly arid. Driven by this increasingly unsustainable dynamic, the World Bank (2016) forecasts a likely decline in several countries’ GDP of up to 6% by 2050 caused by water-induced losses in agriculture, health, income and property, with some regions in the world facing sustained negative growth.

Climate change is projected to significantly increase the population facing water scarcity and/or major river floods in the 21st century. The time frames for the scenario forecasts extend to 2050, 2100, and beyond (IPCC, 2021, 2022). Schewe et al. (2014) calculate that for each degree of global warming, an additional 7% of the global population will be exposed to a decrease in renewable water resources of at least 20%, and on average, 4% of the global land area will see, for each degree of global warming, a decrease in renewable groundwater resources of more than 30%, and 1% by a decrease of more than 70% (Portmann, 2013). Thus, by the mid-century most regions will be deeply affected by such changes.

The above highlights, first, that policy design is transitioning from a paradigm determined primarily by the extrapolation of historical data sets, to one driven by simulated scenario forecasts with a focus on the situation by about 2050. Second, the year 2050 may represent a tipping point with the water sector looking fundamentally different in the second half of the century than in the first. And third, where the conventional paradigm assumed that all demand – by households, agriculture, industry, etc – could be satisfied and that any negative impacts of water appropriation could be mitigated through add-on measures, the new paradigm accepts that the water system has inherent deep constraints and seeks increased efficiency of water use, protection of land and ecological systems, reduction of the ‘water footprint’ of economic activity, and more reliance on a circular economy including efficient water allocation. Capable and well-informed institutions will be essential to all policy and planning, and their implementation.

Water management is also undergoing transformations at institutional level. In the corporate sector complexity and uncertainty are being recognised as new systemic features (McGrath, 2011). For the water sector such transitions are reflected notably in legal and regulatory frames, policy documents and social analyses, which converge in the operational tasks, priorities and goals as perceived by the leadership of water utilities and agencies. These transformations are not well analysed yet. However, the authors conducted an exploratory survey and literature review for the situation of the regional (drinking water) utilities and the regional wastewater and flood management agencies (Water Agencies) of The Netherlands spanning the period 1970–2030⁴. Though confined in scope and methodology the outcomes are considered meaningful and also reflecting trends in higher-income economies as this country is considered to be performing well on governance and overall capability (OECD, 2014). Table 1 provides a qualitative synthesis of the declared perceptions. The leadership is faced notably by expanding sectoral and water-relevant general regulation, including, since the early 2000s, a growing body of EU Directives that together raise requirements (e.g. in drinking and surface water quality, and in financial performance and tariff caps) while narrowing the room for manoeuvring in decision-making. Stricter regulations on land use and spatial management, and nature protection, restrict options for sustained service delivery; many regulations prove simple add-ons rather than streamlined instruments reconciling opposing objectives and trade-offs. Illustratively, some utilities tend to prefer groundwater for supply augmentation over surface water because the former is less tightly regulated. Over the past two decades political and public scrutiny and interference have significantly increased by local governments seeking rents and local communities and vocal citizens expressing demands via informal (twitters) and formal (political) channels. Water Agencies in 2010 were close to being abolished as they were considered not cost-effective in the then-prevailing market liberalisation context. From the 1970s on water management has been recognised as knowledge-intensive: relying on novel technologies and a specialised work force. The sector has had to compete for qualified staff in a structurally tight labour market. Utilities and agencies now increasingly must align with an expanding set of municipal and regional policies such as on saving of water and other resources, decarbonisation, and circular economy. This imposes much closer collaboration and integration with other sectors and closer relationships with local and national-government administrations and industry. Whereas 50 years ago it was challenging to raise scarce capital for ambitious sectoral expansion investments, this constraint subsided in the 1990s (though less so for the agencies) but is now again a management pre-occupation as large physical assets such as sewers, pipelines and plants need to be replaced and measures against droughts put in place while tariffs are capped. Finally, the planning horizons have gradually extended to several decades in line with the growing capacity to forecast sectoral developments and the drive to manage assets and optimise investments. However, this generates significant uncertainty inherent to the forecasting models and the novel nature of many management measures.

Countries in development and in economic transition, on the other hand, arguably are facing a double challenge: complexity (and need for know-how) is growing and changes are rapid. The growth rate of complexity can be illustrated by observing that during the typical career of a water professional most countries are experiencing rapid growth in demography and prosperity. Simultaneously, their water resources and ecosystems are being exploited to, and now often beyond, their sustainability boundaries. A professional in such country, educated in the 1980s using textbooks published in the 1970s has a career spanning the period 1990–2030 and must manage the growing pressures on water in that period resulting from a demographic growth of 40–100%, the transition from a rural to an urban landscape, a 10-fold increase in tap connections, and households and industry turning 3–5 times as wealthy, each consuming 2–10 times more water. The fact that these countries often are heterogenous, comprising richer/urban and poorer/rural parts, adds to the complexity. In addition, they are facing the challenge of simultaneously addressing traditional water challenges (e.g. expanding service to all households) and the new ones of changing climate adaptation, rapid urban expansion, etc. They tend to rather rely on infrastructural approaches to manage supply but have also to start investing in the application of IWRM, institutional development and KCD. At the same time, the need for infrastructure remains large such as, in Africa, for water storage capacity.

The transitions in the physical and institutional realms possess three salient characteristics: complexity, uncertainty, and rapid change. Complexity refers to challenges that are associated with initial ignorance, deep uncertainty and potential contentiousness; cannot be mastered with straightforward measures (which distinguishes complex from complicated challenges); involve many different actors spanning various sectors and jurisdictions; are at least partly in continuing change so that, by the time part of the ignorance is uncovered, the circumstances have changed again; are transaction intensive; and necessitate an experiential discovery pathway of which the ultimate desired outcome is not guaranteed (Glouberman & Zimmerman, 2004; Snyder, 2013; Andrews et al., 2017). This suggests that, looking forward, two main programme lines are unfolding. First, many challenges will project into an uncertain future and need an adaptive, iterative learning approach to uncover realistic pathways towards resolution. Second, the transition from static, bureaucratic approaches to ‘organic’ and dynamic ones able to manage the changes will increasingly necessitate institutional change and policy reform. The water sector needs to explore and equip itself with approaches and instruments to strengthen the knowledge base and institutional capacities required for addressing these transitions while complementing the traditional agendas that remain important.

Whereas typical policies are normative, suggesting full knowledge of what needs to be achieved, several authors have proposed policies that are better able to deal with complexity, uncertainty and rapid change, to design and agree on new appropriate policies and implement them effectively. In uncertain situations, experts do not know, or the parties to a decision cannot agree upon (i) the external context of the system, (ii) how the system works and its boundaries, and/or (iii) the outcomes of interest from the system and/or their relative importance. Deep uncertainty also arises from actions taken over time in response to unpredictably evolving situations (Haasnoot et al., 2013). Large-scale investment programmes, whether infrastructural (roads, energy, water) or social (education, health, etc.) typically span several decades and the design of the investments in the later part depends on fundamental early choices (path dependency) at a time when information is constrained most. The gap between available and needed knowledge will be smaller in countries with an extensive knowledge infrastructure and strong societal trust in knowledge and governance; adaptive management (AM) operates better in environments where cultural, financial and political space exists for ‘learning’ and mistakes. Developing countries, thus, may be at a disadvantage compared to richer economies.

Robust Decision-Making (RDM) (Lempert et al., 2006; Bryant & Lempert, 2010) was proposed as a multiple-scenario evaluation framework for making decisions (in rich economies) on large infrastructural programmes with a large number of highly imperfect forecasts of the future. Rather than relying on improved point forecasts or probabilistic predictions, RDM describes many plausible futures relying heavily on stakeholder involvement, then helps analysts and decision-makers identify near-term actions that are robust across a wide range of futures – that is, actions that promise to do a reasonable job of achieving the decision-makers’ goals compared to the alternative options, no matter what future comes to pass. Pahl-Wostl (2007) and Pahl-Wostl et al. (2007) propose AM to continually adjust water management decisions for Integrated Water Resources Management where positions of multiple stakeholders and longer-term consequences of decisions are uncertain. Kwakkel et al. (2016), Haasnoot et al. (2018) and Haasnoot & Warren (2019), also working mostly in rich economies on water investments to adapt to climate change, devised Dynamic Adaptive Policy Pathways (DAPPs) that continually adjust early adaptation management to environments that are complex, cannot be fully understood up-front, keep evolving and are sensitive to path-dependency. Andrews et al. (2017) propose for developing economies a Problem-Driven Iterative Adaptive Process (PDIA) to prepare effective longer-term development policies (irrespective of the sector) of which the outcomes are uncertain. These AM approaches are long-term iterative processes in which at the onset long-term goals are kept generic and vague but the goals of the imminent iteration well specified, based on the best current insights and making all assumptions explicit. By systematic M&E progress and failure, and assessing the assumptions after each step against expected outcome, uncertainty is reduced step-wise. Thus, the policy is gradually being implemented using best available knowledge and information is steadily accumulated to prepare future decisions. AM is a collection of approaches that help implement programmes or change a system by purposefully learning about the system. Surprisingly, only in the PDIA literature the need for KCD is mentioned explicitly; arguably, as the other AM approaches seem concerned with contexts in richer countries, they may assume knowledge and education systems already in place and sufficiently effective.

AM approaches are knowledge-intensive and draw upon the Arena 6 KCD interventions, and thus also include those of the other Arenas. This intensity may put limits to the applicability of AM. For example, RDM rests on economic valuation of the alternative scenarios for which data may simply be lacking – typical for many developing countries. Because such policies always relate to longer-term programmes spanning decades, broad political support among decision-makers is paramount to ensure sustained commitment, also to the recognition that (partial) failures along the road are likely and represent valuable lessons shaping changes in the course of action that must be embraced. To enhance buy-in at the top, Lawrence & Haasnoot (2016) suggest to ensure opinion diversity by involving public sector actors from multiple governance levels and the private sector; structured interaction between scientists and stakeholders; and dedicated communication on uncertainty.

Once an AM policy is launched learning must be a constituent part, from the individual up to institutional levels (Fabricius & Cundill, 2014; Williams & Brown, 2018; Nikkels et al., 2019; Pasanen & Barnett, 2019). AM requires structured M&E that is oriented towards both learning regarding effectiveness of the preceding steps and their assumptions, and accountability. Which M&E approaches are suitable for use in AM depends on the evidence and data needs to make informed adaptations, the type of programme for which it is used and available resources (Pasanen & Barnett, 2019). A useful format is the Results Framework applied by the World Bank, which identifies medium-term goals, indicators, short-term annual quantitative plan outputs, risks and recourse options. Similarly, the Theory of Change (ToC) is a mechanism to describe the set of assumptions that explain both the phases that lead to a long-term goal, and the connections between these activities and the programme's outcomes (Anderson, 2004). It describes the logic, principles and assumptions that connect what an intervention, service or programme does, and why and how it does it, with its intended results (Ghate, 2018). To be applied well, it demands an institutional willingness to be realistic and flexible in programming responses, both at the design stage and, more importantly, in implementation and performance management (Vogel, 2012).

Knowledge and capacity that are evidently needed here are the acquisition and management of data, the modelling and interpretation of data, and forecasting using mathematical simulation models and artificial intelligence. Rather than concerning a singular device or software, this refers to a system including education and training of modelers, scientists, technicians and decision-makers able to understand what data are required and how to manage it, and well-staffed institutions to finance, develop, maintain and update the specialised equipment.

As described above, once the policy or change programme is designed and agreed upon, its implementation raises challenges, especially in developing countries. The activities involved pertain to all KCD Arenas. Alaerts & Kaspersma (2009) categorise the competences that need to be available for AM and change programmes by the level where they need to be present (individual, organisational, and institutional (enabling) environment) and their nature: technical (in the sense of mono-disciplinary competence), managerial (to manage more complex systems), for governance, and for continuous learning. Based on Kaspersma (2013), Table 2 offers an indicative set of competences for AM and change programmes, dependent on context. Where competences are lacking or weak, KCD may draw from the instruments and tools described earlier.

Agriculture is by far the largest water user across the world; 250–300 million hectares of land are irrigated, supporting about 40% of global food production and several hundreds of millions rural households. Being capital intensive and partly a public function, initially it has often been managed by central technical administrations, raising concern because of moderate to low effectiveness in food productivity and water use, due to low farmer commitment, distorting subsidies and political interference. With rising incomes, many countries started reforms after the 1980s to enhance productivity as well as water security, and lower abstraction. Indonesia, a sprawling archipelago and lower middle-income country, depends heavily on irrigation and used the political and economic recalibration of 1999–2003 to launch a process to reform irrigation from strongly top-down management – with technical capacity, decision power and budgets in the central government – to a participatory-management model with substantial authority and budget shifted to local governments (about 250 districts/counties and 25 provinces) and Water User Associations (WUAs) of farmers. The policy was formulated in 1999–2003 involving all key stakeholders drawing from extensive field pilots and was implemented in the two decades thereafter through an adaptive, sequential approach with wide-scale technical and managerial (institutional) capacity development of local governments and WUAs. Lessons from the field drove stage-wise parallel policy and regulatory reform both at central and local-government levels. Indeed, farmers nor local governments had experience with or regulations for managing their own cropping and irrigation affairs, and at central policy level it was not clear which arrangements would eventually work best without risking embarking on an ineffectual course (Alaerts, 2020). Training of local government staff concerned notably participatory and agronomic practices, coaching of WUAs, water management, social marketing, articulation of local regulations, etc. Training of farmers concerned organisation of farmers, agricultural water use and allocation, agronomic practices, dike and canal maintenance, financial management, conflict resolution, etc. National government staff was concerned primarily with issuance of national regulations and M&E frameworks, and supervising very large irrigation systems. Relatively coherent longitudinal monitoring data sets are available on field-level practices; on economic returns on the six ‘projects’ of the World Bank and ADB that financed this reform and infrastructure; and on rice production at field, project and national levels. Institutional development was monitored by performance indicators. Well over three-quarters of the country are now applying the policy albeit with varying quality. Across the country teams of trainers/field coaches (recruited from NGOs and academia) were established; they had to follow regular (re-)training by Regional Experts, and fed back field experience. At field level water supply and allocation were reported more reliable and less wasteful, and conflicts better managed. Rice production increased significantly year after year after 2005, correlating with policy roll-out.

Figure 1 schematises the adaptive implementation of the change process which applied Arenas 1, 2, 3, 4, and 6 activities in a sequence of stages. In a conventional KCD process, first a problem is identified and knowledge and capacity needs assessed, after which the capacity is developed by a third party, and a result achieved; the institutional environment is assumed fixed. Here, on the other hand, the institutional environment (policies, and national and local regulations) were adjusted stage-wise and in parallel with the expanding body of field experience. Trainers/field coaches coached farmers and local governments and assisted in the structured monitoring and evaluation of outcomes. Academic institutes played an important role in education of new cohorts of professionals and in research. This process consumed much time: trainers had to be identified, trained and re-trained; work in remote areas took long preparation time; experiences needed to be shared and vetted; and with irrigation being seasonal, results could be assessed after one year only. Each stage took 3–7 years. As Indonesia is a large country with constrained resources, early policy roll-out was limited to a few selected districts/counties, with coverage scaled up after each stage to reach over three-quarters of the country after two decades. As institutional capacity develops slowly, full roll-out may require up to three decades. The analysis shows that in the context of ‘wicked’ policy implementation, not only up-front normative policy design matters, but equally importantly, the sequencing of the process of implementation allowing adaptation to physical, institutional and political realities. Still, to be successful a process must build on a credible realistic initial ‘model’. This is especially true in developing and emerging economies with weaker institutional capacities and more fragmented governance structures.

The learning and KCD in the design and implementation stages conformed with theory: they helped inform negotiations and shape the coalitions at national and local levels (Sabatier & Jenkins-Smith, 1993) and facilitated the contextual interaction between the parallel processes (Saleth & Dinar, 2004; Bressers, 2007). They appeared critical for applying an adaptive approach (Pahl-Wostl, 2007; Andrews et al., 2017). At the same time, these process guidelines must be case-specific as the theories do not provide operational guidelines. The policy roll-out confirmed that knowledge dissipation and ensuing administrative change are ‘sticky’ and get absorbed only slowly across the nation's institutions as described by Grindle (2004) and Nickson (2006). Senge (1990), Argyris (1992) and Nonaka & Takeuchi (1995) observed the same stickiness in corporations and relate it to mental and physical barriers to change processes. Structured and experiential learning help to address this (e.g. Alaerts, 1999; Easterby-Smith & Lyles, 2003; Andrews et al., 2017).

Similar irrigation reform processes in other countries followed a less structured approach. In the Indian State of Andra Pradesh, the 1997 Irrigation Act shifted management authority from the centralised Irrigation Department to newly created WUAs, in a top-down reform (Oblitas & Peter, 1999). However, after 2000 the reform faltered; Reddy & Reddy (2005) attributed the failure to weak factual devolution, and absence of institutional capacity in WUAs and Irrigation Department with only cursory training on technical subjects for individuals, not for institutional change. The Indonesian process may have been more ‘organic’, adaptive and sequential supporting structured learning. Vietnam's irrigation sector has remained centralised through regional Irrigation Management Companies under the Ministry of Agriculture and Rural Development. The large-scale Water Resources Assistance Project (2004–2012) applied participatory design but underperformed. Though the concept of participatory irrigation is by now accepted, the operational behaviour of the companies had not adapted. Importantly, the project ambitiously rolled out this new concept but failed to provide a phased design adaptation (IEG, 2019). In Morocco's irrigation reform (1900s–early 2000s) sequential reform was applied reasonably successfully (Doukkali, 2005). Morocco could build on earlier partial liberalisation in the 1970s with a next reform stage with the Maroc Plan Vert of 2008. Here, the iterative process worked well too, with continuous interaction between the implementation projects and formulation of the regulations and an ‘adaptive’ approach as opposed to an ex ante normative one as in Andra Pradesh. Its learning mode ensured that knowledge acquired from outside the country and from the field operations could be absorbed, and that the absence of know-how at the levels of local and central governments and among the farmers were remedied through field piloting, training and exchange between regulators and practitioners.

Saleth & Dinar (2000) recommend that a reform process take a pathway in which law, policy and organisation (administration) are addressed in a sequential yet consistent manner. A progressive step on one component needs to be followed by a consistent step in another, taking advantage of path dependencies, and providing mutual support, to identify and manage political resistance to the changes in a staged way while avoiding overreach. Uphoff (1986) and Ostrom (1990) also call for ‘crafting of institutions’ that responds to circumstances rather than imposing a normative approach. Studying irrigation in Australia, Nikkels et al. (2019) observe that irrigation planning is subject to unforeseeable changes in climate, technologies and societal preferences and, thus, requires DAPP-based management. They conclude that this is to be based on ‘social learning’, though do not elaborate on this.

Knowledge and implementation capacity are critical input factors to economic and social development. Nevertheless, investment in sector-focused KCD is often modest and mostly confined to education and training (Arenas 1–3). Governments’ readiness to invest depends on the expected economic rate of return (ERR). The economic value of a ‘unit’ of knowledge or capacity is hard to quantify as it is part of an economically productive activity and its impact hard to attribute causally to knowledge content. This value emerges through its application and depends on the type and context of the knowledge or capacity. However, for several situations valuation of knowledge/capacity is feasible by isolating the effect of a factor or through simulations. Nordhaus’ seminal experiment (1996) on the value of innovation in lighting over the past three millennia suggests that over 99% of wealth creation derives from knowledge. The effect of most Arena 1 training to enhance specific skills can be causally linked to improved performance, and its economic value assessed. In other cases, especially where knowledge and capacity induce changes in the behaviour of many actors, and where other factors may affect the same outcome, attribution may be tenuous. In the case of education (Arena 3) the value of secondary and higher education has been estimated from the incremental salary enjoyed by graduates and the net benefit to society. In OECD countries, individuals with a secondary school degree tend to earn typically 23% more and those who attain tertiary education – most relevant for the knowledge-intensive water sector – an additional 50%. On average, the gross total return (net present value) of secondary and tertiary education is US$380,000 of which about 91,000 is public (or almost three times the amount of the public investment) and about 300,000 is private for men and 200,000 for women, representing a private ERR on the investment of about 12% for both sexes and about 10–11% for the public investment (OECD, 2011). For urban China, the economic, public return on schooling and higher education in the 1980s–1990s is estimated at 37–50% (Giles et al., 2015). In Australia, the private return for a post-graduate degree is on average 47% and the public return 53% (Deloitte, 2016). These figures are averages across different education orientations and it is likely that disciplines such as IT, MBA, and finance would capture somewhat higher returns than water or civil engineering, which in turn may perform better than sciences such as ecology and microbiology. Thus, significant benefits are accruing to both individual and state. In developing economies, the value of education is arguably higher due to the relative scarcity of graduates but may in some instances be lower as shallow labour markets may not offer enough appropriate jobs to reward the education. In Arenas 2 and 4, knowledge and capacity in the corporate sector are embodied in knowledge products such as patents, goodwill, human capital and networks of clients and suppliers, which are valued for instance for the purpose of the valuation of firms and stocks. Many large corporations nowadays are deriving their value mostly from such intangible assets (Tobin, 1969; Cockburn & Griliches, 1987; Sveiby, 1997; Cummins & Bawden, 2010) though the net value of patents varies significantly across countries and sectors (EC, 2006). The return on private R&D and innovation (Arena 4) has been steadily growing between 1960 and the early 2000s to typically 50–100% as public return and 10–40% as private, but may be also reaching a ceiling in certain sectors; notably public R&D yields very variable outcomes (e.g. OECD, 1996). At least in Scandinavian countries a firm's value appears more related to ‘organisational capital’, or capacity, than to its R&D or patents (Rahko, 2014).

In the public water sector, especially in developing economies, valuing knowledge and capacity is more challenging. For Arena 2 change processes in water utilities, approximations have been proposed in qualitative terms by studying the incremental enhancement of administrative practices and KPIs (Pascual Sanz et al., 2011; Mvulirwenande et al., 2015). For an Arena 6 reform process, Alaerts (2020) derived the ERR from newly developed capacity supporting the implementation of Indonesia's national irrigation reform programme (see above). The projects financed physical rehabilitation plus institutional capacity development of both the local government's and national Irrigation Departments and of the WUAs. The return that can be imputed to capacity development was estimated at about 25%, or about two-thirds in the 36% of the aggregate return. In schemes without need for rehabilitation the ERR rose to 36%, generated solely by investment in capacity development. About one-quarter to one-third of the return remained imputable to the gains from improved institutional capacity. In addition, whereas the return on investment in infrastructure is confined to the lifetime of the physical asset, investment in institutions creates the foundation for further endogenous capacity development and better governance.

Knowledge and capacity have become more broadly accepted since the 1990s in water management and sector development, as being pivotal for defining goals and priorities and implementing the activities to achieve these. They have been demonstrated to cause impact and economic return. Yet, their application still tends to be often simplified to either general education or ‘training’. Admittedly, comprehension of the full scope of KCD is challenging as it necessitates operational understanding of disciplines at the following three levels: (i) how the physical world is under threat and what action needs to be taken; (ii) how the institutional arrangements – policies, organisations, financial flows, etc. – need to be enhanced to achieve what is required under (i); and (iii) how educational, pedagogical and knowledge-management approaches can be applied to achieve the enhancement and change under (ii). Also in other sectors, knowledge and capacity are being placed at the core of development. Management sciences were first to discover the value of knowledge management, and many successful corporations started to roll out these approaches in the 1990s. In several countries the sectoral knowledge base and innovation became the subject of dedicated policies. Furthermore, international development theories have highlighted the deficiency of countries in their implementation capacity, to convert policy goals into effective action. Finally, in developed economies notably the health and environmental regulation communities have started developing an ‘implementation science’ to enhance their capacity to disseminate and embed new know-how faster.

The understanding and application of KCD have advanced substantially: (1) knowledge and capacity must be present in nested levels (individual, organisational, institutional, societal) which need to converge to cause effective action; (2) six arenas/contexts can usefully describe distinct categories of KCD initiatives; and (3) educational, pedagogical, and knowledge-management modes are outlined through which learning occurs and knowledge is imparted. However, too often KCD activities are disconnected from operational work, restricted to short-term project cycles or poorly implemented. The extensive literature on policy analysis tends to overlook the role of knowledge and capacity or treats it as a black box, perhaps because this literature rests mostly on experience in rich economies where ample knowledge and capacity are taken for granted. Many capacity-development guidelines tend to focus on the institutional change analysis, i.e. on ‘what needs to be achieved’ rather than on ‘how to do it’. Yet KCD is a sticky, slow process meaning that it requires long-time engagement of many years or decades and adequate budget.

The water sector is facing growing, urgent challenges in the next three decades: the outstanding challenge of achieving the SDGs, and a new one of adapting to climate change and building a more resilient water-and-land system. To address these, three agendas can be discerned. First, current KCD activities and practices need to be scaled up and designed better based on a deeper understanding of effective approaches and instruments, notably applying longer-term programmatic frames and better embedding them in operational work. Second, more structured KCD must be adopted with medium- to long-term perspectives aiming to support change and reform processes at policy and organisational levels, to address the poor implementation of well-intentioned policies for want of institutional capacity. This challenge pertains to developing and rich countries alike. It calls for the discovery of what constitutes effective implementation through an iterative adaptive process and an implemenatation science. Third, as pressures on the water and land system across the globe are rising rapidly, effective policies must increasingly become pro-active, shaped by modelled forecasts; organisations and institutions will need to become enabled to change and adapt to future scenarios that are more complex, uncertain and evolving rapidly. Enhancing knowledge and (institutional) capacity for designing and implementing policies, and establishing ‘learning organisations’ – and, by extension, learning societies – are intensive and time-consuming processes, following iterative adaptive pathways, for which sustained political commitment and budget are preconditions.

Therefore we recommend the following. Foremost, as change and reform programmes require long-term and sustained political support, peer-learning among leaders (such as senior managers in the public and private sectors, and politicians) needs to be structured and facilitated, perhaps under the auspices of a new UN High-Level Panel of Government Leaders. Governments, water administrations and actors in civil society should prioritise analysis with a long-term horizon (e.g. for climate adaptation), develop capacity to set goals and achieve these, and apply learning approaches and instruments for implementation. However, while long-term programmatic strategies are advocated, the distinct supportive KCD activities should stay realistic and manageable in scope. Third, more dedicated research, knowledge sharing, and advocacy is required, involving practitioners and theoreticians from different disciplines and operating within a global impact monitoring framework; such a framework may develop from the existing framework for SDG6, but needs to be broader in scope and much more effective in shaping policy. To complete the investments and institutional changes in transitions takes 20–30 years; to be ready by 2050 they should be initiated now.

All relevant data are included in the paper or its Supplementary Information.