The United Nations General Assembly's 2010 legal recognition of the human rights to water and sanitation shaped approaches of many actors working to improve access to water, sanitation and hygiene (WASH). Persistent challenges of poor WASH sustainability, scale and inclusion are increasingly being tackled through system thinking and system strengthening. However, little has been written about how participatory system analysis and monitoring can equip and empower WASH actors to apply system thinking, self-assess and course-correct in their own work to improve sustainable WASH for all. WaterAid's Sustainable WASH Services at Scale (SusWASH) programme applies a system approach, underpinned by human rights principles. In this paper, we share perceptions of local stakeholders, engaged in empowerment evaluation in Cambodia and Uganda, and lessons learned for future initiatives that seek to catalyse WASH system improvements for the realisation of the human rights to water and sanitation. We argue that a system approach, underpinned by human rights principles, can help advance progress towards inclusive and sustainable WASH for all. Working in this way fosters inclusive, locally led decision-making about how system blockages can be overcome, strengthening local ownership of a shared vision for change and the capacities and skills required to achieve it.

  • WaterAid's approach seeks to tackle systemic blockages to WASH sustainability, inclusion and scale.

  • This paper illustrates how empowerment evaluation, system concepts and human rights principles can coalesce into a practical, accessible, participatory approach that helps to strengthen WASH systems.

Calls for ‘system change’ are often heard in discussions about the realisation of the human rights to water and sanitation. But how is system change – change to how a system works and what happens as a result (Posthumus et al. 2020) – brought about in practice? ‘System change initiatives are rooted in the assumption that significant improvement in the outcomes of a targeted population … will not occur unless the surrounding system … adjusts to accommodate the desired goals’ (Foster-Fishman et al. 2007, p. 197). System thinking, originating from Complex Adaptive Systems theory, enables a more networked understanding of the environment in which actions are taken (Neely 2019a). WaterAid's definition of the water, sanitation and hygiene (WASH) system aligns with relevant literature and summarises the WASH system as all of the actors (people and institutions), factors (social, economic, political, environmental and technological) and the interlinkages between them that affect the provision of sustainable WASH for all (Neely 2019b; Valcourt et al. 2020). The word ‘system’ can sometimes conjure up thoughts of ‘the establishment’ or government bureaucracy. However, system thinking is much broader than that. In this context, the WASH system includes rights groups and civil society, marginalised people excluded from WASH access, community WASH users, private sector operators, politicians and local government, regulators, and many more, as well as the political economy, and gender and cultural norms which influence how services are provided and progressively improved to achieve the human rights to water and sanitation (Figure 1).

Figure 1

Conceptualisation of the WASH system showing factors, actors and interlinkages (WaterAid).

Figure 1

Conceptualisation of the WASH system showing factors, actors and interlinkages (WaterAid).

Close modal

There is extensive literature on system thinking and approaches, including a growing body of both academic and grey literature on WASH systems, which has emerged largely in response to the sustainability challenges facing WASH services (Lockwood et al. 2010; Huston & Moriarty 2018; Neely 2019b; Valcourt et al. 2020). Much of the academic literature on WASH systems explores the ‘application of complex analytical methods’ to identify and document system factors affecting WASH sustainability and, less frequently, the interactions between factors or the ultimate impact of system approaches on WASH service sustainability (Valcourt et al. 2020). While these persistent sustainability challenges have increasingly informed practitioners’ programming interventions to strengthen systems through which WASH services and behaviours are delivered (WaterAid 2020), there remains a need within the literature of WASH systems for ‘a diversification in the methods, scopes, and applications of system approaches for WASH’ (Valcourt et al. 2020, p. 14), especially methods ‘that are likely accessible to those who are tasked with making critical decisions at the local level around WASH service provision’ (Valcourt et al. 2020, p. 13). While practical, locally accessible tools and approaches are an identified gap in academic system thinking, a number of implementing agencies, including WaterAid, have been exploring ways to fill this gap through their interventions.

The rights to water and sanitation not only ensure accessibility, availability, quality, affordability and acceptability of water and sanitation services, but also promote principles of participation, non-discrimination, accountability, transparency and sustainability in how those services are provided and progressively realised by governments (de Albuquerque 2014). WaterAid (2020) and others have written about the complementarity of system approaches and human rights–based approaches that seek to strengthen the accountability of local government institutions for the realisation of the rights to water and sanitation. Carrard et al. (2020, p. 7) write how ‘a systems-strengthening approach explicitly recognises the key role of local government institutions in water and sanitation service delivery and acknowledges the challenges they face in contexts where the decentralisation process is incomplete. Described as a “philosophy of action” rather than intervention type, a systems-strengthening approach seeks to engage with complexity and work towards meaningful, sustainable solutions’. Grant & Willetts (2019) identify the human rights to water and sanitation as one of the goals of WASH systems and also note that attention to human rights influences whose knowledge is ‘legitimised and drawn upon’ (p. 126), affecting the flow of information and its use in decision-making; key elements that determine how a system functions (Meadows 1999). Despite the recognised complementarity of system approaches and human rights–based approaches, a gap in the literature remains about how to effectively apply human rights principles in practical WASH system-strengthening interventions.

This paper seeks to contribute to the literature by showing how a practical systems approach, underpinned by human rights principles, can help advance progress towards inclusive and sustainable WASH for all. Strengthening WASH systems with a focus on human rights principles and with an emphasis on locally accessible WASH system assessments and regular reflections can help to foster inclusive, locally led decision-making about how system blockages can be overcome, cultivate local ownership of a shared vision for change and build the capacities, skills and mindsets required to achieve it.

WaterAid's approach to strengthening WASH systems: empowerment evaluation

As with many other organisations, the tools, frameworks and approaches which WaterAid uses to define and monitor systems involve participatory exercises that bring together diverse perspectives (Foster-Fishman et al. 2007; Ramalingam et al. 2008; Neely 2019b). WaterAid's repeated use of exercises to engage experts, decision-makers and stakeholders from across the system to monitor system change provides evidence for programme monitoring, but also provides a platform for system change through structured reflection on the state of the system and on the paradigms and mindsets out of which the system arises (Meadows 1999). Such participatory exercises have potential, as ‘social deliberative processes’ to be transformative in the way participating system actors perceive the complex WASH system and their role within it (König 2018). By supporting learning, adaptation and reflection at the individual and collective level (Brown 2019; Grant & Willetts 2019), WaterAid's approach aims to equip and empower WASH actors to apply system thinking, self-assess and course-correct in their own work to provide and improve lasting WASH access for all.

WaterAid employs a version of ‘empowerment evaluation’ to design, implement, monitor and adapt system-strengthening interventions. In this approach, WaterAid plays the role of a ‘critical friend’ (Fetterman 1994) and uses ‘evaluation concepts, techniques and findings to foster improvement and self-determination’ (Fetterman & Wandersman 2007, p. 186). Empowerment evaluation aims to ‘increase the probability of achieving program success by: (a) providing program stakeholders with tools for assessing the planning, implementation, and self-evaluation of their program, and (b) mainstreaming evaluation as part of the planning and management of the program/organization’ (Wandersman & Snell-Johns 2005, p. 422). By applying an empowerment evaluation approach to facilitate reflection and learning and to ask difficult questions, we hope to guide system actors and decision-makers to self-assess the barriers and bottlenecks to WASH sustainability and inclusion within their system and to identify resources and support needs that can help progressively realise their intended change.

Designing system interventions typically involves ‘bounding’ the system, understanding and documenting it, and assessing how its component parts interact in order to identify ‘levers of change’ (Foster-Fishman et al. 2007). Beyond the initial programme design, documenting and measuring system change occurring throughout the programme intervention is necessarily complex, reflecting the dynamic, complex and emergent nature of systems (Brown 2019; Hollander et al. 2020). To capture this dynamism and complexity within WASH system monitoring, Hollander et al. (2020, pp. 6–7) recommend applying a combination of ‘qualitative monitoring methods focused primarily on the process of system change, analytical monitoring and analysis of the overall structure of the system and actor networks and the use of indicators or proxy indicators that adequately capture impacts and results while also considering the complex nature of such impacts’.

WaterAid's empowerment evaluation approach to WASH system strengthening, exemplified in the SusWASH programme, places local decision-makers and stakeholders at the centre of WASH system analysis from the programme design phase, addressing the issue of poor locally accessible system tools identified in the literature. This aims to keep decision-making power about what interventions the programme prioritises in the hands of those who live the reality of local WASH systems rather than external WASH system practitioners (Chambers 1997).

A core component of WaterAid's empowerment evaluation approach is the use of tools and frameworks based on ‘system building blocks’ to facilitate system-wide factor assessments (Hollander et al. 2020). Numerous frameworks for ‘system building blocks’ exist, which have been developed by multilateral institutions and international non-government organisations (NGO)s since they were first documented in 2007 by the World Health Organization (2007). While acknowledging the widely accepted limitations of building block frameworks as a monitoring tool and the need to complement them with other monitoring approaches (World Health Organization 2007; WaterAid 2019a; Hollander et al. 2020), we have found them to be an accessible and practical framing to introduce system thinking to local WASH actors and to facilitate the analysis of various WASH system functions (Figure 2).

Figure 2

WASH system building blocks, contextualised within the political economy and supported by human rights principles (WaterAid 2020).

Figure 2

WASH system building blocks, contextualised within the political economy and supported by human rights principles (WaterAid 2020).

Close modal

In this paper, we present the results of a process review of WaterAid's empowerment evaluation approach to design, support, monitor, adapt and learn about WASH system-strengthening interventions in the SusWASH programme.

In practice, empowerment evaluation involves facilitators guiding participants through three steps to (a) establish their mission or purpose, (b) assess their current state of affairs using an evaluative framework and (c) plan for the future (Fetterman & Wandersman 2007). Empowerment evaluation emphasises the following 10 principles (Fetterman & Wandersman 2005):

  • Improvement: improves programme performance, builds on successes and re-evaluates areas requiring attention.

  • Community ownership: values and facilitates community control.

  • Inclusion: invites involvement, participation and diversity from all.

  • Democratic participation: ensures that decision-making is open and fair.

  • Social justice: addresses social inequities in society.

  • Community knowledge: respects and values community knowledge.

  • Evidence-based strategies: respect and use the knowledge base of scholars.

  • Capacity building: enhances stakeholders’ ability to conduct an evaluation and to improve programme planning and implementation.

  • Organisational learning: uses data to inform decisions and supports organisations to learn from experience.

  • Accountability: supports accountability towards outcomes within a context of existing policies, standards and measures of accountability.

There is a strong alignment between empowerment evaluation principles and the human rights principles outlined above.

During the programme design and periodically throughout implementation, WaterAid used building blocks as an evaluation framework to facilitate and guide empowerment evaluation in participatory workshops with diverse stakeholders to (re)assess and (re)prioritise programming areas and to build collective ownership of issues and a vision for change. To facilitate this, we created outcome mapping tools based on the building block framework (Figure 3) (Earl et al. 2001; Hollander et al. 2020).

Figure 3

Example of a building block assessment framework used in participatory workshops in the SusWASH programme in Cambodia (WaterAid).

Figure 3

Example of a building block assessment framework used in participatory workshops in the SusWASH programme in Cambodia (WaterAid).

Close modal

Methods used in this study

For this paper, we applied a process review method involving questionnaires administered online to understand the extent to which our engagement with system actors in the SusWASH programme upheld the empowerment evaluation principles and led to improvements in the WASH system strength. The SusWASH programme has been implemented in Cambodia, Ethiopia, Pakistan and Uganda since 2017. Two of the four country contexts, Cambodia and Uganda, were selected based on the consistency of the empowerment evaluation approach used in the programme in those contexts to engage actors within the system and the countries’ application of participatory building block frameworks to drive cycles of assessment, action and self-reflection. Review participants were selected based on their involvement in the SusWASH programme in Kampong Chhnang province, Cambodia and Kampala city, Uganda, and included representatives from sub-national government, WASH line departments, and community and civil society representatives. A total of 15 participants from Cambodia and 12 participants from Uganda took part in the review. Participants were invited to respond to a series of Likert-scale statements and open questions. The Likert-scale questions sought to assess participants’ knowledge and understanding of applying building block frameworks to assess their local WASH system, their perception of changes in the strength of the WASH system building blocks during the course of the programme and the extent to which they felt WaterAid's programming approach embodied the 10 principles of empowerment evaluation. Question responses were rated on a scale from −2 to 2; whereby −2 indicated a strongly negative result (e.g. deterioration), while on the other end of the scale, 2 indicated a strongly positive result (e.g. improvement). Open questions provided opportunities for further qualitative descriptions of the changes they had perceived in their organisation and WASH sector during the 4 years of SusWASH programme interventions.

The process review results are complemented by a snapshot of stakeholder building block assessment results in which WASH stakeholders assessed the state of building blocks of their local WASH system and identified actions to progressively strengthen the system's functionality.

In the following section of this paper, we discuss the findings while highlighting the complementary nature of empowerment evaluation approaches, human rights principles and system concepts outlined in the aforementioned literature.

This section presents the results of the process review and selected supporting evidence gathered through empowerment evaluation activities held throughout the SusWASH programme. In Tables 1 and 2, we present the results of the process review, showing local stakeholders’ perceptions of the empowerment evaluation approach to system strengthening used in the SusWASH programme and the relative change in each WASH system building block over the life of the programme. In Table 3, we present selected results from participatory WASH system building block assessments as examples of stakeholders’ analysis and assessment of their local WASH systems.

Table 1

Strength of the application of empowerment evaluation principles in the SusWASH programme

PrincipleNumber of indicatorsCambodiaUganda
n=15n=12
1. Improvement 1.67 1.33 
2. Community ownership 0.91 1.67 
3. Inclusion 1.20 1.83 
4. Democratic participation 1.27 1.92 
5. Social justice 0.94 1.75 
6. Community knowledge 1.43a 1.92 
7. Evidence-based strategies 1.33 2.00 
8. Capacity building 0.95a 1.67 
9. Organisational learning 0.90b 1.50 
10. Accountability 1.40 1.42 
PrincipleNumber of indicatorsCambodiaUganda
n=15n=12
1. Improvement 1.67 1.33 
2. Community ownership 0.91 1.67 
3. Inclusion 1.20 1.83 
4. Democratic participation 1.27 1.92 
5. Social justice 0.94 1.75 
6. Community knowledge 1.43a 1.92 
7. Evidence-based strategies 1.33 2.00 
8. Capacity building 0.95a 1.67 
9. Organisational learning 0.90b 1.50 
10. Accountability 1.40 1.42 

The ratings in the table are based on the following scale: 2, principle strongly demonstrated; 1, principle weakly demonstrated; 0, principle neither demonstrated nor lacking; −1, principle weakly lacking; −2, principle strongly lacking.

aOnly 14 responses to one or more indicators.

bOnly 13 responses to one indicator.

Table 2

Participants’ perception of change in each WASH system building block over the life of the SusWASH programme

Building blockNumber of indicatorsCambodiaUganda
n=15n=12
1. Active empowered people and communities 1.20 1.83 
2. Gender and social inclusion 0.84 1.71 
3. Institutional arrangements 0.90a 1.58 
4. Coordination and integration 1.13 1.92 
5. Monitoring 1.21ba 1.38 
6. Strategic planning 1.22 1.79 
7. Financing 0.94 1.25 
8. Government leadership 0.91 1.67 
9. Service delivery and behaviour change 1.67 1.33 
10. Accountability and regulation 1.22 1.63 
11. Environment and water resources n/a n/a 
Building blockNumber of indicatorsCambodiaUganda
n=15n=12
1. Active empowered people and communities 1.20 1.83 
2. Gender and social inclusion 0.84 1.71 
3. Institutional arrangements 0.90a 1.58 
4. Coordination and integration 1.13 1.92 
5. Monitoring 1.21ba 1.38 
6. Strategic planning 1.22 1.79 
7. Financing 0.94 1.25 
8. Government leadership 0.91 1.67 
9. Service delivery and behaviour change 1.67 1.33 
10. Accountability and regulation 1.22 1.63 
11. Environment and water resources n/a n/a 

The ratings in the table are based on the following scale: 2, strong improvement perceived in this building block; 1, weak improvement perceived in this building block; 0, no change perceived in this building block; −1, weak deterioration perceived in this building block; −2, strong deterioration perceived in this building block.

aOnly 14 responses to one indicator.

bOnly 13 responses to one indicator.

Table 3

Selected results of participatory building block assessments

 
 

In Table 1, we present the extent to which local stakeholders in Cambodia and Uganda feel the approach used in the SusWASH programme upheld the 10 empowerment evaluation principles. In general, scores in Uganda were more positive than the scores in Cambodia. On average, stakeholders in Uganda perceived all 10 principles to have been demonstrated to some extent, with ‘Evidence-based strategies’ scoring the highest (2.0). The principle the least well demonstrated in Uganda was ‘Improvement’ in the system goal of providing WASH services (1.33). Conversely, in Cambodia, the highest scored principle was ‘Improvement’ (1.67) followed by ‘Community knowledge’ (1.43). The lowest scored principle in Cambodia was ‘Organisational learning’ (0.94).

In Table 2, we present the results of participants’ perception of the change in each WASH system building block over the life of the SusWASH programme. We see that stakeholders in both Cambodia and Uganda consider that there have been improvements, of some kind, across all 10 rated building blocks. The lowest scored building block in Cambodia was ‘Gender and social inclusion’ (0.84). While in Uganda, the lowest scored was ‘Financing’ (1.25), although this was still rated as having had a ‘weak improvement’. The highest scored building block in Cambodia was ‘Service delivery and behaviour change’ (1.67), while in Uganda, the highest scored building block was ‘Coordination and integration’ (1.92).

Table 3 presents selected results of participatory WASH system building block assessments conducted by sub-national WASH stakeholders during the SusWASH programme. These results reflect the second and third step of the empowerment evaluation process (Fetterman & Wandersman 2007); stakeholders who have already set their overall mission and goal for WASH in their context assessed the current state of the system through the building block framework (Figure 3) and identified actions that they could collectively take to respond to weaknesses.

Following building block assessments and regular reflection sessions, WaterAid worked with others to support the operationalisation of the action points identified by local stakeholders.

In Uganda, WaterAid supported the Ministry of Health and the Kampala Capital City Authority (KCCA) to align their WASH monitoring indicators with the global Joint Monitoring Programme service level indicators. WaterAid also built the capacity of government staff and community-level Village Health Teams (VHTs) to collect service level data using electronic data capture tools. The transition from paper-based data collection to electronic has helped ensure the accuracy and reliability of the data collected. This new electronic data collection approach is being tested by the KCCA for learning and potential scale-up.

Faced with the high cost of water, public schools often struggled to pay their water bills. This challenge grew as the government began promoting the installation of water-borne toilets in all public schools. To address this, WaterAid, in collaboration with the KCCA, developed the Single Flush Female-Friendly Toilet. The toilet design was built with water-saving technology, integrating the pit latrine model with pour-flush technology thus significantly reducing water usage by over 10 times compared to a normal flush toilet. The five target schools that previously struggled to pay their bills can now afford their water bills and have access to water for flushing, handwashing, cleaning and drinking at all times. The toilet design also provides for a safe disposal of menstrual hygiene materials for girls.

To build government leadership in Cambodia, WaterAid worked in partnership with a local NGO to strengthen local authority leadership through the Civic Champions Programme1; helped to facilitate sector coordination and multi-stakeholder collaboration; promoted sector learning and peer-to-peer learning exchange, including the sharing of best practices through mass media and media engagement; promoted good sanitation and hygiene behaviours through the family champion initiative which identified and publicly recognised families practicing good WASH behaviours and gender equality; and encouraged high-level sector and political leaders to discuss and share their strategic vision with the public through social media.

To strengthen gender, equity and social inclusion (GESI) in Cambodia, WaterAid engaged with and influenced sector and political leaders to allocate and mobilise resources to support most marginalised households gain access to clean water and sanitation; empowered vulnerable groups by building their capacity and confidence to share their WASH experiences with local authorities through storytelling on social media and in public forums; challenged gender norms by promoting women's leadership, facilitating women's participation in WASH networks and working with male leaders to change their mindsets towards gender relations.

In the subsequent paragraphs, we discuss the outcomes and lessons learned from the combination and practical application of empowerment evaluation principles, human rights principles and system concepts outlined above.

Strengthening organisational learning and capacity to improve the system by identifying and responding to blockages

The routine participatory evaluation of the WASH system has helped to strengthen the awareness and capacity of system actors, and the processes they use, so that they may continue to practically assess their system and respond to emerging issues beyond the life of WaterAid's intervention (Table 4). On average, survey respondents from both Cambodia and Uganda felt that their engagement with the SusWASH programme had improved their personal and organisational capacity to evaluate WASH sector needs, rating the capacity building principle as 0.95 and 1.67, respectively. They felt better equipped to use data for evaluating their practice and improve their decision-making, rating the organisational learning principle as 0.9 and 1.5, respectively. Respondents also felt that WASH services had improved in their area over the corresponding period, rating the improvement principle as 1.67 and 1.33 in Cambodia and Uganda, respectively. These perceptions are supported by government monitoring data which show that between 2017 and 2020, 15,667 households (13% of total households) gained access to basic sanitation in SusWASH programme areas in Cambodia and sanitation access rates have increased in SusWASH programme areas of Uganda by 2% from 21% in 2018 to 23% in 2020.

Regular reflection and system assessments have supported local stakeholders to build on their successes and re-evaluate areas requiring further effort. As explained in the process review by a government stakeholder in Kampong Chhnang, Cambodia, the results from WASH sector evaluation workshops are used to ‘compare what has been done, compare between the work plan and the real practice, to check if it responds to the requirement or not. Then we consider what areas we need to improve and how to do it better for the next steps’. The collective identification of blockages within the system each year informed tangible actions for strengthening the system (examples in Table 3) as per the three steps of the empowerment evaluation approach (Fetterman & Wandersman 2007). The following are two examples of actions taken as a result of collective system blockage identification. The idea to create a knowledge management unit within the Kampala City Council Authority in Uganda was proposed to strengthen the facilitation of WASH monitoring and reporting systems. In Cambodia, the identification of limited meaningful participation of women and marginalised people in WASH decision-making (participation being one of the human rights principles) led to collective agreement to create safe and effective spaces for those groups to raise their voices and share their concerns.

Alongside these strengthened capacities have been tangible improvements occurring in WASH system functions, many of which have arisen from the cycles of system assessment and action planning. These include:

  • Formation of district WASH committees and open defaecation plans under the leadership of district administrations in Cambodia.

  • Roll-out of community WASH monitoring systems in Uganda.

  • A joint assessment of WASH accessibility and sustainability and life-cycle costing of WASH services in healthcare facilities across Kampala, Uganda.

  • Creation of cross-department district WASH committees in Kampong Chhnang, Cambodia.

  • A piloting of public–private partnerships for toilet pit emptying and partial subsidies for marginalised households in Kampala, Uganda.

  • Peer-to-peer sharing between communes who had used their own budget to promote sanitation and hygiene to inspire other communes in Kampong Chhnang, Cambodia.

  • Development of local leaders’ leadership skills and engagement in WASH motivated increases in sanitation coverage in Kampong Chhnang, Cambodia.

Regular reflections in Cambodia and Uganda provided a platform for raising and problem-solving issues among the participating stakeholders. Collectively assessing system problems from a range of different perspectives leads to participants re-assessing their own ways of thinking (König 2018) and reinforces the self-organising nature of complex systems (Ramalingam et al. 2008). One civil society stakeholder in Cambodia highlighted that the programme's assessments contributed to clarifying unclear sector roles: ‘Private Water Operators and sub-national authorities get better understanding on their roles and responsibilities on WASH’. In some instances, involving and engaging non-traditional WASH stakeholders in the discussions to assess the WASH system has created feedback loops that have shaped how information is shared within the WASH system (Meadows 1999). One government survey respondent from Uganda noted that the discussions encouraged different actors to engage in the system: ‘the political leadership has more ownership of WASH planning, delivery and monitoring’. In Uganda, the generation and use of data (empowerment evaluation principle 9), which is accessible to all, helps inform decisions and investments in WASH, while also strengthening transparency. Building the capacity of Kampala authorities to collect, store, visualise and use JMP service level data (the ‘Monitoring’ building block scored 1.38 in Table 2) as part of sanitation monitoring has improved decision-making and transparency. These examples further demonstrate how the human rights principles of transparency and accountability have been operationalised within system-strengthening approaches.

Through regular cycles of reflection and action, the approach tries to model a culture of learning, reflection and adaptation among WASH actors (Grant & Willetts 2019). WaterAid deliberately did this through facilitating regular participatory ‘pause and reflect’ sessions, and the use of online knowledge hubs and real-time learning and sharing networks through platforms such as Facebook, Telegram and WhatsApp. Learning activities and cycles of action, reflection and learning draw on WaterAid's own adaptive programming which encourages planned learning activities to intentionally deepen the understanding of the context and system dynamics.

Our reflection and learning approaches during the SusWASH programme prompted review and updates to the building block framework itself to ensure it remained relevant (WaterAid 2019b) to the changing programme contexts over time. For example, at the beginning of the interventions, we did not have a specific building block articulated for government leadership, active and empowered people or GESI. Adding these helped to focus the discussions and awareness among stakeholders of the importance of these aspects of the system. The definitions used in frameworks and outcome mapping also evolved according to changes in context – such as when policy or regulation changes created new mechanisms for coordination, financing or monitoring. While such changes over time complicate the use of building block frameworks for programme monitoring, they reflect the dynamic nature of the WASH systems in which they were being applied and improve their value as tools for empowerment evaluation.

WaterAid and several of the stakeholders responding to the process review survey have noted the value of the building block framework for categorising, assessing and planning to respond to problems in WASH systems. We have begun to see small examples of how these practical, locally accessible tools have potential for take-up by those making critical decisions around WASH service provision and scaling beyond WaterAid's direct intervention areas. In Cambodia, several district administrations have fixed their latest building block assessments to the office wall and refer to it in their monthly WASH progress meetings and action planning. In Uganda, the KCCA have used their own budget to mentor two municipality authorities through a municipal WASH investment planning process that draws on participatory system assessment processes. At the national level, we also see potential for institutionalisation of participatory building block assessments within government and sector efforts to strengthen WASH systems. Sector working groups in Cambodia engaging with Sanitation and Water for All and Agenda for Change have drawn on the SusWASH experience in Kampong Chhnang and are discussing and framing sector priorities using participatory building block assessments, and in Uganda, elements of the work in Kampala such as training and WASH assessment and planning processes in healthcare facilities are being scaled nationally by the Ministry of Health through their own investment. While we have not transitioned to sector stakeholders using the framework to facilitate participatory discussions independently of WaterAid, the above examples indicate that actors, or groups of actors, other than large NGOs like WaterAid could potentially scale this way of working at either sub-national or national level. Based on our experience in these two countries, scaling and institutionalising system approaches could be possible if there is an adequately resourced organisation or institution that can play a convening and facilitating role, and that the tools and frameworks used are appropriately contextualised and locally owned, though we note that this takes time even when led by an organisation like WaterAid. Achieving the outcome of participatory WASH system building block assessments being led and institutionalised by either civil society or government actors would provide further evidence of the usefulness of the approach and the confidence and capacity of users to self-assess and course-correct (Fetterman & Wandersman 2007). It would also contribute to the human rights principle of sustainability in both service sustainability and sustained embedment of human rights principles in the WASH system.

Critiques and limitations of these lessons

WaterAid is cognisant that building blocks do not represent the WASH system in its entirety and should be complemented by other monitoring approaches (WaterAid 2019a; Hollander et al. 2020). In one of the formative documents which first described the building blocks of the health system, the World Health Organization (2007, p. v) notes that ‘while the building blocks provide a useful way of clarifying essential functions, the challenges facing countries rarely manifest themselves in this way. Rather, they require a more integrated response that recognises the inter-dependence of each part of the health system’. Hollander et al. (2020), building on previous analysis by WaterAid (2019b), highlight that building blocks are incomplete as a full programme monitoring tool because they ‘neglect the full complexity of systems engagement and change’ (p. 4) and therefore capture only a simplified and static view of predefined categories. Monitoring indicators and corresponding interventions may become siloed and deprioritise the interlinkages and emergent properties of the system (WaterAid 2019a; Hollander et al. 2020). For these reasons, and the inherently subjective nature of the participatory assessment process, WaterAid has separate internal processes and tools for monitoring programme impact which draw on the participatory assessments alongside other evidence sources. The analysis must also be considered within the context's political economy and influencing factors such as power structures, relationships and norms (Foster-Fishman et al. 2007). In practice, during the programme start-up phase, for the purposes of ‘bounding the system’ (Foster-Fishman et al. 2007), WaterAid complements the building block assessments with simple stakeholder mapping tools for local sector stakeholders to identify which actors have willingness, ability and influence to advance improvements in WASH, while simple power and gender analysis exercises help to prompt reflection on issues of marginalisation and power exclusion.

While we accept the above critiques of building block frameworks, we see their value in the way in which they can introduce, and make accessible, the concept of the WASH system to local stakeholders. Similarly, we see the value in the way in which they can draw diverse stakeholders together to share perceptions of persistent problems and discuss strategies for tackling them. Indeed, it is this process which helps to uncover issues of power, attitudes and behaviours, even if the building block framework itself does not explicitly focus on these components; it is nearly impossible to discuss aspects of leadership, community empowerment, GESI and accountability without considering power, agency and the dynamic nature of rights holder–duty bearer relationships. We also recognise that identified system changes cannot always be attributed to WaterAid, particularly in areas crowded with other WASH system actors (Ramalingam et al. 2008; Hollander et al. 2020). For this reason, we are mindful of clearly stating whether or not WaterAid has played a leading role, or more of a supportive role, in bringing about any identified system change.

Inclusive participation to re-shape WASH sector power relations

Through an empowerment evaluation approach, regular reflections among sector stakeholders about changes in their system over time went beyond system monitoring to become a tool for facilitating dialogue about system issues in which diverse voices and perspectives could be heard and engaged (Table 5) (Neely 2019b). As one government stakeholder from Uganda stated, ‘the WASH agenda needs everyone's involvement and at my level I can contribute to this change’. Respondents from Cambodia and Uganda felt that WaterAid's programme approach demonstrated the principles of ‘Inclusion’ (1.20 and 1.83, respectively), ‘Democratic participation’ (1.27 and 1.92, respectively) and ‘Community knowledge’ (1.43 and 1.92, respectively). Stakeholders from both countries attributed the engagement of diverse stakeholders in WASH activities to the system assessment process.

Each stakeholder's understanding of the systems they are part of will differ; such subjectivity reinforces the importance of involving multiple and diverse stakeholders in exploring and assessing systems (Foster-Fishman et al. 2007). WaterAid does this through facilitating participatory multi-stakeholder discussions because we see the value in the process of stakeholders exchanging and challenging each other's perspectives. Using participatory approaches inevitably invites questions of power and perceptions of power relations. In the words of Chambers (2005, p. 290), ‘participation is about power relations. It is about much else as well; but power relations are pervasive: they are always there, and they affect the quality of process and experience’. Gathering political leaders, technical experts, community representatives and social and business leaders to discuss the state of the WASH system and create shared plans to improve WASH services is a process for encouraging participants to acknowledge their own power and build awareness about who is often absent from decision-making within their local WASH system (thus linking to the human rights principle of participation and non-discrimination). The process review identified several examples of changing power relations that are helping to resolve system blockages. One government bureaucrat in Uganda reported that ‘the political leaders have become the champions of WASH interventions which makes the work of us, the technical people, easy’. Another government stakeholder from Uganda noted ‘some of the research has helped us engage different groups in our intervention’, while one civil society stakeholder from Cambodia observed that the participatory process enabled ‘multiple partners to work on WASH, includ[ing] private sector and political group’. We have observed additional early signs of shifting power dynamics as a result of SusWASH programme interventions:

  • In Uganda, WaterAid mapped power dynamics and incentives of local actors to identify leverage points for system change. In Kampala, power is highly centralised. Little community participation in government planning processes can mean community WASH needs are not met or prioritised. The SusWASH programme has created spaces for citizen engagement (scoring 1.83 in Table 2) through the WASH Mayors Forum and community-level Participatory Budget Clubs. Community hygiene promotion groups locally called ‘Weyonje’ volunteers have become more empowered to engage service providers including public hospitals in their area about expenditure of public funds in support of WASH, something they had never done before and thought was not their responsibility.

  • In Cambodia, discussions with men and women about their roles and capabilities helped build a safe and open dialogue on gender equality and non-discrimination among district government staff, enabling fulfilment of WASH roles and responsibilities while beginning to address barriers to women's participation. Participatory barrier analysis and public storytelling also helped marginalised people to identify their own power in driving WASH improvements. Despite these efforts, it is interesting to note that stakeholders from Cambodia participating in the process review rated a change in the ‘Gender and social inclusion’ building block the weakest (0.84). This could be linked to the level of attention paid by WaterAid to increase local stakeholder understanding of gender and social inclusion issues and local stakeholder recognition of the efforts required to address them.

These examples support the observations of Foster-Fishman et al. (2007) that simply ‘the act of having stakeholders explore and accommodate differences among competing worldviews can serve to create transformative shifts in stakeholders’ understandings of the problems’ (p. 201).

Table 4

Key lessons learned and relevant empowerment evaluation principles regarding strengthening organisational learning and capacity

Key lessons learnedRelevant empowerment evaluation principles
  • Building blocks can provide an accessible framework through which local decision-makers can begin to analyse the WASH system and progressively strengthen it.

  • Monitoring frameworks can and should evolve with the context.

  • Efforts should be made to clarify the role played by any implementing agency in bringing about a system change for accountability and transparency purposes. Often system changes are the result of collective action.

  • Institutionalising system thinking and scaling systems assessments beyond NGO programming may be possible, if an adequately resourced institution can play a convening role and tools are adequately contextualised.

 
1. Improvement
7. Evidence-based strategies
8. Capacity building
9. Organisation learning 
Key lessons learnedRelevant empowerment evaluation principles
  • Building blocks can provide an accessible framework through which local decision-makers can begin to analyse the WASH system and progressively strengthen it.

  • Monitoring frameworks can and should evolve with the context.

  • Efforts should be made to clarify the role played by any implementing agency in bringing about a system change for accountability and transparency purposes. Often system changes are the result of collective action.

  • Institutionalising system thinking and scaling systems assessments beyond NGO programming may be possible, if an adequately resourced institution can play a convening role and tools are adequately contextualised.

 
1. Improvement
7. Evidence-based strategies
8. Capacity building
9. Organisation learning 
Table 5

Key lessons learned and relevant empowerment evaluation principles regarding inclusive participation

Key lessons learnedRelevant empowerment evaluation principles
  • Taking time to build and maintain rapport and trust is critical to locally owned and accessible system analysis.

  • Participatory multi-stakeholder assessments of the system can contribute to building a collective understanding about where power in the system does, and should, reside.

 
2. Community ownership
3. Inclusion
4. Democratic participation
6. Community knowledge 
Key lessons learnedRelevant empowerment evaluation principles
  • Taking time to build and maintain rapport and trust is critical to locally owned and accessible system analysis.

  • Participatory multi-stakeholder assessments of the system can contribute to building a collective understanding about where power in the system does, and should, reside.

 
2. Community ownership
3. Inclusion
4. Democratic participation
6. Community knowledge 
Table 6

Key lessons learned and relevant empowerment evaluation principles regarding shifting mindsets

Key lessons learnedRelevant empowerment evaluation principles
Empowerment evaluation's emphasis on participation and community knowledge creates opportunities for individual and collective self-reflection and self-improvement. In time, this can help to shift deeply held attitudes and beliefs, helping to strengthen accountability and tackle issues of social injustice. 5. Social justice
10. Accountability 
Key lessons learnedRelevant empowerment evaluation principles
Empowerment evaluation's emphasis on participation and community knowledge creates opportunities for individual and collective self-reflection and self-improvement. In time, this can help to shift deeply held attitudes and beliefs, helping to strengthen accountability and tackle issues of social injustice. 5. Social justice
10. Accountability 

As an NGO temporarily participating in the WASH systems we seek to strengthen, and with no mandate or responsibility for ongoing WASH delivery, WaterAid has learned that it takes time to build trust and acceptance of this way of working among local stakeholders. It took years for local stakeholders from government, civil society and the private sector to value the system-focused contributions of WaterAid teams in Cambodia and Uganda in contexts where NGOs typically fill a role in delivering services outside of government processes. It took at least one annual cycle for a shared vocabulary and understanding of the WASH system to begin to emerge and even longer before stakeholders expressed the appreciation of the value of honest and open reflection in order to identify and address weaknesses in the system. Such openness is a necessary step towards developing a collective vision and a space in which diverse actors feel able to meaningfully direct and own the process; as Chambers (1997, p. 133) puts it ‘relaxed rapport between outsiders and local people, and some measure of trust, are minimum conditions for [participatory appraisals]’. This reflection raises questions about the viability of this approach in more closed contexts where suspicion of civil society and international NGO activity is particularly high and trust is low.

Critiques and limitations of these lessons

Although small examples of positive progress have been observed, complex and inequitable power dynamics persist in both of the systems described and will continue to persist for a long time. Despite WaterAid's efforts to engage and facilitate meaningful participation of rights groups, representatives of marginalised communities, actors who possess formal power and authority, and other actors in between, not all groups have been represented, and group discussions may continue to favour the voices of some actors over others (Chambers 1997). We have observed that deep-seated distrust, prejudices and suspicion (mentioned above) can perpetuate harmful attitudes that prevent the safe spaces needed for meaningful participation or shifts of power within the system (Foster-Fishman et al. 2007). WaterAid is not immune from blindness to power; only four of the 27 process review responses were from representatives of civil society and rights groups. More can and must always be done in system-strengthening programmes to continuously and consciously create space for empowerment of those who experience marginalisation and ensure that where momentum for change builds among actors who possess formal power that it is accompanied by accountability, non-discrimination and transparency (de Albuquerque 2014).

Shifting mindsets to focus on equitable WASH services for all

Paradigms, which Meadows (1999, p. 18) describes as ‘the sources of systems’, are the shared ideas and beliefs about how the world works. ‘Normative elements’ such as people's attitudes, values, beliefs and expectations maintain the system's existence, and explain why and how the system and its members operate, and ‘can often be identified as the root causes of systems problems’ (Foster-Fishman et al. 2007, p. 205). To an extent, all system-strengthening interventions, therefore, should aim to identify and leverage change in such paradigms, and this lies at the core of WaterAid's participatory approach based on empowerment evaluation (Table 6). Process review respondents from Cambodia and Uganda felt on average that WaterAid's approach emphasised social justice principles (0.94 and 1.75, respectively) and encouraged accountability within roles and responsibilities between rights holders and duty-bearers (accountability principle rated 1.4 and 1.42, respectively). Notably, civil society representatives from Uganda felt that the social justice and accountability principles were less well demonstrated than other respondents (rated 1.0 and 1.5, respectively), and civil society representatives from Cambodia also rated the accountability principle lower than the average (1.0). This finding could be explained by the role and focus of civil society organisations in holding government to account and tackling issues of social injustice; they may have higher expectations and a higher level of understanding about what is required to meet them.

The process review identified examples of how participants’ beliefs had changed through their involvement in the SusWASH programme, especially regarding the rights to water and sanitation services. One Government stakeholder from Uganda reported, ‘at first I thought it was someone else's task to demand for better wash services but now I know and I have learnt that policies are influenced rightly from the grass roots from the lowest citizens as a right’. Though changing individual mindsets, attitudes and beliefs may happen quickly, changing them across an entire system takes continuous engagement, champions and a variety of tactics (Meadows 1999). WaterAid's partnerships and regular activities bringing together Mayors, government bureaucrats, community groups and disabled people's organisations have sought to contribute to this process (in line with the human rights principles of participation and transparency). A district government respondent from Cambodia shared their revelation that ‘everyone has the right[s] to water and sanitation. To meet the basic needs of water supply, [we] collaborate with institutions, NGOs and the private sector to expand WASH services and encourage participation from all relevant stakeholders’. Changes in mindset such as this may have come from storytelling and photography exhibitions we arranged for groups who experience marginalisation to share their experience and WASH needs with decision-makers, from visits by government officials to households of those groups to help them better understand people's situation and their access to WASH, and from ‘early-adopter’ decision-makers championing rights and equality among their peers (Meadows 1999).

Alongside experiential and relationship-building activities, we also facilitated targeted training and awareness raising sessions with disabled people's organisations on the rights to water and sanitation and on issues of inclusion and non-discrimination. Working to strengthen the skills and processes of government, private sector and civil society actors in areas of planning, budgeting, monitoring and coordination helped to open opportunities for discussing equality and sustainability in WASH service planning and delivery and make rights part of duty-bearers’ everyday practice (Carrard et al. 2020). Indeed, following participatory life-cycle costing assessments, we observed a shift in some duty-bearers’ mindsets around the use of resources for WASH and who should be involved in decisions regarding resource use. One government stakeholder in Uganda shared that they previously ‘had beliefs that women had no option in making decisions concerning sanitation and hygiene’, while another shared that they had ‘learnt that all engagements need to be sustainable, hence community ownership is key’. A third Ugandan government stakeholder had come to realise that ‘schools can do a lot more and create a positive WASH impact within their available resources’. These findings illustrate that participatory system-strengthening approaches, designed based on a detailed understanding of the incentives and motivations of the target system actor(s), can shift and alter deeply embedded attitudes, beliefs and norms.

Critiques and limitations of these lessons

WaterAid's WASH system monitoring approach has not been designed to systematically document changes in mindsets over time; the above lessons are based on programme participants’ reflections at a single point in time. We continue to evolve our monitoring framework and its accompanying guidance to better capture these changes and to understand how these changes manifest in improvements in WASH system strength and service levels. One means of facilitating this could be to have more regular personal reflections and use of semi-structured interviews to draw out the changes in mindsets, attitudes and norms, as has been done for gender power analysis and power mapping. Complementary tools already in use include simple stakeholder mapping and political economy analysis.

This paper illustrates how empowerment evaluation, system concepts and human rights principles can coalesce into a practical, participatory approach that helps to strengthen WASH systems for the realisation of sustainable WASH for all. System thinking, framed through simple frameworks like system building blocks, can help local actors to understand the interconnected nature of how the people, institutions and resources required for the realisation of the human rights to water and sanitation interact, behave, flow and function. Approaching system-strengthening interventions with an emphasis on human rights principles can help to guide efforts that seek to strengthen how a WASH system functions in a way which empowers and builds capacity among local actors, shifts power relations and challenges people's attitudes towards equity and social justice. System perspectives do not presuppose a pathway to change but rather develop pathways from initial analysis and continuously adapt over time. Systems analyses, when undertaken in an empowerment evaluation approach with the participation of diverse voices, can help to identify potential leverage points to target and effect change in the system depending on the context. What is unique to this approach, and which we think is critical to the realisation of the human rights to water and sanitation, is taking system actors on a journey to deepen understanding of which parts of the system, which principles, in which context and when, it makes most sense to target and prioritise. Other WASH agencies working to realise the human rights to water and sanitation should continue to test, share lessons, and where appropriate, work with others to scale practical approaches that combine these concepts together.

The funding for WaterAid to undertake the SusWASH programme was provided by the H&M Foundation.

1

More information on the Civic Champions Programme can be found in WaterAid's website: https://washmatters.wateraid.org/publications/civic-champions-leadership-development-programme-for-strengthening-sub-national-wash and on local partner WaterSHED's website: https://watershedasia.org/civic-champions/.

All relevant data are included in the paper or its Supplementary Information.

Brown
C.
2019
Measuring impact in WASH from a complex adaptive systems perspective
. In:
Systems Thinking and WASH Tools and Case Studies for a Sustainable Water Supply
(
Neely
K.
, ed).
Practical Action Publishing
,
Rugby
, pp
17
28
.
http://dx.doi.org/10.3362/9781780447483.008
.
Carrard
N.
,
Neumeyer
H.
,
Pati
B. K.
,
Siddique
S.
,
Choden
T.
,
Abraham
T.
,
Gosling
L.
,
Roaf
V.
,
Alvarez-Sala Torreano
J.
&
Bruhn
S.
2020
Designing human rights for duty bearers: making the human rights to water and sanitation part of everyday practice at the local government level
.
Water
12
,
378
388
.
https://doi.org/10.3390/w12020378
.
Chambers
R.
1997
Whose Reality Counts?
Practical Action Publishing
,
Rugby
.
Chambers
R.
2005
Ideas for Development
.
Taylor & Francis Group
,
London
.
de Albuquerque
C.
2014
Realising the Human Rights to Water and Sanitation: a Handbook by the UN Special Rapporteur
.
United Nations Office of the High Commissioner for Human Rights
,
Lisbon
.
Earl
S.
,
Carden
F.
&
Smutylo
T.
2001
Outcome Mapping: Building Learning and Reflection Into Development Programs
.
International Development Research Centre
,
Ottawa
.
Fetterman
D. M.
1994
Empowerment evaluation
.
American Journal of Evaluation
15
(
1
),
1
15
.
doi:10.1177/109821409401500101
.
Fetterman
D. M.
&
Wandersman
A.
2005
Empowerment Evaluation: Principles in Practice
.
Guildford Press
,
New York
.
Fetterman
D. M.
&
Wandersman
A.
2007
Empowerment evaluation: yesterday, today and tomorrow
.
American Journal of Evaluation
28
,
179
198
.
doi:10.1177/1098214007301350
.
Foster-Fishman
P.
,
Nowell
B.
&
Yang
H.
2007
Putting the system back into systems change: a framework for understanding and changing organizational and community systems
.
American Journal of Community Psychology
39
,
197
215
.
doi:10.1007/s10464-007-9109-0
.
Grant
M.
&
Willetts
J.
2019
Learning for adaptive management: using systems thinking tools to inform knowledge and learning approaches
. In:
Systems Thinking and WASH Tools and Case Studies for a Sustainable Water Supply
(
Neely
K.
, ed.).
Practical Action Publishing
,
Rugby
, pp.
107
132
.
http://dx.doi.org/10.3362/9781780447483.007
.
Hollander
D.
,
Ajroud
B.
,
Thomas
E.
,
Peabody
S.
,
Jordan
E.
,
Javernick-Will
A.
&
Linden
K.
2020
Monitoring methods for systems-strengthening activities toward sustainable water and sanitation services in low-income settings
.
Sustainability
12
,
7044
7060
.
doi:10.3390/su12177044
.
Huston
A.
&
Moriarty
P.
2018
Understanding the WASH System and its Building Blocks
.
IRC WASH
,
Delft
,
The Netherlands
.
König
A.
2018
Sustainability science as a transformative social learning process
. In:
Sustainability Science: Key Issues
(
König
A.
&
Ravetz
J.
, eds).
Earthscan/Routledge
,
London
.
Lockwood
H.
,
Smits
S.
,
Schouten
T.
&
Moriarty
P.
2010
Providing Sustainable WASH Services at Scale, International Symposium on Rural Water Services – Background Paper, Kampala. Available from: https://www.ircwash.org/sites/default/files/Lockwood-2010-Providing.pdf.
Meadows
D.
1999
Leverage Points: Places to Intervene in a System
.
The Sustainability Institute
,
Hartland, VT
.
Neely
K.
2019a
Introduction: WASH projects and complex adaptive systems
. In:
Systems Thinking and WASH Tools and Case Studies for a Sustainable Water Supply
(
Neely
K.
, ed.).
Practical Action Publishing
,
Rugby
, pp.
1
16
.
http://dx.doi.org/10.3362/9781780447483.001
.
Neely
K.
2019b
Systems thinking and transdisciplinarity in WASH
. In:
Systems Thinking and WASH Tools and Case Studies for a Sustainable Water Supply
(
Neely
K.
, ed.).
Practical Action Publishing
,
Rugby
, pp.
17
28
.
http://dx.doi.org/10.3362/9781780447483.002
.
Posthumus
H.
,
Shah
R.
,
Miehlbradt
A.
&
Kessler
A.
2020
A Pragmatic Approach to Assessing System Change
.
Available from: https://beamexchange.org/resources/1334/ (accessed 10 December 2021)
.
Ramalingam
B.
,
Jones
H.
,
Reba
T.
&
Young
J.
2008
Exploring the Science of Complexity: Ideas and Implications for Development and Humanitarian Efforts
, 2nd edn.
Overseas Development Institute
,
London
.
Valcourt
N.
,
Javernick-Will
A.
,
Walters
J.
&
Linden
K.
2020
System approaches to water, sanitation, and hygiene: a systematic literature review
.
International Journal of Environmental Research and Public Health
17
,
702
720
.
doi:10.3390/ijerph17030702
.
Wandersman
A.
&
Snell-Johns
J.
2005
Empowerment evaluation: clarity, dialogue, and growth
.
American Journal of Evaluation
26
(
3
),
421
428
.
doi:10.1177/1098214005278774
.
WaterAid
2019a
Beyond Building Blocks? Identifying and Monitoring Dynamic Drivers of System Performance
. .
WaterAid
2019b
Glossary of System Strengthening and Empowerment Terms
.
WaterAid
2020
System Strengthening for Inclusive Lasting Water, Sanitation and Hygiene (WASH) That Transforms People's Lives: Practical Experiences From the SusWASH Programme
. .
World Health Organization
2007
Everybody's Business: Strengthening Health Systems to Improve Health Outcomes: WHO's Framework for Action
.
World Health Organization
,
Geneva
,
Switzerland
. .
This is an Open Access article distributed under the terms of the Creative Commons Attribution Licence (CC BY 4.0), which permits copying, adaptation and redistribution, provided the original work is properly cited (http://creativecommons.org/licenses/by/4.0/).