Scorecard diagnostic tool for the enabling regulatory environment of faecal sludge management

Achievement of the sustainable development goal (SDG), target 6.2.1, on safely managed sanitation services for all by 2030 requires an all-inclusive sanitation approach that combines both sewered and non-sewered sanitation services (UNICEF & WHO 2020). However, 46% (3.6 billion) of the global population lacked access to these services by 2020 and at the current rate of acceleration, more than 2.8 billion people might miss this target (WHO | UNICEF JMP 2021). Nevertheless, with more than 3.1 billion people using onsite sanitation technologies worldwide, non-sewered systems are driving considerable progress in achieving this goal especially in developing countries. The number of people using onsite sanitation technologies is expected to grow to 5 billion by 2030 (Strande et al. 2014).

However, poorly managed non-sewered sanitation facilities pose great human and environmental health risks along the whole faecal sludge management (FSM) chain. The FSM chain consists of the user interface (toilet), containment, emptying, transport, treatment, reuse and/or disposal of faecal sludge (Strande et al. 2014). Therefore, the measures needed to protect human and environmental health should be clearly laid out in an effective regulatory framework and firmly enforced along the whole FSM chain (Bassan 2014). Most developing countries lack such clear regulatory frameworks to address each step along the FSM chain. Others lack specific guidelines, regulations or national standards to specifically address FSM (ESAWAS Regulators Association 2019). In other countries where multisectoral-based regulatory frameworks exist, standards and the regulations are often ignored in practice due to unclear mandates among the several regulatory agencies (Burr & Waititu 2018). Thus, FSM services end up neglected both in delivery and in regulation along the entire chain. In addition, lack of rules, reporting mechanism and enforcement procedures have led to difficulties in tracing the past trends and the current outcomes of FSM in most developing countries (Jayathilake et al. 2019).

An enabling regulatory environment outlines the who, the how and the what to regulate for inclusive service provision. It consists of the policy, legal and institutional arrangements including the regulatory tools and instruments required (Coglianese 2012; ESAWAS Regulators Association 2019). For an enabling regulatory environment to successfully promote sustainable FSM services, it needs to be effective to achieve its planned goals (Kirkpatrick & Parker 2004). The World Health Organization (WHO) recommends a situational analysis of existing policies, legislation and practice as the first step towards achieving safely managed sanitation services for all (WHO 2018). The policy, the legal and the institutional considerations should be evaluated holistically for the entire FSM chain but singly and exhaustively for each step (Bassan 2014). In addition, rapid assessment of sanitation policies is recommended for every country in order to assess their adequacy and effectiveness in responding to the prevailing sanitation situations (African Ministers’ Council on Water (AMCOW) 2021).

Unfortunately, only a few developing countries apply some form of assessment on the enabling regulatory environment and where it is applied, the methods adopted are usually partial and unsystematic (Kirkpatrick & Parker 2004). Measuring effectiveness involves comparing a given phenomenon to the best practice or standard of high performance (Stake & Schwandt 2011). Diagnostic tools facilitate the comparison between current and preferred status in order to identify areas of improvement. They utilize a set of indicators to measure the relevant outcomes of concern and a research design to support the conclusions (Olivier 2017). There exist innumerable diagnostic tools for evaluating different aspects of the enabling environment of water, sanitation and hygiene (WASH). Some are described in the sanitation tool compendium (Dey et al. 2016) and others in the mapping of WASH sustainability tools (Schweitzer et al. 2014). However, there isn't a sanitation sector-wide consensus on a common framework or the number or scope of aspects to be evaluated (Gensch & Tillett 2019).

Some of the existing tools such as World Bank's City Services Delivery Assessment (CSDA) (Ross et al. 2016) and Sector Functionality Framework (SFF) (WSUP 2019) employ broad indicators such as enabling, developing and policy/mandates, which subsume the critical elements of the enabling regulatory environment into the broader indicators. Others such as the Global Analysis and Assessment of Sanitation and Drinking Water (UN-Water (GLAAS) and World Health Organization 2019) and the WASH Bottlenecks Analysis Tool (UNICEF 2020) integrate FSM into the general WASH provision. This neglects the detailed assessment of the policy, legal and institutional frameworks, particularly the quality and adequacy of the policy framework as guided by the Guidelines for the Assessment of National Sanitation Policies (Elledge et al. 2002; Water Supply and Sanitation Collaborative Council and World Health Organization 2005; ESAWAS Regulators Association 2019; AMCOW 2021). Other tools fail to include all the steps of FSM from the user interface to reuse. Others combine disposal and reuse as a single step overlooking the fact that disposal and reuse of faecal sludge and its products should be guided and regulated differently since they pose differing human and environment health risks (Strande et al. 2014; WHO 2018).

Therefore, this paper presents the contents, development and web-testing process of Faecal Sludge Management; Policy, Legal and Institutional (FSMPLI) Scorecard tool. This is an integrated web-based diagnostic tool for evaluating the effectiveness of the three critical elements of the enabling regulatory environment along the entire FSM chain. The paper also presents the results of a pilot study in Kenya where the scorecard tool was tested for applicability.

Indicators are the factors used to denote the degree to which the three elements of the enabling regulatory environment meet their objectives based on their existing state or output. The factors were defined by identifying the suitable attributes and the expected standards of each element based on the literature (Coglianese 2012). A three-point Likert-type of scale was applied where the ideal condition of each indicator formed the stem statement, while the three-response scale formed the scoring criteria (Johns 2010). Each criteria was assigned a value and colour code as 1 (green) for effective, 0.5 (yellow) for limited and 0 (red) for poor (Ross et al. 2016) as in Table 1.

A policy is a statement of intent, a commitment, a principle or a rule that guide decision making to achieve a given outcome. It serves as key stimulus for local action and create conditions within which FSM stakeholders operate (Elledge et al. 2002; AMCOW 2021). The policy framework was assessed through five of the seven key elements applicable for the entire sanitation service chain (Water Supply and Sanitation Collaborative Council and World Health Organization 2005; ESAWAS Regulators Association 2019).The five indicators included equity, targeting of resources, quality of service, financial considerations and institutional roles and responsibilities (Elledge et al. 2002). Health and environmental considerations were incorporated into quality of service because quality of any sanitation service affects human and environmental health (Bassan 2014). The regulation and monitoring aspects of the policy were also incorporated into the quality of service since monitoring assesses progress while regulation ensures quality and efficiency of sanitation services. The capacity development aspects were incorporated into institutional roles and responsibilities since capacity development involve improving the ability of different institutions to perform their assigned roles (AMCOW 2021).

The legal framework is a broad set of constitutional, legislative, regulation, jurisprudential and managerial rules that facilitate the implementation and the enforcement of policies. For it to be effective, it should include clear roles and responsibilities for each stakeholder (ACE Project 2021). It should also incorporate requirements for compliance and means for enforcement. These include licensing procedures, monitoring and reporting mechanisms, performance indicators, penalties and sanctioning procedures for non-compliance as well as incentive for outstanding performance (Kameri-Mbote & Akech 2011). The legal framework was evaluated by the existence and operationalization of laws, standards and regulations with an emphasis on incentives, penalties and sanctioning procedures employed (SNV and ISF-UTS 2016; WHO 2018; ESAWAS Regulators Association 2019).

The institutional framework comprises the different organizations and other stakeholders involved in FSM services provision. Effective institutional frameworks facilitate the design, delivery, regulation and consumption of FSM services (Mumssen et al. 2018). It was evaluated based on operationalization of the roles and responsibilities of the actors of FSM (investors, service providers, regulators and consumers) in implementing the policy framework and enforcing the legal framework (Chua et al. 2005).

The web-based scorecard diagnostic tool was built using the Laravel Framework Hypertext Preprocessor (PHP) for server-side processing. Laravel (PHP) is one of the most popular open-source web development frameworks. It is flexible, secure and it supports rapid application development (McCool 2012). The data were stored in MySQL which is also a popular open-source Structured Query Language database management system. MySQL allowed addition, access and processing of the data stored in it, while offering a flexible programming environment (Christudas 2019). The tool was organized into models as described in Table 2.

The results in FSMPLI-Scorecard were displayed in either a sunburst or matrix chart developed on the Chart.Js library. Chart.Js is also a free open-source data visualization library used to create a wide range of responsive web-based visualization (Da Rocha 2019). The tool was hosted on an Apache web server with security-enhanced Linux which enhances the mandatory access control policy. The mandatory access control denies unnecessary access to the web application but allows required access. This provides the web server functionality while limiting potential damage to it (Gosselin & Schommer 2001). A user is required to register as an administrator to set up a diagnostic survey for any predefined area of study. The administrator then invites the relevant stakeholders to give their opinion on the effectiveness for each indicator for each step along the FSM service chain. The tool has a user guide video developed at fiverr.com: a freelance platform offering digital services (Green et al. 2018). Figure 1 shows a flow chart for FSMPLI-Scorecard.

FSMPLI-Scorecard was tested for functionality using the PHP unit. Test protocols were created for each application unit including the category, location, result, section, survey and user. This was done by creating scenarios such as insert data, validation, update data, view data and delete data in each unit (Alrashed 2018). It was tested for browser compatibility with Internet Explorer, Edge, Firefox, Safari, Opera, Chrome, iOS and Android using power mapper (Mesbah 2014). The server IP address for FSMPLI-Scorecard was tested for loading and stress using artillery (Howe et al. 2021). It was also tested for security in terms of package vulnerabilities, Secure Socket Layer (SSL) access and directory access using a Laravel package to assess the ease of access by unauthorized users (Dahle 2020).

Purposively selected stakeholders and FSM experts were invited to test the functionality of tool in application through a pilot study of the effectiveness of the regulatory environment of FSM services in Kenya. They included officials from government ministries, regulators, academia, non-state actors and service providers, as shown in Table 3. After an exposure to the tool in a stakeholders' workshop, the participants assigned each indicator a score of 1 for effective, 0.5 for limited or 0 for poor for each step along the FSM service chain based on their individual experiences and best of knowledge. The tool calculated the average score for each indicator, the average score for each element and the overall score as a single index.

The FSMPLI-Scorecard numerically evaluated the effectiveness of the three elements of the enabling regulatory environment to show areas of strength and weakness as well as identify areas of priority for each indicator along the FSM chain (Ross et al. 2016). It considered each step individually from the user interface, containment, emptying, transport, treatment, disposal and reuse (Strande et al. 2014). Considering each step ensures that the different actors, infrastructure and procedures applied per step are adequately addressed in the policy, legal and institutional frameworks (Bassan 2014). This consideration expanded the SFF (WSUP 2019) which evaluates the sanitation sector in general without narrowing down to the specific steps of FSM. It also expanded the World Bank's CSDA (Peal et al. 2014; Ross et al. 2016; Scott et al. 2019) which leaves out the user interface stage and also considers disposal and reuse as a single step.

FSMPLI-Scorecard not only examined the existence of a policy framework but assessed its quality based on the predefined key elements of sanitation policies (Elledge et al. 2002) and the African guidelines for sanitation policy (AMCOW 2021). By employing predefined and already published indicators, the tool allows local and contextualized interpretation of the indicators while maintaining their universal applicability. They are narrow in scope compared to other existing diagnostic tools (Schweitzer et al. 2014; Dey et al. 2016), to effectively facilitate a detailed assessment of each element. Table 4 shows the alignment and differences between FSMPLI-Scorecard and similar tools.

The open-access web-based FSMPLI-Scorecard diagnostic tool found at http://fsmpli-scorecard.com/ facilitated an integrated participatory evaluation of the policy, legal and institutional frameworks of the FSM service chain based on predefined indicators and scoring criteria (Mansourian et al. 2011). It covered all the steps of FSM from the user interface (U), Collection (C), Emptying (E), Transport (Tr), Treatment (Tt), Disposal (D) and Reuse (R). The tool's front page is shown in Figure 2.

The tool can be used by any registered administrator at any internet supplied geographical location across the globe (Rogers et al. 2020). By integrating the different aspects of measurement, the tool facilitates a comprehensive situational analysis of the enabling regulatory environment. It rates the quality of each element according to the predefined indicators and scoring criteria along the entire FSM service chain. This holistic diagnosis could reduce the time, resources and expertise required for the evaluation (Ross et al. 2016).

The three Likert-type scale applied in FSMPLI-Scorecard sufficiently identified the indicators in need of improvement for each step, without necessarily breaking them into finer categories (Johns 2010). The predefined scoring criteria standardizes the meaning of the scores thus reducing ambiguity by providing a common understating of the scores among diverse stakeholders with diverse interest across the sector (Rogers et al. 2020). The tool calculates the average score for each indicator, then the average score for each element under diagnosis and the overall score as a single index. The overall score indicates the effectiveness of the enabling regulatory environment categorized in terciles as the following: 0–0.33 (poor), 0.34–0.66 (limited) and 0.67–1 (effective) (Meyer et al. 2019). This three-level numerical scale aims to qualitatively indicate areas of priority to stakeholders and not necessarily a quantitative evaluation (Ross et al. 2016).

The results are presented in charts using different colour codes of green for effective, yellow for limited and red for poor. This speeds up the visualization and interpretation of the outcome for each indicator. The presentation helps to establish the probable cause–effect relations for the score attained by a given indicator to the specific step of FSM (Burr & Waititu 2018). The tool is balanced in the sense that, the overall score is an average score of each element per indicator per step of FSM. This denotes an even distribution of weight among the elements and across the range of indicators (Kaplan & Norton 1992). The sunburst presentation of the results enhances the perception of each element and each indicator as important and the interrelationship between the elements is implicitly understood (Schulz et al. 2011). Figure 3 shows an example of a sunburst presentation of results in FSMPLI-Scorecard tool. However, due to the chain nature of FSM services, the results can be presented in a matrix format as shown in Figure 4 depending on user preference.

All the units in the tool passed for functionality in all the scenarios tested. Therefore, the tool is expected to perform its functions such as inserting data, validation of data, updating data, viewing data and deleting data as it was designed to (Alrashed 2018). The tool was found compatible with all the browsers tested as shown in Table 5. Therefore, it can be accessed and applied through any of the stated browsers.

For the loading test, a ping to the server address retuned with 0% loss of data in an approximate average round-trip of 158 ms. It was able to support 496 concurrent virtual users in 820.7 ms in the stress testing. Therefore, it was found reliable and capable of performing its functions satisfactorily regardless of a high number of concurrent users (Subraya & Subrahmanya 2000) In the security test, none of packages were found vulnerable to external attack and only the public directory is accessible from the browser since the projects code is stored in in accessible a directory. Therefore, the tool is secure in terms of access from unauthorized users and external attacks (Kundu 2012).

The enabling regulatory environment of FSM in Kenya received an average score of 0.47 through the multisectoral FSM stakeholders' opinion survey. The policy framework was rated as limited with an average score of 0.45, across the five indicators. However, financial considerations were found poor for most FSM services with an average score of 0.28. The legal framework was also rated as limited with an average score of 0.5 across the three indicators. The institutional framework was also found limited with a low score of 0.39. It was particularly found poor in terms of the investors for most FSM services as well as service providers for emptying services. The results as shown in Figure 4 are in a matrix format.

The survey was carried out based on each participant's inherent knowledge and experience soon after the initial exposure to the scoring framework. It is therefore likely that they all did not have the necessary in-depth knowledge or prior experience with each element or similar understanding to give identical scores for each indicator per step of FSM services. They also probably did not have time to access and assess available evidence to score the indicators patently. However, the tool linked each stakeholder's opinion to give an average score per indicator per step and also an overall score for the enabling regulatory environment in Kenya. By linking these opinions together, the tool would improve the quality of the diagnosis and also avoid biased conclusions (Rogers et al. 2020).

However, there was no moderation for consensus building since it was a pilot test on the tool, but stakeholders' participation formed a basis for a sector-wide dialogue towards consensus building (Rogers et al. 2020). Therefore, the participants requested for a subsequent all-inclusive stakeholders' forum to discuss the outcome of the diagnosis towards a common ownership of the results. They also suggested to be provided with the scoring criteria prior to the survey, so as to consider available evidence for each indicator and to allow consultations among colleagues so as to present institutional-based perspectives.

FSMPLI-Scorecard evaluated the effectiveness of the enabling regulatory environment of FSM based on the indicators described in the Regulation Strategy and Framework for Inclusive Urban Sanitation Service Provision Incorporating Non-Sewered Sanitation Services developed by the Eastern and Southern African Water and Sanitation (ESAWAS) Regulators Association. However, it adapted, improved and expanded some broad indicators from several existing diagnostic tools such as the World Bank's CSDA and WSUP's SFF in order to specifically assess the quality of the policy framework according to the Guidelines for the Assessment of National Sanitation as provided by the Water Supply and Sanitation Collaborative Council and WHO. It also considered the African Sanitation Policy Guidelines provided by the African Ministers' Council on Water.

The tool was found functional in its application by successfully facilitating a participatory diagnosis of the effectiveness of the policy, legal and institutional frameworks of FSM services along the entire service chain. It proved effective in identifying areas of improvement across all the indicators along the entire chain. For instance, the financial considerations in the policy framework, the investors and service providers in the institutional framework were identified as the main priority areas of consideration in the Kenyan pilot study. However, the tool does not provide the specific details leading to a given score assigned to each indicator per step. Therefore, its application should precede a detailed evaluation of each indicator in a given context in order generate specific details per indicator per step of FSM. This would lead to development of evidence-based priorities towards an effective enabling regulatory environment of FSM services.

The open-access web-based FSMPLI-Scorecard can be reliably and safely applied to diagnose the enabling regulatory environment of any predefined study area such as a region, a country, a city or even a smaller geographical extent. However, it should be evaluated against a user's opinion and preferences.

Participation of the sector stakeholders, revealed diverse perspective and opinions on the current status of each indicator. This initiated the identification of priorities towards a joint commitment among the stakeholders for accelerated safely managed sanitation services for all. However, the tool does not dictate the specific remedial actions to be taken to improve the low scoring elements but it establishes the expected goal such that each respective arm or actor should adopt the most appropriate action plan to arrive to those goals.

To draw a more comprehensive and conclusive diagnosis, more and representative stakeholders should be invited to participate in the opinion survey. An all-inclusive stakeholders' forum should be held to discuss the outcome of the diagnosis towards a consensus building and a common ownership of the results. This would be essential for the development of coherent framework of actions for strengthening the policy, legal and institutional framework of FSM services in a given context.

Ethical aspects considered in this research included the following: informed voluntary participation, right to refuse or withdraw, confidentiality and privacy, risks and benefits and payment for participating in the study. They were presented in consent forms that were signed by each participant prior to data collection: The research was approved by the Jomo Kenyatta University of Agriculture and Technology Institutional Ethics Review Committee (JKUAT IERC); Ref. No. JKU/2/4/896B and was licensed by the National Commission for Science and Technology and Innovation (NACOSTI) Ref. No. 932560.

FSMPLI-Scorecard was developed under a research (partially) funded by the African Doctoral Dissertation Research Fellowship (ADDRF) award offered by the African Population and Health Research Centre (APHRC) and funded by the Bill and Melinda Gates Foundations through a project grant to APHRC. We wish to appreciate our colleagues at the Jomo Kenyatta University of Agriculture and technology for their positive critique and suggestions throughout the study. We appreciate all the key interview informants, the validation workshop participants and their institutions for their invaluable contributions towards the development and testing of this tool. We acknowledge the contributory comments and suggestions of Rémi Kaupp of WaterAID UK during the development of the indicators and scoring criteria. We also appreciate, Mbitrix Technologies Limited for their partnership in the tool development.

All relevant data are included in the paper or its Supplementary Information.