SIASAR: a country-led indicator framework for monitoring the rural water and sanitation sector in Latin America and the Caribbean

Currently, an estimated 663 million people worldwide still use unimproved drinking water sources, while 2.4 billion still use unimproved sanitation facilities (Joint Monitoring Programme 2015b). Against this background, the 2030 Agenda for Sustainable Development adopted in September 2015 includes a specific sustainable development goal (SDG) dedicated to water and sanitation (United Nations General Assembly 2015). This goal (SDG 6) seeks to complete the unfinished business of the millennium development goals (MDGs), specifically setting out to ‘ensure availability and sustainable management of water and sanitation for all’ (United Nations 2014; United Nations General Assembly 2015). In light of the investments envisaged to reach this ambitious goal, updated and comprehensive data are required for sector performance assessments and promote decision-making. In turn, indicator frameworks are needed to allow the organization of an increasing amount of available data. Furthermore, the relevant communities of sectorial practitioners need to be consulted, and field pilots need to be carried out to test the monitoring framework and to ensure that they are scientifically valid and contextually salient (Kayser et al. 2013).

Numerous approaches have been developed to provide a coherent monitoring framework for addressing specific problems, ranging from improving data availability, to facilitating access to information and encouraging the use of this information in decision-making (Giné-Garriga 2015). Water, sanitation and hygiene (WaSH) data have been typically collected through different methods (Bostoen 2007; WaterAid and ODI 2005; United Nations Children's Fund 2006; Jiménez Fdez. de Palencia & Pérez-Foguet 2012; Giné-Garriga et al. 2013). Likewise, the sector has witnessed the development of a variety of approaches for monitoring and evaluating service delivery (Sullivan et al. 2003; Cohen & Sullivan 2010; Giné-Garriga & Pérez-Foguet 2011, 2013a; Flores-Baquero et al. 2013; Kayser et al. 2013; Luh et al. 2013; Bartram et al. 2014). Ultimately, for data use, important efforts have been made to provide decision-makers with reliable information that can support planning, targeting and prioritization, particularly in decentralised contexts (Ghosh & Rao 1994; Jiménez Fdez. de Palencia & Pérez-Foguet 2011; Giné-Garriga et al. 2015).

At the international level, the WHO/UNICEF Joint Monitoring Programme (JMP) has taken on the role of producing and facilitating national, regional and global estimates of populations using improved facilities. Data have been analysed using a binary categorisation of those households that use an ‘improved’ or an ‘unimproved’ drinking water source and sanitation infrastructure (Joint Monitoring Programme 2006). In both cases, service ladders have been defined to allow a disaggregated analysis through four rungs, which represents the evolution from the worst scenario (surface drinking water sources and open defecation practice) to the optimal one (on-premise piped water and sanitation facilities that ensure hygienic separation of human excreta from human contact). Admittedly, the indicators employed by the JMP have fallen short of measuring progress in some key areas, and more precise and complete measurements are required to drive the sector forward (Giné-Garriga & Pérez-Foguet 2013b; Joint Monitoring Programme 2015a).

In Latin America and Caribbean (LAC) countries, a regional sector information system (IS) has been developed, namely the Rural Water and Sanitation Information System (SIASAR; Sistema de Información de Agua y Saneamiento Rural). SIASAR seeks to support decision-making of a variety of stakeholders involved in the sector (such as policy makers, national and local planners and sector practitioners). Among others, one aim of this initiative is to improve resource allocation by influencing targeting and prioritization; thus, it promotes evidence-based planning (Pena et al. 2014).

This research describes SIASAR as a country-led monitoring initiative. We aim to highlight the way in which this monitoring system can be used to more fully help involved stakeholders with the decision-making process, by providing a focus for the stakeholder engagement process. Specifically, this study assesses those aspects regarding data collection, analysis and use within the SIASAR initiative, paying special attention to the role of all member countries in these steps. In the introduction, we provide a short description of the IS origin, organization and functionality, followed by the two-fold perspective upon which this research is conducted in Section 2. The main aspects regarding data collection, analysis and use, and the results associated to this monitoring framework tested in the field, are provided in Section 3. We conclude that SIASAR provides an adequate IS for monitoring the rural water supply and sanitation (RWSS) subsector, despite its increasing decentralization, by focusing on service level and sustainability aspects. Finally, we highlight the advantages of the participatory process and the development of a large-scale IS.

The SIASAR initiative was initially launched in 2011 by the governments of Nicaragua, Honduras and Panama. Although these countries already had their own IS, these were out-of-date and overly focused on the water component, largely neglecting sanitation and hygiene issues. In this context, strategic partners such as the World Bank (through the former Water and Sanitation Program, currently known as Water Global Practice), the Inter-American Development Bank, the Spanish Agency for International Cooperation and the Swiss Agency for Development and Cooperation joined efforts to support these countries in the development of SIASAR. This IS focuses not only on service delivery but also integrates sustainability aspects as a core element. This last aspect was identified by and agreed upon by country experts as the main challenge to be addressed in the RWSS subsector.

In 2012, the first version of the SIASAR conceptual model, questionnaires and Information and Communications Technology (ICT) tools became functional. This milestone allowed countries to first collect and process information and then to display it publicly through the SIASAR website and other dissemination mechanisms (e.g., automatic reports, dynamic maps and defined indicators). Since 2013, with a consolidated system and more than 5,000 registered communities, SIASAR has expanded, mainly through the natural framework of the Central American Forum and the Dominican Republic of Drinking Water and Sanitation (FOCARD-APS; Foro Centroamericano y República Dominicana de Agua Potable y Saneamiento). In 2014, the first Regional Agreement was signed, and SIASAR was adopted as the regionally approved and free affiliation IS for the FOCARD-APS countries as well as for those who wanted the convenience of using these instruments. Thus, Dominican Republic, Costa Rica and the Mexican state of Oaxaca joined the initiative in 2014, and Peru, the Brazilian state of Ceará, Bolivia, Paraguay and Colombia joined between 2015 and 2016. Today, SIASAR is in use in eleven countries.

As previously mentioned, the SIASAR initiative is rapidly expanding within the LAC region. This expansion clearly challenges the dynamics of the organizational structure and work sessions. Nevertheless, these challenges are balanced by the strong governmental ownership and continuous participation of involved members, which generate positive aspects in terms of sustainability and stakeholder empowerment.

Moreover, for functionality, several instruments are in place to strengthen coordination. First, both sectorial and ICT issues have been discussed during innumerable virtual working sessions. To do this, different groups have been created, taking advantage of their capabilities and specific interests. While sectorial teams focus on more conceptual aspects, ICT groups make the resulting ideas and agreements tangible and functional. These parallel efforts are coordinated through the very frequent communication among all SIASAR stakeholders (i.e. monthly videoconferences), during which main issues and improvements are discussed, shared and formally approved by all members. Among these aspects, key indicators and data collection processes have been defined (and revised as needed), a comprehensive conceptual framework has been developed (and updated as required) and several instruments for data use have been implemented. As a result, SIASAR was developed natively on a highly practical and interactive web platform that draws on the strengths of open source and mobile technology (Pena et al. 2014). Further, a SIASAR General Assembly takes place annually, bringing together all working groups – from the executive to the technical levels—, which allows participating countries to debate current and future issues and increases a sense of community. In addition to this, field missions and pilot studies are implemented as needed to test the required aspects in practice. Finally, regional events take place when new countries officially join SIASAR, to: (i) test in field the validity of the survey instruments; (ii) share good practices for data collection; and (iii) promote the sense of community that characterizes this initiative.

It is important to stress that this entire initiative, from the organizational to the functional levels, has been created ad hoc by the SIASAR community itself. It is not dependent on any organization or international structure. SIASAR has designed its own coordination and work space and has provided budget and distributed costs among country members. This level of self-management achieved by these countries is one of the strengths that allows SIASAR to be considered a successful case of a collaborative and effective network among teams from 11 countries.

Within this study, we will analyze SIASAR from a two-fold perspective. First, we focus on major aspects related to the information cycle, including the issues of (i) data collection, (ii) data analysis and (iii) data use. Second, we highlight the existing participatory processes of the initiative. This is especially relevant when addressing: (i) stakeholder engagement and appropriation of the IS; and (ii) the medium- and long-term sustainability of the IS. The three information challenges (data collection, analysis and use) are addressed from this perspective. Finally, we provide a range of illustrative examples to foster a better understanding by the reader about SIASAR's scope and performance.

Public investment in the RWSS subsector of LAC countries has been traditionally biased toward the construction of new infrastructure, and little attention has been paid to other factors that compound sustainable water and sanitation service delivery (Lockwood et al. 2010). Understanding those factors is critical for addressing sustainability gaps and for improving policy development, sector planning, priority setting, budget allocation, project design and technical assistance provision. This core idea led countries to design a set of survey instruments to analyse the quality and sustainability of services from different perspectives: (i) the community; (ii) the water system; (iii) the service provider; and (iv) the technical assistance provider (see Table 1). In parallel, the alignment of SIASAR with international monitoring standards (e.g. indicators related to use of water and sanitation infrastructure, as those proposed by the JMP during the MDG period) was taken into account. Briefly, one specific questionnaire was elaborated to collect data from each information source, as shown in the following table.

For data collection processes, the SIASAR community paid attention to three key stages. The first stage involves those preliminary tasks needed to ensure successful field work. From these activities, special focus is given to: (i) previous contacts with local institutions and sector practitioners in a given area, as this aspect favours field planning, initiative dissemination and generation of synergies; and (ii) survey team training given by the SIASAR-responsible entity in each country.

The second stage represents one of the salient aspects of SIASAR, as it combines data collection from households, the water systems and the service provider. This is especially relevant as widely used household surveys (e.g., Multiple Indicator Cluster Surveys of UNICEF, and the Census of Population and Housing) are not sufficient to monitor many relevant questions related to the RWSS subsector (Giné-Garriga et al. 2013). A separate questionnaire was therefore developed to assess these different information sources. Data are then collected through different mechanisms: a closed-question survey, direct observation and water quality testing. For a given geographical area, the methodology consists of visiting all communities in the area. In each community, all existing water systems are inspected, members of the organizations in charge of providing the service are interviewed and a tour around the community (including schools and health centres (SHC)) takes place. Associated technical assistance providers are identified as well; it should be pointed out that these actors generally have a different scale of intervention. Finally, sanitation and hygiene information is obtained through household visits and direct observation. The selected sample of households is defined according to each country's resources and/or community participation degree. Additionally, the use of ICT-based devices are noteworthy, as these facilitate the survey teams' ability to collect, store and transmit related data.

Finally, the third stage integrates the validation process, which is crucial for ensuring data quality. Validation is carried out through: (i) an instant check facilitated by the survey device (internal programming and detection of logical errors) and an agreement with the interviewed players; and (ii) a final verification executed by municipal or national authorities. Once the information is validated, data are published on the public website. As can be seen in Table 2, the majority of the SIASAR countries have already collected a significant amount of data. The case of Nicaragua is noteworthy, as it carried out a baseline of all rural communities, systems and service providers. This milestone was officially certificated by its government. Furthermore, and as expected during any data collection process, Nicaragua has also finished updating this baseline.

Undoubtedly, the SIASAR initiative has grown over the past few years. This growth brings the challenge of constantly reviewing survey instruments, with the aim of integrating each country's realities and of ensuring comparability. Under a principle of simplicity – use as little information as necessary, but not less—, each country's particularities are analysed, discussed and finally harmonised and approved by all countries. Additionally, SIASAR visualizes the need of alignment and collaboration with other international monitoring systems. In this sense, a thorough revision of survey instruments is currently taking place, in order to tailor them to post-2015 key monitoring elements.

One salient aspect of SIASAR is the manner that collected data are organised and analysed (Pérez-Foguet & Flores-Baquero 2015). Six aggregated dimensions are defined to assess water and sanitation services from different and complementary points of view. The aim of this structure is to keep different aspects in focus that characterise an increasing decentralized RWSS subsector, as in practice, institutional roles and responsibilities of sectorial issues are assumed by different stakeholders (Giné-Garriga & Pérez-Foguet 2013a, 2013b). This characteristic is common to most of the SIASAR countries. The dimensions mentioned have been proposed to measure: (i) the water service level (WSL); (ii) the community sanitation situation and various hygiene issues at the household level (SHL); (iii) the condition of water system infrastructure (WSI); (iv) the service provider performance; (v) the technical assistance provider performance; and (vi) the WaSH situation in public institutions (SHC). Additionally, at a higher level, the cited dimensions are aggregated into two partial indices: (i) water, sanitation and hygiene service level (WSHL), and (ii) water services sustainability index (WSSI). These partial indices aim to keep the focus on aspects related to quality and sustainability of services identified by all member countries. Finally, a last level is represented by an aggregated water and sanitation service performance (WSP) index. These last indices provide a means of initiating discussion and stimulating public interest. All presented elements are listed in Table 3.

Each dimension comprises four components. In turn, each component is fed by a short list of single indicators, with a total of 109 indicators. In terms of method and technique, index construction relies on a simple step-by-step procedure: (i) the selection and combination of key indicators into their corresponding subindices, using an equal and dimensionless numeric scale; (ii) the determination of weights for each subindex and their aggregation to yield an overall index; and (iii) the dissemination of index values by means of a grading system, classifying communities in four levels.

First, indicators are classified according to the previously described conceptual framework. As collected data are frequently represented on different scales (such as percentage of systems with adequate water treatment, distance-to-source in metres, service continuity in hours per day, and so forth), they have to be normalized prior to their analyses. A score between 0 and 1 is assigned for each parameter, whereby 1 represents the best performance and 0, the worst-case scenario. Components are then defined by simple and easy-to-use multi-attribute utility functions. Table 4 represents an example in which the operation and maintenance (O&M) management of the service provider is assessed through discrete values (note that punctuation is linear in some cases).

Next, different components of each dimension are aggregated into a single value. Two major issues need to be addressed: (i) the choice of weights should reflect the relative importance of each component, and (ii) the aggregation function should be consistent with the theoretical framework (Nardo et al. 2008; Giné-Garriga & Pérez-Foguet 2010; Flores-Baquero et al. 2016). For weight assignment, two approaches were compared: (i) equal weights, and (ii) weights according to expert opinion (analytical hierarchical process). Several experts from all member countries were involved in this process. Based on comparative analysis, equal weights are assigned to all components. Similarly, partial indices are calculated by using an equal weighted grouping of all dimensions, and a general index (the WSP index) is created by giving both partial indices the same relative importance. For the aggregation method, two different alternatives were tested: (i) linear (compensatory), and (ii) geometric (partial compensatory). According to the analysis carried out, partial compensatory methods (geometric) might favour the appearance of null values, hampering the duty of discrimination among results. Thus, all dimensions are constructed by allowing the compensation of their components (additive aggregation). However, due to the first compensation, posterior aggregations for partial and general indices are geometric.

Finally, in order to foster prioritization and decision-making, the achieved results were made more understandable for final users and stakeholders by linking components, dimensions and index values to a defined set of categories (A, B, C or D, whereby A represents the best result and D, the worst). This representation can be done in several ways, and two alternatives were tested here: (i) equal intervals, and (ii) different intervals. The results of the two tested methods differed significantly from each other. Overall, however, the different interval method was the preferred method: despite being less simple conceptually, the fact that it requires a higher overall score in order to move up to a better category sets a higher service threshold, which can be seen as a positive aspect. Thus, the different interval method is the classification adopted for visualizing results. Specifically, the intervals are defined as follows: A, [1–0.9], with both limits included; B (0.9–0.7]; C (0.7–0.4]; and D (0.4–0].

The involvement of the member countries in conceptual framework design has been indispensable. First, preference levels (punctuation) of a component's utility functions were agreed by common consent at the sectorial level. The same procedure was carried out to define an aggregation methodology, while the assignment of weights was based on a wider consultation (as mentioned previously). There is no doubt about the need to preserve the consensus among countries in relation to the SIASAR concept and its monitoring framework. However, it is also necessary to validate the conceptual model against real data. The validation showed that the model is sufficiently flexible to evolve and to be adapted to future requirements. This principle of flexibility allowed different versions of the conceptual model to be developed. As a result, and in terms of calibration, survey tools as well as the definition of the different components have been improved. The aim was to favour (i) the refinement of the preference levels, and (ii) the duty of discrimination. One illustrative example relates to the definition of grading system thresholds. As shown in Table 5, there is a notable difference when equal intervals or different intervals are used. According to the former, 62% of communities obtain a favourable value, which might direct attention to other locations. In contrast, using different intervals places 85% of communities in lower qualifications, thereby increasing the range of more needed settlements. As the regional goal is to reach better service levels, this latter grading system is currently implemented.

The ultimate goal of sound sector-related data is to facilitate its use and to improve decision-making. To do this, two elements are necessary (Grosh 1997): (i) the data must be analyzed to produce outcomes that are relevant to the policy question, and (ii) the analysis must be disseminated and transmitted to policy makers. Assuring that data are easily accessible and are presented in a user-friendly format will stimulate their use. Otherwise, decision-making will be carried out without considering the information.

To promote data use, SIASAR has designed and developed a battery of harmonized reports. They provide a comprehensive assessment of key issues reflected in the conceptual model. As the aim is to guarantee the usefulness of these tools, reports have been tested in field with key stakeholders, paying attention to the local level. The following examples show the practical use of SIASAR as a decision-making tool. In Nicaragua, during the period 2013–2016, 64 rural municipal plans for water supply and sanitation were developed using data from the system, supporting the prioritization of the most vulnerable communities to receive investments and technical assistance. Since 2014 in Honduras, sectorial profiles of 28 municipalities have been developed based on SIASAR information that target technical assistance activities and inform the development of Municipal Development Plans. This is especially relevant as these plans are an instrument by which Honduran municipalities formally request the transfer of national funds for local investments. In Panama, SIASAR data were used in 2016 to better target investments and technical assistance in indigenous communities. In Dominican Republic, 33 water systems were rehabilitated during 2016 based on information collected through SIASAR.

In this paper, we have presented the SIASAR initiative, which has been implemented in several countries to provide updated and reliable information and to support sector decision-making. Key messages from this initiative are:

• - Although a participatory process is time and resource-consuming, the benefits from such a strategy are remarkable in terms of government ownership, stakeholder engagement and sustainability. The processes presented here have stimulated pedagogic spaces where country members have learnt by doing, collaborating in the development of the IS. This fact facilitates a constant improvement of the tool.

• - Clear advantages come from the development of a large-scale IS. The use of a harmonized monitoring architecture provides regional understanding, collaboration and comparison. In this sense, special attention is paid to the celebration of regional meetings aiming to (i) test in field the validity of the survey instruments, (ii) share good practices for data collection, and (iii) promote the sense of community that characterises this initiative.

• - SIASAR has emerged as an adequate monitoring framework for the RWSS subsector for numerous reasons:

  • • It exploits relevant data to assess dispersed rural water systems, from mere technical aspects to sustainable ones. It combines data at household, community and water infrastructure and service provider levels. Its conceptual model promotes identification of priority elements of the assessment, and the technological platform facilitates this analysis at different scales, allowing informed strategies and interventions to be designed.

  • • Resulting from the participatory process for conceptual model development, country members have increased their technical capacities and their understanding of the implications of index construction, allowing them to delve deeper into their own sector needs, test proposed alternatives in field and takes consensual decisions.

  • • SIASAR provides easy-to-use and meaningful tools to better identify needs and priority segments of the population, and so to target future investments more effectively. Already tested in field, existing tools have supported the development of sectorial diagnosis, the design of sectorial plans, the implementation of specific actions and, in summary, those relevant decision-making processes for which SIASAR was envisaged and created.

The authors would like to thank all country members of the SIASAR community for their dedicated and future work in all processes carried out. They are the ones who make all this possible.

Further thanks go to The World Bank, which has believed on this research group's capability and commitment and which has provided support for the research in numerous ways.