Embracing data uncertainty in water decision-making: an application to evaluate water supply and sewerage in Spain

Analyses of complex water management decision-making problems, involving tradeoffs amongst multiple criteria, are often undertaken using multi-criteria decision analysis (MCDA) techniques. Various forms of uncertainty may arise in the application of MCDA methods, including imprecision, inaccuracy or ill determination of data. The ELECTRE family methods deal with imperfect knowledge of data by incorporating ‘pseudo-criteria’, with discrimination thresholds, to interpret the outranking relation as a fuzzy relation. However, the task of selecting thresholds for each criterion can be difficult and ambiguous for decision-makers. In this paper, we propose a confidence-interval-based approach which aims to reduce the subjective input required by decision-makers. The proposed approach involves defining the uncertainty in the input values using confidence intervals and expressing thresholds as a function of the interval estimates. The usefulness of the approach is illustrated by applying it to evaluate the water supply and sewerage services in Spain. Results show that the confidence interval approach may be interesting in some cases (e.g. when dealing with statistical data from surveys or measuring equipment), but should never replace the preferences or judgments of the actors involved in the decision process.

Decision-making in water management is inherently complex. Water decisions often involve large numbers of alternatives, competing objectives, and participation of multiple stakeholders with conflicting interests (Hyde et al. 2005). Consequently, a formal framework for water resources decision-making is required. Multi-criteria decision analysis (MCDA) provides a structured approach for analyzing decision problems with multiple objectives and criteria (Mutikanga et al. 2011). MCDA can assist decision-makers in identifying critical issues, assigning relative priorities to those issues, selecting the best compromise solutions, and enhancing communication in the evaluation of decision problems (Flug et al. 2000).

Numerous MCDA methods have been developed over the years, and are commonly classified in three classes: full aggregation approach, outranking approach, and goal, aspiration or preference-level approach (Ishizaka & Nemery 2013). The ELimination and Choice Expressing the REality (ELECTRE) methods developed by Roy (1991) belong to the group of outranking approaches and are one of the best known and widely applied methods, especially in Europe (Wang & Triantaphyllou 2006). This is evident from their broad use in wide-ranging decision-making situations, from natural resources and environmental management to structural engineering, logistics and supply chain management, and public planning and policy decisions (Govindan & Jepsen 2016). In water management, the specific application areas include ranking water allocation strategies (Bella et al. 1996; Zardari et al. 2010), assessing projects for river basin planning and development (Duckstein et al. 1982; Raj 1995), selecting alternative strategies for managing irrigation systems (Raju et al. 2000; Pedras & Pereira 2009), choosing operation rules for reservoir systems (Ko et al. 1994; Malekmohammadi et al. 2011), prioritizing pipe rehabilitation projects in water and sewer networks (Carriço et al. 2012; Tscheikner-Gratl et al. 2017), comparing watershed management schemes (Tecle et al. 1988; Ceccato et al. 2011) or identifying priority water users or regions for future inversions (Roy et al. 1992; Morais & Almeida 2006). However, despite their extensive application, the drawbacks of ELECTRE methods are still discussed by researchers (Figueira & Roy 2009; Figueira et al. 2013), mainly what their theoretical limitations are and whether they aid the decision-making process.

In addition, as in every other MCDA method, uncertainty is ubiquitous in the ELECTRE decision-making process. According to French (1995), different forms of uncertainty may arise in decision analysis from imprecision, ambiguity or lack of clarity. One form is the uncertainty about the selection of criteria that adequately represent the objectives of the decision problem. Another is the uncertainty surrounding the assignment of criteria weights. There is also uncertainty related to the numerical accuracy of input data. Data uncertainty (i.e. the degree to which data are inaccurate, imprecise or unknown) can be due to many factors, such as inherent variability (from the natural processes that continually affect water resources), measurement errors (caused by equipment or random sampling effects) and boundary conditions (from external factors that cannot be accounted for explicitly) (Klauer et al. 2006). However, as stated by Xu & Tung (2008), MCDA methods are often applied without much consideration given to the uncertainty in the input data and their propagation into the problem solution. As can be expected, data uncertainty may have an important influence on the ranking of alternatives (Eastman et al. 1993), which thus casts significant doubt on the decision analysis results.

Dealing with inaccurate, imprecise, uncertain or ill-determined data is one of the foremost strong features of the ELECTRE family methods (Figueira & Roy 2005). Instead of ‘true criteria’, ELECTRE methods include ‘pseudo-criteria’, with discrimination thresholds, to account for the imperfect knowledge of the data (Figueira et al. 2013). However, fixing the discrimination thresholds for each criterion can be a difficult and ambiguous task for decision-makers, and remains a problematic issue (Govindan & Jepsen 2016). A number of researchers have addressed the need for more comprehensive approaches for selecting appropriate threshold values. Rogers & Bruen (1998) described a methodology for choosing realistic threshold values for use in environmental appraisal systems. The method took into account the effect on human beings of the difference between criterion scores. Hokkanen & Salminen (1997) provided another approach for selecting thresholds in the context of solid waste management systems. It associated thresholds with the possible error range in criteria, which was inferred with the help of regression analyses. On the other hand, Banias et al. (2010) overcame the subjectivity issue by connecting the thresholds to the performance values range (i.e. the difference between the maximum and minimum values), divided by the number of alternatives. The idea behind this was to emphasize the discrimination power of the method: the more alternatives there were, the more necessary it was to have finer thresholds to discriminate among them. This approach, which echoed others in the literature (Haralambopoulos & Polatidis 2003; Polatidis & Morales 2016), provided a simple way for determining the thresholds, but ignored the uncertainty underlying the data. More work needs to be done in order to assist decision-makers in choosing thresholds in a rational and defendable manner.

In this paper, we introduce an extension of the ELECTRE III method to address the issue of fixing discrimination thresholds. We propose a ‘confidence interval-based’ approach, where uncertainty in the input data is defined using confidence intervals and thresholds are expressed as a function of the interval estimates. Our objectives are to: (i) introduce a new approach for threshold determination, which provides a means of reducing the degree of subjectivity; and (ii) test the proposed approach by applying it to a priority ranking of water supply and sewerage services in Spain.

The ELECTRE III method is based upon developing a preference relation, called the ‘outranking relation’, among alternatives evaluated on several criteria. The outranking relation is defined as a binary relation, S, between two alternatives, a₁ and a₂, such that a₁Sa₂ if there are enough arguments to declare that alternative a₁ is ‘at least as good as’ alternative a₂ (Bouyssou 1996). To build the outranking relation, a series of pairwise comparisons of the alternatives is done using the concordance–discordance principle. It represents, in a sense, the reasons for and against an outranking situation (Roy 1996): a₁ outranks a₂ if a majority of criteria support this assertion (concordance condition) and if the opposition of the other criteria is not ‘too strong’ (non-discordance condition). The method, in the second phase of outranking relation exploitation, derives two pre-orders: downward, Z₁, and upward, Z₂. Both pre-orders Z₁ and Z₂ are constructed through descending and ascending distillation procedures, respectively (for details of these procedures, see Roy 1996). A final pre-order of alternatives is suggested at the intersection of Z₁ and Z₂. Figure 1 illustrates a summary of the method.

The construction of the concordance and discordance indexes requires the definition of three discrimination thresholds for each criterion:

• The indifference threshold, q_i, beneath which the decision-maker is indifferent to two alternatives.

• The preference threshold, p_i, above which the decision-maker shows a clear preference for one alternative over the other.

• The veto threshold, v_i, above which the decision-maker negates any possible outranking relationship indicated by the other criteria.

Choosing realistic values for each threshold involves a high degree of subjectivity. In order to facilitate this task for decision-makers, we propose an approach that allows for less subjective input through defining thresholds as a function of the confidence intervals of the alternatives' performances. Hence, we address two concerns that may affect the validity of the rankings: (i) the subjectivity in choosing threshold values and (ii) the imprecision in performance values due to measurement error. The idea behind the approach is explained in Figure 2.

We selected a real case study to test the proposed approach. It consisted in a priority ranking of water supply and sewerage services in Spain. The objective was to prioritize the different regions of Spain according to their need for better water supply and sewerage services. This prioritization could be used to support current or future political actions regarding water management in Spain.

The alternatives in the decision problem were the 17 Autonomous Communities of Spain (Andalucía, Aragón, Asturias, Baleares, Canarias, Cantabria, Castilla y León, Castilla-La Mancha, Catalunya, Comunitat Valenciana, Extremadura, Galicia, Madrid, Murcia, Navarra, País Vasco and Rioja). The 11 criteria used to rank the regions consisted of water supply, wastewater, economic and structural factors. A description of each criterion is contained in Table 1.

Data on the regions were obtained from the ‘Survey on Water Supply and Sewerage’ done by the Spanish National Institute of Statistics. The survey is framed within the National Statistic Plan 2013–2016 (INE 2014), and aims to provide access to reliable and regular data regarding water management in Spain. The survey consists in a questionnaire on the collection, purchase, sale, supply and distribution of water, as well as collection and treatment of wastewater, by companies or institutions in the same Autonomous Community. The sample for the survey is extracted based on geographical coverage: it covers all municipalities with a population of more than 15,000 inhabitants, which is nearly two-thirds of the Spanish population. The sampling error is estimated to be 5%.

The data for year 2014 are shown in Table 2. These data constituted the performance values for ELECTRE III (note: we considered that all criteria had the same importance, and thus the same weight coefficients). The application of the mathematical model was undertaken with the use of R software (v3.3.1).

The discrimination thresholds are introduced to allow the correct interpretation of the differences between the alternatives' performances. One way for giving numerical values to such thresholds would be by coming back to their definition and analyzing the main sources of imprecision and uncertainty (Roy 1991). Thus, in this context of water supply and sewerage services, we can value the thresholds as follows:

• C1: volume of drinking water available per inhabitant and day. The variation in volume between regions was 208 l/inhab/d. In light of this variation, a difference of 100 l/inhab/d was not considered convincing evidence, while a difference of 200 l/inhab/d or more was taken to imply strict preference.

• C2: volume of drinking water supplied to the network per inhabitant and day. The variation in volume was 130 l/inhab/d. We assumed that indifference remained up to 50 l/inhab/d and strict preference started from 100 l/inhab/d.

• C3: percentage of water losses. In Spain, the mean values for losses were 16.5%. We thus considered that a difference of 15% was not an indication for preference, while a difference of 25% showed strict preference.

• C4: volume of treated wastewater per inhabitant per day. The variation in volume was 0.358 m³/inhab/d, so we selected 0.15 and 0.25 m³/inhab/d as an indication for indifference and strict preference, respectively.

• C5: volume of wastewater reused per inhabitant per day. The variation in volume was 0.138 m³/inhab/d. We assumed that differences below 0.05 m³/inhab/d were not evidence for preference, while differences above 0.15 m³/inhab/d showed strict preference.

• C6: unit cost of water supply. The mean value for the cost of water was 1.005 EUR/m³. We considered that indifference remained under 1 EUR/m³ and strict preference began from 2 EUR/m³.

• C7: unit cost of sewage. In this case, the mean value for the cost was 0.725 EUR/m³, so we fixed the indifference and preference levels as 0.75 and 1.5 EUR/m³, respectively.

• C8: length of the water network per inhabitant. The length of the water network ranged from 2.8 km/inhab in Madrid to 8.1 km/inhab in Asturias. A difference of 2.5 km/inhab was not seen as convincing evidence, while a difference of 5 km/inhab was seen to imply strict preference.

• C9: length of the sewerage network per inhabitant. The length of the water network ranged from 1.9 km/inhab in Catalunya to 4.9 km/inhab in Asturias and Galicia. We considered that differences in length below 1.5 km/inhab were not significant, but differences above 3 km/inhab were a sign of strict preference.

• C10: volume of water leaked per kilometre and year. The variation in volume was 3,875 m³/km/y, so we chose 2,000 and 3,500 m³/km/y as levels of indifference and strict preference, respectively.

• C11: number of storm tanks. The number of storm tanks ranged from 0 in various regions (Asturias, Baleares and Rioja) to 63 in Madrid. We decided that differences in the number below 20 were not indicative of preference, while differences above 40 were a sign of strict preference.

The veto values for all 11 criteria were determined in reference to the value of the preference threshold. As Roy et al. (1986) pointed out, unless there are good reasons for adopting another choice, the ratio v/p can be fixed as a constant for each criterion. We selected a ratio of 2, as shown in Table 3.

As seen, fixing the thresholds involved a significant subjective input by us. Although we did not pick threshold values in an arbitrary manner but by examining the data, a certain amount of arbitrariness was inevitable. Roy et al. (1986) emphasized the need for a sensitivity analysis, using extreme values of q–p–v, to verify that this subjective input did not significantly affect the final ranking of alternatives.

In the approach we propose, we attempt to reduce the degree of subjectivity when choosing the thresholds by expressing them in terms of the confidence intervals of the performance values (see Equations (1)–(3)). This approach can be interesting in some cases, when working with statistical data. Let us remember that the indifference threshold describes the largest difference between the performance values so that the decision-maker is indifferent between two alternatives, while the preference threshold describes the largest difference that makes him prefer one over the other. Consequently, it is reasonable to say that two alternatives could be considered indifferent if their confidence intervals overlap; otherwise, one would be preferred over the other. The veto threshold, on the other hand, is not associated with the sources of imprecision and uncertainty, but with a base principle of the outranking relation: the discordance concept. However, as explained by Roy et al. (1986), the size of the veto threshold is generally fixed in terms of the preference thresholds (i.e. v/p ratio). That is why we computed the veto thresholds as twice the preference values.

We would like to emphasize that this approach is not designed to ‘estimate’ the value of the discrimination thresholds. These thresholds are not experimental values to be estimated, but rather values used to model the decision-maker's preferences. Our confidence interval approach only aims to assist decision-makers in selecting numerical values for thresholds in specific cases, but should never replace the preferences of actors in the decision process.

After the determination of the discrimination thresholds (either with our subjective input or using the confidence interval approach), the mathematical model for the ranking is resolved. Two complete pre-orders are first constructed, through descending and ascending distillation procedures. The descending distillation ranks the alternatives from the best available to the worst, while the ascending does it in the reverse manner. Figure 3 presents both pre-orders in graphs where the axes are the positions of the Autonomous Communities.

Distillations with the first set of thresholds (those fixed with our subjective input) show Catalunya (A9) as the region most in need of better water supply and sanitation services, followed by Aragón, Extremadura, Madrid and Navarra (A2, A11, A13 and A15). This can be interpreted as a result of the bad performances of Catalunya in the majority of evaluation criteria (C1, C2, C3, C4, C6, C7 and C9). The outcome from distillations with the confidence interval approach is, however, different. Whereas Aragón, Extremadura and Navarra remained at the bottom of the ranking, Catalunya and Madrid occupied a higher rank. This is a consequence of the uncertainty in the performance values. As seen in Figure 4, although Catalunya (A9) occupied the bottom ranks in almost all criteria, the confidence intervals for its performance values overlapped with other regions. Our approach considers two alternatives to be indifferent if their confidence intervals overlap. That is why it resulted in a different ranking of regions.

It is important to draw attention to the fact that both rankings are equally relevant and valid. It would be wrong to say that one ranking is good or bad only by referring to a mathematical model. As Roy (2005) stated when explaining the purpose of MCDA, these models should not be viewed as being conceived to discover a pre-existing truth. It is not possible to know which is the ‘right’ ranking and which is not, because the ‘right’ ranking does not exist. Decision-aiding based on MCDA models is only meant to guide the decision-making process.

In the same way, discrimination thresholds are not ‘real values’ that exist somewhere. They are merely numbers designed to reflect a system of preferences. Consequently, there should always be room for a substantial degree of subjectivity/flexibility in their determination (Roy et al. 1986). Our confidence interval approach may be interesting in some cases (e.g. when dealing with statistical data), but only to guide the decision-maker in this inevitably arbitrary process. Robustness analyses will still be needed to assess the extent of the influence of this arbitrariness on the final results, as well as to better define the choice of numerical values in view of this effect.

ELECTRE outranking methods are one of the best known and widely applied in the context of decision aid. The output of ELECTRE depends critically on the input information, hence the data input should ideally be precise. Yet, in reality, the available data are often uncertain. Discrimination thresholds (indifference, preference and veto) were incorporated in ELECTRE methods to take into account the imperfect knowledge of the data. Fixing these thresholds for each criterion can be, however, a difficult and ambiguous task for analysts and decision-makers, as it involves a substantial element of subjectivity.

We propose an approach that allows for less subjective input in the determination of thresholds. This is achieved by characterizing the uncertainty in the performance values by defining the confidence intervals of the available data, and expressing the discrimination thresholds as a function of these interval estimates. Ranking of alternatives is therefore provided to the decision-maker without his subjective input. The illustration of the proposed approach using the water and sewerage case study demonstrates how uncertainty in the data can be used to define the discrimination thresholds. It also highlights the significant difference in rankings when thresholds were set with and without our subjective input.

However, the confidence interval approach should not be viewed as ‘better’ than basing the thresholds on our judgments. Thresholds are not experimental values that need to be estimated, but rather values that we use to model our, or the decision-maker's, preferences. The only aim of the proposed approach is to guide him in some cases, with specific data: statistical data.