This paper explores use of the Delphi technique on regulatory impact assessment (RIA) in order to select criteria as well as to analyze the non-neutrality of stakeholders in the Portuguese case study. Although the decision-making process has been supporting a different prescriptive approach, there is no neutral decision, which can reflect on the (in)efficiency of the government's action. To cope with imperfect knowledge, we have built the link between the objectives from Law no. 194/2009, which determines the regulatory framework in the Portuguese water sector and its potential criteria. Moreover, the elicitation weights for each criterion previously selected were framed in an innovative way, under a different perspective, either customers, municipalities or concessionaires. Evidence advises that there are relevant myopic, omission, splitting, and insensitivity biases for decision analysis, because of the distortion of input. Thus, the Delphi technique enables the decision makers to obtain reliable information before taking a decision. The results in terms of a different perspective for each criterion enable us not only to identify the non-neutrality of decision analysis, but also to (re)think the stakeholder's participation into the context of the Law referred to. Finally, this approach could consolidate our understanding concerning the potentialities of the Delphi technique in RIA, especially in policies with several objectives.

Introduction

The ‘holy grail’ of research is establishing methodological rigor in order to obtain more reliable and robust results. This refers to a researcher, analyst and decision-makers’ responsibility to ensure that procedures have been adhered to and confounding factors eliminated to produce dependable outcomes (Hasson & Keeney, 2011).

Traditionally in quantitative research the decision process is based on stakeholder's participation (SP), especially in the water agenda, where specific expertise is inevitable for decision making (Van Ast & Gerrits, 2016) and the practical application of SP remains problematic (Mostert, 2003). Although experts and decision makers need to provide judgment in decision modeling as contributions to estimate the possible impact of decision alternatives, one has to be concerned with the self-interest of experts who may have a stake in the outcome of the analysis (Montibeller & von Winterfeldt, 2015).

Particularly crucial for assessing the possible impacts of ongoing or new regulations, regulatory impact assessment (RIA) has become an important rational policy tool. Specifically, for the water sector, where interventions by governments have taken place without a clear idea of why or how or when (Jacobs, 2016), RIA might be required in order to improve the efficiency, transparency and accountability of regulatory decisions encouraging good governance and contributing to better business enabling environments (Kirkpatrick, 2016).

Although several RIA definitions have been found in the literature, hereafter RIA could be understood as a policy tool composed of four steps: (i) status quo, (ii) consultation, (iii) assessment and (iv) review. In this case study, the exercise of RIA is still limited to the status quo and consultation. Both are vital for the decision process influencing the inputs’ context into an assessment step. Here, different types of methods could be used, e.g., decision conference, nominal group technique (NGT), public consultation, Delphi technique and so on.

Since the 1950s, the usage of the Delphi survey method has undergone different stages of development, i.e., (i) secrecy/obscurity, (ii) novelty, (iii) popularity, (iv) scrutiny and (v) continuity. After a time of stagnation in the 1980s, the Delphi technique received increasing interest in the early 1990s (von der Gracht, 2012). More recent applications concentrate on the web-based implementation of the Delphi procedure, but still follow the technique's fundamental rationale and consider consensus measurement to be a crucial component of analysis (Gnatzy et al., 2011). To measure the consensus, four key features may be regarded as necessary for defining a procedure as a ‘Delphi’, i.e., (i) anonymity, (ii) iteration, (iii) controlled feedback and (iv) the statistical aggregation of group response (Rowe & Wright, 1999).

Moreover, comparative advantages of the Delphi technique have facilitated its applicability because it enables a group of experts to be canvassed rapidly and inexpensively (eliminating printing and postage costs) without geographical limitations (Reppa, 2007), although the feedback time could be a disadvantage.

Despite the relevance of the topic of biases for decision analysis and its effects on the impartiality of government actions, there is a lack of articles discussing the trade-off between SP in a problem with several and controversial objectives. This situation has become more problematic in the last two decades, where water utilities in many countries have undergone radical evolution which has transformed the basis of operation (Massarutto et al., 2013) and also the regulatory system with no measured impacts on the society.

Bearing this in mind, this paper investigates the case study of Portugal through Law no. 194/2009 which had several aims, one of them, maybe the main one, to improve the public-private partnership (PPP) model for the water sector. However, the expected outcome of this law has not provided the envisioned value for money.

Unfortunately, an ex-post analysis of policies with several objectives, as in our case, is frequently neglected in the evaluation of the decision-making process or done in a strict way, i.e., without a clear definition of the criteria, methods and robustness, and sensitivity analysis. Notwithstanding this, we develop a ‘modified and argument’ Delphi design (Hasson & Keeney, 2011) as a consult or participatory step in the RIA approach.

This proposed consultation step (modified and argument) Delphi design was used in two stages. Initially, we built the link between the Law no. 194/2009 objectives and their possible impact in order to select the more adequate criteria and reduce biases, e.g., myopic problem representation and omission bias (Montibeller & von Winterfeldt, 2015). Secondly, argument-Delphi was used to capture the weighting coefficients (swing-weight), typically used with multi-attribute utility theory (MAUT) methods, in order to reduce the splitting, equalizing and insensitivity bias (Montibeller & von Winterfeldt, 2015).

The present paper contributes to the literature by proposing a consistent evaluation methodology and the practical implications for managing the implementation of RIA in primary or subordinate laws, for which the case study of the Portuguese water sector (PWS) offers an excellent example. Furthermore, this paper not only intends to propose how Law no. 194/2009 might be improved, but also how this frame can support governments to develop new and/or ongoing regulations beginning through the status quo. This proposal is innovative concerning the use of Delphi in the water supply sector and also in terms of reducing bias in decision analysis.

The remainder of this paper is organized as follows: the second section presents the importance of RIA in the regulatory system. Section three outlines the Delphi theory. The Portugal case study is detailed in the fourth section. Section five discusses the results. Concluding remarks are presented in section six.

RIA in brief

According to Alemanno (2016), RIA carries the possibility to render the policy process not only transparent and participatory, but also more rational, through the increased availability of evidence support policy initiatives enhancing the scrutiny of the preparatory process, leading to the adoption of the final rule.

Although RIA has become an important tool over the past few years, with evidence based on decisions made by policy-makers worldwide, the academic literature on RIA, as regards its processes and application, is still concentrated on central government's agenda and on social sciences. Nowadays, debates around the world about RIA have intensified, not only on state capacity and governance, but also concerning its better understanding and applicability at different regulatory levels and in different sectors.

Here, RIA can be understood (by the authors) as a policy tool systematically and consistently examining the selected potential impact arising from government action, under ex-ante or ex-post time perspective. Also, the following RIA framework steps can be considered to evaluate laws or regulatory acts: (i) status quo, (ii) assessment, (iii) consultation, and (iv) review or final step.

The first step, named ‘status quo’, is composed of: (i) the problem and context, (ii) purpose, (iii) scenarios, and (iv) potential impact (criteria). The ‘assessment’ comprises the analysis of scenarios, the assumption of the level and analysis of the options in each impact (criteria) as regards profits and losses. The ‘consultation’ step is the appreciation of the need for transparency and openness in the process of SP. The ‘final’ step consists of periodically reviewing the regulations to evaluate the extent to which they are achieving the objectives/intended benefits.

Currently, the hybridization of RIA practices results also from the diffusion of several methods of policy appraisal (De Francesco, 2016) and their uses depending on the depth of the RIA and the complexity and the dimension of the problem.

Here, concerning the RIA framework, we focused on the ‘consultation step’, which involves the stakeholders to get more reliable information as the input of our analysis. As for the RIA method, we focused on Delphi, which can enable identification of criteria and similarly aid the analytical methods (mono, multicriteria and risk methods) in a different way.

Finally, implementing the Delphi technique as a useful method in the RIA (ex-post) approach in the PWS (Law no. 194/2009) may enable the construction of more robust and transparent links between this law's objectives and also may enable clear understanding concerning the weighting for each criterion in a more rational way. This proposal is innovative regarding the uses of a different perspective in eliciting weighting coefficients, either by customers, municipalities or concessionaires’ representatives.

Delphi technique

The Delphi technique was developed during the 1950s by workers at the RAND Corporation (a research institution that initially focused on national security issues and later concentrated on scientific, educational, and charitable endeavors for public welfare) while involved in a US Air Force sponsored project (von der Gracht, 2012).

Delphi is understood as a social research technique for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem (Rowe & Wright, 1999). Nowadays, the Delphi technique is also used widely in the fields of public policy and strategic decision making in companies (Makkonen et al., 2016). Also, four key features may be regarded as necessary for defining a procedure as a ‘Delphi’, respectively: anonymity, iteration, controlled feedback and the statistical aggregation of group response (Rowe & Wright, 1999).

Furthermore, the comparative advantages of the Delphi technique as a consultation tool, especially in this case, are related to four aspects: (i) it allows a number of experts to be called upon to provide a broad range of views, (ii) it does not require face-to-face meetings, (iii) it helps keep attention directly on the issue, and (iv) it is inexpensive. On the other hand, Delphi requires adequate time and participant commitment and is also more time-consuming than group process methods, e.g., NGT, conference decision, public consultation and so on. Although there is no register of Delphi in RIA in the water services (WS) agenda, its use is prominent rule on this matter, because it can help decision makers to achieve consensus and also reduce cognitive and motivational bias.

Concerning Delphi designs, Hasson & Keeney (2011) identified ten main categories of Delphi, which are classical, modified, decision, policy, real time, e-Delphi, technological, online, argument and disaggregative policy. Table 1 highlights the main types of Delphi design in policy problems.

Table 1.

Delphi characteristic (Hasson & Keeney 2011, modified).

Design type Objective Target panelist Mechanism Rounds (no.) Round 1 design 
Classical To elicit opinion and obtain consensus Experts selected based on aims of investigation Traditionally postal May employ fewer than 3 rounds Open qualitative first round, to allow panelists to record responses 
Modified To be different according to project design in order to achieve consensus Experts selected based on aims of investigation Varies, postal, online and so on May employ less than 3 rounds Panelists provided with pre-selected items, drawn from several sources, within which they are asked to consider their responses 
Decision To build decision-making and create the future in reality rather than forecasting it Decision makers, selected according to hierarchical position and level of expertise Varies Varies Can assume similar process to classical Delphi 
Policy To produce opposing interpretations on policy and potential resolutions Policy makers selected to obtain divergent opinions Can adopt a number of formats including bringing members together in a group meeting Varies Can assume similar process to classical Delphi 
Argument To develop pertinent arguments and expose underlying reasons for different opinions on a specific single issue Panelists should represent the research issue from different perspectives Varies Varies Can adopt similar process to modified Delphi 
Disaggregative policy To conceptualize future scenarios in which panelists are asked about their probable and the preferable future Panelist selection can vary depending on the aim of the research Varies Varies Adoption of modified format using cluster analysis 
Design type Objective Target panelist Mechanism Rounds (no.) Round 1 design 
Classical To elicit opinion and obtain consensus Experts selected based on aims of investigation Traditionally postal May employ fewer than 3 rounds Open qualitative first round, to allow panelists to record responses 
Modified To be different according to project design in order to achieve consensus Experts selected based on aims of investigation Varies, postal, online and so on May employ less than 3 rounds Panelists provided with pre-selected items, drawn from several sources, within which they are asked to consider their responses 
Decision To build decision-making and create the future in reality rather than forecasting it Decision makers, selected according to hierarchical position and level of expertise Varies Varies Can assume similar process to classical Delphi 
Policy To produce opposing interpretations on policy and potential resolutions Policy makers selected to obtain divergent opinions Can adopt a number of formats including bringing members together in a group meeting Varies Can assume similar process to classical Delphi 
Argument To develop pertinent arguments and expose underlying reasons for different opinions on a specific single issue Panelists should represent the research issue from different perspectives Varies Varies Can adopt similar process to modified Delphi 
Disaggregative policy To conceptualize future scenarios in which panelists are asked about their probable and the preferable future Panelist selection can vary depending on the aim of the research Varies Varies Adoption of modified format using cluster analysis 

Within each Delphi type, the features of the Delphi may also vary in terms of the number of rounds, the level of anonymity and feedback given, as well as the insertion criteria, sampling approach or method of scrutiny (Hasson & Keeney, 2011).

Traditionally, the Delphi application follows nine steps, i.e., (i) prepare the questionnaire and select the experts (hereafter, panelists), (ii) test the questionnaire, (iii) 1st round, (iv) analysis of the feedback, (v) check the convergence, (vi) prepare a new questionnaire, (vii) 2nd round, (viii) analysis of the new feedback and (ix) publish final descriptions (3rd round).

A number of authors claim that the Delphi approach enhances reliability and is a validity instrument, i.e., personal bias can also influence the accuracy of a policy decision aid application. Thus, to reduce the splitting bias, the analyst or decision maker ought to obtain objectives and attributes from multiple stakeholders, which offer different degrees of detail to different parts of the problem analyzed (Montibeller & von Winterfeldt, 2015).

In terms of stability, three rounds are optimal in Delphi. With two rounds, homogeneity cannot be established (Boulkedid et al., 2011). Also, many Delphi studies have used discretionary criteria or descriptive statistics for the determination of consensus and the quantification of its degree. The criteria have, however, sometimes been chosen rather arbitrarily. The literature review revealed that researchers have actually used all kinds of descriptive statistics in order to measure consensus. One can find applications of measures of association as well as measures of central tendency and dispersion (von der Gracht, 2012).

Finally, the decision makers, very often find that they need to apply models or techniques which are required to be fed with data that cannot be obtained directly from present or past reality, data which do not exist, or data which are not reliable or that are insufficient (Landeta et al., 2008). In line with these authors and keeping in mind that joining individual judgments may lead to ‘process gain’, where groups may perform better than their best participants with some adaptation that can reduce bias, we justify the Delphi applicability in the RIA approach into the Portuguese case study.

Portugal case study

Big numbers, water management model and legal framework

In mainland Portugal, 15 operators in the bulk service and 267 operators in retail services are responsible for WS to approximately 9.6 million inhabitants and 95% of households served.

The institutional model assumed in Portugal is based on the French regulatory and development model, i.e., the responsibility for WS belongs to the municipalities, which can delegate their operation to the private sector by public tender (Marques, 2008). Nevertheless, nowadays the scope of the regulation is broader (both in the number of regulated companies and in the intervention of regulation); initially its aim was only to supervise the quality of service of the private concessionaires and to provide some technical assistance to the contracting stage and when renegotiation of the contracts took place (Marques & Berg, 2010). The common model of bulk services are public concessions (73%) and are responsible for serving 71% of the population in Portugal.

Since 2009, and effective as of 2011, the Water and Waste Services Regulation Authority (ERSAR) has been regulating all delivery models; however, this development needs an increased maturity and expertise. Indeed, ‘an improved compliance and standardization in reporting is needed, mainly by in-house models, related to the information requested by ERSAR, in order to enable prompt and efficient analyses, enhancing the regulatory procedure’ (Pinto et al., 2015).

Usually the private investors involved in PPP arrangements at the local level consist of construction companies or specialized sub-holdings owned by them (Da Cruz & Marques, 2012). Private sector participation is involved in almost 25% of the retail market (ERSAR, 2015). It only had a slight increase in the last decade, but even so it can represent an opportunity to study this specific participation model in Portugal's water sector. According to ERSAR (2015), about 18% of the population are served by municipal companies, 30% by direct provision, 23% by direct provision but with some autonomy, and approximately 20% by concessions (Figure 1).
Fig. 1.

Geographical distribution of WS providers (retail) in Portugal by management models (Source:ERSAR, 2015).

Fig. 1.

Geographical distribution of WS providers (retail) in Portugal by management models (Source:ERSAR, 2015).

In terms of the legal framework in the PWS, Law no. 194/2009 regulated these possible arrangements, ranging from fully private to fully public, the current legal and regulatory framework of the water sector in Portugal.

Nowadays, there are no clear issues concerning the impact of this law, whether positive or negative, regarding the performance of private sector participation in the PWS. This case provides a chance to build a link between the objectives of this law and selected criteria and also to evaluate the non-neutrality of SP in the RIA approach which can result in greater public acceptance of decisions and a constructive dialogue (Mostert, 2003).

Methodology

In order to select criteria (1st stage) and check the non-neutrality of SP in the PWS (2nd stage), the authors’ RIA approach (Figure 2) was carried out. Hence, our RIA framework considers only the ‘consultation step’.
Fig. 2.

RIA framework and consultation step.

Fig. 2.

RIA framework and consultation step.

Vital objectives must follow a set of properties: (i) value relevance – essential objectives involved in decision, (ii) understandability – clear mining, (iii) operationality – in terms of performance of strategic options, (iv) non-redundancy – should not measure a similar concern twice, (v) preferential independence – in terms of performance of strategic options on one objective disregarding their performance. For each essential objective placed at the bottom level, an associate criterion should be specified. Also, the criteria might have direct quantitative and indirect quantitative attributes, and qualitative attributes (Belton & Stewart, 2002). Independently, each criterion should have five properties (Keeney & Gregory, 2005) to be employed in a preference model, each criterion should be: (i) unambiguous, (ii) comprehensive, (iii) direct, (iv) operational and (v) understandable.

A total of 24 criteria were pre-selected based on data available and their capacity to measure the objectives from Law no. 194/2009. These criteria and also their descriptors and indicators are available in Appendix 1 (available with the online version of this paper). Thus, in order to implement the RIA framework proposal ‘status quo versus consultation’, considering the PWS context, Figure 3 displays the potential relationship between the objectives of Law no. 194/2009 and its potential representative criteria.
Fig. 3.

Objectives and pre-selected criteria by Law no. 194/2009.

Fig. 3.

Objectives and pre-selected criteria by Law no. 194/2009.

To build this first matrix (options versus impacts) the dimensions suggested by ERSAR and the data available (ERSAR’ reports, concessionaires websites and concessions contracts) were considered.

To build a reliable link between the objectives and pre-selected criteria (1st stage) as well as to check the SP in different perspectives, either customers, municipalities or concessionaires (2nd stage), we follow nine steps: (i) prepare the questionnaire and select the panelists, (ii) test the questionnaire, (iii) 1st round interaction, (iv) analysis of the feedback, (v) check the convergence (or new questions), (vi) prepare a new questionnaire, (vii) 2nd round interaction, (viii) analysis of the new feedback and (ix) 3rd round interaction (publish final descriptions). Finally, in each stage a different design of Delphi was used.

First stage: modified-Delphi design

The first stage analyzed the relation between the objectives of Law no. 194/2009 and the impact through the criteria and their descriptors pre-selected by the data available. The questionnaire was developed by Google docs®. A total of 40 questions were allocated in four blocks: (i) introductory letter, (ii) data from the panelist, (iii) technique questions and (iv) additional comments. The questionnaire pre-test was carried out by members of the University of Lisbon, University of Brasília, concessionaires and water sector associations in Portugal.

The panelists were invited to categorize the pre-selected 24 criteria and objectives of Law no. 194/2009 based on five levels: (i) indispensable, (ii) very important, (iii) important, (iv) low important and (v) unnecessary (Mendonça, 2009). The panelists were selected by their representativeness in the PWS, e.g., concessionaires, municipalities, regulatory authority, academics, consulting and consumers’ associations. The analysis of the feedback was carried out by statistical assessment. More details regarding the first stage of the Delphi technique are provided in Appendix 2 (available with the online version of this paper).

The number of rounds is one simple way of conducting consensus measurement and in many situations it is sufficient. The interquartile range (IQR) is the measure of dispersion for the median and consists of the middle fifth of the observations. In fact, it is a frequently used measure in Delphi studies, and it is generally accepted as an objective and rigorous way of determining consensus (von der Gracht, 2012). Finally, the impacts were selected based on statistical parameters (mode, median and average).

Second stage: argument-Delphi design

In multicriteria decision analysis (MCDA), classically one of three weighting methods are used, the first two being direct rating and pairwise comparison (used in the analytical hierarchy process, in outranking methods such as PROMETHEE and also in swing-weights). However, swing-weights has been used as a classical support with MAUT methods (Belton & Stewart, 2002). With this in mind, we rejected the use of these two objective methods because of their limited focus on variance from target and costs, respectively, and their complexity given the limited scope of this work. The third major approach, the weighting coefficient from experts or stakeholders (panelist), includes a wide variety of methods, such as public opinion surveys, facilitated group consensus-based procedures (e.g., the Delphi technique) and various weighting procedures used in MCDA modeling methods.

However, there are many common mistakes in defining weights and several appropriate methods to elicit weight protocols (see Montibeller & von Winterfeldt, 2015). One innovative method that enables reduction of the splitting bias is swing-weights through the argument-Delphi design.

In line with Montibeller & von Winterfeldt (2015), we considered in our paper the swing-weight approach, which refers to the variation of the weights as the value measure swings from the lowest value to highest value level in the scale (by the value function for each criterion). The swing steps were: (i) the decision maker was asked which ‘swing’ from the worst to the best outcome would result in the largest, second largest, etc., improvement; (ii) the criterion with the most preferred swing is most important, and given 100 points; (iii) the magnitudes of all other swings are expressed as percentages of the largest swing; and (iv) again, the derived percentages are the raw weights that are normalized to yield final weights (Zardari et al., 2015). Appendix 3 (available with the online version of this paper) provided a detailed description of the value function for each criterion selected from the first stage.

The panelists were selected by their representativeness on PWS. The weight coefficients for each perspective, concerning the customers, the owner (municipality) and the water utilities (concessionaires) were obtained by the panelists and the normalization was calculated using Equation (1): 
formula
1
where m corresponds to the number of panelists and wi is the median of each criterion. More details regarding the swing-weighting by the argument-Delphi technique are provided in Appendix 4 (available with the online version of this paper).

Results

Whole Delphi technique

In the test phase of the questionnaire, three panelists were invited to participate to evaluate and improve the form and content of the draft version. The feedback was 100% and a little change was made in terms of form. The Delphi technique (modified and argument design, i.e., 1st and 2nd stage) took 120 days although the total of 100 days was previously defined by the authors.

In the first stage (83 days), the period of 40 days initially granted for the ‘1st round’ was extended by 10 days to obtain a greater adhesion of the panelists. However, the 2nd round was ‘compensatory’ concerning the time required. In the second stage 37 days were sufficient to complete the process.

Modified-Delphi

The 1st round feedback was from 42% of the panelists, i.e., from the total of 48 questionnaires sent, 20 were answered. In the 2nd round, the feedback was 70%, i.e., from a total of 20, there were 14 feedbacks. The total procedure participation was 29%. Wright & Giovinazzo (2000) and Gordon (1994) suggest levels (for non-feedback rate) of about 30–60% (1st round) and 20–30% (2nd round). According to those authors’ parameters, the feedback was sufficient for our proposal. Nevertheless, some particularities should be pointed out: (i) the method was carried out via electronic contact, (ii) the difficulty of the matter, and (iii) the time required to answer the questionnaire was more than 10 minutes, which could contribute to the amount of ‘no answer’ responses. Figure 4(a) shows the participants’ representativeness.
Fig. 4.

(a) Preselected panelist, (b) 1st round and (c) 2nd round.

Fig. 4.

(a) Preselected panelist, (b) 1st round and (c) 2nd round.

In terms of representatives, the main stakeholders identified in the whole modified-Delphi, involving about 80% of the total, were: (i) concessionaires, (ii) municipalities, and (iii) regulatory authorities (Figure 4(b) and 4(c)). The academic, consultation and water sector association maintained their participation during the process. In this way, we conclude that the stakeholders who answered the questionnaire are representative of the PWS.

In terms of convergence, the ‘number of rounds’ and IQR were considered. Three iterations are typically sufficient to identify points of consensus and also the IQR ≤ 1.5. Figure 5 shows the IQR (1st and 2nd interaction).
Fig. 5.

IQR by the 1st and 2nd feedback. Please refer to the online version of this paper to see this figure in colour: http://dx.doi.org/10.2166/wp.2017.131.

Fig. 5.

IQR by the 1st and 2nd feedback. Please refer to the online version of this paper to see this figure in colour: http://dx.doi.org/10.2166/wp.2017.131.

In the 1st round only three criteria were superior to the pre-stablished limit (1.5) (red arrows) which forced us to invest in the 2nd round in order to achieve the pre-established goal. However, five others (blue arrows) decreased in terms of IQR and the remainder kept at the same level as before. As a result, the measure of consensus was acceptable and reliable, which let us follow on to the next step in the direction of the most relevant criteria through statistical analysis. Based on the parameters presented in the modified Delphi subsection, 13 criteria (black arrows, Figure 6) were selected from the total of 24. Figure 6 shows the average, mode and median as a final result of the modified Delphi. More information concerning all criteria are available in Appendix 1 and selected criteria in Appendix 5 (available with the online version of this paper).
Fig. 6.

Descriptive statistics of the criteria.

Fig. 6.

Descriptive statistics of the criteria.

Argument-Delphi

The argument-Delphi was developed to allow evaluation of the non-neutrality of SP in the PWS. Thus, the elicitation weights coefficient Wi, i = 1 to 13 associated with each Ci-th criterion by swing-weights might be a convenient method for weighting. Individual utility functions are assessed using the range of attribute values for the alternatives being considered.

Feedback in the 1st round was received from 53% of the panelists, i.e., from the total of 15 questionnaires sent, eight were answered. In the 2nd round, the feedback was 88%, i.e., seven answered. The feedback in the total process was 40%. The representatives identified in the argument-Delphi (second part) were: (i) municipalities, (ii) customers, and (iii) concessionaires. Based on Wright & Giovinazzo (2000) and Gordon (1994) parameters, we considered that the feedback was sufficient for our proposal. The ‘number of rounds’ were achieved in our analysis.

Figure 7 provides the median in terms of customers’, municipalities’ and concessionaires’ perspectives (1st and 2nd round). The median was calculated by Equation (1).
Fig. 7.

Outputs (1st and 2nd round) from argument-Delphi in different perspectives: (a) customers, (b) municipalities and (c) concessionaires.

Fig. 7.

Outputs (1st and 2nd round) from argument-Delphi in different perspectives: (a) customers, (b) municipalities and (c) concessionaires.

First, the results show an interesting aspect of changing perspective. The ‘concessionaires’ perspective’ was more reluctant in terms of modifying its answer between the rounds. In fact, it can be associated with the following hypothesis: (i) time to (re)valuate, and (ii) more conservative assumptions. The customers and municipalities’ perspectives were more flexible in the round interaction. In a specific way, four criteria were not modified, these were financial autonomy, mains failure, real water losses, and sludge disposal from the customers’ perspective, and affordability level, governance, non-revenue water, and risk sharing from the municipalities’ perspective. Second, the group of graphics shows ‘how the preference is heterogeneous and unbalanced’. From the customers’ perspective, there is a smoother variation between criteria. On the other hand, the concessionaires’ and municipalities’ perspectives highlighted ‘coverage of total costs, safe water, affordability level, non-revenue water and financial autonomy’ and ‘physical accessibility, safe water, fulfillment licensing and sludge disposal’, respectively.

Additionally, based on our data analysis, from the customers’ perspective, the following criteria were highlighted: coverage of total costs, physical accessibility and at the same level the affordability, safe water and coverage of total costs. From the municipalities’ perspective, the following criteria were highlighted: physical accessibility, safe water and licensing. Finally, from the customers’ perspective, the gap was stronger in coverage of total costs and safe water. In fact, this analysis shows in a clear way the trade-offs between stakeholders.

To summarize, this innovative approach allows us to understand the different perspectives of each stakeholder, as well as the balance of preferences. Here, the non-neutrality of elicitation weights in a multicriteria problem is demonstrated. In fact, these results show how Law no. 194/2009 might be improved or reviewed regarding the water agenda. In this sense, this consult method can reinforce the need to check the potential distributive effects of any regulation act from the government.

Concluding remarks

Firstly, this research identified the potential use of RIA as a policy tool in an innovative and unprecedented case in the water agenda, showing the clear relation between the steps – objectives from Law no. 194/2009, criteria and their descriptors, and also the preferences of stakeholders – and their contribution to the decision-making process.

Secondly, consult methods in this RIA approach become an exercise in sharing responsibility and social learning and can promote the governance of the water system. In this sense, by searching through and reviewing the literature, we have been able to confirm that the Delphi in the RIA approach could be used as a valid instrument for forecasting (Landeta, 2006) and supporting decision making providing, in this case study, crucial links between the objectives of this Law referred to and the perception of each representative studied.

Thirdly, the first part of Delphi (modified design) does not only allow us to build the link between the objectives of Law no. 194/2009 and 24 criteria pre-selected based on data available, but also to improve the reliability and the validity of our analysis and reduce some of the bias, e.g., myopic problem representation and omission bias. The second part of Delphi (argument design) highlighted how the perception changes, either under the customer, municipality or concessionaire perspectives. Here, while the customers’ perspective is well distributed between the four objectives, municipalities focused on customers and environment protection, and providers focused on economic sustainability. The stability of the modified-Delphi was achieved in both parts according to the literature parameters.

Fourthly, in general the methods that are envisioned to pick up, improve and process the subjective information most relevant for a complex problem are a valuable complement for the application of techniques and construction of models mainly stemming from objective data. The experiences presented here provided us with evidence that the Delphi method enhances the institutional participation and communication in the RIA approach. Moreover, this exercise presents not only the frame that can support policy decision, but also indispensable elements that need to be considered in Law no. 194/2009. This is a typical academic exercise with clear contributions to policymaking circles.

However, the aforementioned experiences have allowed us to learn certain lessons about effectively running Delphi exercises in the water agenda. Thus, some lessons deserve particular attention in both stages in order to support the RIA consultation step:

  • having institutional support facilitates, e.g., expert collaboration;

  • selecting a panelist who can make effective contribution to knowledge area and sector, with his/her degree of motivation to take part of the research;

  • putting oneself, and the test-team, in the place of the panelist, generally it is necessary to sacrifice questions and rounds in order to guarantee panel participation and continuity;

  • giving simple and adequate information in the attendance of experts;

  • being creative in order to encourage contribution and sending or requesting the feedback;

  • giving the same level information in multicriteria problems to the stakeholders in order to reduce bias (first stage) and also trying to convince them to answer according to their perspective when possible (second stage);

  • looking for alternatives that not only enable the reduction of bias from subjective judgements in support of decision policies, but also provide the analyst with more reliable information;

  • the study must not finish for the panelist when the last completed questionnaire has been sent; the expert must be aware when the study has finally finished that his or her contribution has been of some use.

Finally, in the following lines there are some thoughts in terms of consultation methods for RIA that can improve the regulatory system:

  • (a)

    This type of qualitative technique, properly applied, may contribute to improving the efficiency of the quantitative techniques, e.g., MCDA modeling methods in RIA, by allowing them access to a new type of information which is relevant for understanding and modeling the problem studied.

  • (b)

    With consultation/participatory methods on RIA relatively high levels of reliability and validity for a technique of these characteristics could be achieved.

  • (c)

    In PPP arrangements, which has been noteworthy worldwide, this approach enables the capture of the real perception of stakeholders in order to design or review laws or regulation in a more effective way.

  • (d)

    RIA consultation in the water agenda might enable more consistent decisions on the conditions that need to be met when accountability and more effectiveness actions of public policy are required by society.

Acknowledgements

The authors wish to thank the Ministry of Developing and Planning in Brazil and also the Regulatory authority (ERSAR) in Portugal. Furthermore, the mobility of this study has been possible with the Erasmus Mundus SMART2 support by project reference (No. 552042-EM-1-2014-FR-ERA MUNDUS-EMA2) coordinated by CENTRALESUPELEC. The authors would also like to thank the editor of Water Policy and the anonymous reviewers for their precious time and constructive comments on the paper.

References

References
Alemanno
A.
(
2016
).
Courts and regulatory impact assessment
. In:
The Handbook of Regulatory Impact Assessment
.
Edward Elgar Publishing
,
Cheltenham
,
UK
. pp.
127
141
.
Belton
V.
Stewart
T. J.
(
2002
).
Multiple Criteria Decision Analysis: An Integrated Approach
.
Kluwer Academic Publishers
,
Norwell, MA
, p.
400
.
Boulkedid
R.
Abdoul
H.
Loustau
M.
Sibony
O.
Alberti
C.
(
2011
).
Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review
.
PLoS One
6
,
1
9
.
doi:10.1371/journal.pone.0020476
.
Da Cruz
N. F.
Marques
R. C.
(
2012
).
Mixed companies and local governance: no man can serve two masters
.
Public Administration
90
,
737
758
.
doi:10.1111/j.1467–9299.2011.02020.x
.
De Francesco
F.
(
2016
).
Diffusion across OECD countries
. In:
The Handbook of Regulatory Impact Assessment
(
Dunlop
C. A.
Radaelli
C. M.
, eds).
Edward Elgar Publishing
,
Cheltenham
,
UK
, pp.
271
285
.
ERSAR
(
2015
).
Relatório Anual do Setor de Águas e Resíduos em Portugal (Annual report of water and waste water). Vol. 1. Lisbon
.
Gnatzy
T.
Warth
J.
von der Gracht
H.
Darkow
I. -L.
(
2011
).
Validating an innovative real-time Delphi approach – a methodological comparison between real-time and conventional Delphi studies
.
Technological Forecasting and Social Change
78
,
1681
1694
.
Gordon
T. J.
(
1994
).
The Delphi Method
.
United Nations University
,
USA
.
Hasson
F.
Keeney
S.
(
2011
).
Enhancing rigour in the Delphi technique research
.
Technological Forecasting and Social Change
78
,
1695
1704
.
doi:10.1016/j.techfore.2011.04.005
.
Jacobs
S.
(
2016
).
Towards a simpler and practical approach
. In:
The Handbook of Regulatory Impact Assessment
(
Dunlop
C. A.
Radaelli
C. M.
, eds).
Edward Elgar Publishing
,
Cheltenham
,
UK
. pp.
78
90
.
Keeney
R. L.
Gregory
R. S.
(
2005
).
Selecting attributes to measure the achievement of objectives
.
Operations Research
53
(
1
),
1
11
.
Kirkpatrick
C.
(
2016
).
Developing countries
. In:
The Handbook of Regulatory Impact Assessment
(
Dunlop
C. A.
Radaelli
C. M.
, eds).
Edward Elgar Publishing
,
Cheltenham
,
UK
. pp.
381
396
.
Landeta
J.
(
2006
).
Current validity of the Delphi method in social sciences
.
Technological Forecasting and Social Change
73
,
467
482
.
doi:10.1016/j.techfore.2005.09.002
.
Landeta
J.
Jesús
M.
Ruíz
V.
Galter
J.
(
2008
).
Results of a Delphi survey in drawing up the input-output tables for Catalonia
.
Technological Forecasting and Social Change
75
(
1
),
32
56
.
doi:10.1016/j.techfore.2007.01.005
.
Makkonen
M.
Hujala
T.
Uusivuori
J.
(
2016
).
Policy experts’ propensity to change their opinion along Delphi rounds
.
Technological Forecasting and Social Change
1
,
1
8
.
doi:10.1016/j.techfore.2016.05.020
.
Marques
R. C.
(
2008
).
Comparing private and public performance of Portuguese water services
.
Water Policy
10
,
25
42
.
doi:10.2166/wp.2007.033
.
Marques
R. C.
Berg
S.
(
2010
).
Revisiting the strengths and limitations of regulatory contracts in infrastructure industries
.
Journal of Infrastructure Systems
16
,
334
342
.
doi:10.1061/(asce)is.1943–555x.0000029
.
Mendonça
E. C.
(
2009
).
Metodologia para avaliação de desempenho de sistemas de drenagem urbana (Methodology Evaluation for Urban Drainage Systems)
.
Master Thesis at University of Brasilia
,
Bras–lia, Brazil
.
Montibeller
G.
von Winterfeldt
D.
(
2015
).
Cognitive and motivational biases in decision and risk analysis
.
Risk Analysis
35
,
1
22
.
doi:10.1111/risa.12360
.
Mostert
E.
(
2003
).
The challenge of public participation
.
Water Policy
5
(
2
),
179
197
.
Pinto
F. S.
da Cruz
N. F.
Marques
R. C.
(
2015
).
Contracting water services with public and private partners: a case study approach
.
Journal of Water Supply Research and Technology-Aqua
64
,
194
210
.
doi:10.2166/aqua.2014.121
.
Reppa
G. P.
(
2007
).
Implications for health education
.
Science Educacional International
18
,
217
233
.
Rowe
G.
Wright
G.
(
1999
).
The Delphi technique as a forecasting tool: issues and analysis
.
International Journal Forecasting
15
,
353
375
.
doi:10.1016/S0169-2070(99)00018-7
.
Van Ast
J.
Gerrits
L.
(
2016
).
Public participation, experts and expert knowledge in water management in the Netherlands
.
Water Policy
16
,
1
13
.
doi: 10.2166/wp.2016.253
.
Von der Gracht
H. A.
(
2012
).
Consensus measurement in Delphi studies. Review and implications for future quality assurance
.
Technological Forecasting and Social Change
79
,
1525
1536
.
doi:10.1016/j.techfore.2012.04.013
.
Wright
J. T. C.
Giovinazzo
R. A.
(
2000
).
Delphi: uma ferramenta de apoio ao planejamento prospectivo (Delphi: a support tool for prospective planning)
.
1st Report in Administration Research
1
(
12
),
12
16
.
Zardari
N. H.
Ahmed
K.
Shirazi
S. M.
Yusop
Z.
(
2015
). .

Supplementary data