Approaches for the evaluation of future-oriented technologies and concepts in the field of water reuse and desalination

The need for targeted and safe water reuse solutions is increasing worldwide. However, the implementation of water reuse solutions is still very limited compared to their potential due to a number of factors, e.g. low economic attractiveness and lack of public acceptance of reuse solutions, low awareness of technological advantages and poor coordination of actors from industry, authorities and the water companies. Decisions on the implementation of innovative technologies and concepts for water reuse and desalination are regularly marked by conflicting goals between economic, technical, environmental and socio-political considerations. In order to inspire decision-makers to use new technologies and to convince sceptics of the viability of innovative approaches, transparent, sound decision criteria are required.

In the joint projects of the funding measure ‘Future-oriented Technologies and Concepts to Increase Water Availability by Water Reuse and Desalination (WavE)’ by the German Federal Ministry of Education and Research (BMBF), suitable evaluation approaches were developed for different user groups and local target settings, taking into account the respective decision-making situation. The internationally established applications range from the treatment of saline groundwater to the reuse of municipal and industrial wastewater for various purposes in the municipal, agricultural and industrial sectors (Table 1). Depending on the application case and object of evaluation, different evaluation methods are applied, to assist operators, customers and plant constructors in various phases of the planning process in making decisions and to provide target group-oriented results.

The evaluation in the projects WaReIp and MULTI-ReUse addresses the early conceptual design of process chains for water reuse in industry or agriculture. The focus of the project DiWaL is on the development of optimised water management concepts, including the application of Pulsed Electric Field (PEF) treatment for efficient reduction of microorganisms in process water and paints in pre-treatment and dip coating plants for automotive series painting. In the project WaKap, innovative desalination processes and raw water treatment processes for drinking water supply in areas with a poorer infrastructure are evaluated and compared to alternative water supplies. In the project WEISS, innovative treatment concepts for cooling water recycling in the steel industry are assessed.

The time horizon of the evaluation as well as the system boundaries are chosen differently in the individual projects: In the DiWaL and HighCon project, the entire life cycle of a plant is considered as in the framework of Life Cycle Assessment (LCA). In contrast, in MULTI-ReUse and WaKap, the focus is primarily on the operating phase of the systems. In WaReIp, plant construction and operation are considered.

Nevertheless, the individual evaluation procedures have some aspects in common, which result from the value system of the decision-makers and/or the respective object of evaluation. Context-specific factors, such as the region of application, can also lead to overlapping procedures.

The aim of this paper is to assist decision-makers in identifying the best possible source of water supply by providing a generic approach for comparative process assessment (section Valuation issues in wave) and providing a structured overview of assessment criteria with high relevance to the topic of water reuse and desalination (section Evaluation framework). Based on the generic flow chart and the list of criteria, which were derived from six multi-criteria evaluation concepts used in selected WavE projects, it is possible to compile an evaluation methodology together with a set of criteria, tailored to a specific subject of evaluation and the decision-maker(s). If necessary, further application- or country-specific criteria can be added to the list. In section Evaluation criteria, Multi-criteria decision analysis (MCDA), cost–benefit analysis (CBA) and LCA are presented as exemplary evaluation methods for identifying preferred solutions. Finally, possible data sources and methods for addressing uncertainty are presented and discussed with regard to their applicability (section Evaluation methods and selection of preferred solutions).

Despite project-specific differences, a common generic structure was identified, which applies to all six WavE projects involved. Three levels of assessment can be distinguished: prerequisites, minimum requirements and evaluation criteria (Figure 1).

Examining that prerequisites (here n = 4), such as fulfilling legal boundary conditions for the intended purpose of use, local availability of the raw water flows required for reuse, general openness of the decision-makers towards the solutions and a ascertained demand for water reuse, are fulfilled, represents the first step of this process. These four prerequisites lie outside the sphere of influence of the project. Failure to meet them can significantly impede the implementation and use of the investigated technologies or systems and thus leads to the exclusion of the alternative in question from the decision-making process.

Furthermore, before the actual evaluation begins, the general suitability of the alternatives is examined on the basis of various minimum requirements (here: n = 3). In contrast to the prerequisites, the test criteria, which are used to check if the minimum requirements are met, can be improved by adapting the evaluated process or system until it is generally suitable for the purpose at hand. Based on the test criteria, a technical process selection and refinement can be made. Unfulfilled minimum requirements indicate the need for action or optimisation potential of the respective solutions. Alternatives for which the minimum requirements cannot be met even after revisions are discarded as unsuitable. The minimum requirements essentially refer to the conformity of the intended technical process with operational requirements, its integrability into existing infrastructure or building and plant technology, as well as the feasibility of synchronising supply and demand of the treated water. Selected test criteria can also be included in the comparative assessment in the form of evaluation criteria: the degree to which they are (over)achieved represents the assessment basis for the characteristic concerned. An example of this is the integrability, which needs to be given in principle (test criterion) but should also be evaluated qualitatively on the basis of the effort involved (evaluation criterion).

The remaining alternatives can then be subjected to a comparative evaluation for decision-making, which can be based on a quantitative evaluation (e.g. using material or energy balances, Environmental Impact Assessment, life cycle costing) or a qualitative evaluation of individual evaluation aspects (e.g. using stakeholder analyses) or can combine elements of both evaluation methods in the form of a holistic, multi-criteria decision support approach (see section Evaluation criteria).

If the examination of some test criteria is more complex than determining the evaluation criteria, a screening for promising alternatives and/or rejection of undesirable ones can be carried out first based on the evaluation criteria, before compliance with the (remaining) test criteria is ensured.

The aim of the evaluation is to uncover strengths and weaknesses of the compared alternatives and to identify the system solution that is most suitable for a specific site. Often, however, there is no clear best solution, so that a suitable compromise has to be found. It should be noted that the evaluation criteria usually have different units or scales and are of varying importance, which makes comparison difficult.

In order to evaluate technical alternatives within the six WavE projects under consideration, qualitative, semi-quantitative and quantitative criteria from the fields environment, social issues and economy are combined in accordance with the triple bottom line (TBL) model of sustainable development and the Sustainable Development Goals adopted by the UN. Against the background of the particularities of water supply technology (van Leeuwen & Marques 2012), the assessment dimension technology is included as a fourth evaluation dimension.

The number of evaluation criteria asked for in each project varies between 20 and 30. In total, around 50 different criteria were compiled from the participating projects covering the following aspects.

Interactions between water treatment plant and environment are recorded on the basis of resource and land use (including land use, energy and water requirements, chemical consumption), emissions into soil, water and air, and the contribution to maintaining or increasing ecosystem services (e.g. volume of additionally provided water).

The social acceptance and compatibility of a water supply alternative can be described, among other things, based on the transparency and acceptance of the decision-making process (qualitative), the acceptance of water reuse and the products manufactured (qualitative), the creation of new jobs, political legitimacy, the potential for raising awareness for sustainable resource use, nuisances caused by the operation of the system (noise, aesthetics), conflict potential due to competition over resources and land, user friendliness and affordability.

In the economic dimension, the economic profitability, local economic development potential, economic risk potential, as well as market competition, are characterised by the following criteria: possibilities of generating profits, operating and maintenance costs, specific total costs, costs for investment and commissioning, contribution to local value creation, competitive advantages through improved market positioning, technology transfer, potential for transfer to other markets, technological readiness level (TRL), risk management, as well as damage costs caused by plant failure.

In the fourth assessment dimension of technology, the TRL, minimum level of training required to operate the plant, robustness, efficiency, flexibility and level of automation of the plant, operating and maintenance costs, achievable yield, technical integrability in existing infrastructures and processes, degree of dependence on other infrastructure systems, mean time to failure, fault tolerance and occupational health and safety during operation of the plant are assessed. Based on these criteria, technical feasibility, as well as process safety, can be accounted for in the evaluation.

In general, all criteria can be divided into universal and specific criteria. Examples of universal criteria, which are used in almost all WavE projects considered, are operating costs, emissions, operating and maintenance expense and political legitimacy. In contrast, specific criteria are used in only a few projects, such as the potential for opening up additional markets, the potential for automation and the potential for the creation of ecosystem services or the creation of jobs. An overview of the different criteria is given in Figure 2.

Criteria that are important primarily from an economic/business perspective or from the perspective of public services of general interest are indicated by corresponding symbols.

Adopting the different criteria presented in Figure 2, it should be noted that from different perspectives, some criteria may have partly opposite implications for the evaluation. One example is the criterion ‘creation of new jobs’, which is seen positively from a social perspective but is viewed rather negatively from an economic perspective due to the associated personnel costs. In addition, the allocation of criteria is not always clear-cut, as criteria can have different implications in the context of different evaluation categories. This will be illustrated using the example of the criterion resource and space requirements, which can be reflected in the three categories ‘environment’, ‘social’ and ‘economy’:

• The use of natural resources and land is increasing constantly worldwide. It is associated with emissions and other environmental impacts throughout the life cycle of technical equipment and products. At the same time, the existing competition for land, for example, means that biological diversity, the uniqueness of landscapes, natural or archaeological monuments, fertile soils and an appropriate proportion of mineral resources must be protected (German Environment Agency 2018). Due to the limited availability of resources and land, competition situations arise in which the various types of use need to be weighed up within the ‘Environment’ category.

• In the context of dwindling resources and land, questions of fair distribution arise in order to avoid social inequalities and, as a result, social upheavals. The evaluation of these impacts of a project is recorded in the category ‘Social’.

• From a business point of view, the acquisition of resources and land is directly linked to costs. Scarcer resources and land can be expressed in the form of fluctuating or rising prices. Accordingly, this aspect needs to be considered in the category ‘Economy’.

Depending on the decision-makers' perspective, different criteria can be used for evaluation. Two distinct positions within the WavE projects under consideration are the entrepreneurial/business perspective, focused particularly on business and marketing aspects, and the public service perspective, in which social and environmental aspects are more important. In addition, depending on the decision context, country-specific, application-specific and/or user-specific criteria may be added.

Technical infrastructure systems in urban areas ensure the supply of water, energy, heat, the disposal of wastewater and waste and the provision of traffic and green spaces. In many countries, the provision of technical infrastructure is organised as part of public services or regulated by the state (Dominguez et al. 2009). Water supply is a grid-bound investment-intensive infrastructure system, which ties up considerable amounts of capital. However, it has a low ‘return on investment’ and is accordingly described as a natural monopoly in the micro economy. Despite the great importance of public actors in providing and financing these infrastructure services, the private sector often plays an important role as well (OECD 2007). Since water resources are public goods, the complex implications of collective action should be taken into account: even if every actor benefits when action is taken, it is not worthwhile for any of the actors to invest in a solution on their own (cf. ‘Prisoner's Dilemma’). Against the background of climate change, the issue of ‘sustainability’ for infrastructure systems is increasingly coming into focus. The public administration has a supporting, regulating, but also monitoring role in the implementation of these criteria (OECD 2007). Taking these framework conditions into account, the WavE collaborative projects have included evaluation criteria in their assessment tools that take into account important aspects of water supply in the context of public services of general interest.

The goal of the WaKap project, for example, is the development and piloting of an innovative, energy-efficient, modular combination process for the desalination of seawater and brackish or groundwater treatment. The background of the project is that the water supply in Vietnam and other Southeast Asian countries is increasingly facing challenges, such as the influence of climate change, the strong regional population growth and additional water demand due to economic development. For a permanent, sustainable use, safety-relevant and social aspects play an outstanding role in this project, in addition to ecological, technical and economic aspects.

• Transparency and acceptance in the decision-making process: The decision-making process should be communicated to the public in a transparent and comprehensible manner and should be carried out with the involvement of the relevant stakeholders. The selection of the actors considered relevant plays an important role in this process.

• Participation and co-determination in the decision-making process: In a further step, it should be examined to what extent the decision-making process is carried out and completed with the involvement of the relevant actors and to what extent equal rights for all social groups are taken into account (see here, for example, Schmitter (2000) and Grote & Gbikpi (2002)).

• Competition for resources and land: The use of natural resources and land has a variety of social impacts in addition to the consequences for the environment. Competition for land and resources with the existing environment should therefore be carefully examined, e.g. with regard to natural areas, areas for local recreation or areas for alternative uses. In this context, attention should be paid to the quality and quantity of resources and land required for a planned facility. In terms of land quality, for example, this means that natural areas with a high availability of biodiversity must be protected in particular.

The business perspective is typically focused mainly on economic and technical aspects. Apart from costs and revenues, the stable and failure-free operation of processes is often of paramount importance. This is especially true for the supply of utilities, such as water, which do not directly generate any revenue but are required for most production processes. A reliable water supply, both in terms of quantity and quality, is crucial to ensure undisturbed production and thus prevent substantial financial losses due to production downtimes and impaired product quality. Depending on the available water sources, water reuse can offer advantages but may also pose a risk in this regard. As the cost of water supply and wastewater treatment often accounts for only a fraction of the production costs and revenues, the indirect (financial) implications of water supply alternatives should be considered in the assessment.

Other criteria, e.g. the potential to improve the corporate image or to open up new markets can increase the attractiveness of a technical solution and justify possible additional expenditure. These additional aspects are usually closely related to the strategic goals of the company and may thus differ substantially in different cases. Generally, ecological aspects have gained importance in this regard in recent years. More eco-friendly technologies and processes may not only promote a green image but also facilitate compliance with future legal standards and the cooperation with local authorities. Social aspects, however, usually play a subordinate role, which is also reflected in the business-oriented assessments of the WavE projects WEISS and WaReIp. However, even though they are not as apparent in the evaluation in a corporate setting, ecological and social aspects are reflected in (external) requirements (prerequisites), such as laws and regulations that must be complied with.

Two different situations that companies may face in terms of technology selection are represented in the involved WavE projects:

• Companies that provide water reuse and desalination technologies need to decide which technologies to include in their portfolio and/or develop. This is considered in the projects WEISS and DiWaL.

• Companies that want to implement water reuse and desalination on their own site need to decide on a suitable technical solution – as in the project WaReIp.

The selected criteria and system boundaries can differ between these two situations. In the first case, the evaluation is focused mainly on the concerned technologies and assesses associated market opportunities. As this involves how well the technology satisfies the needs and preferences of potential clients, many criteria are relevant to both situations. In the second case, the question might not just be which technology to choose, but if and how to implement water reuse at all. This requires a broader view on existing water supply options (and wastewater treatment and discharge options) for the industrial site to properly compare alternatives. New market possibilities (e.g. through expansion of the production site) and competitive advantages can also be assessed, but in this case, they usually derive rather indirectly from the employed water treatment technologies.

In order to select a preferred solution on the basis of the various evaluation criteria, advantages and disadvantages of different alternatives have to be compared and weighed against each other. The solutions to be evaluated usually have different strengths and weaknesses, so it is not obvious which one is the best. Different scales and units of the evaluation criteria can make the comparison more difficult. In addition, resource and water management usually involves several stakeholders and decision-makers, who may have different preferences (Hajkowicz & Higgins 2008). Consequently, compromises have to be found between conflicting objectives that are advocated by different stakeholders.

There is a variety of methods that can support structured and transparent decision-making. They differ essentially in the selection of the evaluation aspects taken into account, the procedure for comparing and weighting the evaluation criteria and the determination of the best solution(s). Table 2 provides an overview of the advantages and limitations of the assessment methods used within the six projects that build the basis for this paper.

The aim of multi-criteria evaluation is to identify the best solution or a manageable number of equally good alternatives. A complete ranking of all alternatives is not necessarily required for this, but may be the result, depending on the evaluation method used (Figueira et al. 2005). Common to all multi-criteria evaluation methods is the general procedure of (i) specifying the objective and the alternatives to be considered, (ii) defining the criteria, (iii) measuring the relative importance of the criteria and (iv) aggregating the judgements (Belton & Stewart 2002). A detailed analysis of the different methods, their foundations, as well as strengths and weaknesses, can be found, for example, in Belton & Stewart (2002).

MCDA is used, for example, in the projects MULTI-ReUse and WaKap in the form of a utility value analysis. The sustainability assessment tool developed within MULTI-ReUse supports stakeholders and decision-makers in evaluating different water recycling solutions against the current system configuration in order to identify the most sustainable water supply system for the future. In this project, utility value analysis has been selected as favourable multi-criteria assessment methodology because it is comprehensive enough to deal with the diverging objectives attached to such decision cases and at the same time offers the highest level of flexibility to be applicable in various contexts. Including experts from different disciplines in the compilation and review process of criteria list, served to ensure a common understanding of terminology and consideration of contradicting stakeholders’ viewpoints of water reuse implementation. The application of the decision support tool for the sustainability assessment of the two case studies provided its developers with key insights about advantages as well as limitations that were necessary to transform it into a practice-oriented tool.

Within MCDA, the weighting of the individual criteria is usually done in dialogue with the decision-makers, whereby disparate preferences of different actors can also be taken into account. It is not uncommon that persons – even in similar positions – express different opinions when asked about their preferences. In the WEISS project, for example, the importance of the category ‘technology’ was assessed very differently by different experts. While most experts rated it just behind cost, once it was classified as completely unimportant. With regard to environmental compatibility, the spectrum even ranged from almost unimportant to the most important criterion. In addition to the diverging preferences of the respondents, it cannot generally be ruled out that a different understanding or different ideas regarding the aspects to be classified may have contributed to the different results. Determining the evaluation criteria and their relative importance is a complex task that requires a great deal of sensitivity in the dialogue with the decision-makers. Moreover, the importance of the weights depends on the method chosen (Belton & Stewart 2002). Therefore, the influence of the choice of different weights on the resulting ranking needs to be investigated and discussed with the decision-makers (Figueira et al. 2005; Pajer et al. 2017) (see also section Evaluation methods and selection of preferred solutions).

Overall, the multi-criteria evaluation is a very open method that can be adapted to specific problems. Any evaluation criteria can be taken into account, even with different scales and units. The procedure aims to reflect as accurately as possible the preferences and value systems of the individuals and groups involved indecision-making. Thus, multi-criteria evaluation enables tailor-made, transparent decision support but also requires appropriate design in order to deliver relevant and reliable statements. The concrete formulation of preferences and discussion of the compromises to be made contributes to a common understanding among the parties involved and helps to comprehensively illuminate the advantages and disadvantages of various alternatives. Thus, multi-criteria evaluation can offer added value even beyond the sole choice of a preferred solution.

As a rule, it is not only decisive what costs are caused by different decision options but also the benefits have to be seen in relation. In the so-called cost–benefit analysis (cf. Atkinson & Mourato 2015) internal costs are compared with the monetised benefits. In addition to direct costs, in particular, technical assessment criteria, such as maintenance expenditure or downtimes caused by system failure can be expressed in terms of the expected associated costs as part of the operating costs. In addition, external costs (e.g. due to greenhouse gas emissions) and benefits (e.g. increased regional water availability) can also be taken into account from an economic point of view within the framework of a so-called extended CBA. However, the approach to monetisation is ambiguous and can be a sensitive and controversial issue, especially for social aspects.

If the benefits cannot be monetised, a cost-effectiveness analysis can be carried out. This can be useful if, for example, two measures lead to the same result or if the consideration is focused on one parameter. However, if side effects are to be included in the evaluation, it is necessary to weight different effects implicitly or explicitly. For example, effects are implicitly weighted with zero if they are not included in the comparison. If alternative A (with low internal costs but higher emissions) is chosen instead of alternative B (with higher internal costs but lower emissions), the emissions are implicitly (even without LCA or other sustainability assessments) valued with the difference in internal costs, since it is implicitly assumed that the losses are lower than the difference in internal costs.

Overall, CBA is well suited to reflect economic and, in part, technical evaluation criteria. By explicitly evaluating the benefits, even solutions with different benefits can be compared without difficulty. The monetary evaluation fits well into traditional, economically influenced decision-making processes and evaluation concepts. Ecological and social criteria can only be considered in the CBA on the basis of their economic impact or by assigning a financial value. To avoid the monetisation of social and ecological criteria, the CBA can be embedded in a broader multi-criteria evaluation.

In the context of technology assessment, LCA serves to compare the environmental performance of different technologies that produce the same product or have the same purpose (e.g. wastewater treatment or the supply of fresh water) on a global scale. Descriptions of the LCA methodology can be found, e.g. in Klöpffer & Grahl (2012), ISO 14044 (2018) or UNEP LCI (2018). The focus of LCA is on the environmental and health impacts caused directly (e.g. exhaust gases of a boiler) or indirectly (e.g. emissions caused by upstream processes such as production and transport of building materials) during different life cycle phases of a product or a service. Various impact categories, such as climate change, land occupation, acidification or use of non-renewable resources, are considered in order to provide a comprehensive picture of the environmental impacts caused per functional unit. Within each impact category, the effects of all processes considered in the LCA are quantified and summed up by means of an impact category indicator (e.g. kg CO₂-eq per functional unit). Effects are related to a so-called functional unit. When comparing different alternative technologies, products or services, a comparison is possible. When mapping water recycling within closed system boundaries, the choice of functional unit may well differ from that in an open system. Since in the project DiWaL, two technologies for efficient reduction of microorganism in the pre-treatment for dip painting of the car body are compared, the functional unit is ‘one car body without complaints’. In the WavE project HighCon, for example, the wastewater volume to be treated in m³ was defined as a functional unit. The treated water flows back into the process and thus reduces the fresh water withdrawal and the effort for further treatment for production. In this case, the material and energy required to treat the fixed volume of wastewater is thus offset by savings on the fresh water side, reduced waste flows and recovered inorganic salts as raw materials. Similarly, in projects for water reuse, the m³ of water provided can also be selected as a functional unit, which facilitates a comparison of potential water resources. The associated implications for wastewater treatment and discharge need to be considered in a closed system.

ISO 14044 (2018) does not specify how to identify the preferred solution based on the environmental and health impacts identified. It does, however, provide a very rigorous and comprehensive assessment of the environmental and health aspects of different alternatives using a transparent, standardised approach. As a basis for decision-making, the impact categories identified in the LCA can be used as evaluation indicators, supplemented by other relevant evaluation criteria, in a multi-criteria evaluation.

To apply ISO 14044 (2018) is especially important for cross-study comparisons, e.g. for similar products or ‘functional units’ from different companies.

Car body painting plants have a high production volume where a great deal of water is consumed. In the DiWaL project, the environmental performance of PEF treatment is compared with the surge dosing of biocides which is applied to reduce the microbial contamination of paint and process water. The environmental analysis conducted within the DiWaL project combines elements of the LCA with a parametrised energy and material flow model and further input from stakeholder consultation via different dialogs and workshops. This approach is called Integrated Innovation and Sustainability Analysis (Gasde et al. 2019, 2020). Applying simplified approaches similar to LCA (e.g. using expert guess and stakeholder interviews for data assumptions), scenario and hotspot analysis, allowed an evaluation of the potential environmental impact of the PEF treatment early in the development phase of the innovation. As far as possible, quantitative results regarding environmental and economic sustainability have been derived. However, more importantly, the analysis helped to generate feedback loops in the early innovation phase. By addressing these, the relevant sources of negative impacts can be indicated. Consequently, the design of the corresponding product was adjusted. There are different types of so-called prospective or ex-ante LCAs which try to take aspects into account which will change in the future (Cucurachi et al. 2018). The PEF treatment will start to operate only in the near future (ca. 5 years). Moreover, once such a device is installed, it will continue to operate for maybe 15 years and more. Further devices will go into operation even later. Hence, in order to make a fair comparison of this innovation with its current alternatives, the LCI data were adjusted to future conditions. For example, the underlying electricity generation mix is very relevant for the evaluation of the PEF treatment. To select a reasonable time horizon is one of the conventions which have to be decided in the frame of the definition of the goal and scope of the analysis.

The data required for the evaluation depend on the project and on the evaluation criteria, dimensions and methods chosen. The following types of data are found in several WavE projects:

• 1.

  System- or plant-specific operating data (energy/chemical consumption, cleaning performance, personnel and maintenance requirements, etc.)

• 2.

  Geometry, material properties, mass and volume of materials used (e.g. for building a device)

• 3.

  Technical boundary conditions: inlet quality, water quantities and requirements, information on existing systems

• 4.

  Costs, tariffs

• 5.

  Characterisation factors for the assessment of environmental impacts (impact factors for LCA), e.g. ReCiPe (Goedkoop et al. 2013)

• 6.

  Legal and organisational frameworks

• 7.

  Decision-maker and stakeholder goals and preferences

Not all the data required are specific to the project or the system under evaluation. Therefore, these data may be available elsewhere and do not have to be collected in the course of the project. Depending on the data source, different challenges arise for its use (Table 3).

Empirical values, experience (e.g. from projects already implemented) and literature data are usually readily available for established technologies and processes. However, it should be examined whether the transferability of such data to innovative approaches is guaranteed. Increasingly, literature data on studies and experiments that have been carried out are also being questioned as to their reproducibility.

Data from laboratory tests and pilot plants carried out in the project offer a very high reliability, provided that sufficient attention is paid to careful data preparation. The scalability to industrial scale (up-scaling) should already be taken into account in the design of experiments.

Modelling enjoys unbroken popularity as a supplement or alternative to laboratory tests or piloting. Here, the conflict of objectives between the level of detail and the practicability represents the main challenge. The quality of a model prediction is always related to the quality of the input data.

One tool for obtaining qualitative or semi-quantitative data is stakeholder and/or expert consultation. Here the aim is to bring together experiences and complementary knowledge from different perspectives. In order for this method to be successful, a common understanding of the facts to be evaluated and a common terminology is crucial.

Statistical data are used, for example, to estimate population figures, water demands or price indices – often also to be able to forecast future developments. In addition to the challenge typical for statistical data of using a suitable sample size and distribution as a basis, extrapolations of past developments into the future need to be understood and should be presented as hypotheses.

As a basis for LCA, very large amounts of data on the environmental impacts of production, transport and disposal processes are required, which are usually not available or cannot be collected within a project. Proprietary databases such as the ecoinvent data base offer extensive information on these topics. If the data are to be extended by proprietary data sets, there is a risk that these will not be consistent with the existing data.

Laws and regulations supply information on effluent or discharge values and other emission limits which need to be complied with. These can vary greatly from region to region. In addition to the transferability from one site to another, which needs to be checked, this also means that the data are often not physically consistent. This can lead to the fact that an alternative that is considered to be sensible on the basis of the state of the art cannot be implemented or that legal requirements are physically contradictory.

Since the evaluation for decision support is carried out in the planning/design phase, the required data can usually not be collected on the real system. Therefore, the data used for the evaluation are subject to uncertainties that may also vary for different alternatives. How reliably prerequisites and minimum requirements can be fulfilled and evaluation attributes estimated, both for the expected use of the technology and with regard to future developments or unexpected boundary conditions, can therefore play an important role in the decision-making process. This is also reflected in some of the evaluation criteria, such as the TRL or the robustness and flexibility.

Further uncertainties or ambiguities arise in the selection of preferred solutions. For example, the choice of weighting factors in classical MCDA methods is highly subjective and not necessarily clear to decision-makers (see also section Evaluation framework) (Figueira et al. 2005).

A simple and intuitive way to consider uncertainties of the input data and/or the evaluation is to consider different predefined scenarios (i.e. different sets of input parameters or data sets). This method is often used to describe possible future developments, but it can also be used to map different plausible, mutually exclusive conditions for planning and evaluation (e.g. good/bad biodegradability of organic substances in the treated water, low/high population growth). In the MULTI-ReUse project, for example, scenarios with different weightings of the evaluation criteria are compared in order to investigate the influence on the resulting ranking of the alternatives.

The definition of the scenarios significantly influences how robust and comprehensive the additional knowledge gained is. To keep the uncertainty analysis manageable, only a limited number of scenarios can be considered. This limits the possible combinations of uncertain factors that can be investigated. However, a limitation of the considered contingencies can be quite useful if not all combinations are likely or possible. An advantage of the scenario analysis is that it is easy to implement and the results are straightforward to understand and interpret.

Alternatively or in addition, a systematic investigation of the influence of uncertainties in the underlying data or the assessment steps on the evaluation result can be carried out by means of a comprehensive (global) uncertainty and sensitivity analysis (see Saltelli et al. (1999)). Depending on the question posed and the evaluation framework, the possible intervals (or probability distributions) of the input data are examined to see how they affect the criteria specification, aggregated evaluation indicators or the ranking of the alternatives. The sensitivity analysis also allows identification of the most relevant uncertainties in the input data as well as their interactions, so that a targeted refinement and expansion of the data basis can be carried out. Possibilities for integrating uncertainty and sensitivity analysis into the model-based planning of water reuse concepts in industrial parks are being investigated in the WaReIp project.

The prerequisite for a broad sensitivity analysis is a formalised and, as far as possible, automated calculation of the target variables, since many calculation runs are usually required. In addition, the uncertainty or possible bandwidths of the input data should be estimated as realistically as possible in order to obtain meaningful results. The implementation, evaluation and interpretation of the results tend to be more complex than when considering individual parameter sets/alternative data sets and may not be entirely trivial. On the other hand, global sensitivity analysis provides a more complete picture of the possible effects by simultaneously considering the existing uncertainties and allows a well-founded prioritisation of the influencing factors.

Within the framework of the WavE joint research projects, a variety of evaluation approaches have been adopted and developed to assess the advantages of innovative process technologies over conventional supply structures in the context of water reuse and desalination.

The working group on evaluation methods in WavE has succeeded in developing a generic approach for the comparative evaluation of procedures, technologies and system solutions. Via checking prerequisites and test criteria, compliance of the proposed solutions with the existing legal framework and other prerequisites can be achieved by optimisation of the proposed solutions in an iterative process. Furthermore, a structured overview of evaluation criteria was developed, which, in addition to the three evaluation dimensions environmental, social and economic of the TBL model of sustainable development, also includes the field of technology. Considering all these evaluation criteria in the assessment enables the users to find the most preferable of the feasible solutions for a specific decision case. Together, the evaluation procedure and the evaluation criteria grouped in the form of individual clusters offer intuitively usable assistance for preparing individualised, multi-criteria evaluations. The results of the working group thus show a high relevance with regard to manifold questions of water supply and reuse in practice.

The choice of criteria and methods for the evaluation of sustainable technologies and concepts is always application-specific and should be oriented towards the overall objectives, the actors involved in the decision (e. g. perspective of public services or business perspective) and the decision-making framework (scope, implications, duration of the decision-making process) in order to identify the most suitable solution for the individual decision-making situation. The differences between the evaluation approaches developed in the WavE projects underline this need for adaptation to the specific situation. However, many similarities were identified both in the general approach and methodology as well as the chosen criteria. Nevertheless, it should be noted that each evaluation method goes hand in hand with limitations that have to be considered in the selection process.

MCDA is a family of decision-making methodologies that may include any criteria defined by the stakeholders and can be shaped very specifically to any project. However, no out-of-the-box application is possible. LCA allows very rigorous assessment of global ecological impact but does not include other technical, economic or social criteria. In contrast, CBA can encompass aspects of all four dimensions as long as they are monetised, i.e. expressed in terms of financial revenues or costs. LCA and CBA do not directly point out the most preferable solution but can be valuable complements in an MCDA framework.

Practical applicability of an evaluation methodology is strongly dependent on the availability of reliable and robust data. Assessing technical, economic, ecological and social aspects of water reuse and desalination solutions requires data from many different sources. Especially in planning, data availability and reliability is a crucial issue for prognosis as the required information concerns concepts and technologies that are not yet implemented. As the data form the basis on which decisions are taken, data quality deserves close attention. Considering the uncertainty of the underlying data and assumptions can improve the robustness of the developed alternatives and the taken decisions, thus minimising risks (or abating negative consequences).

A thorough consideration of technical, economic, ecological as well as social criteria are recommended to be considered in any decision case. In both public and corporate projects, ecological aspects have become increasingly important, although the motivation for including respective criteria in the decision-making process may not always be the same. In public projects, well-being and preservation of nature and resources seem to be more focused, whereas in corporate projects, green image and compliance are the environmental aspects that highest priority is assigned to. In general, social and ecological aspects are less important for the assessment in corporate projects but are reflected in prerequisites that stem from laws and regulations.

A multi-criteria evaluation, which follows the generic approach of the WavE projects presented, helps to implement a clearly structured decision-making process. A detailed examination of the objectives and requirements, exchange of viewpoints and preferences, as well as the development of a common understanding among the project participants during the decision-making process, can promote the development of sustainably satisfactory solutions, especially when closely linked to process development. Through appropriate, accompanying documentation of the selected procedure and the methodology applied, the evaluation approach offers a high degree of transparency for all those involved. The evaluation results support the decision-makers in communicating the decisions made internally and externally.

Multi-criteria evaluation methods cannot liberate the actors of the final decision, because none of the presented methods can fully reflect all aspects to be considered. Therefore, it is important to facilitate a profound understanding of the pros, cons, potentials and risks of the available alternatives. To this end, the presented evaluation methods are a helpful communication and decision support tool.

The research leading to these results has received funding from the German Federal Ministry of Education and Research as part of the funding measure ‘Future-oriented Technologies and Concepts to Increase Water Availability by Water Reuse and Desalination (WavE)’.

All relevant data are included in the paper or its Supplementary Information.