The CLARA Simplified Planning Tool (CLARA-SPT) supports economic planning of water supply and sanitation system alternatives by applying life-cycle cost (LCC) principles. The cost functions used in the tool were developed based on standard designs, bills of quantities and a database of country-specific unit rates. In order to make reasonable economic comparison among system alternatives, the simulations of the tool should be able to predict costs of actual systems efficiently. In this paper, various statistical validation measures were employed to evaluate the simulation accuracy of the CLARA-SPT for eight water supply and nine sanitation technologies commonly implemented in Ethiopia. After evaluating the simulated outputs against observed cost data, appropriate corrective measures were introduced for technologies that did not replicate reality fairly. Costs of seven water supply and six sanitation technologies were able to be represented by the CLARA-SPT with good accuracy. Disinfection was however mimicked fairly. In addition, poor simulation accuracy was identified for three sanitation technologies (i.e. UDDT, Composting toilet and Faecal sludge collection). Therefore, the tool developers are advised to correct internal design assumptions of these three sanitation technologies to improve the simulation accuracy of the CLARA-SPT.

INTRODUCTION

More than 700 million and 2.5 billion of the global population lived without access to improved sources of drinking water and sanitation facilities in 2012, respectively, from which Ethiopia shared about 44 million and 70 million people, respectively (WHO/UNICEF 2014). Strategically, sound economic assessment of water supply and sanitation (WS&S) system alternatives is crucial to improve the service coverage (Haller et al. 2007). The cost of increasing access to improved WS&S services varies considerably with the types of technology selected in a system that largely influence the decision-making process. The CLARA Simplified Planning Tool (CLARA-SPT) was developed to estimate the cost of WS&S system alternatives and to bridge life-cycle cost based WS&S system planning (Casielles Restoy et al. 2014; Lechner et al. 2014). The economic comparison and selection of WS&S options is done based on the system's total costs within the defined planning horizon. The total costs comprise initial investment costs (IIC), operation and maintenance costs (O&MC), reinvestment costs (RIC) and revenues. However, the accuracy of the simulation tool needs to be proven before it is widely applied for actual project planning, which can be addressed through model validation. Model validation is a process of checking how accurately the simulated output represents the real system, hence it is a principal step in model development and application (Sargent 2005). Actually, it is too expensive and time consuming to develop an absolutely correct model for its intended purpose. However, it is possible to create a model that can represent reality within an acceptable range of accuracy, where it is considered as valid for the intended application (Sargent 2011).

The selection of the appropriate validation technique is defined by the level of accuracy demanded, the purpose of the model characteristics and the availability of data (Mayer & Butler 1993). Generally, model validation techniques can be grouped into graphical and statistical methods. Relatively, statistical methods are more robust and very much dependent on types and distribution of available data (Mayer & Butler 1993). Statistical methods include goodness-of-fit measures, confidence interval, hypothetical test, etc., which are used to evaluate the range of resemblance between simulated and actual data (Toledo & Koutsopoulos 2004). The goodness-of-fit measures are principally based on the techniques developed for the validation of economic forecasting and econometric models (Power 1993), which can evaluate the overall performance of a simulator. Graphical methods provide a visual comparison of simulated and observed data to assess the performance of the model (Moriasi et al. 2007) and include histogram, box plot and behaviour graph, etc. (Sargent 1996).

In this paper, we aimed to validate the CLARA-SPT for the Ethiopian context using actual observed WS&S project data from Bahir Dar and Arba Minch towns. Since the tool had recently been developed, the comparison of the tool's results with real costs had not yet been assessed. Simultaneously we targeted the checking of design assumptions, correcting them if required to improve the prediction accuracy of the tool, and planned to provide remedial information for further development of the tool.

METHODS

Study area description

The study was conducted in the vicinities of Bahir Dar and Arba Minch towns, which are located at the north-western (11°37′N and 37°10′E) and south-western (06°00′N and 37°30′E) parts of Ethiopia, respectively. Bahir Dar, capital of the Amhara National Regional State (ANRS), is the most important administrative and economic centre of the region, where many infrastructure developments are currently going on. Arba Minch is the administrative town of the Gamo Gofa zone of the Southern Nations, Nationalities and Peoples Region (SNNPR). According to Ethiopia's town socio-economic classification Bahir Dar and Arba Minch are categorized under grade ‘A’ (i.e. towns enjoying high living standard with very high development potential) and grade ‘B’ (i.e. towns having high development potential but lower living standards at the present), respectively (MoWR 2006). Thus, construction material and labour cost price indexes in Arba Minch town vicinity are lower than in Bahir Dar. Because of the geomorphological formation, a shallow ordinary soil underlain by a hard soil stratum is dominant in the surroundings of Bahir Dar, which adversely influences excavation costs.

On the other hand, electric power interruption is a common phenomenon at most places in Ethiopia. For example, in the vicinity of Arba Minch the electric power is typically interrupted for 3–4 hours per day, while around Bahir Dar the duration ranges from 1.5 to 2 hours. Regarding the technology-specific construction materials concerned, the masonry wall type of septic tanks was commonly observed in the vicinity of Arba Minch, while both concrete and masonry wall types are commonly found in the Bahir Dar areas.

Data collection

Out of 38 WS&S technologies available in the CLARA-SPT, it was possible to collect design and cost-related data for 17 technologies: Spring development, Borehole, Disinfection, Surface reservoir, Elevated reservoir, Pumping station, Transport main, Distribution network, Urine diversion dry toilet (UDDT), Composting toilet, Faeces collection, Urine collection, Faecal sludge collection, Septic tank, Sanitary sewer, Sludge drying bed and Composting.

Even though we faced difficulty in accessing enough recorded information about recurrent costs, we succeeded in finding data for energy and/or chemical-intensive technologies such as Disinfection, Pumping station and Borehole. We only found aquifer hydraulic conductivity (Kf) values for a few spring sites at which the Kf values were found to be highly variable. However, recently accomplished research by Ketema & Langergraber (2015a) has shown the non-influential behaviour of Kf on the costs. During CLARA-SPT validation, therefore, an average value of Kf was fixed from the given range of 3 × 10−6 to 10−3 m/s, whenever a recorded Kf value was missing.

To evaluate the applicability of the tool under different circumstances one should consider technologies implemented at various levels of geographic and socio-economic diversity. All cost data were collected from contract agreement documents of already completed or ongoing projects. Noticeable deviations between contracted and completion costs of projects were recognised. According to the Ethiopian construction contract administration rule BaTCoDA (1987), a total cost variation of ± 15% from the contracted price is acceptable with the approval of project supervisor engineers. A cost variation from 15% to 25% is still acceptable with the additional approval of the project owners. The rule declares 25% as the maximum permissible variation in a valid contract.

The CLARA Simplified Planning Tool

The CLARA-SPT is designed to estimate and compare the life-cycle cost (LCC) of water supply and sanitation alternatives during pre-planning. An individual WS&S alternative comprises a set of technological options under the functional groups of ‘Water Source’, ‘Water Purification’, ‘Water Distribution’, ‘Waste Collection’, ‘Waste Treatment’ and ‘Reuse’. The technologies’ IIC, O&MC, RIC and revenues are linearly aggregated to estimate total LCC of the specific alternative. Thus, the LCC of an alternative is the sum of the present values of all individual cash flows of technologies included within it (Casielles Restoy et al. 2014). The tool estimates the LCC of technologies based on standard designs, bills of quantities (BoQs) and country-specific unit prices for which the database was created in 2013. The tool users (i.e. planners) are in charge of providing reliable information for global and technological input parameters of proposed alternatives in general and for each involved technology in particular. Recently, Ketema & Langergraber (2015a, b) identified important and non-influential input parameters by applying sensitivity analysis methods.

Model validation methods

From the structural setup of the CLARA-SPT, a technology-cost-focused validation was found to be the easiest and most efficient way to assess the overall performance of the tool. A number of goodness-of-fit measures were applied: Root-Mean-Square-Error (RMSE), Mean-Error (ME) and Mean-Percent-Error (MPE). Additionally, Theil's inequality coefficient was used (Theil 1961). The coefficient of determination (R2) was also computed to describe the proportion of the total variance in the observed data that can be represented by the simulator.

The RMSE quantifies the overall error of the simulation. Since the errors are squared before averaging them, RMSE gives relatively high weight to large errors compared to small errors (Cao et al. 2012). It is calculated by 
formula
1
where n is total number of observed samples, and XOi and XPi are the ith observed and simulated (predicted) values, respectively.
Both ME and MPE can indicate the existence of under- or over-estimation by the simulation of reality (Toledo & Koutsopoulos 2004). They are useful statistical indicators to measure the simulation error separately at each sample point rather than estimating aggregated value. MPE also provides an error magnitude of simulation with respect to observed data. ME and MPE are calculated by 
formula
2
 
formula
3
R2 is the square of the correlation coefficient (r), that describes the degree of collinearity between the observed and simulated values (Legates & McCabe 1999). It ranges from 0 to 1, with higher values indicating better agreement. It is given by 
formula
4
where and are the average of XO and XP, respectively.
Theil's inequality coefficient (U) is a standardized RMSE originally suggested as a systematic means of evaluating econometric model forecasts (Theil 1961). U provides information on the relative error of the tool simulation with respect to observed data and can be calculated by 
formula
5
U ranges from 0 to 1. U = 0 indicates the perfect fit of the model simulation to observed data whereas the worst possible fit is denoted by U = 1.

Setting a threshold value for simulation efficiency

As clearly stated in the user manual for CLARA-SPT (Casielles Restoy et al. 2014), the tool is developed to assist decision-making processes at the early stage of WS&S project planning. Often planners have struggled to estimate the economic value of various possible WS&S solutions created as the result of the pre-planning process. Actually, cost estimation at this early stage of planning can hardly be expected to be certain. Based on the intended purpose of the CLARA-SPT and cost variation provision in the Ethiopian contract administration rule BaTCoDA (1987), the accuracy level of the simulation result can be categorized into good (|MPE| ≤ 15% and U ≤ 0.15), fair (15% < |MPE| ≤ 25% and 0.15 < U ≤ 0.25) and poor (|MPE| > 25% and U > 0.25). For example, less than ± 15% magnitude of MPE and U between observed and simulated values of specific technology indicates a good simulation efficiency of the tool. This should be supplemented with relatively lower values of bias (RMSE and ME) and a higher value of R2 within its significance level.

RESULT AND DISCUSSION

Initial investment costs

The statistical analysis results of the IIC of water supply and sanitation technologies in Tables 1 and 2 reveal the good simulation accuracy of the CLARA-SPT (i.e. |MPE| ≤ 15% and U ≤ 0.15) for four water-supply technologies (i.e. Spring development, Borehole, Elevated reservoir and Distribution network) and six sanitation technologies (i.e. Faeces collection, Urine collection, Septic tank, Sanitary sewer, Sludge drying bed and Composting). On the contrary, the IIC simulation accuracy for Disinfection, Surface reservoir, Pumping station, UDDT, Composting toilet and Faecal sludge collection was not satisfactory. The significance level of correlation between observed and simulated values of these WS&S technologies were also estimated and are shown in Tables 1 and 2. For instance, for Spring development the correlation was found significant at 99% and 95% confidence intervals for Bahir Dar and Arba Minch vicinities, respectively.

Table 1

Statistical parameters and sample number (n) of observed and simulated water supply technology IICs for Bahir Dar and Arba Minch (bold italic: poor estimation accuracy)

Statistical parameters of simulation and observation
TechnologyAreaRMSE (€/person)ME (€/person)MPE (%)R2Un
Spring development BD 0.96 0.02 3.13 0.92** 0.05 10 
AM 0.83 0.35 11.28 0.56* 0.12 
Borehole (8 inch) BD 5.30 −0.13 5.80 0.97** 0.06 10 
AM 4.58 −0.45 −12.19 0.82* 0.07 
Disinfection BD + AM 1.27 1.10 89.45 0.73** 0.41 8 
Surface reservoir BD 1.86 1.36 20.80 0.66** 0.10 24 
AM 4.47 4.13 89.43 0.90ns 0.32 4 
Elevated reservoir BD + AM 6.30 0.95 7.69 0.87** 0.10 10 
Pumping station BD + AM 8.84 7.63 54.60 0.70** 0.55 14 
Transport main (PVC, HDPE & GI) BD 2.66 0.41 8.00 0.83** 0.13 14 
AM 2.30 −0.43 −14.11 0.99** 0.06 
Distribution network BD 3.04 0.08 11.23 0.96** 0.01 16 
AM 1.95 −0.58 −3.51 0.98** 0.05 11 
Statistical parameters of simulation and observation
TechnologyAreaRMSE (€/person)ME (€/person)MPE (%)R2Un
Spring development BD 0.96 0.02 3.13 0.92** 0.05 10 
AM 0.83 0.35 11.28 0.56* 0.12 
Borehole (8 inch) BD 5.30 −0.13 5.80 0.97** 0.06 10 
AM 4.58 −0.45 −12.19 0.82* 0.07 
Disinfection BD + AM 1.27 1.10 89.45 0.73** 0.41 8 
Surface reservoir BD 1.86 1.36 20.80 0.66** 0.10 24 
AM 4.47 4.13 89.43 0.90ns 0.32 4 
Elevated reservoir BD + AM 6.30 0.95 7.69 0.87** 0.10 10 
Pumping station BD + AM 8.84 7.63 54.60 0.70** 0.55 14 
Transport main (PVC, HDPE & GI) BD 2.66 0.41 8.00 0.83** 0.13 14 
AM 2.30 −0.43 −14.11 0.99** 0.06 
Distribution network BD 3.04 0.08 11.23 0.96** 0.01 16 
AM 1.95 −0.58 −3.51 0.98** 0.05 11 

ns: a not significant correlation.

** and *: significance levels of P < 0.01 and P < 0.05, respectively.

BD and AM: Bahir Dar and Arba Minch, respectively.

The same notation is used for all tables.

Table 2

Statistical parameters and sample number (n) of observed and simulated sanitation technology IICs for Bahir Dar and Arba Minch (bold italic: poor estimation accuracy)

Statistical parameters of simulation and observation
TechnologyAreaRMSE (€/person)ME (€/person)MPE (%)R2Un
UDDT BD 22.81 21.08 58.24 0.85** 0.20 10 
AM 34.91 33.19 146.82 0.65** 0.37 40 
Fossa alterna (Composting toilet) BD 22.15 11.01 11.90 0.01ns 0.18 15 
AM 21.84 17.31 54.70 0.01ns 0.23 15 
Faeces collection (small & big truck) BD + AM 0.41 −0.22 −6.99 1.00** 0.01 73 
Urine collection (Vacuum truck) BD + AM 8.57 −3.28 −4.51 0.96** 0.08 51 
Faecal sludge collection (Vacuum truck) BD + AM 2.05 1.66 35.07 1.00ns 0.04 46 
Septic tank BD 9.11 1.73 2.63 0.96** 0.03 10 
AM 28.96 24.86 103.69 0.95ns 0.38 10 
Sanitary sewer BD 1.22 −1.27 −9.20 0.98** 0.03 23 
AM 0.48 −0.29 −3.92 0.99** 0.01 24 
Sludge drying bed BD + AM 0.48 0.16 5.45 0.36ns 0.08 
Composting BD 0.37 −0.09 −0.07 0.99** 0.01 26 
AM 3.42 −0.08  0.40 0.99** 0.02 26 
Statistical parameters of simulation and observation
TechnologyAreaRMSE (€/person)ME (€/person)MPE (%)R2Un
UDDT BD 22.81 21.08 58.24 0.85** 0.20 10 
AM 34.91 33.19 146.82 0.65** 0.37 40 
Fossa alterna (Composting toilet) BD 22.15 11.01 11.90 0.01ns 0.18 15 
AM 21.84 17.31 54.70 0.01ns 0.23 15 
Faeces collection (small & big truck) BD + AM 0.41 −0.22 −6.99 1.00** 0.01 73 
Urine collection (Vacuum truck) BD + AM 8.57 −3.28 −4.51 0.96** 0.08 51 
Faecal sludge collection (Vacuum truck) BD + AM 2.05 1.66 35.07 1.00ns 0.04 46 
Septic tank BD 9.11 1.73 2.63 0.96** 0.03 10 
AM 28.96 24.86 103.69 0.95ns 0.38 10 
Sanitary sewer BD 1.22 −1.27 −9.20 0.98** 0.03 23 
AM 0.48 −0.29 −3.92 0.99** 0.01 24 
Sludge drying bed BD + AM 0.48 0.16 5.45 0.36ns 0.08 
Composting BD 0.37 −0.09 −0.07 0.99** 0.01 26 
AM 3.42 −0.08  0.40 0.99** 0.02 26 

The result also shows poor simulation accuracy for the masonry wall type of Septic tank which has commonly been constructed in the vicinity of Arba Minch (i.e. MPE = 103.69% and U = 0.38). This is because CLARA-SPT internally assumed a concrete wall type of Septic tank as the standard design to minimize seepage risk. Hence, due to the higher cost of concrete the simulation overestimates the IIC of masonry-wall Septic tanks for Arba Minch. In Bahir Dar the actual cost of the concrete-wall type of Septic tank was represented well by the tool.

The tool underestimates the IIC of Transport main when the observed sample set contained ductile cast iron (DCI) pipes, where MPE was −29.2% and −46.0% for Bahir Dar and Arba Minch, respectively (Figure 1). However, the simulation performed much better by excluding the DCI pipes from the pipe network, where MPE was improved to 8.0% and 14.1% for Bahir Dar and Arba Minch, respectively. In view of this, it has been understood that the higher unit price of DCI pipes significantly influences the performance of the simulation, since CLARA-SPT neither internally includes various pipe materials nor provides a chance for planners to choose the pipe material. On the other hand, the tool internally assumes ordinary soil excavation unit price as a default value, but in the excavation history of projects already implemented around Bahir Dar about 20% and 5% of the total trench excavation volumes have been soft and hard rocks, respectively. From implemented projects, unit costs of soft rock and hard rock excavation were about four times and nine times the ordinary soil excavation rate, respectively. Therefore, for Bahir Dar areas an IIC increment factor of 200% was applied for Transport main, Distribution network and Sanitary sewer in order to include the extra excavation cost of soft and hard rock. In the vicinity of Arba Minch ordinary soil excavation was commonly experienced, where there was no need to apply a correction factor.
Figure 1

Comparison of observed and simulated IICs of Transport main without DCI pipe and DCI pipe Transport main.

Figure 1

Comparison of observed and simulated IICs of Transport main without DCI pipe and DCI pipe Transport main.

The reason for the poor simulation accuracy of UDDT and Composting toilet is that the tool developer internally fixed Persons per UDDT and Persons per composting toilet at 15 persons, but these parameters were already identified as important input parameters (Ketema & Langergraber 2015b). In reality, the UDDT and Composting toilet were designed for 5 to 20 PE per unit. This discrepancy made the tool overestimate the IIC of both technologies (Table 2). Similarly in Faecal sludge collection, an important parameter of Volume per pick-up point, which is equal to Cesspit volume, is internally fixed to 18 m3, but cesspits show a wide range of volume in reality. As a result, the simulation accuracy of Faecal sludge collection was poor. Therefore, the tool developer should make these above-mentioned important parameters available for tool users to provide their own accurate values instead of internally fixing them.

The smaller values of RMSE = 0.41 and ME = −0.22 of Faeces collection gave the relatively higher value of MPE = −6.99% than Urine collection's RMSE = 8.57, ME = −3.28 and MPE = −4.51%. Similarly, the smallest RMSE = 0.64 and ME = −0.11 values of Disinfection were not supported by the lowest values of MPE = 14.19% and U = 0.25. From these we can say that the measurements of MPE and U are relatively conservative compared with other goodness-of-fit indicators. Moreover, it is challenging to set some standard threshold values for RMSE and ME, since they are subject to the unit (e.g. €/person). On the other hand, a low value of RMSE, MPE or U never guarantees a large value of R2 (e.g. Sludge drying bed), but in the reverse case the largest R2 values were always supported by smaller values of U. These indicated that the precision of statistical measures is much better when the value of R2 is higher. Nevertheless, a high value of R2 does not always indicate the correctness (e.g. Faecal sludge collection). Based on this experience, we conclude that it is difficult to evaluate the tool's simulation accuracy based only on one of these statistical indicators.

From the results, we saw different levels of error propagation and simulation performance for Bahir Dar and Arba Minch. Relatively, the simulation accuracy of the tool is better for Bahir Dar than Arba Minch, except for those technologies that have considerably higher excavation volume at extended distance (e.g. Distribution network and Sanitary sewer). That is because of the extra excavation cost of the hard soils usually presented around Bahir Dar town.

Correction of cost functions

Even though the original IIC simulation performance of Disinfection, Surface reservoir, and Pumping station were poor (Tables 1 and 2), their efficiency could be improved by applying appropriate corrective actions. In the following paragraphs, the main causes of poor simulation and the kinds of corrective measures being taken are explained briefly.

The CLARA-SPT's cost function of Disinfection overlooked the construction costs of the ‘chlorination house’ and ‘electro mechanical accessories’, despite their being vital for operating the system. In addition, the cost of a ‘Chlorine solution preparation plastic tank’ was taken as similar to that of an ‘Ordinary plastic tank’. Even though both are plastic tanks, a chemical storage tank costs more than the ordinary plastic tank. By including these missed components in the BoQ of Disinfection a new cost function was generated and applied. As the result, the modified cost function improves the IIC simulation efficiency of Disinfection from a poor (i.e. MPE = −89.93% and U = 0.41) to fair level (i.e. MPE = 14.91% and U = 0.25) (Figure 2 and Table 3).
Table 3

Statistical parameters and sample number (n) of observed and simulated water supply technology IICs for Bahir Dar and Arba Minch using the corrected cost functions

Statistical parameters of simulation and observation
Technology nameAreaRMSE (€/person)ME (€/person)MPE (%)R2Un
Disinfection BD + AM 0.64 −0.11 14.91 0.73** 0.25 
Surface reservoir BD 1.48 −0.99 −12.34 0.66** 0.11 24 
AM 1.20 0.81 13.08 0.90ns 0.10 
Pumping station BD + AM 3.67 −0.79 7.63 0.60** 0.15 14 
Statistical parameters of simulation and observation
Technology nameAreaRMSE (€/person)ME (€/person)MPE (%)R2Un
Disinfection BD + AM 0.64 −0.11 14.91 0.73** 0.25 
Surface reservoir BD 1.48 −0.99 −12.34 0.66** 0.11 24 
AM 1.20 0.81 13.08 0.90ns 0.10 
Pumping station BD + AM 3.67 −0.79 7.63 0.60** 0.15 14 
Figure 2

Comparison of observed and simulated IICs of Disinfection for original and corrected cost functions (CF).

Figure 2

Comparison of observed and simulated IICs of Disinfection for original and corrected cost functions (CF).

The internal design assumptions for the Surface reservoir includes an extra 65 m3 reservoir volume for two hours firefighting, which is never considered in the implemented projects in Ethiopia. By omitting this additional capacity from the Surface reservoir BoQ, the high MPE of 20.80% and 89.43% for Bahir Dar and Arba Minch were reduced to −12.34% and 13.08%, respectively, which represent reality better for both sites (Figure 3 and Table 3).
Figure 3

Comparison of observed and simulated IICs of Surface reservoir for original and corrected cost functions.

Figure 3

Comparison of observed and simulated IICs of Surface reservoir for original and corrected cost functions.

In the CLARA-SPT, the Pumping station cost function was developed for 24/7 electric power availability, which does not happen in reality in Ethiopia. Rather, the additional power source of a diesel generator is commonly installed to operate the pumping station during electric power interruptions. To include the extra cost incurred due to the presence of a diesel generator, cost increment factors were introduced by Ketema et al. (2014) as a function of generator power and daily working hours. On the other hand, the tool adopted the unit rate price of pumps from European pump suppliers’ catalogues; this is because of the lack of country-specific data at the time of the tool's development. These assumed price data were examined and found to be about half of the actual equivalent pump price in Ethiopia. Accordingly, the assumed unit prices of all pumps in the tool were corrected. Costs of standby pumps were not included as part of the cost estimation, even though it is mandatory to have at least one additional pump at every pumping station for better working efficiency. This needs to be included to represent the actual pumping station setup. The Pumping station cost function was corrected considering the above-mentioned modifications and thus the poor simulation performance (i.e. MPE = −54.60% and U = 0.56) was improved to good (i.e. MPE = 7.63% and U = 0.15) (Figure 4 and Table 3).
Figure 4

Comparison of observed and simulated IICs of Pumping station for original and corrected cost functions.

Figure 4

Comparison of observed and simulated IICs of Pumping station for original and corrected cost functions.

Operation and maintenance (O&M) costs

Design assumptions in the tool to estimate annual O&M cost and reinvestment cost were evaluated and verified based on expert knowledge. We faced difficulty in finding exact costs spent to repair unforeseen damage of WS&S infrastructure. Statistical comparison between observed and simulated data for the three relatively energy- and/or chemical-intensive technologies like Disinfection, Pumping station and Borehole are summarized in Table 4. Increment correction factors developed by Ketema et al. (2014) were applied for a pumping station in order to include the diesel generator's operation cost.

Table 4

Statistical parameters and sample number (n) of observed and simulated values for annual O&M costs

 Statistical parameters of simulation and observation
Technology nameAreaRMSE (€/person)ME (€/person)MPE (%)R2Un
Borehole (8 inch) BD 0.60 −0.21 −6.03 0.53** 0.15 22 
  AM 0.51 −0.32 9.50 0.10ns 0.07 
Disinfection (Ca(OCl)2BD + AM 0.11 0.11 24.82 0.95** 0.11 
(Na(OCl)) BD + AM 0.15 0.15 21.83 0.93** 0.10 
Pumping station BD + AM 0.37 0.04 6.32 0.73** 0.11 14 
 Statistical parameters of simulation and observation
Technology nameAreaRMSE (€/person)ME (€/person)MPE (%)R2Un
Borehole (8 inch) BD 0.60 −0.21 −6.03 0.53** 0.15 22 
  AM 0.51 −0.32 9.50 0.10ns 0.07 
Disinfection (Ca(OCl)2BD + AM 0.11 0.11 24.82 0.95** 0.11 
(Na(OCl)) BD + AM 0.15 0.15 21.83 0.93** 0.10 
Pumping station BD + AM 0.37 0.04 6.32 0.73** 0.11 14 

We can see the good simulation performance of the tool for the O&M cost values of Borehole and Pumping station. However, insignificant correlation has been estimated for Borehole for the vicinity of Arba Minch vicinity that, which might be due to the small sample size (n = 5). The tool fairly represents the reality of Disinfection's O&M cost of using either calcium hypochlorite (Ca(OCl)2) or sodium hypochlorite (Na(OCl)). In addition the values of RMSE and ME of the three technologies are favourable for the good simulation performance of the tool.

Reinvestment cost assumption evaluation

Reinvestment costs are directly linked with the lifetime of the individual technologies' major components and the planning horizon of WS&S alternatives. The tool assumes a lifetime of 50 years for concrete work, 25 years for civil work, 10 years for electrical and mechanical equipment like pumps; and 15 years for valve fittings and vehicles. For example, a Pumping station has two reinvestment periods, the first one involving the replacement cost for the pump and its accessories after 10 years of service and the second involving the replacement cost of the house, which can serve for 25 years. In contrast to the regular assumption, the tool assumes a lifetime of 25 years for concrete works for Spring development, Borehole, and River water extraction, which leads to illogical computational agreement. Thus, to maintain the computational consistency of the tool, the developers should correct the identified limitations and maintain uniform reinvestment timing for identical items and works. Apart from these limitations, the tool fairly estimates the net present value of reinvestment costs according to the selected planning horizon, the net interest rate and the lifetimes of technologies and their components.

CONCLUSION

From the study, we concluded that validation of CLARA-SPT cost estimation is an essential prior step to training and improving the computational efficiency of the tool with respect to the real costs. Model validation is also very critical for further development of the tool. Unfortunately, there is no specific set of validation tests that can be easily applied to determine the ‘correctness’ of the simulation, rather as many measures as possible should be applied for better crosschecking and confidence.

The simulation accuracy of the CLARA-SPT is acceptable for most technologies except for the poor accuracy of three sanitation technologies (Table 5). Poor simulation performance for a technology is reflected in the LCC of systems, where errors of all involved technologies accumulate. Therefore, in order to estimate how well representative are the LCC of WS&S alternatives, it is important to validate the simulation of each involved technology within definite boundaries. Hence, the authors strongly recommend performing remedial action on the named internally fixed input parameters, which should be available for users. By doing so, the poor simulation accuracy of UDDT, Composting toilet and Faecal sludge collection (Table 5) can be improved. Generally, the CLARA-SPT provides valuable lifetime economic information on WS&S alternatives to support the early-stage planning process of the sector.

Table 5

Summarized validation results for tested WS&S technologies

Water supply
Sanitation
TechnologyOverall accuracyTechnologyOverall accuracy
Spring development Good UDDT Poor 
Borehole (8 inch) Good Composting toilet Poor 
Disinfection Fair Faeces collection Good 
Surface reservoir Good Urine collection Good 
Elevated reservoir Good Faecal sludge collection Poor 
Pumping station Good Septic tank Good 
Transport main (excluding DCI) Good Sanitary sewer Good 
Sludge drying bed Good 
Distribution network Good Composting Good 
Water supply
Sanitation
TechnologyOverall accuracyTechnologyOverall accuracy
Spring development Good UDDT Poor 
Borehole (8 inch) Good Composting toilet Poor 
Disinfection Fair Faeces collection Good 
Surface reservoir Good Urine collection Good 
Elevated reservoir Good Faecal sludge collection Poor 
Pumping station Good Septic tank Good 
Transport main (excluding DCI) Good Sanitary sewer Good 
Sludge drying bed Good 
Distribution network Good Composting Good 

ACKNOWLEDGEMENTS

The authors acknowledge APPEAR (Austrian Partnership Programme in Higher Education and Research for Development) for providing financial support for the first author. We are thankful to the CLARA project for providing full access to the CLARA-SPT database.

REFERENCES

REFERENCES
BaTCoDA
1987
Standard Conditions of Contract for Construction of Civil Work Projects
.
Ethiopian Building and Transport Construction Design Authority
,
Addis Ababa
,
Ethiopia
.
Cao
H.-X.
Hanan
J. S.
Liu
Y.
Liu
Y.-X.
Yue
Y.-B.
Zhu
D.-W.
Lu
J.-F.
Sun
J.-Y.
Shi
C.-L.
Ge
D.-K.
Wei
X.-F.
Yao
A.-Q.
Tian
P.-P.
Bao
T.-L.
2012
Comparison of crop model validation methods
.
Journal of Integrative Agriculture
11
(
8
),
1274
1285
.
Casielles Restoy
R.
Lechner
M.
Langergraber
G.
2014
CLARA Simplified Planning Tool v1.5 User Manual, http://clara.boku.ac.at/index.php/planning-tool-2 (accessed 24 November 2014)
.
Ketema
A. A.
Dalecha
T.
Assefa
E.
2014
Water supply and sanitation system pre planning and CLARA Simplified Planning Tool application for Arba Minch, Ethiopia
.
Sustainable Sanitation Practice
19
,
44
52
,
http://www.ecosan.at/ssp (accessed 24 November 2014
).
Ketema
A. A.
Langergraber
G.
2015a
Sensitivity analysis for water supply parameters of CLARA Simplified Planning Tool using three complementary methods
.
Journal of Water Supply: Research and Technology – AQUA
64
(
4
),
391
403
.
Ketema
A. A.
Langergraber
G.
2015b
Sensitivity analysis of CLARA Simplified Planning Tool using the Morris screening method
.
Water Science and Technology
71
(
2
),
234
244
.
Lechner
M.
Pressl
A.
Langergraber
G.
2014
The CLARA simplifing planning tool
.
Sustainable Sanitation Practice
19
,
29
35
,
http://www.ecosan.at/ssp (accessed 24 November 2014
).
Mayer
D. G.
Butler
D. G.
1993
Statistical validation
.
Ecological Modelling
68
,
21
32
.
Moriasi
D. N.
Arnold
J. G.
Liew
M. W. V.
Bingner
R. L.
Harmel
R. D.
Veith
T. L.
2007
Model evaluation guidelines for systematic quantification of accuracy in watershed simulations
.
Transactions of the ASABE
50
(
3
),
885
900
.
MoWR
2006
Urban Water Supply Design Criteria
.
Ethiopian Ministry of Water Resources
,
Addis Ababa
,
Ethiopia
.
Sargent
R. G.
1996
Some subjective validation methods using graphic display of data
. In:
Proceedings of the 1996 Winter Simulation Conference
(
Charnes
J. M.
Morrice
D. J.
Brunner
D. T.
Swain
J. J.
, eds).
IEEE
,
New York
,
USA
, pp.
345
351
.
Sargent
R. G.
2005
Verification and validation of simulation models
. In:
Proceedings of the 2005 Winter Simulation Conference
(
Kuhl
M. E.
Steiger
N. M.
Armstrong
F. B.
Joines
J. A.
, eds).
IEEE
,
New York
,
USA
, pp.
130
143
.
Sargent
R. G.
2011
Verification and validation of simulation models
. In:
Proceedings of the 2011 Winter Simulation Conference
(
Jain
S.
Creasey
R. R.
Himmelspach
J.
White
K. P.
Fu
M.
, eds).
IEEE
,
New York
,
USA
, pp.
183
198
.
Theil
H.
1961
Economic Forecast and Policy
.
North Holland Publishing Company
,
Amsterdam
,
The Netherlands
.
Toledo
T.
Koutsopoulos
H. N.
2004
Statistical validation of traffic simulation models
.
Transportation Research Record: Journal of the Transportation Research Board
1876
(
1
),
142
150
.
WHO/UNICEF
2014
Progress on Drinking Water and Sanitation 2014 Update
.
World Health Organization and UNICEF
,
Geneva
,
Switzerland
.