Support tools to predict the critical structural condition of uninspected pipes for case studies of Germany and Colombia

Urban drainage systems present alarming rates of aging and deterioration in both developed and developing countries (Osman 2012; Ferguson et al. 2013). Traditionally it has been economically feasible to apply reactive management strategies, mainly repairing when failures occur; however, this strategy will become less viable as the system age and the funding gap increase (Rokstad & Ugarelli 2015). Therefore, urban system stakeholders are facing critical challenges to apply proactive management strategies while simultaneously considering the diversity of actors and constraints involved (budget limitations, environmental regulations and municipal water infrastructure benefits) (Baik et al. 2006; Cardoso et al. 2012; Younis & Knight 2012; Caradot et al. 2018). The lack of information about sewer condition limits rehabilitation strategies. Therefore, deterioration models have been developed to forecast the evolution of the system according to its current and past state (Caradot et al. 2015). Most of the developed deterioration models in sewer asset management are based on statistical (e.g., Wright et al. 2006) and machine learning (e.g., Harvey & McBean 2014) approaches. In the previous investigation by the authors (Hernández et al. 2017), Random Forest (RFs) and Logistic Regression (LR) were the methods more suitable (among models based on five classification and regression methods) to be used as prediction tools to support sewer asset management in Bogota. The exploration of these two models were carried out in other cities: according to Wright et al. (2006), models based on LR were suitable to predict the pipes in deficient and acceptable conditions in Vallejo (California, USA); while Harvey & McBean (2014) have explored models based on RF with an excellent area under the ROC curve of 0.81 for Guelph (Ontario, Canada). As well as these two experiences, the majority of deterioration models have been explored for a specific case study. The questions that lead to the development of this work are: (i) Could the application of these methods or models be useful for the prediction of the structural condition in case studies with different topographic conditions, soil uses, and demographics (Chornet 1994), as well as different quality and quantity datasets, inspection guidelines and the level of technical expertise (Caradot et al. 2018)?; and in the right case, (ii) Are the prediction results of these models appropriate for the same management objectives? Therefore, this work focuses on applying two deterioration models based on different approaches (RF and LR) for two case studies with different characteristics.

For the city of Germany (250,000 inhabitants), after data cleanup, 37,506 consistent CCTV inspections (representing 1,476 km) were linked to 23,958 sewer pipes. The structural condition of the inspected pipes has been evaluated using the French classification methodology RERAU (Ahmadi et al. 2014). A structural condition class is assigned to each sewer segment (from manhole to manhole) on a four-grade scale (1 to 4, with 4 being the worst condition where immediate rehabilitation is needed).

The database of the Colombian city (6.7 billion inhabitants) contains, after data cleanup, 4,633 consistent CCTV inspections (representing 245 km) linked to 4,327 sewer pipes. The structural condition of the inspected pipes has been evaluated using the local sewer assessment methodology NS–058 (EAAB 2001). It attributes a score from 1 to 5 to each inspected pipe, 5 being the worst condition. The original scores 3 and 4 were grouped to compare the results of both cities with the same number of conditions.

According to the literature, some experiences (Wright et al. 2006; Harvey & McBean 2014) have shown that RF and LR are methods with a high performance in predicting the sewer condition from the physical characteristics of assets.

RF is a machine learning algorithm based on decision trees. A decision tree consists of paths and nodes, with each node using a rule to decide between two or more paths. A rule is typically in the form of ‘If A then do B’, where A is a condition related to the descriptors of the input data and B is the step on the path through the trees. The last rule gives the classification of the input data example. Several decision trees are developed using a random selection of inputs and random feature selection at each node to grow the trees. In the end, the output of each tree is a category of classification, so the output of RF is the category most often classified among all trees (Breiman 2001).

The deterioration models RF and LR have been calibrated and validated using CCTV and GIS data of the German and the Colombian cities respectively. Two thirds of the data were chosen randomly, such as calibration data, with the rest being validation data.

The sewer characteristics, taken into account as input variables in the models (age, material, type of effluent, depth, diameter, slope, and type of road), were the same for both cases: these variables were chosen according to the suggested physical sewer characteristics in Davis et al. (2001). Likewise, the output variable was the structural condition (four-grade scale) which is described in the sub-section ‘Case studies’ (see Figure 1).

Three techniques were used to analyze the prediction results from these two methods: (i) ROC space (Brown & Davis 2006); (ii) Performance Curve (Kästner et al. 2015); and (iii) deviation analysis (Caradot et al. 2016). The first one measures the True Positive Rate (TPR) vs. False Positive Rate (FPR) of the structural condition's predictions, and from these, it is possible to calculate the PLR (Positive Likelihood Ratio), which quantifies how likely it is to have a positive prediction than a negative one (TPR/FPR). The second one identifies which percentage of pipes predicted as being in a critical condition was actually found to be in that condition (the first's pipes' percentages have the highest probability to be on it). The last one analyses the deviation between the proportion of observed and predicted pipes of the critical structural condition for each age period.

According to the ROC space's analysis of the critical condition's prediction results by each method (Table 1), it was found that the models of both approaches have a PLR value higher than 1 for the Colombian city, which means that both models give better predictions than a random model (TPR > FPR). However, for the German city, only the RF model gives a PLR value higher than 1; the PLR value was assumed to be zero for the LR model. In addition, for each model it was found that: (i) LR prediction shows that the PLR value is higher for the Colombian city (5.43) than the German one (0.02); and (ii) RF prediction shows the opposite result: 3.8 and 5.2 for the Colombian and German cities, respectively. The RF results are interesting in that they highlight the importance of PLR, because even though the proportion of pipes predicted correctly for the Colombian city's case is higher (TPR = 0.57) than the German city's one (TPR = 0.26), likewise the proportion of pipes predicted wrongly is also higher for the Colombian city (FPR = 0.15) than for the German city (FPR = 0.05).

Therefore, ROC space analysis of the critical condition predictions shows that LR is more adapted for the prediction of the Colombian city's sewer system's critical conditions, and RF for the German one.

The performance curves, presented in Figures 2 and 3, show that both methods (RF and LR) exhibit similar behavior, but produce different results for each case study.

According to the performance curve analysis of the RF predictions, shown in Figure 2 for both cities, RF correctly predicts 63% of pipes in a critical condition (or 83% of pipes in critical and poor conditions) for the first 10% of pipes with a high probability to be in this condition, for the Colombian city's case (Figure 2, left). Whereas, for the German city, RF predicts 33% of the first 10% of pipes (or 63% of pipes in critical and poor conditions) with a high probability to be in critical condition, correctly (Figure 2, right). Although the curves', behavior concurs to the fact that the probability of a pipe being in a critical condition decreases along with the percentage of success, the prediction results show a greater accuracy in identifying those pipes in a critical condition for the Colombian city's case.

On the other hand, as can be seen in Figure 3, the performance curves of LR prediction results for both cities show lower accuracy than the RF prediction results: according to the bar plots on the right side of each performance curve (Figure 3), which represent the 10% of pipes with the highest probability to be in a critical condition, the accuracy is around 62% and 27% for the Colombian and the German cases, respectively.

For the Colombian city's case, the accuracy of identifying those pipes with a high probability to be in a critical condition that are actually in a critical condition are similar for both RF and LR. However, for the German city's case, this accuracy is worse than a random selection for both models (RF and LR), which matches with the TPR results shown in Table 1.

The above performance curves' analysis shows that the models based on RF and LR could be useful in identifying the critical conditions correctly for the Colombian's case: the Colombian city's stakeholders could make strategic plans for rehabilitation, choosing the 10% of pipes with the highest probability of being in a critical condition, ensuring a success rate of over 60% in finding pipes in a critical condition and more than 83% (according to RF) of pipes in a critical or poor structural condition (red and orange stripes, Figure 2 left).

The deviation analysis of the prediction results of both methods is shown in Figure 4 and 5 (RF and LR, respectively). According to the analysis of RF, the Colombian's city's case results show higher deviation in the prediction of critical condition for each period of 10 years (Figure 4, left) than the German city (Figure 4, right). It is important to observe that for the Colombian city, for pipes aged between 40 and 50 years, the model overestimates the critical condition, predicting these pipes to be in a better condition (orange, yellow and green stripes), but for the younger pipes (aged <20 years) and 50–60-year-old pipes the model underestimates the critical condition, predicting some pipes to be in a critical condition when they are not. According to the distribution conditions of both case studies, it is possible to observe (top graphs of Figures 4 and 5) that the deterioration depends on the age, making it directly proportional to the criticality of the assets. This behavior is observed for the German case (deviation lower than 5%) and the Colombian case for assets younger than 60 years old (deviation lower than 7%). Therefore, the model represents the same behavior in the prediction. However, in the Colombian case, for pipes older than 60 years old this behavior changes, and the model underestimates the prediction of these pipes. The authors assume that the atypical behavior of the pipes older than 60 years for the Colombian case depends on the reliability of the data, the old construction methods, the lack of information about the rehabilitation dates and if these assets have been rehabilitated but not reported. These gaps should be explored in future research works.

On the other hand, the deviation analysis of LR (Figure 5) shows higher deviations for both cities (Figure 5) compared with those obtained with RF (Figure 4). Nevertheless, the deviation is still higher for the Colombian city's case. For the Colombian city's case, LR also tends to overestimate pipes whose ages are between 40 and 50 years, while pipes younger than 30 years and older than 60 years-old pipes are underestimated. For the German city's case, the deviation on each period is lower than ±5%, except for 70-year-old pipes.

Figure 5 shows a clear relationship between the distributions of conditions and the pipes' age, in particular for the German case. However, even if LR's prediction represents this behavior, it is not as accurate as RF's prediction.

According to the general prediction's approach, both methods are suitable to predict the structural condition; however, it depends on each case study. According to the ROC space analysis, RF was the method with a higher effectiveness rate (PLR) to predict the critical structural condition for the German city's case, while LR was the suitable one for the Colombian city's case. Nevertheless, in the analysis with the performance curve and deviation analyses, RF achieved adequate results in both cases: based on the performance curve analysis, RF was appropriate for identifying the pipes in a critical condition with an accuracy of 63% for the Colombian city's case, and based on the deviation analysis, RF had the lowest deviation (<± 5%) on each 10-year period's pipes for the German city's case.

The reason that RF prediction results are more appropriate in both cases lies in the fact that RF does not expect linear features or direct interactions, as LR does. Although the PLR's index showed that LR achieves a more accurate predictability for the Colombian city's case, there were fewer pipes predicted to be in a critical condition (TPR = 0.38), while for the same case study, the TPR was around 0.57 for RF.

Since the RF prediction results are different for both case studies, it is possible to analyze in which way the stakeholders could take advantage of these predictions: (i) for the German city's case, the stakeholders could design strategic budget plans that have an approach based on the number of pipes that will be in critical condition taking into account their age; and (ii) for the Colombian city's case, the stakeholders could develop support tools to predict the current structural condition of uninspected sewers and to prioritize the management of those in the worst condition.

Two methods (RF and LR) were chosen to predict the critical condition of sewer pipes of two cities with different contexts (a Colombian city and a German city) to analyze their prediction behavior.

Three techniques were used to examine the results obtained: (i) a general prediction approach showing the rate of likelihood to predict the critical condition correctly (PLR index); (ii) the accuracy in identifying which pipes are actually in a critical condition (performance curve); and (iii) a deviation analysis of critical condition prediction for pipes according to their age.

RF was the model that could be useful as a support tool in the sewer asset management of both cases studies. The difference lies in the benefits that these predictions can give when making decisions for different management objectives: (i) for the German city, the prediction results could be useful for designing strategic investment plans in order to know the quantity of pipes that the utility should rehabilitate each year, while (ii) for the Colombian city, the predictions are appropriate to make decisions concerning inspection or rehabilitation plans, since the probability to identify the sewer's assets in a critical condition (C4) correctly (according to the analysis of the sample of the 10% of sewers with highest probability to be in this condition) is around 63% and could be 83% if the stakeholders also consider in these plans the misclassification of those pipes in a bad structural condition (C3).

The capacity of prediction could be higher for both cities if previous studies of the sewer characteristics or environmental characteristics that could influence the structural condition were taken into account, as well as an optimization of the hyperparameters for the RF method.

This research is supported by the program PROCOL, bilateral convention between DAAD in Germany (Proposal title: ‘Development of innovative tools to support efficient sewer asset management strategies in Germany and Colombia) and COLCIENCIAS in Colombia (Colciencias –Pontificia Universidad Javeriana ‘Contrato de Financiamiento de Recuperación Contingente de Movilidad Internacional No. 646 del 2015’ and Project ID: 6725 - PRY ID – 6853 - Proposal title: ‘Herramientas de gestión proactiva de alcantarillados adapatadas a diferentes contextos de gestión patrimonial’) including funding from the German Federal Ministry of Education and Research (BMBF).

Furthermore, the authors would like to thank COLCIENCIAS and PUJ for supporting one of the authors in her Ph.D. studies (‘Convocatoria 727 del 2015- Apoyo doctorados nacionales’).