Validation of wastewater data using artificial intelligence tools and the evaluation of their performance regarding annotator agreement

To limit the pollution of natural resources, the stakeholders in the water sector must be mobilized for pragmatic management of wastewater facilities and rigorous control of wastewater discharges. Hence, it is important to know the functioning and structural state of the wastewater system and to prevent or identify malfunctions in order to orient and implement an operating and/or investment programme that responds to a logic of continuous improvement and sustainable management (van Daal et al. 2017). Overflow monitoring allows sustainable management based on continuous metrology (flow measurements) or discontinuous metrology with periodicity (sampling to establish pollution balances, etc.). Information is thus obtained on the direct discharges of the wastewater system during rainy events. Considering the number of equipped points, a large and diversified park of sensors should be operated and maintained. The plethora of sensors in place with a high sampling frequency and acquisition systems with innovative and wireless data transfer technologies leads to the integration of wastewater systems in a smart city perspective (Dogo et al. 2019b). The goal is therefore to optimize the strategies for processing and exploiting the massive amount of measurements made within the framework of the Internet of Things (IoT) in order to get the most out of them. This approach is part of a quality process whose aim is to ensure that the results obtained through the instrumentation of the various facilities are appropriate.

Thus, the reliability of the data is essential, and its validation is indispensable. Despite the progress made in the field of sensors and acquisition systems, the charged and aggressive nature of wastewater leads to problems of inconsistency or loss of data, which invalidates some of the measures. Possible failures include clogging and fouling of the probe, displacement of the sensor by the flow, transmission errors, and others. These problems affect the quality of the stored data. They impact the recorded measurement and can occur in different aspects: missing data, measurement blockage or saturation, sensor drift, noisy signal, or others (Methnani 2012). In order to ensure the quality of the measured data, they must be systematically validated.

In the field of artificial intelligence (AI), data validation refers to fault or anomaly detection. It is a discipline of data mining that is widely studied and deployed in different domains, such as intrusion detection, fraud detection, health monitoring, and industry manufacturing. It consists in identifying elements or events that are significantly different from usual data.

Therefore, the objectives of this study are manifold:

• discuss the use of an automatic AI-based validation system for large sewer measurement data sets;

• provide a comparison between two state-of-the-art algorithms;

• highlight the shortcomings and limitations of the ‘ground truth’ validation;

• recommend future research directions in this field.

This paper is organized as follows. A review of relevant work from the literature is discussed in Section 2. Sections 3 and 4 describe our proposed methodology for the validation of wastewater quality data and the different used approaches. Experiments and results are discussed in Section 5, and Section 6 presents the study's conclusion and perspectives for future work.

Invalid data is a straightforward consequence of system failures and harsh operating conditions. To address the problem at the source, the measuring device must be adapted to the measurement range and placed in a position that allows accurate measurements without significant disturbances (Campisano et al. 2013). An acquisition system capable of functioning properly under different weather conditions is also necessary. In addition, a preventive maintenance procedure and self-cleaning systems are required whenever possible. Nevertheless, experiences have shown that despite all these routines and efforts to optimize the reliability of sensors in sewerage networks, they remain insufficient, failures are inevitable and consequently corrupted data as well (Saberi 2015).

Several ‘trivial’ anomalies such as sampling problems, out of range, missing data, sensor blockage, or saturation can be detected via simple parametric tests. These tests can be implemented on specific supervision software or spreadsheets (Van Bijnen & Korving 2008). This (pre-)validation can be completed by statistical techniques or physically based models allowing to create and exploit a virtual redundancy (Piatyszek et al. 2000; Alferes et al. 2013). Most of these statistical models rely on hypotheses made on the characterization of the data and assume a stationary process. This assumption is far from being valid in sewerage networks which are highly dynamic on a daily and seasonal basis (Mourad & Bertrand-Krajewski 2002).

Thus, the pre-validation process is often completed by an expert validation. The latter requires the intervention of an operator to evaluate the overall likelihood of the results obtained (Mourad & Bertrand-Krajewski 2002; Branisavljević et al. 2010; Therrien et al. 2020). This manual procedure is mainly based on numerical models or visualization tools. The use of numerical simulation models requires the presence of an explicit model that allows to statistically evaluate the deviation between the measurement and the expected value based on the model-driven behaviour of the system (Venkatasubramanian et al. 2003). Such models are used for the validation of hydrometric data using the network's 1D models, but their calibration is difficult for pollution data. Moreover, these models are time-consuming and require a thorough knowledge of the operating conditions of the network and the correlation between these different parameters. In contrast, visualization-based validation requires specific validation criteria that rely on prior knowledge of the measurand dynamics and available exogenous information. Mourad & Bertrand-Krajewski (2002) formulate and evaluate seven criteria for the validation of hydrometric rainfall intensity and flow data. Zidaoui et al. (2022) provide validation procedures for turbidity data with material redundancy. This approach rapidly reaches its limits, especially with the increase in database volume to be processed. This task is also daunting for the experts involved, given the time it takes and the repetitive process. A reorientation of the workforce towards engineering tasks where its added value is valuable is more judicious. Moreover, this approach is also subject to human error and subjectivity which are difficult to overcome (Mourad & Bertrand-Krajewski 2002). This subjectivity will be discussed and evaluated later in this work (cf. Section 3.3).

All these arguments plead for the investigation of new approaches that allow to optimize the process of data validation in sewerage networks by automating this task and reducing the time needed.

AI has already proven its efficiency for data validation in different disciplines. Consequently, it is naturally one of the levers for action and one of the paths to be investigated. There are several examples of automatic data validation based on AI in the field of hydrology, particularly for the assessment of the quality of rivers and drinking water (Zhang et al. 2017; Muharemi et al. 2019). Support vector machine (SVM), logistic regression (LR), and artificial neural networks (ANN) are some of the most commonly used AI approaches for anomaly detection in water quality data (Dogo et al. 2019a). One-Class Support Vector Machine (OC-SVM) is a reviewed version of the SVM algorithm, adapted to the imbalanced database where the anomalies are a minority. This algorithm was tested on data from a drinking water treatment testbed with 51 measured parameters (Inoue et al. 2017). The results are encouraging, especially for their limited computational time. Furthermore, the latest research in the field of temporal data mining has led to the development of the Matrix Profile model which is deemed to be ‘the state-of-the-art anomaly detection technique for continuous time series’ (Feremans et al. 2020). This model has so far never been evaluated for wastewater quality data. Nevertheless, being domain agnostic and given that its results in other applications have proven its performance and its ability to identify anomalies, it seems important to test it in our context.

To summarize, this work aims to investigate AI approaches and evaluate their ability to optimize the data validation process. The evaluation will focus on two algorithms that result from the state-of-the-art analysis, namely One-class SVM and Matrix Profile. Their performance is compared to the results of the expert validation applied to quality data from a wastewater network.

OC-SVM was proposed by Schölkopf et al. (2001) for the detection of anomalies by considering that the majority of the training data are in a first class while the anomalies form a second class. Zhang et al. (2008) adapted this model to temporal data and evaluated its performance on telecommunication network data.

Matrix Profile is a relatively new algorithm for time series analysis, introduced in 2016 by Eamonn Keogh (University of California Riverside) and Abdullah Mueen (University of New Mexico) (Yeh et al. 2016). Its principle is to compare subsequences of the time series to itself by computing the Euclidean distance between each pair of subsequences of a given length. A naive way would be to compute the complete distance matrix of all pairs of subsequences in a time series and then to evaluate the minimum value of each column. However, this approach is not feasible due to the computational complexity and storage capacity. Hence, the interest of Matrix Profile algorithm which optimizes these calculations and adapts the method to large databases.

In order to evaluate the AI algorithms, an urban wastewater network database was used. This database comes from a city in France of about 46,000 PE. For confidentiality reasons, the site is anonymized. The instrumentation operation consisted in equipping six overflow chambers since February 2021 with continuous measurement probes for turbidity and conductivity.

The main objective of the operation is to progressively achieve the most reliable evaluation of the rejected flows by systematically validating the collected data which constitute the database of this work with the help of an automatic pre-validation combined with a manual expert validation.

The evaluation of the performance of different AI algorithms and their efficiency in identifying anomalies implies the presence of a reference validation. This is referred to as ground truth. For this, domain experts are contracted to validate the data manually. This validation consists in visualizing the data acquired by the different probes while varying the analysis scale from a few hours to several weeks. Multivariate analyses can also be performed in order to compare the different measurements with each other and to evaluate their coherence and their representativeness of the network functioning. Validating a month's chronicle of data can take up to 2 h full time for a trained expert.

Choosing the right metrics is a crucial step in evaluating the performance of a machine learning model. Often, a single metric is not enough to truly judge a model. The most classical is accuracy. However, it makes no difference between correctly classified valid and invalid data. In our problem, data are highly unbalanced, meaning that anomalies are a minority which leads to high accuracies even if invalid data is not identified.

Therefore, metrics suitable for binary classification of unbalanced databases (anomalies being minor) are used. The confusion matrix compares the results of the algorithm (predicted class) to the reference (true class). A positive label denotes an anomaly (invalid data). The standard template for binary confusion matrix is shown in Table 1 as a 2 × 2 matrix with four types of results.

Table 1

Confusion matrix definitions

View Large

From the confusion matrix, different metrics can be calculated:

• Precision is the ratio of correctly predicted positive observations to the total predicted positive observations.

  
• Recall (sensitivity) is the ratio of correctly predicted positive observations to all observations in actual class.

  
• F1 score is the weighted average of precision and recall. Therefore, this score takes both false positives and false negatives into account.

  

In real-life scenarios, time series coming from sensors are particularly subject to noise and outliers. The aim of anomaly detection models is to identify outliers. However, time series data requires some groundwork before it can be mapped by machine learning algorithms. The crucial issues are temporal irregularity and missing data. When it comes to time series analysis, it is important to have a constant timestamp. Resampling is then an important technique to ensure a round-the-clock frequency. Consequently, before any modelling, time series are aligned to get fixed timestamps, here a frequency of 5 min is chosen, which corresponds to the pre-set frequency for data acquisition.

Moreover, most anomaly detection algorithms are sensitive to the presence of missing data. Time series models require that there are no gaps in data along the time index. Consequently, missing data need to be replaced with judiciously chosen values before fitting a model. The replacement of values in this context is known as ‘imputation’. There are multiple imputation algorithms in the literature. The aim here is to replace the gaps so that the algorithms that are sensitive to their presence can run but still identify it as an anomaly later on. Replacement with zeros is then chosen.

As mentioned before, the original OC-SVM model was not designed for time series. Thus, a pre-processing of the data must take place. This consists in segmenting the data into sequences of predefined length L and feeding the model with subsequences instead of single points. This operation adds a parameter to the OC-SVM model which is the size of the sequence to be injected. Matrix Profile performs this operation internally without the need for pre-processing since the window size is already one of its parameters.

Model fitting is used to identify the optimal parameters of the model which is used for anomaly detection. An optimization procedure consists in defining a search space with the possible values for each hyperparameter. Different optimization algorithms can be used to explore this space. Here, we use a grid search. A finite combination list of values for each hyperparameter is specified, and all combinations are evaluated to identify the optimal combination (see Table 2).

The algorithms are all implemented in Python. The processing time depends on the hardware and the processing optimizations. The training and testing of the models were performed on the hardware listed in Table 3.

In addition, to evaluate the inter-variability among experts, we calculated the pairwise metrics (Section 3.4) between labellers in order to take into account the imbalanced nature of the database (Lampert et al. 2016). Table 4 synthesizes the results. The overall F1 score was 0.658.

The imbalance between precision and recall and the low score of the former showed that one expert was more likely to invalidate than the other. It is a consequence of the so-called validation bias, which is the tendency of an annotator to prefer one decision over another when there is doubt. Another cause of discrepancy is the random variation of an expert in relation to the other or even regarding himself. As a matter of fact, a revalidation of the same data by the same expert later does not lead to exactly the same result.

Moreover, great variability was observed between different measure sites (see Table 5).

For the evaluation of the performance of machine learning models, we considered the consensus between the two experts, in such a way as to keep only the most accurate defects and to avoid introducing a human bias. For further use, this approach would impose a permanent double validation to tune the AI model, which is very costly.

In the case of an unsupervised approach (which is the case for the two proposed models), this bias must be considered when interpreting the performance results of the model. By contrast, the problem is more critical in the case of a supervised approach, since the model will embed the bias introduced by human subjectivity. Indeed, supervised models use labelled data as input for learning and adapt their parameters in order to obtain the same results as output, whereas unsupervised models deduce functions to describe the internal structures from unclassified data. Thus, in view of the invalidity of a gold standard validation free of subjectivity, it is more appropriate to go for unsupervised approaches.

We realized that the F1 score of the model was of the same order of magnitude as the inter-variability between the two experts. It could be said that Matrix Profile obtained results equivalent to the two experts while considering their subjectivity. But moreover, the model allowed us to make objective decisions based on a rational metric which is the Euclidean distance between the subsequences, and this was within a much shorter duration than that required for an operator: we went from over 60 min to a few seconds.

On the other hand, the false negatives were related to several factors, starting with the delimitation of the defects, just like false positives. Moreover, we noted the sensitivity of Matrix Profile to its parameters. By setting a window size of 48 h, we were unable to identify short defects. And with a fixed anomaly rate, we forced the algorithm to choose the most flagrant anomalies, and which may not be the one identified by the expert. Moreover, it should be noted that the expert performs his validation in a multivariable approach by analysing the measurements of the two turbidimeters as well as the conductometer. However, the model Matrix Profile only has access to the turbidity reconstructed from the two sensors (Zidaoui et al. 2022).

In view of the environmental stakes, the reliability of the data measured in a wastewater network is crucial. However, given the operating constraints, the remediation of this problem is tedious and costly; hence, the interest to optimize it. Today, data validation is mainly done via automatic validation at the supervision level and/or manual validation performed by an expert. The comparison of two AI algorithms was carried out on pollution sensor data sets. The OC-SVM model proves to be unsuitable for the nature of the turbidity data in sewerage networks where different groups can be observed depending on the meteorological conditions (dry weather and rainy weather with different intensities). Thus, multi-clustering models can be better adapted to this type of data. On the other hand, the use of the Matrix Profile algorithm leads to promising results and an F1 score of 68%, comparable to the score of one expert compared to another. In that sense, in addition to being fast, the use of this model allows it to free from the subjectivity of the domain experts and to objectify the data validation process. Being unsupervised, the Matrix Profile model is able to provide accurate results. However, the evaluation of its performance is biased by the ground truth validation bias which is the result of a subjective approach. Thus, in reality, and given its principle, the model can be considered to have more encouraging results.

Such results can be further optimized by considering a validation pool with more experts. The objective is to have more consistency in the anomalies. The idea is to consider an odd number of experts in order to avoid having middle sequences where we cannot decide on their nature: valid or invalid. Moreover, the validation of the model has been performed on a typical site, and a generalization of the approach must be performed for different rates of anomalies. How will the model behave on a site where the sensors are often malfunctioning? And on the other hand, would it be able to identify very few anomalies; an anomaly rate of around 1% for example? Finally, a multivariate approach by taking advantage of conductivity and redundancy data is also conceivable: the goal is to identify certain contextual anomalies.

This work is part of a doctoral thesis involving the company 3D EAU and the ICube laboratory of the University of Strasbourg. We are therefore particularly grateful to these two entities for their funding and scientific support.

Data cannot be made publicly available; readers should contact the corresponding author for details.

The authors declare there is no conflict.