Pre-processing of imbalanced samples and the effective contribution in fault diagnosis in wastewater treatment plants

Monitoring and correcting the possible operation faults during wastewater treatment is of great importance, despite the fact that the process can be affected by various factors such as complex biological reaction mechanisms, highly time-varying and multivariable aspects (Hong et al. 2003). An optimal real-time fault diagnosis method can help maintain the stability of the water processing flow in wastewater treatment plants (WWTPs) and benefit the industry with adequate water output, lower operation cost and less secondary pollution to the environment (Hu & Hu 2002; Hamed et al. 2004). However, due to the imbalance of data numbers between the situation under faulty states and ground states, misclassification and inaccurate diagnosis during water processing often occurs and leads to inadequate output and even damage to the equipment as the traditional classifiers bias to the majority classes due to more sample data compared with that of the minority. The classification between imbalanced classes is considered as the main challenge of fault diagnosis using machine learning techniques.

Research on classifying imbalanced data began in the 1960s (Cover & Hart 1967). One way to improve the classification is to improve the algorithm of traditional machines that are used for balanced data classification or develop new algorithms because the previous machines assume or expect balanced class distributions or equal misclassification costs (He & Garcia 2009). For instance, the improvement of support vector machines (SVM) (Raskutti & Kowalczyk 2004; Hwang et al. 2011), and the development of back-propagation neural networks (BPNNs) (Chen et al. 2008) and extreme learning machines (Zong et al. 2013) are effective ways to compensate or increase the weight of data from minority classes. However, the weight of data strongly depends on the distribution of given datasets and needs to be adjusted manually according to different raw data because the imbalance of data numbers in different classes still exists in these methods. Another way is to minimize the gap between majority and minority classes through data pre-processing by either under-sampling the datasets of majority classes (Polat & Güneş 2008; Yen & Lee 2009) or over-sampling datasets from minority classes (Chawla et al. 2002; Nguyen et al. 2011). Data pre-processing has been recognized as an effective method for imbalanced data classification. However, the presented methods for data under-sampling have a risk of losing the representativeness of datasets, suggesting that more study is needed to understand how to minimize the loss of representative datasets during under-sampling. In the industry of wastewater treatment, unfortunately most of the previous studies use ideally equivalent datasets for fault diagnosis (Fuente & Vega 1995; Lee et al. 2005; Motamarri & Boccelli 2012; Tao et al. 2013) and neglect the imbalance of practical data. It should be noted that a previous work by Qian et al. (2014) has enhanced the accuracy of minority data classification to some extent, which implies that data pre-processing can also be an applicable route to increasing the accuracy of fault diagnosis in WWTPs.

Based on the existing studies regarding data pre-processing in other areas and the present status of fault diagnosis in wastewater treatment, this study proposes a novel method for imbalanced data classification and applies it in a case study of wastewater treatment, where the synthetic minority over-sampling technique (SMOTE) algorithm is used for over-sampling and Fast RVM for under-sampling. The fault diagnosis performance with different classifiers is shown to be enhanced compared with the results without pre-processing.

The raw data in this study are obtained from the UCI (University of California Irvine 1987) Machine Learning Repository (http://archive.ics.uci.edu/ml/index.html), which is primarily obtained from the daily measurements in an urban WWTP in Manresa, a town located near Barcelona, Spain. The single plant treats a daily wastewater flow of some 35,000 m³ with pretreatment, primary treatment by clarification, secondary treatment by means of activated sludge, and finally chlorination. The same raw data has been used for wastewater plant data processing elsewhere (Sànchez et al. 1997; Fan 2003; Zhang & Zhu 2011; Liu & Han 2013; Qian 2014). In total, 527 samples with 38 attributes were collected daily over approximately 2 years. Definitions of the 38 attributes are shown in Appendix A. The 527 samples are classified into 13 classes named by an Arabic numeral, as shown in Appendix B. (Appendix A and Appendix B are available with the online version of this paper.) Each class represents one type of wastewater treatment operation. The distribution of raw dataset numbers under the 13 classes is shown in Table 1. The data show that at faulty situations (i.e. class 2, 3, 4, 6, 7, 8, 10), the number of datasets is limited to no more than four, compared with normal situations (i.e. class 1, 5, 9 and 11) at more than 53. Based on this dramatic imbalance in data between faulty and normal situations, more sampled datasets will tend to represent the normal situations and cause inaccurate diagnosis of classifiers without pre-processing.

Method for under-sampling of the majority class data. Considering the possible loss of useful data caused by the reduction of datasets in the majority classes, it is important to choose a proper model that can reduce the number of datasets and maintain the maximum representativeness of the remaining data. This study employed Fast RVM rather than SVM and RVM for the following reasons.

Generally, all of the three methods share common steps to reduce the datasets, as shown in Figure 1. The main difference lies in choosing representative datasets (i.e. support vectors (SVs) for SVM and relevant vectors (RVs) for RVM & Fast RVM). For SVM, the machine learns and updates the data through minimizing the gap between the samples and hyperplane based on the Structural Risk Minimization Principle. However, the sparsity of data could become weaker with the increase of SVs, if the training datasets had been increased (Burges 1996; Burges & Scholkopf 1997). The RVM is a sparse Bayesian learning framework for producing sparse models, in which a set of hyperparameters is placed based on the principle of automatic relevant data points. After training, the weights of most samples tend to zero and only a few samples with non-zero weights will be left as RVs. Thus for a certain class of datasets, the number of RVs for RVM should be less than the number of SVs for SVM, which suggests that RVM can collect fewer samples with good representability after data reduction (Tipping 2001; Yang 2006), even though both methods can reduce the loss of representative datasets. Based on RVM, Fast RVM keeps the sparsity of RVs and reduces the computational complexity that can reduce the time complexity of training of a RVM classifier (Tipping & Faul 2003). Thus Fast RVM is chosen as a fast and proper model for majority class data reduction.

In data reduction, the majority classes are first divided into two clusters by K-means algorithm (Hartigan & Wong 1979). Then, the data from two clusters are input to a Fast RVM to build up a binary classifier. RVs are obtained from the established Fast RVM model. The number of RVs is fewer than the original database, but is representative of the sparsity of the original data, and plays a key role in generalizability (Liu 2011). Thus, we replace the majority class data with these RVs. In this way, the majority class is reduced and the effective information is retained. The reduction procedures are as follows:

• (1) cluster the majority class samples with K-means algorithm;

• (2) build up a binary classifier on Fast RVM;

• (3) replace the majority class samples with RVs.

The steps of Fast RVM classification are introduced as follows (Tipping & Faul 2003):

Based on Equation (5), an iterative analysis is followed to find the set of weights that maximize the function ⁠, in which the hyperparameter α is associated with the corresponding updated weight w. After updating, the trained classifier is built up with samples that have weight w unequal to 0, and the corresponding samples are RVs.

where is a random number between 0 and 1.

The modeling process of the Fast RVM based binary classifier is the same as mentioned above under ‘Methodology’, and the modeling process of SVM and RVM classifiers are introduced elsewhere (Vapnik 1995; Tipping 2001).

According to the description of the 13 classes, the dataset is further classified into four classes (one faulty class and three normal classes) as shown in Tables 2 and 3. Similar classification methods have been conducted elsewhere (Fan 2003; Zhang & Zhu 2011; Liu & Han 2013; Qian 2014).

It is clear from Table 2 that the dataset is still imbalanced with the ratio of four classes at 23.7:8.3:4.6:1. It is worth noting that these wastewater datasets present an identical imbalanced distribution.

Initially, 147 data with incomplete attributes are eliminated. The remaining 380 original samples are defined as input-target pairs ⁠, where ⁠, ⁠, i is the number of samples ⁠. For each of the 380 samples, the 38 attributes are normalized according to the formula ⁠, where j is the numbering of attributes ⁠. For the training and testing, the 380 original samples are divided into training set and testing set at a ratio 2:1 by stratified random sampling, where n₁ and n₂ are the number of examples in training set and testing set respectively ⁠.

As shown in Figure 3, when the number of RVs is approximately 80, the G-mean and total classification accuracy are both at their highest. In Figure 4, when the number of RVs is approximately 80, the classification accuracy of class 1 and class 4 are relatively higher than the classification accuracy of class 2 and class 3. The results suggest that the model performs well for class 1, when the number of RVs is at approximately 80. Distribution of the datasets is shown in Table 4.

From Table 4, it can be observed that the distribution of the original training datasets is ⁠. After pre-processing, the new distribution of the training datasets is ⁠, which suggests that the imbalance of datasets is minimized.

To analyze the classification performance on the pre-processed data, four classifiers, including a single multiple classifier BPNN, and three multiple binary classifiers based on SVM, RVM and Fast RVM, are chosen to classify the same data and the results are presented in Table 5. The single multiple classifier, the BPNN, is constructed with a three-layer structure, 38 input nodes and four output nodes, and the optimal hidden layer node number is defined by searching from 1 to 30 with a grid search mechanism on the training set. Regarding SVM, RVM and Fast RVM, radial basis function function is chosen as the kernel function, the penalty factor C in SVM and the kernel width σ in SVM, RVM and Fast RVM are determined from −10 to 10 through 5-fold cross-validation with a grid search mechanism on the training set. The OAO algorithm, a method combining several binary classifiers, is used to combine SVM, RVM and Fast RVM to yield a multiple classifier. The experimental classification performances are evaluated by the average values of 10 trials through 5-fold cross-validation. ‘Time’ in Table 5 indicates the training time. All experiments are simulated in the MATLAB 2013a simulation environment. The PC environment comprises: Core i3 2.1G, 2G ROM, more than 150 G remaining space of hard disk.

As shown in Table 5, majority classes using the BPNN classifier have high classification accuracies (i.e. class 1 accuracy R1 acc = 92.88%, class 2 accuracy R2 acc = 78.33%) because the accuracy of the BPNN relies heavily on the number of datasets. Therefore, the lack of samples in minority classes leads to poor classification accuracies, especially in class 4. Accuracy on class 4 is only 5%. The G-mean value of the BPNN classifier is 0.07. The results match the characteristics of the training principle of the BPNN, i.e. the empirical risk minimization principle, which strongly relies on the number of the training samples. Thus the imbalance between minority classes and majority classes leads to unsatisfactory results.

Compared with the BPNN, the SVM classifier has reduced dependence on training samples and obtains better G-mean values. The accuracy of class 4 is higher than that of the BPNN. However, the optimal hyperboundary surface in the SVM classifier favors the majority class (i.e. class 1 accuracy R1 acc = 92.63%) and results in the huge difference between major and minority classes. The better performance of the SVM over the BPNN indicates that the Structural Risk Minimization Principle for the SVM training can reduce the dependence of classifiers on the balance of data.

However, the application of RVM results in highest accuracy on class 4 compared with other classifiers, which matches the character of sparse probability models in the Bayesian learning framework, and the G-mean value is better than that for other classifiers.

Unfortunately, the performance of Fast RVM is comparably poor with the G-mean at 0.29 and class 4 accuracy at 22.22%, although the training time has been substantially reduced by approximately 50% compared with other classifiers.

The simulation results in Table 5 suggest that, for the four classifiers, the classification accuracies in class 4 are worse than the classification accuracies in class 1 because of the imbalance between the majority classes and the minority classes, indicating an unsatisfactory classification of the raw data. Among all of the classifiers, RVM performs best, but the G-mean remains quite low.

The classification performances of classifiers on the pre-processed data are presented in Table 6.

Regarding the BPNN, the class 3 and class 4 accuracies have been raised from 60 to 76.25% and 5 to 30% compared with Table 5; while class 1 accuracy has been decreased from 92.88 to 85.38% and class 3 accuracy remains almost the same. However, the G-mean value has improved remarkably from 0.07 to 0.42 because of the obvious increased classification accuracies in expanded minority classes (class 3 and class 4).

Regarding the SVM classifier, the accuracies of class 2, class 3 and class 4 have been increased from 67.33, 61.25 and 30% to 76.67, 73.75 and 50%, respectively, with class 1 accuracy being declined from 92.63 to 83%, and the G-mean value has been raised from 0.39 to 0.52.

Regarding the RVM, the accuracies of class 2, class 3 and class 4 are all improved, but there is an obvious decrease of class 1 accuracy (from 91.88 to 79.88%) because of the dramatic decrease in data number after pre-processing. The G-mean of RVM has also been improved to 0.57.

Lastly, regarding the Fast RVM, class 4 accuracy has increased from 22.22 to 45%. Notably, class 1 accuracy is the best among all of the classifiers because the data reduction with Fast RVM has considered the fitness of classifying data onto Fast RVM classifiers. The G-mean value has been improved to 0.63 as the best among all of the classifiers; while the processing time is decreased to less than 1 second at 0.92 s.

We found the G-mean values of all of the classifiers improved notably as shown in Table 7, which highlights the value of data pre-processing. In the present case study, the multiple binary classifiers (i.e. SVM, RVM and Fast RVM) perform better than the multiple classifier BPNN and further study is needed for more in-depth discussions.

This study demonstrates the application of a pre-processing method for imbalanced data classification with Fast RVM for under-sampling of majority class data and the SMOTE algorithm for over-sampling of minority class data. The method is applied to fault diagnosis in a WWTP. The case study suggests that this pre-processing method can neutralize the data imbalance between the majority and minority classes to increase the share of the minority classes among the whole database. For this case study, improvements for the four chosen classifiers (i.e. BPNN, SVM, RVM and Fast RVM) in G-mean values, classification times, and accuracy can be observed. The Fast RVM multi-classifier performs better compared with other classifiers in this certain case with time at 0.92 s and G-mean value at 0.63.

However, the present case study is limited, and it does not assert a general claim. Additionally, further research (including more case studies, data splitting, as well as models optimization) is required for more general conclusions. Finally, other data-preprocessing methods for imbalanced data should also be compared to develop recommendations for industrial application.

This work was supported by Science and Technology Planning Project of Guangdong Province, China (2016A020221008, 2016B090918028, 2016B090927007) and Science and Technology Planning Project of Guangzhou, China (201604010032). The authors are grateful to the anonymous reviewers and the editor for their helpful comments and suggestions which greatly helped us to improve the quality of the paper.