Using computer vision to monitor ice conditions in water supply infrastructure: a study of salient image features

During the break-up period, the flowing ice in man-made open channels can have disastrous consequences. For example, the large ice blocks can damage hydraulic structures (e.g., sluices and bridges) built over the channel, threatening their safety and stability. The flowing ice chunks can get stuck in narrow passages of the channel, forming ice jams that will block the natural flow and raise the water level. This will eventually induce flooding in the surrounding area. It was reported that ice jam-induced floods account for up to 80% of all historical floods in the Schenectady areas of the United States (USGS 2019). Economic loss caused by ice jams is over a million every year in Canada (CBC-News 2014). In the face of the catastrophic outcomes, effective ice condition monitoring (ICM) is of paramount importance as it can inform situation-aware decisions to proactively plan for adverse ice conditions.

Other than disaster prevention and remediation, ICM also provides critical information for guiding the operation of man-made hydraulic infrastructure (e.g., water channels and dams) in cold regions (Norvanchig & Randhir 2021; Zhang et al. 2021). For example, ice chunks stuck behind a water dam can cause a loss of water head, significantly reducing power production (Gebre et al. 2013). Accordingly, the water dam needs to dynamically adjust its water release schedule to accommodate the changing ice condition. In addition, the occurrence of ice-related incidents (i.e., ice flow or ice cover) can undermine the water supply capability of water channels. Hence, related engineering and hydraulic measures need to be taken to fulfill basic water demand based on the detected ice condition.

Despite its importance, precise and effective ICM is challenging to implement, especially for the many water channels located in remote areas (Zhao et al. 2012). Prevalent solutions are to deploy water level gauges and flow meters along the channels to estimate ice condition from water level and discharge; however, these indirect approaches are error-prone and usually cannot accurately identify the ice condition. Shallow water ice profilers (SWIPs) can provide high-resolution ice cover measurements but are difficult to install and maintain in an underwater environment. Satellite-based methods (e.g., scatterometers) require daytime and cloud-free conditions, and relevant satellite images are expensive to access (Vuyovich et al. 2009).

The rapid advancement of computer vision (CV) provides a promising opportunity to tackle the challenges of ICM. The rationale is three-fold. First, the increasing affordability of image-capture devices such as webcams (Vuyovich et al. 2009; USGS 2019) and unmanned aerial vehicles (UAVs) makes it possible to acquire a large amount of visual data (e.g., photos and videos) with ease. Second, the nature of imaging allows it to provide high-solution, full-coverage representations of the surface ice conditions in the waterway, from which direct, precise, and undistorted measurements can be derived. Thirdly, the powerful CV algorithms nowadays, e.g., feature extraction and machine learning (ML) models, have made automatic image processing possible. In fact, the use of images for ICM is not new – many projects have installed webcams to stream site video footage remotely. However, current practice relies on manual efforts to identify ice conditions from a massive number of images, which is extremely laborious and time-consuming.

This paper performs a comprehensive study of salient image features to classify different ice conditions in water supply infrastructure, which paves the way to automating ICM using CV. The contribution is two-fold. First, multiple color and texture descriptors have been developed to characterize channel ice. Their effectiveness in differentiating ice conditions is investigated. It generates new insights in terms of how different ice conditions manifest different visual appearances and provides empirical evidence in the correlation between image features and the states of ice in the water channel. Second, ML models have been trained for automatic ice condition classification based on the developed image features. It demonstrates the feasibility of shallow ML models in classifying ice conditions with a limited amount of data as long as distinctive ice features are provided. From a practical perspective, the study lays a foundation for incorporating CV into emerging image collection platforms such as UAVs to automatically monitor ice conditions in water infrastructure.

Early studies attempted to monitor ice conditions by sensors embedded in the channel or from satellite imagery. For the embedded sensor-based approaches, flow regime parameters such as water level and discharge are used to reflect the change in ice conditions. For instance, water level gauges (Kowalczyk & Hicks 2003; Beltaos et al. 2011) and pressure loggers have been deployed to monitor long-term flow status evolution. As an increase in water level along with a decrease in discharge may signify the formation of ice cover (Beltaos & Burrell 2010; Beltaos et al. 2011), empirical models can be established to predict ice conditions with the flow data collected. However, it is usually error-prone to use indirect indices (e.g., water level and discharge) to infer ice conditions. Therefore, new devices have been developed to directly characterize the ice status. The SWIP is one such device for ice measurement in shallow water scenarios (Ghareh Aghaji Zare et al. 2016). Gunn et al. (2015) demonstrated the feasibility of using X- and Ku-band scatterometers in measuring freshwater lake ice thickness by decoupling the influence of snow cover. Cui et al. (2015) presented an apparatus for freshwater ice thickness monitoring based on electrical resistance and temperature. However, the cost and efforts of installing and maintaining these devices are considerable, which limits their large-scale applications (King et al. 2013; Gunn et al. 2015).

The other line of research intends to develop methodologies for ice monitoring from satellite images. Liu et al. (2009) reported an application of the high spatial-temporal-resolution satellite data in ice condition observation. Unterschultz et al. (2009) demonstrated the potential of using synthetic aperture radar (SAR) satellite images to identify ice jams, intact ice, and open water during breakup. Chaouch et al. (2014) proposed a method that can distinguish between ice cover and flowing water from remote sensing photography. Vasil'ev et al. (2022) studied the possibility of navigation satellites in monitoring ice cover dynamics. As pointed out by Vuyovich et al. (2009) and USGS (2019), satellite-based methods are heavily dependent on weather conditions (requiring daytime and cloud-free condition), and can only obtain relevant images periodically due to the satellite cycle revolving around the earth.

The latest development in CV shows promising potential in exploiting near-ground images for ICM. Compared with satellite images, near-ground photos are easily accessible. For many projects, webcams are standard equipment that have already been installed (Vuyovich et al. 2009). UAVs, since they entered the commercial market years ago, are becoming increasingly affordable. Liu et al. (2019) reported an application of UAVs in water channel inspection. Pioneering efforts have been made to apply CV in processing the near-ground images collected by webcams or UAVs for ice study. Jedrzychowski & Kujawski (2014) proposed an automatic method to assess the percentage of ice coverage from webcam images. Ansari et al. (2017) developed a shore-based imagery processing algorithm for ice cover quantification. Rødtang et al. (2021) explored the potential of CV techniques such as Structure from Motion (SfM) in drone-based ice growth surveying, while Ansari et al. (2021) developed a convolutional neural network called IceMaskNet for river ice detection and characterization from aerial photography. Existing studies tend to focus on quantitatively characterizing the ice cover area at different ice stages for research purposes instead of ICM. Some efforts (Ansari et al. 2021; Pei et al. 2023) attempted to apply deep learning (DL) models to directly classify ice conditions. However, DL entails a large annotated dataset to enable end-to-end model training, which is difficult to obtain in such a domain-specific scenario as ice monitoring in water infrastructure. In addition, DL is usually deemed as a black box, making it difficult to interpret and develop confidence in the results (Rudin & Radin 2019). The limitations call for a need to develop useful high-level image features to reduce data volume needed, and on the other hand, understand features that matter in ice classification.

The study of handcrafting and selecting suitable image features for object recognition is called image feature engineering. It refers to a process of transforming raw data (e.g., RGB pictures) into feature representations (e.g., color or texture descriptors) that can be used in an object classification or detection task (Brownlee 2014). In domain-specific problems where available data are too scarce to train deep learning models, the technique has been an effective and useful means for building condition assessment (Torok et al. 2014), bridge inspection (Abdel-Qader et al. 2003; Yeum & Dyke 2015), structural damage detection (Ramana et al. 2019), and post-disaster search and rescue (Rudol & Doherty 2008). In terms of ice condition monitoring, researchers have been attempting to extract information on the ice evolution process from certain image features. Jedrzychowsk & Kujawski (2014) found that the combined use of a canny edge detector and a morphologic algorithm can effectively extract ice float on the water surface. Ansari et al. (2017) developed a workflow to characterize spatial and temporal features of ice cover, which includes preprocessing, image registration, image rectification, target detection and ice characteristics calculation. Kalke & Loewen (2018) investigated the use of a support vector machine (SVM) for the prediction of total surface ice concentration. However, few have attempted to systematically investigate image features that are distinctive and salient in differentiating various ice conditions, and thus enable automatic ICM.

The literature review reveals three knowledge gaps. First, existing ICM approaches based on embedded sensors or satellites are either too expensive to operate (e.g., SWIP), or are of insufficient resolution to ensure consistent monitoring (e.g., SAR). Second, the data-greedy and black box nature of DL-based methods make them difficult to scale-up in real life. Thirdly, although image feature-based ML represents an explainable learning approach that only requires small data, few studies have systematically investigated ice features for enabling automatic ICM. Different from existing methods, our study puts forward a low-cost, robust, and all-weather approach to monitor ice conditions in water supply infrastructure based on near-ground imagery. By a systematic investigation of salient ice features, our approach does not need big data to train, and is interpretable to both humans and machines. It boosts humans' confidence in the model and can be deployed at scale for its low training cost.

The objective is to find suitable descriptors to characterize image features of different ice conditions and realize automatic ice condition classification. To achieve the objective, a three-step methodology is developed. First, the freeze–thaw cycle of channel water is studied to determine a categorization of different ice condition stages. Second, a set of feature descriptors in terms of color and texture are defined to characterize and differentiate the various ice conditions. Finally, based on the proposed feature descriptors, ML models are developed and compared for automatic ice condition classification. Experiments will be implemented to evaluate the developed methodology from two aspects. On the one hand, correlation between the proposed descriptors and the ice condition will be studied to identify the most salient features for effective ice condition differentiation. On the other hand, several ice condition classifiers based on ML will be trained and quantitatively evaluated.

In this section, a color descriptor and three texture descriptors are developed to characterize image features of different ice conditions.

Texture can also be a strong indicator of different ice stages. For example, ice flow and ice jam tend to have a coarser texture compared with those of open water and ice cover. In addition, under the influence of water flow, the texture of ice flow is likely to distribute along a single direction while the ice jam might present a multi-directional pattern. Based on these observations, three texture descriptors have been proposed.

In order to evaluate the proposed methods, a total of 340 images were collected by field investigations and online searching. The collected data cover different ice stages and ice flow strength, and were collected in various illumination conditions. The dataset was outsourced for annotation using our developed web-based annotation tool (see Supplementary material, Section S1). We posted the link to the website on social media and used personal connections to reach out to potential annotation workers. Based on a majority vote, the class and subclass of each image were decided. Supplementary material, Section S2 shows the composition of the obtained dataset after annotation, and its key parameters. Overall, the numbers of images depicting open water, ice cover, ice flow, and ice jam scenarios are 64, 86, 120, and 70, respectively, and in the ice flow category, there are 67 and 53 images showing moderate and strong ice flow, respectively. Resolution of the image ranges from 190 × 128 to 657 × 420. The average brightness and contrast after normalization to [0,1] are 0.61 and 0.67, respectively.

To determine suitable descriptors to input to the classification models, correlations between the descriptors and the ice condition have been investigated. Eta-squared (η²) is adopted to quantitatively measure the correlation between a discrete variable (ice condition) and a continuous variable (feature descriptors). A rule of thumb on interpreting the value of η² is given here: The larger η², the stronger the correlation, and a η² over 0.14 is considered acceptable. In our study, descriptors with η² lower than 0.14 are deemed irrelevant to the ice condition, and thus will be excluded from the SVM models.

When calculating δ-EHD, a threshold (i.e., Th) is applied to ensure the algorithm only considers the edges with intensity over the Th. Figures 7(c) and 7(d) show the distribution of δ-EHD values under different Th. In the second row of the table in Figure 7(c), the η² value regarding ‘ice flow’ and ‘ice jam’ is presented to reflect the capability of the index in separating these two ice stages. The highest η² of δ-EHD regarding the two ice stages and all ice stages is, respectively, observed at Th = 0.1 and Th = 0.05. Given that the value of η² regarding the two ice stages at Th = 0.05 (i.e., 0.325) is quite close to the peak at Th = 0.1 (i.e., 0.333), the Th = 0.05 is adopted to attain a better separation of all ice stages. In terms of ice flow strength, the threshold value of 0.001 is selected due to the strong correlation between ice flow strength and δ-EHD under the threshold.

In the HOG algorithm, the parameter ‘cell size’ decides the level of detail to which the feature is extracted: A large cell size tends to extract the big-picture features in an image while a small one focuses on the detailed features. As shown by Figures 7(e) and 7(f), the correlation of δ-HOG using different cell sizes with ice condition is analyzed. The η² value regarding ‘ice flow’ and ‘ice jam’ is also presented at the bottom. When the cell size is 48 × 48 pixels, the highest η² values over all ice stages and the two ice stages are both observed at the same time. Hence, the δ-HOG using the 48 × 48 cell size is adapted to the ice stage classification model. As for ice flow strength, since η² values under different cell sizes are all lower than our baseline (i.e., 0.14), δ-HOG is excluded from the corresponding classification model.

By the analysis in the last section, the input to the SVM classifiers is determined, as listed in Table 1. The dataset consisting of 340 images was randomly divided into a training set, a validation set and a test set. The training set was used to train the SVM classifiers; the validation set was used for hyperparameters tuning and the selection of the optimal model; and the test set was used to evaluate the performance and generalization ability of the optimal model determined by the validation set. The bottom three rows of Table 1 list the constitution of the training set, validation set, and test set used by classifiers #1 and #2.

A Python package, scikit-learn, was used to implement the training and testing process. Three important hyperparameters need to be specified in the training process, i.e., kernel, C, and gamma. The ‘kernel’ designates the type of hyperplane used to separate the data. For example, the ‘linear’ kernel is effective in linear-separable data while the other kernels (e.g., ‘poly’ and ‘rbf’) are normally used to deal with nonlinear data. C is the regularization parameter that defines the tolerance level of misclassification. The gamma defines the range of influence a training example should reach. After parameter tuning on the validation set, it was observed that classifier #1 achieved the best performance when a ‘poly’ kernel with a degree of 2, C = 300, and γ = 10 were adopted. Classifier #2 yielded the best outcomes when a ‘rbf’ kernel, C = 15, and γ = 30 were adopted. The confusion matrices of the optimal models on the validation set are shown in Supplementary material, Figure A4.

The optimal models were used to classify the image samples in the test set, and the resulting confusion matrices are presented in Supplementary material, Figure A5. Supplementary material, Table A1 lists some key performance indexes of the models on the test set. The accuracy, defined as the portion of correctly predicted samples accounting for all the samples, is 0.719 and 0.773 for classifier #1 and classifier #2, respectively. The results indicate that the obtained classification models can yield reliable prediction results for ice condition recognition.

The two ice condition classifiers yielded reliable prediction performance on the test set, which can be attributed to the following factor. Effective feature descriptors have been defined and selected as the input to the SVM classifiers. As shown by the box plots from Figures 6 and 7, the collective use of the StV, EP, δ-EHD, and δ-HOG can effectively differentiate among the different ice stages, i.e., ‘open water’, ‘ice cover’, ‘ice flow’, and ‘ice jam’, while the integrated adoption of the StV, EP, and δ-EHD can be used to differentiate different levels of ice flow strength, i.e., ‘moderate’ and ‘strong’. The correlation analysis in Experiment One successfully optimized the hyper-parameters of the descriptors (i.e., i and j for StV, the threshold for EP and δ-EHD, and cell size for δ-HOG) and excluded irrelevant variables from the models (e.g., δ-HOG for classifier #2), which ensured that only the most correlated variables are considered.

The time performance of the proposed method is listed in Supplementary material, Section S4. Note that all the algorithms were run on a laptop, ASUS VivoBook S15, with an Intel Core i7-8550 U processor, and Nvidia GeForce MX150 GPU. The adopted inputs to the SVM models are a few high-level features extracted based on prior human knowledge; hence, the training process is quite efficient compared with models based on pixel-wise low-level features, which usually require a large amount of time and data to learn high-level features from individual pixels. Given a newly captured photo, it only takes 0.9257 s to calculate its corresponding feature descriptors (0.9189 s) and identify the ice stages (0.001362 s) and ice flow strength (0.005475 s) it depicts. This efficiency allows relevant professionals to gain rapid awareness of potential safety issues once they are captured in the images.

Although this research has demonstrated the viability of CV in ICM by an exploratory study of salient image features, future research is suggested to further enhance the method from the following two aspects:

• First, preprocessing techniques should be incorporated in future work. Without proper preprocessing, the performance of image classification might be undermined by a series of factors, such as blurring, varying lighting, and water reflection. The negative effect of these factors can be potentially addressed by the following preprocessing techniques: A high-pass filter can be applied to blurred images to enhance the contrast and strengthen the edges; image normalization should be adopted to adjust the illumination and shadow condition; as for water reflection, Hough line transform can be used to detect and remove tree limbs reflected on the water.

• Second, a solution should be devised to apply the feature engineering results and the developed classifiers in practical ice monitoring. Two possible solutions are suggested. The first one is based on shore-based cameras. The cameras can be pre-calibrated to ensure that they only focus on water and (or) ice surfaces in the channel. This solution can realize long-term monitoring of ice conditios In the other solution, UAVs can be applied to perform periodical inspection of the water channel. By using the region of interest (ROI) extraction method introduced by Chen et al. (2019) and Chen et al. (2021), ROI can be extracted from the images captured by the UAV for the subsequent classification tasks. Compared to the shore-based camera solution, this solution can fulfill a full-coverage ice condition assessment of the entire channel.

This study performed a systematic image feature analysis for ice condition classification during the whole freeze–thaw cycle of water supply channels. The study was focused on four ice stages (i.e., open water, ice cover, ice flow, and ice jam) and two levels of ice flow strength (i.e., moderate and strong). Using a newly developed web-based tool, an ice condition dataset has been generated by outsourcing the annotation task to different annotation workers. A color-based descriptor (i.e., StV) and three texture-based descriptors have been proposed, and their correlations with different ice conditions have been quantitatively investigated. Using the correlated descriptors as input, two SVM classifiers have been trained to automatically identify the ice stage and ice flow strength from images. Experiments have been implemented to validate the efficacy of the proposed algorithms. The suitability and the optimal hyperparameters for the proposed descriptors have been investigated. The StV_65-180 and StV_25-180 are the most correlative with ice stages and ice flow strength, respectively, among the different channel intervals. EPs with thresholds of 0.6 and 0.2 are, respectively, most relevant to the types of ice stages and ice flow strength while the optimal threshold for δ-EHD is, respectively, 0.05 and 0.001. The δ-HOG using the 48 × 48 cell size can best differentiate the ice stages while the index is not suitable for ice flow strength classification. The SVM classifiers with the above descriptors as inputs can achieve reliable prediction (with an accuracy of over 70%) with near real-time calculation efficiency (less than 1 s). Future work should focus on further improving the classification performance by adopting image pre-processing and exploring application solutions based on webcams or UAVs.

This research was supported by the National Key Research and Development Program of China (No. 2018YFC0406903) and the National Natural Science Foundation of China (No. 51979189).

All relevant data are included in the paper or its Supplementary Information.

The authors declare there is no conflict.