https://orcid.org/0000-0002-6588-2206
Abstract
Vision-based analysis of waterbodies can provide important information required for monitoring, analyzing, and managing water resource systems, such as visual flood detection, delineation, and mapping. Water, however, is an ornery object in image processing, as it can be found in different forms and colors in nature. This makes the detection, classification, and tracking of water in images and videos difficult for computer vision models. There are still visual differences resulting from water texture and its inherent optical properties associated with different waterbodies which can be recognized and extracted to support computer models to better analyze water images. This study aims to utilize a set of early, mid-level, and high-level vision techniques, including Gabor kernels, local binary patterns (LBPs), and deep learning (DL) models to extract and analyze water texture and color of different waterbodies in digital images. For this purpose, ATLANTIS TeXture (ATeX), an image dataset for waterbodies classification and texture analysis, was used. Models were trained for the task of classification on ATeX. Then, the performance of each model in extracting texture features was evaluated and compared. Results showed that the classification accuracy achieved by the Gabor magnitude tensor, LBP, and DL model (ShuffleNet V2 × 1.0) are 29, 35, and 92%, respectively, and thus the DL model outperforms traditional vision-based techniques. Moreover, the classification results on raw images represented by different color spaces (e.g., RGB, HSV, etc.) emphasized the importance of color information for digital image processing of water. Analyzing representative visual features and properties of different water types and waterbodies can facilitate designing a customized Convolutional Neural Networks (CNNs) for water scenes, as CNNs recognize objects through the analysis of both texture and shape clues and their relationship in the entire field of view.
HIGHLIGHTS
The role of texture and color information in the analysis of water images is important to provide water feature properties.
Large-scale water dataset makes it possible for researchers to develop data-driven models on water resource problems.
Investigation of computer vision techniques for water can provide useful information for developing vision-based models customized for water-related extreme events.
INTRODUCTION
Water is reflective, (a) shows water reflection in an urban setting while (b) shows the reflection effect in the natural environment. Turbulence and turbidity are two other important factors, (c) shows rapids (turbulent water) in a river, (d) visually compares the clear and muddy appearance of water on two different sides of an earth dam, and (e) shows estuary where a freshwater river or stream meets the ocean.
Water is reflective, (a) shows water reflection in an urban setting while (b) shows the reflection effect in the natural environment. Turbulence and turbidity are two other important factors, (c) shows rapids (turbulent water) in a river, (d) visually compares the clear and muddy appearance of water on two different sides of an earth dam, and (e) shows estuary where a freshwater river or stream meets the ocean.
Different constituents of the water column and their inherent optical properties have direct effects on light absorption and scattering of different water types. These particles influence the color of light that is detected by the camera sensor. For example, colored dissolved organic matter (CDOM)-rich water tends to look black because CDOM absorbs light strongly, but it does not reflect much light backward. Moreover, tidal forces cause a re-suspension of sediments, and non-algal particles (naps) making the color of water light brown. External factors such as light and temperature, as well as nutrients derive phytoplankton productivity which reflects green color possibly due to chlorophyll concentrations (Palacios 2020). These constituents, which vary from one waterbody to another, have a direct effect on the texture and color of the water. Texture and color both are important within the digital image processing domain. The texture appears to be a very strong cue to object classification, and the color is a powerful descriptor that often simplifies object identification and extraction from a scene (Gonzalez 2009). In this study, we address challenges associated with the visual sensing of water, using vision-based models and techniques customized for texture and color analysis in digital images.
In computer science, texture analysis has been widely applied for different purposes such as facial recognition and expression analysis (Liu et al. 2014). Facial expression analysis refers to developing models for automatically analyzing and recognizing facial motions and feature changes from visual information (Tian et al. 2005). These features can be extracted either by hand-designed filters (Zhang et al. 1998; Zhao & Pietikainen 2007) or trained ML models (Meng et al. 2017). So far, many studies have attempted to improve the feature extraction techniques to provide better facial recognition (Tong et al. 2007). Another application is texture mapping in which a two-dimensional (2D) surface, called a texture map, is wrapped around a three-dimensional (3D) object or 2D mask (Park et al. 2019). Moreover, texture analysis can be used in image super-resolution (i.e., photo enhancement) task which aims to recover a high-resolution image from a low-resolution one (Wang et al. 2018). A generative adversarial network (GAN) is normally used for recovering realistic texture.
Large-scale datasets provide the opportunity for researchers to develop and train ML-based models, either for temporal or spatial analysis, for real-world applications (Lin et al. 2014; Mottaghi et al. 2014; Zhou et al. 2019). For instance, CAMELS dataset (Addor et al. 2017; Hao et al. 2021) containing hydrometeorological time series and catchment attributes for 671 watersheds in the U.S. led to developing and training long short-term memory (LSTM) models to improve streamflow predictions for ungauged basins (Kratzert et al. 2018; Kratzert et al. 2019). Other publicly available datasets, such as multispectral Landsat analysis ready data (ARD) imagery, synthetic aperture radar (SAR) data, and LiDAR-derived coastal digital elevation models (DEM) have been used together to develop a Convolutional Neural Network (CNN) model for multi-class land cover classification (Feng et al. 2019; Xu et al. 2019; Muñoz et al. 2021).
Samples of ATeX patches.
This study aimed to investigate vision-based texture and color analysis techniques on the water to find visual features which can distinguish water in different waterbodies. Texture information is applicable to configuring the architecture of the CNN-based models as the recognition strategy of CNNs follows local to global features in different layers of the forward pass. In other words, objects are recognized through the analysis of texture and shape-based clues – local and global representations and their relationship in the entire field of view. In section 2, three different approaches, including two conventional methods (based on the hand-craft features) and a Deep Learning (DL) model, were built to extract texture features of water. The quality of extracted features was then evaluated using K-Nearest Neighbors (KNNs) in section 3. The role of color and color space in the task of classification was discussed in section 4. Finally, in the last part of section 4, the role of ATeX in improving the performance of DL-based semantic segmentation models in water sciences was discussed.
METHODOLOGY
Texture representations
Facial recognition and expression task are very similar to water detection and classification. In face recognition, texture representations and spatial patterns of face components (eyes, nose, lips, etc.) provide valuable information for recognition. For water, however, spatial information provides no significant information, as pattern subelements (textons) (Julesz 1981) resulting from filter-based texture representations do not follow any specific spatial coordinates. Thus, in this case, the texture of the water is the only source of information for the accurate classification of waterbodies. In this section, Gabor kernels filter and local binary pattern (LBP) descriptors are used to extract texture information. The quality of the resulting texture representations is then evaluated and compared with the KNN classification method in the following section.
Gabor filters
Gabor wavelets have been commonly used for the task of face recognition (Shen & Bai 2006; Vinay et al. 2015). Frequency and orientation representations of Gabor filters are claimed to be very similar to those of human visual system (Olshausen & Field 1996). They have been found to be particularly appropriate for texture representation and discrimination. In the spatial domain, a 2D Gabor filter is a Gaussian kernel function that is modulated by a sinusoidal plane wave. The Gabor wavelet representation facilitates the recognition without correspondence (hence, no need for manual annotations) as it captures the local structure which corresponds to spatial frequency (scale), spatial localization, and orientation selectivity. As a result, the Gabor wavelet representation is not sensitive to changes caused by illumination changes and subtle nuances (Liu & Wechsler 2002). The texture representations resulting from Gabor filters on ATeX waterbody patches are shown in Figure 3(a).
and 
define the orientation and scale of the Gabor kernels, 
, and 
denote the norm operator. Wave vector 
is defined as follows:
and 
. Maximum frequency, 
, is the spacing factor between kernels in the frequency domain. Five different scales, 
, and eight orientations, 
are applied with the following parameters:
, 
, and 
. The kernels exhibit desirable characteristics of spatial frequency, spatial locality, and orientation selectivity. Figure 3(b) shows different combinations of frequency and orientation of the filters applied in this study. 
(a) The visual results of four different scales and orientation Gabor filters on three waterbody patches. (b) The real part of the Gabor kernels at five scales and eight orientations with the following parameters: 
, 
, and 
.
(a) The visual results of four different scales and orientation Gabor filters on three waterbody patches. (b) The real part of the Gabor kernels at five scales and eight orientations with the following parameters: , , and .
to decrease the size of the kernels and alleviate computational complexity, because 2D Gabor filters are Gaussian-based, so the values of a Gaussian function at a distance larger than 
from the mean are small enough to be ignored (Gonzalez 2009). The experiments have been separately run for RGB, grayscale, and HSV color space of ATeX patches to compare texture representations within each color space. The following steps discuss the experimental procedure for grayscale images as shown in Figure 4: - 1.
First, ATeX patches are imported as gray values (N
32
32) where N represents the number of patches. - 2.
Multi-scale and multi-orientation Gabor filters are applied and corresponding Gabor magnitude responses are obtained. Then, all responses are concatenated to build an augmented feature tensor for each patch N
32
32
40. - 3.
Each augmented feature tensor is downsized in space from 32
32 to 16
16 using ‘MaxPooling’ operator which results in size of N
16
16
40. - 4.
Finally, all 10,240 features, resulting from 16
16
40, for each patch, are reduced to the top 500 dimensions of the data with the highest variance using Principal Component Analysis (PCA).
Augmented feature tensor pipeline for Gabor Kernels.
The same procedure is repeated for RBG and HSV color spaces, while the first step is just different to represent the color channels of images. In HSV (RGB) experiments, input patches have an additional dimension N
32
32
3, which represents Hue, Saturation, and Value (Red, Green, and Blue) of each patch. Accordingly, Gabor filters are constructed dimensionally compatible, but Gabor magnitude responses and resulting augmented feature tensor have the same dimensions as described for the grayscale patches.
Local binary patterns
LBP is a type of visual descriptor used for classification in CV. The original LBP operator is introduced by Ojala et al. (1996). The operator labels the pixels of an image by comparing the 3 × 3 surrounding neighborhood of each pixel with the center value using a binary system. The corresponding location of the pixel on the binary map gets 1 if the value of the surrounding pixel is more than the center pixel and it gets 0 vice versa. Then, the histogram of the labels can be used as a texture descriptor. Figure 5(a) shows the basic LBP operator (Ahonen et al. 2004).
The LBP operator can be extended to include more neighbor pixels (Ojala et al. 2002). The circular approach and bilinearly interpolating the pixel values enable any radius and number of surrounding pixels. In this approach, we will use the notation 
which means P sampling points on a circle of radius of R. Figure 5(b) shows different sampling points for different radii.
(a) The basic LBP operator (Ahonen et al. 2004) and (b) circularly symmetric neighbor sets for different 
(Ojala et al. 2002).
(a) The basic LBP operator (Ahonen et al. 2004) and (b) circularly symmetric neighbor sets for different (Ojala et al. 2002).
. The subscript represents using the operator in a 
neighborhood, and superscript 
stands for using only uniform patterns and labeling all remaining patterns with a single label. Figure 6 shows an example of 
on the river delta patch from ATeX, where the flat, edge-like, and corner-like regions of the image are highlighted on images and histograms. 
Different patterns are highlighted on both image and histogram resulting from an LBP response.
Different patterns are highlighted on both image and histogram resulting from an LBP response.
is defined as
Different parameters, i.e., sampling points 
, radius 
, and uniform or non-uniform pattern, offer different resulting histograms. So, it is important to adjust the parameters based on the problem. In this study, sampling points, radius, and pattern are considered 8, 1, and ‘uniform,’ i.e., 
, respectively. Another consideration is that histograms cannot preserve the spatial information across the image. So, for the face recognition problem, it is suggested to implement regional LBP for different parts of the face to preserve the spatial information (Ahonen et al. 2004). In the case of water, however, due to the irregularity of water features in spatial coordinates, regional LBP would not be effective. Moreover, several possible dissimilarity measures have been proposed for histograms. Log-likelihood, χ2, and Kullback–Leibler Divergence (KLD) are provided in this study according to the following equations:
- Log-likelihood statistic:(5)
- χ2 statistic:(6)
- KLD:where(7)
and Q are two probability distributions.
DL-based representations
Preliminary classification results on low-level vision-based methods (section 2.1) showed the important role of texture information in the better performance of KNNs in classifying different waterbodies. In this section, we investigate the performance of the DL-based model in feature extraction. DL models are capable of automatically learning patterns from raw data through multiple layers of processing (LeCun et al. 2015). It is because in these models, each layer transforms the input representation into a higher-level representation, which let the deeper layers learn more important aspects of the raw data and discard irrelevant variations (higher-level representation) (Eltner et al. 2021).
We train ShuffleNet V2 × 1.0 (Ma et al. 2018), a DL-based classification model designed for mobile devices with very limited computing power, on ATeX dataset. The PyTorch pre-trained ShuffleNet V2
is fine-tuned on 32
32 patches with 30 training epochs, we use SGD optimizer with a momentum of 0.9 and weight decay of 0.0001, and the learning rate and batch size are set to 1.0
10−2 and 64, respectively.
ShuffleNet V2 is an efficient CNN architecture inspired by ShuffleNet (Zhang et al. 2018). ShuffleNet is a network architecture widely adopted in low-end devices such as mobiles. In ShuffleNet architecture, ‘bottleneck’ building block (He et al. 2016) is modified by two new operations, pointwise group convolution and channel shuffle, to greatly reduce computation cost while maintaining accuracy. Pointwise group convolution is introduced to reduce computation complexity of 1 × 1 convolution (bottleneck). Channel shuffle operation is also provided to overcome the side effects brought by group convolutions (Figure 7(a)).
1 convolutions are no longer group-wise (unlike Zhang et al. 2018) and the same ‘channel shuffle’ operation as in (Zhang et al. 2018) is used to enable information communication between the two branches. 
(a) Channel shuffle with two stacked group convolutions. GConv stands for group convolution (Zhang et al. 2018). (b) ShuffleNet Units. (1) bottleneck unit (He et al. 2016); (2) ShuffleNet unit with depthwise convolution (DWConv) (Chollet 2017; Howard et al. 2017), pointwise group convolution (GConv) and channel shuffle (Zhang et al. 2018); (3) ShuffleNet V2 unit (Ma et al. 2018) using channel split operator.
(a) Channel shuffle with two stacked group convolutions. GConv stands for group convolution (Zhang et al. 2018). (b) ShuffleNet Units. (1) bottleneck unit (He et al. 2016); (2) ShuffleNet unit with depthwise convolution (DWConv) (Chollet 2017; Howard et al. 2017), pointwise group convolution (GConv) and channel shuffle (Zhang et al. 2018); (3) ShuffleNet V2 unit (Ma et al. 2018) using channel split operator.
(a) Neural network architecture of Autoencoder. (b) The loss function decay rate of the linear Autoencoder on ATeX training set over 300 epochs.
(a) Neural network architecture of Autoencoder. (b) The loss function decay rate of the linear Autoencoder on ATeX training set over 300 epochs.
1.0 is used to extract features. The extracted features are plotted to provide a better understanding of DL-based model performance in feature representation. To do this, features are first fed into t-Distributed Stochastic Neighbor Embedding (t-SNE) (Van der Maaten & Hinton 2008; Van Der Maaten 2014) and linear Autoencoder (AE) for dimensionality reduction. Both methods are particularly well-suited for the visualization of high-dimensional datasets. Final results are then plotted on 2D graphs (Figure 9). 
Visualization of reduced 2D representations resulted from t-SNE (left) and AE (right) using features extracted from ShuffleNet V2
1.0.
Visualization of reduced 2D representations resulted from t-SNE (left) and AE (right) using features extracted from ShuffleNet V21.0.
t-Distributed Stochastic Neighbor Embedding
under a Gaussian centered at 
over pairs of high-dimensional objects to calculate the similarity of datapoint 
to datapoint
. The formula to calculate 
is as follow:
. A binary search is performed for the value of variance that produces probability distribution with a fixed perplexity that is specified by the user using the following equation:
Perplexity, learning rate, and the number of iterations are set to 20, 200, and 1,000, respectively. The gradient calculation algorithm uses ‘Barnes-Hut’ approximation, which is the fastest t-SNE implementation. Input 32
32
3 holds the raw pixel values of the image in RGB color space. The image is passed through the feature layer of ShuffleNet V2
1.0 resulting in 1,024 extracted features. Then extracted features are fed into t-SNE for dimensionality reduction in low-dimensional space of two dimensions.
Autoencoder
AEs are unsupervised learning techniques commonly used to learn efficient data encoding. Basically, autoencoders can learn to map input data to the output data, while they learn how to encode the data. The output is the compressed representation of the input data. In fact, the main objective of training an AE neural network is dimensionality reduction. In the other words, ‘Autoencoding’ is a data compression algorithm where the compression and decompression functions are (i) data-specific, (ii) lossy, and (iii) learned automatically from examples rather than engineered by a human. The principles behind AEs are as follows:
- 1.
The encoder takes the input and encodes it,
. - 2.
Between the encoder and the decoder, there is an internal hidden layer, h, which learns the coding of the input that is defined by the encoder,
. - 3.
The decoder function tries to reconstruct the input data from the hidden layer coding. Considering the decoder function as
, the reconstruction (
) can be defined as 
or, i.e., 
.
Figure 8(a) shows the architecture of the AE used in this study. The encoder units take 1,024 features extracted by ShuffleNet V2 × 1.0, it includes three hidden layers which consist of 128, 64, and 12 nodes, respectively. Over the encoding path, features decrease from 1,024 to 2D representations in the latent space. The same architecture is adopted for the decoder unit. Mean Squared Error (MSE) and ‘Adam’ (Kingma & Ba 2014) are used as loss function and optimizer, respectively. The network is trained for 300 epochs. The initial learning rate is set to 1.00 × 10−3, and the multiplicative factor of learning rate decay is set to 0.1 which is scheduled for every 90 epochs. The loss function decay rate on ATeX training set is shown in Figure 8(b).
EXPERIMENTAL RESULTS
Features extracted from different methods are fed into customized KNNs using different dissimilarity metrics (Euclidean [L2N], χ2, Log-likelihood [LLV] and KLD) for estimating the accuracy of classification on the validation set. For raw images with different color spaces, and texture features captured from Gabor magnitude responses, L2N is used as the distance metric. In the case of LBP, the dissimilarity of histograms of ‘patterns’ resulting from LBP operation is compared using three different dissimilarity measures including χ2, LLV, and KLD. In the case of the DL model, and in order to achieve fair results, images of the validation set are first passed into the feature layer of the pre-trained ShuffleNet V2
1.0 (on ATeX training set), then the extracted features were fed into the KNNs.
Despite the fact that KNNs have considered ML tools, it is a simple data-driven method that just compares the validation patches with all training ones and reports the dissimilarity vector for each validation input. So, the final results show just the performance of filters in extracting the features of patches, and the classification method does not have any significant effect on the performance of filters.
The accuracy results are reported in Table 1. The results are categorized based on image color space, feature extraction methods, model parameters, and metrics for dissimilarity evaluation. The number of neighbors (K) for KNNs ranges from 1 to 500. Results show the highest performance of DL model features ranging from 88 to 92% depending on the number of neighbors. After the DL model, the raw images in HSV color space, and LBP methods (independent of dissimilarity measurement method) offer the best performance with 50 and 35% accuracy, respectively. On the other hand, raw images in RGB color space provide the lowest performance with 6% accuracy. For grayscale images, Gabor magnitude responses show 29% accuracy, while the highest performance for raw grayscale images does not exceed 24% accuracy.
The experimental results on ATeX validation set
| Image | Gabor wavelets | LBP | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| K | RGB | G | HSV | RGB | G | HSV | KLD |  | LLV | DL |
| 1 | 6 | 20 | 50 | 8 | 23 | 18 | 24 | 24 | 24 | 92 |
| 3 | 6 | 20 | 48 | 9 | 25 | 20 | 26 | 27 | 27 | 92 |
| 5 | 6 | 20 | 49 | 9 | 27 | 21 | 28 | 29 | 29 | 92 |
| 8 | 6 | 21 | 49 | 9 | 28 | 22 | 31 | 31 | 30 | 92 |
| 15 | 6 | 19 | 47 | 9 | 28 | 22 | 33 | 32 | 33 | 91 |
| 50 | 6 | 21 | 44 | 9 | 29 | 22 | 35 | 34 | 34 | 91 |
| 70 | 11 | 20 | 43 | 10 | 29 | 22 | 35 | 35 | 35 | 90 |
| 100 | 11 | 21 | 43 | 10 | 27 | 22 | 34 | 34 | 33 | 90 |
| 200 | 11 | 22 | 40 | 10 | 27 | 23 | 32 | 33 | 32 | 89 |
| 300 | 11 | 22 | 38 | 10 | 27 | 23 | 32 | 32 | 32 | 89 |
| 500 | 11 | 24 | 36 | 9 | 27 | 23 | 30 | 30 | 30 | 88 |
| Image | Gabor wavelets | LBP | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| K | RGB | G | HSV | RGB | G | HSV | KLD | | LLV | DL |
| 1 | 6 | 20 | 50 | 8 | 23 | 18 | 24 | 24 | 24 | 92 |
| 3 | 6 | 20 | 48 | 9 | 25 | 20 | 26 | 27 | 27 | 92 |
| 5 | 6 | 20 | 49 | 9 | 27 | 21 | 28 | 29 | 29 | 92 |
| 8 | 6 | 21 | 49 | 9 | 28 | 22 | 31 | 31 | 30 | 92 |
| 15 | 6 | 19 | 47 | 9 | 28 | 22 | 33 | 32 | 33 | 91 |
| 50 | 6 | 21 | 44 | 9 | 29 | 22 | 35 | 34 | 34 | 91 |
| 70 | 11 | 20 | 43 | 10 | 29 | 22 | 35 | 35 | 35 | 90 |
| 100 | 11 | 21 | 43 | 10 | 27 | 22 | 34 | 34 | 33 | 90 |
| 200 | 11 | 22 | 40 | 10 | 27 | 23 | 32 | 33 | 32 | 89 |
| 300 | 11 | 22 | 38 | 10 | 27 | 23 | 32 | 32 | 32 | 89 |
| 500 | 11 | 24 | 36 | 9 | 27 | 23 | 30 | 30 | 30 | 88 |
Note: Raw images, Gabor responses, and features extracted from ShuffleNet are evaluated based on L2N measurement (numbers are in percent).
It is worth mentioning that considering the high computational complexity and processing time needed for convolution operation, Gabor operation is too expensive compared with LBP operation.
DISCUSSION
Color significance
The KNNs results on raw images in Table 1 show the significance of color and color space in different waterbodies.
Violin plot of (randomly selected) samples of raw pixel intensity values.
According to Figure 10, samples collected for each label have entirely different statistics (e.g., data distribution, minimum, maximum, median, and interquartile range). This proves that color plays an important role and can provide additional information for each waterbody.
We believe the reason for the features extracted by ShuffleNet to be more informative compared to other methods in this study and other types of CNN-based models (Erfani & Goharian 2022) is that the ShuffleNet units can directly take advantage of color information in each channel by the ‘channel split’ operator.
Color space significance
Clustering results (DBSCAN and K-means) on 2D representations of glacier images in the ATeX training set. Please refer to the online version of this paper to see this figure in color: http://dx.doi.org/10.2166/hydro.2023.146.
Clustering results (DBSCAN and K-means) on 2D representations of glacier images in the ATeX training set. Please refer to the online version of this paper to see this figure in color: http://dx.doi.org/10.2166/hydro.2023.146.
According to Figure 11, the first and second rows, respectively, are the results of DBSCAN and K-means on ‘glacier’ 2D representations concluded from t-SNE analysis. The third and fourth rows are the same analysis on the glacier class concluded from linear AE analysis. Figure 11(a) plots the DBSCAN clusters on 844 images of the glacier in the train set. The DBSCAN algorithm views clusters as areas of high density separated by areas of low density. Due to this rather generic view, clusters found by DBSCAN can be any shape, as opposed to K-means which assumes that clusters are convex shaped. The central component of the DBSCAN is the concept of core samples, which are samples that are in areas of high density (Pedregosa et al. 2011).
There are two hyper-parameters in DBSCAN, min_samples and eps that are considered 5 and 0.3, respectively. Figure 11(b) shows the Silhouette for the various clusters. The Silhouette score is used because the ground truth labels are not known (all images belong to the glacier class). The Silhouette coefficient is an evaluation that is performed using the model itself, where a higher Silhouette coefficient score relates to a model with better-defined clusters. Figure 11(c) compares statistics of sub-clusters by (violin) plotting the corresponding 50 pixels which are randomly sampled before, and Figure 11(d) compares pixel intensity values of samples existing in each sub-clusters. Three consistent color codes (gray, cyan, green) used in Figure 11 represent the same sub-cluster in different subplots. Moreover, in the DBSCAN method, the outliers are indicated by the ‘Red’ color in both the scatter and violin plots.
The right column Figure 11(d), 11(h), 11(l) and 11(p)) shows the pixel intensity values (which are randomly sampled from all three RGB channels) existing in each sub-clusters. As it is evident, there is a noticeable difference between pixel intensity values between sub-clusters for both t-SNE and AE 2D representations. This difference is more obvious in the sub-clusters determined by the DBSCAN method (Figure 11(d) and 11(l)).
Investigation on information provided by different color codes for glacier sub-clusters resulted by DBSCAN. Please refer to the online version of this paper to see this figure in color: http://dx.doi.org/10.2166/hydro.2023.146.
Investigation on information provided by different color codes for glacier sub-clusters resulted by DBSCAN. Please refer to the online version of this paper to see this figure in color: http://dx.doi.org/10.2166/hydro.2023.146.
ATeX for transfer learning
In this section, we pre-trained the backbone of a CNN-based model for semantic segmentation of waterbodies on ATeX to investigate the performance of ATeX for transfer learning. For this purpose, we pre-trained ResNet-101 (He et al. 2016) on ATeX as the backbone of PSPNet (Zhao et al. 2017), a semantic segmentation network. Then, we trained PSPNet on ATLANTIS, a benchmark for semantic segmentation of waterbody images (Erfani et al. 2022). The results of this experiment are compared with those of PSPNet using pre-trained ResNet-101 on ImageNet and reported in Table 2.
The per-category results on the ATLANTIS test set by PSPNet-ATeX and PSPNet-ImageNet
| Labels | canal | ditch | fjord | flood | glaciers | spring | lake | puddle | rapids | reservoir | river | delta | sea | snow | pool | waterfall | wetland | A-mIoU | A-acc | mIoU | acc |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PSPNet-ATeX | 54 | 23 | 43 | 40 | 51 | 58 | 27 | 55 | 47 | 26 | 31 | 64 | 61 | 52 | 47 | 54 | 52 | 46 | 61 | 41 | 73 |
| PSPNet-ImageNet | 56 | 29 | 48 | 35 | 52 | 54 | 31 | 55 | 41 | 26 | 29 | 54 | 61 | 49 | 45 | 58 | 48 | 45 | 64 | 41 | 73 |
| Labels | canal | ditch | fjord | flood | glaciers | spring | lake | puddle | rapids | reservoir | river | delta | sea | snow | pool | waterfall | wetland | A-mIoU | A-acc | mIoU | acc |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PSPNet-ATeX | 54 | 23 | 43 | 40 | 51 | 58 | 27 | 55 | 47 | 26 | 31 | 64 | 61 | 52 | 47 | 54 | 52 | 46 | 61 | 41 | 73 |
| PSPNet-ImageNet | 56 | 29 | 48 | 35 | 52 | 54 | 31 | 55 | 41 | 26 | 29 | 54 | 61 | 49 | 45 | 58 | 48 | 45 | 64 | 41 | 73 |
Note: The best results of PSPNNet-ATeX are highlighted with bold font.
The PSPNet is implemented using PyTorch. During training, the base learning rate is set to 2.5 × 10−4 and it is decayed following the poly policy (Zhao et al. 2017). The network is optimized using SGD with a momentum of 0.9 and weight decay of 0.0001. In total, we train the network for 30 epochs, around 80K iterations with a batch size of 2. The training data are augmented with random horizontal flipping, random scaling ranging from 0.5 to 2.0, and random cropping with the size of 640 × 640. The proposed network is trained in a fully supervised fashion constrained by the cross-entropy loss function on both final prediction P (main loss) and the intermediate feature produced by the fourth block of the ResNet-101 (auxiliary loss). Following (Zhao et al. 2017), the weights of the main and the auxiliary loss are set to 1 and 0.4, respectively.
For performance evaluation, we take the mean of class-wise intersection over union (mIoU) and the per-pixel accuracy (acc) as the main evaluation metrics. To further evaluate the performance of the waterbodies, we calculate the mean IoU and the accuracy just for the aquatic categories – A-mIoU and A-acc, respectively. Aquatic categories include 17 labels showing just water content in different forms and bodies, e.g., sea, river, lake, etc. In Table 2, PSPNet equipped with a pre-trained ResNet backbone on ATeX is called PSPNet-ATeX, while PSPNet using pre-trained ResNet backbone on ImageNet is called PSPNet-ImageNet. According to the results, and considering mIoU as a more informative metric for semantic segmentation, PSPNet-AteX provided better performance over 10 labels out of all 17 aquatic labels. PSPNet-AteX achieved 40.40% mIoU on Flood, as the most important label in ATLANTIS, 5.0% more than what was achieved by PSPNet-ImageNet. Totally, PSPNet-ATeX outperformed PSPNet-ImageNet over aquatic labels by 0.81%. Considering accuracy, however, PSPNet-ImageNet performed better by achieving 63.74% accuracy, 2.49% better than PSPNet-ATeX.
CONCLUSION
This study investigated vision-based texture and color analysis techniques on the water to find textural properties of water in different aquatic (eco)systems. ATeX dataset, a benchmark for texture analysis of water in different waterbodies, made it possible to implement and apply different CV techniques on the water. Three different methods including Gabor kernels, LBP, and DL-based model were applied in this study. LBP results showed that water texture analysis is a more difficult problem compared with the task of facial recognition in CV. The classification results proved that the high-level image analysis method (i.e., DL-based model), outperformed much better than early-vision techniques for water feature extraction. KNN results on raw images emphasized on the important role of color and color space for water. Using HSV color space increased the accuracy of classification results to 50%. Among early-vision techniques, LBP offers better results, depending on the parameters which should be adjusted based on the problem. Finally, and beyond our preliminary development of datasets and models, this research aims to call the community to start building and sharing new robust water and hazard detection models using advances in image processing.
The results of this study give DL model developers insight to design customized CNN-based models for detecting and segmenting water in surveillance imageries. These models then can be deployed during flash or nuisance flooding in urban areas, for real-time monitoring of flood events. Surveillance imagery networks can provide spatial dynamics of the surface water extent in the monitored region. Such a vision-based framework is capable to provide disaster prevention agencies with actual field information such as water stage and discharge for over-bank flow states.
DATA AVAILABILITY STATEMENT
The ATeX dataset, models, and codes developed and used in this study are publicly available in the GitHub repository (https://github.com/smhassanerfani/atex).
CONFLICT OF INTEREST
The authors declare there is no conflict.
%20cropped.png?versionId=7142)