Assessing the impact of PET estimation methods on hydrologic model performance

Potential evapotranspiration (PET) is a key input to hydrological models. Yet, there is no consensus on which the PET estimation method is most applicable for hydrological modeling (Xu & Singh 2002; Oudin et al. 2005a). Verstraeten et al. (2008) presented a comprehensive overview of the scientific literature on methods for estimating PET and stated that the selection of one method from many is primarily dependent on the objectives of the study and the type of data available. However, if a perfect PET input was needed to run a hydrologic model successfully, such a model would be rarely used in the real world, as evapotranspiration is challenging/expensive to measure accurately. Therefore, understanding how models respond to PET inputs is critically important.

The influence of the applied PET method on rainfall–runoff models has been subjected to several studies. Parmele (1972) observed varying sensitivity to PET biases of +10% and −20% on three watershed models in nine US catchments, indicating that the relative importance of PET accuracy is likely to vary according to location, as well as how model performance is assessed (e.g., whether it focuses more on high or low flows). Vázquez (2003) used three different PET inputs to run the Système Hydrologique Européen (SHE) model and found that different PET inputs had significant impacts on the accuracy of streamflow simulation. Xu & Singh (2002) compared six PET estimation methods in Switzerland and found that radiation-based methods performed better than temperature-based methods. Evans (2003) tested several regional climate models associated with the CMD-IHACRES model on a catchment in the central USA and obtained optimal results when estimating evapotranspiration with temperature as a surrogate for PET. In contrast, Roderick et al. (2009a) discussed the downside of using a PET formula that does not adequately represent all relevant processes by highlighting recent trends in pan evaporation data. Over the past several decades, evaporation from Class-A pans has decreased, while at the same time, annual temperatures have risen (Fu et al. 2009; Roderick et al. 2009b). Therefore, they suggested that pan evaporation would have increased if rising temperature leads to rising evaporation; the opposite of what has been observed from recent pan data. Much of the observed declines in pan evaporation was attributed to declines in radiation and/or wind speed. This implies the potential importance of including other factors that are affecting evaporation, such as wind speed and radiation in the calculation of PET.

Although different PET methods may give significantly different results, some studies showed that hydrological models tend to be insensitive to differences in PET inputs (Andersson 1992; Andréassian et al. 2004; Oudin et al. 2005a). Andersson (1992) compared the output of the HBV model calibrated to seven different PET methods, including daily Penman PET values, mean monthly Penman PET values, and temperature-based PET models. The HBV model coped with the biases of PET methods by adapting its precipitation correction factor (i.e., a precipitation adjustment factor) without noticeable efficiency loss. While the differences between PET methods in terms of model efficiency were marginal, temperature-based methods slightly improved the performance of the model. According to Andersson (1992), mean Penman PET produced better runoff simulations than time-varying Penman PET. Nandakumar & Mein (1997) found that random PET errors could lead to significant runoff errors, but the sensitivity of watershed models depended on the dominant hydrological processes. While the bias in PET showed a linear relationship with bias in the predicted runoff, the bias in PET had a lower effect than bias in rainfall. Andréassian et al. (2004) tested the impact of a regionalized Penman PET on the performance of two rainfall–runoff models on a sample of 62 mountainous catchments. They found that the same average PET input for all catchments (i.e., a very simple assumption on PET) yielded the same result as a more complex input obtained from regionalization. Oudin et al. (2005b) compared different PET estimation methods on four conceptual rainfall–runoff models and found that both simplistic and complex PET estimation methods can achieve similar model performances. Oudin et al. (2005a) found no systematic improvements in rainfall–runoff model efficiencies when using temporally varying PET instead of mean PET and concluded that the rainfall–runoff models were poorly responsive to detailed PET inputs.

However, none of these studies has addressed rainfall–runoff model sensitivity to PET inputs based on catchment classification of long-term energy demands of watersheds (e.g., Budyko classification). We hypothesize that catchment energy availability represents a primary control on hydrologic model sensitivity to PET inputs. This study explicitly explores the importance of PET time series to hydrologic models in 57 catchments across diverse conditions, including climate, land cover, and energy/water-limitations due to differences in the dominant factors that influence the changes in the actual evapotranspiration (AET).

A random subset of 57 catchments from the Model Parameter Estimation Experiment (MOPEX) data set (Duan et al. 2006) has been selected for analysis to provide broad geographical coverage with reasonable computational time. Test catchments are located within eight aggregated US EPA Level II eco-regions (Omernik 1987) in the USA (Figure 1) and have different climatic conditions (based on Köppen classification) ranging from humid subtropical to cold semi-arid (Beck et al. 2018). Study catchments range in the area from 80 to 10,000 km² (average ≈ 3,000 km²), mean elevation from 10 to 2,600 m (average ≈ 590 m), and mean slope from 0.1 to 50% (average ≈ 9.7%). A number of recent studies have been based on MOPEX catchments (e.g., Sivapalan et al. 2011; Wang & Hejazi 2011; Carmona et al. 2014). This data set contains daily time series of precipitation, streamflow, maximum temperature, and minimum temperature, as well as a climatologic PET estimate that is based on pan evaporation data. Some of the catchments included in the MOPEX data set contain periods of missing streamflow data. Therefore, the most recent 7 years of data with no missing streamflow days were selected for this analysis.

Four PET products were used in this study and are described below: (1) daily MOPEX, (2) mean monthly MOPEX, (3) Hargreaves and Samani (Hargreaves & Samani 1985), and (4) Hamon PET (Allen et al. 1998; Lu et al. 2005) (Figure 2). The daily PET data included in the MOPEX data set used for this analysis were calculated based on the National Oceanic and Atmospheric Administration (NOAA) Evaporation Atlas (Farnsworth et al. 1982). Daily PET data were produced by fitting monthly pan data taken from the NOAA Evaporation Atlas to a Fourier series with a single annual cycle (repeating each year). Daily values of mean PET data included in the MOPEX data set were averaged by month to obtain mean monthly PET data.

Temperature-based methods (i.e., Hargreaves-Samani (H-S) and Hamon methods) were chosen as temperature readings are often the most readily available meteorological data that relates to PET. Although the H-S method only uses a daily measurement of maximum and minimum temperatures as inputs, it effectively incorporates radiation indirectly. Relative humidity and cloudiness are not explicitly contained in the equation, but the difference in maximum and minimum air temperature is related to relative humidity and cloudiness (Samani & Pessarakli 1986). The H-S method has proven to produce good results and has great resilience in diverse climates around the world (Shahidian et al. 2012). Hamon's method (Allen et al. 1998; Lu et al. 2005) is a simple, empirical method that uses temperature as the major driving force for evapotranspiration, but also includes other variables such as daytime length and saturated vapor pressure. In this method, the daytime length is used as an index of the maximum possible incoming radiant energy and the saturated vapor pressure is the moisture-holding capacity of the air at the prevailing air temperature. It has been observed that Hamon's method produces similar estimates of PET as other more sophisticated methods (Vörösmarty et al. 1998).

GR4J (Perrin et al. 2003), one of the most commonly applied, simple, conceptual rainfall–runoff models (e.g., Vaze et al. 2011; Coron et al. 2012; Smith et al. 2018), was selected for use in this study. The four-parameter GR4J model (Table 1) has been demonstrated to perform well in comparative studies (Perrin et al. 2001) and has been extensively tested on the range of climate conditions in the USA, Australia, France, etc. (Le Moine et al. 2007; Coron et al. 2012). The GR4J model involves two components for runoff simulation, a production component that regulates the water balance and a routing component that regulates the water transfer. The soil storage, characterized by its maximum capacity (X₁), stores rainfall and loses water as evapotranspiration and percolation. The production component classifies each day as either wet (P > PET; producing net rainfall) or dry (P < PET; producing a net evapotranspiration). Evapotranspiration from the production store relies on the wet/dry status of a day. For wet days, AET is set to be equal to PET, whereas for dry days, AET is modeled as the sum of the precipitation and the evapotranspiration from the store, which is modeled as a fraction of the net evapotranspiration depending on the water level in the production store relative to X₁. If there is net rainfall, a part of the rainfall will end in the production store. Percolation from the production store, together with the remaining net precipitation, reaches the routing component of the model. The total water available for routing is divided into two fractions: 90% comprise slow runoff that is routed through a unit hydrograph (UH₁) and a routing store, 10% are attributed to fast runoff that is routed through a unit hydrograph (UH₂). UH₁ and UH₂ utilize the same reference time base (X₄), which characterizes the rising time of the unit hydrographs. The amount of water that is stored in soil pores is defined by routing storage (X₃). The value of the routing store varies with the humidity and type of soil. The groundwater exchange coefficient (X₂) is a function of groundwater exchange that influences the routing store. While water inflow to the store from groundwater gives a positive value for X₂, water leaving from the store to groundwater gives a negative value.

The GR4J model was used in combination with the degree-day snowmelt module to account for snow in northern latitudes (Table 2; Kollat et al. 2012). The degree-day method assumes that precipitation occurs as snow when the air temperature drops below a threshold value (TT) and as rain otherwise. If precipitation occurs as snow, it is added to the dry snow component within the snowpack unless it ends up in the free water reservoir, which represents the liquid water content of the snowpack. The snowpack is capable of retaining melt water, expressed as a proportion of the total snowpack storage capacity (CWH). Thus, not all liquid water immediately results in runoff, rather the liquid water holding capacity of snow must first be exceeded. The melt water within the snow can also refreeze (if temperatures are below threshold TT) according to the refreezing parameter (CFR), which is expressed as a fraction of the degree-day factor (CFMAX).

Due to the conceptual nature of the GR4J model, calibration is required to determine appropriate parameter values. Candidate parameter sets were sampled under a Monte Carlo (MC) sampling framework (from uniform distributions) to generate 50,000 parameter sets, and the top 10% of parameter sets (i.e., ‘behavioral’ parameter sets), based on hydrograph fit, were retained for model parameter/performance analysis.

Split-sample validation testing (Klemes 1986) was used to test model skill beyond the calibration period. For this study, 7 years of streamflow data for each catchment were used, with the first 4 years held for calibration and the last 3 years for validation. The model was validated using both direct-validation (i.e., calibrated and validated using the same PET forcing data) and using cross-validation (i.e., calibrated and validated using each PET forcing data).

Test catchments were classified as energy-limited or water-limited based on their α value. The parameter α represents catchment similarity, while it is closely associated with the partitioning of water and energy availability of catchments (Gerrits et al. 2009; Carmona et al. 2014).

The GR4J model was tested with four different PET inputs: daily and mean monthly PET data from the MOPEX data set, daily H-S, and daily Hamon PET forcings. Budyko classification of the selected MOPEX catchments resulted in two hydrologically similar groups on the basis of magnitudes of the similarity parameter α (Carmona et al. 2014). In Figure 3, a catchment is energy-limited when the evaporative fraction (AET/PET) reaches 1 and AET approaches PET; a catchment is water-limited when AET/PET asymptotes to a value less than 1.

The impact of PET inputs on the GR4J model under energy-limited and water-limited conditions was explored. The performance of the GR4J model in calibration and validation periods (in terms of KGE) were analyzed when directly validated with parameters obtained from the same PET forcing data set, as well as when cross-validated with parameters obtained from the other PET forcing data sets. Figure 4 shows the variation in the moving average of the median model performance of the top 10% of parameters (based on KGE) in cross-validation of each PET product for catchments fall along the water- to energy-limited continuum. The median model performance emphasizes the broader pattern along the water- to energy-limited continuum that is obscured by considering only the optimum parameter set. The spread between the lines (i.e., red and green) in Figure 4 is proportional to the impact of PET input data on model performance. Both daily MOPEX and mean monthly MOPEX PET data sets are based on the same actual pan PET data where mean monthly data is simply the monthly based averages of daily MOPEX data. Both H-S and Hamon methods are temperature-based methods where H-S is the most complex PET method among the two methods. Although the H-S method produces slightly higher PET volumes (Figure 2), in general, both PET methods exhibited similar patterns across test catchments (Figure 4). Thus, the model performance of the GR4J model for the most (i.e., H-S) and least (i.e., mean monthly) complex PET data sets are presented in detail (Figure 5). In Figure 5, the model performance for the top 10% of parameter sets is presented for three extremely water-limited (e.g., α = 1.26) catchments and for three extremely energy-limited (e.g., α = 5) catchments. The variability among the four boxplots (median, interquartile range) corresponds to the impact of different PET products on model performance. In addition to the model performance (in terms of KGE), modeled and observed hydrograph comparisons for representative catchments are presented for a detailed comparison of model performance of the GR4J model (Figure 6).

The GR4J model was largely unaffected by the differences in PET forcings under water-limited conditions (Figures 5 and 6). It can be noted that model performance (as KGE) was nearly constant under both direct- and cross-validation tests. KGE was very similar regardless of PET input used, with median values and interquartile ranges showing little variation. Additionally, the spread between modeled and observed hydrographs for two cross-validated PET products showed small differences in water-limited catchments (Figure 6). In contrast, the model was impacted by the differences in PET inputs in most energy-limited catchments (Figures 4–6). The model was more strongly impacted by PET forcing inputs when cross-validated than direct-validated, as expected. KGE varied significantly (both in terms of median values and the interquartile ranges) between the four different PET inputs when cross-validated (Figures 4 and 5). The model performance varied significantly in many catchments under energy-limited conditions, when daily and monthly averaged MOPEX PET parameters were cross-validated to daily Hamon and H-S PET (Figures 4 and 5). Furthermore, model performance was lower (Figures 4 and 5) and the uncertainty (spread of boxplots in Figure 5) was larger when H-S or Hamon parameters were cross-validated on daily/mean MOPEX PET data sets than when daily/mean MOPEX parameters were cross-validated on H-S or Hamon PET data sets.

In many energy-limited catchments, the model performance exhibited less variability when cross-validated between H-S and Hamon PET data, as well as between daily MOPEX and monthly MOPEX PET (Figure 4). This can be attributed to the lack of significant difference between the PET products being cross-validated. Although we displayed model performance in detail for extreme cases (i.e., extremely water-limited and extremely energy-limited), the catchments classified in between the extreme endpoints exhibited intermediate sensitivity to PET reflective of their position along the energy- to water-limited continuum (Figure 4).

The variability of GR4J model parameters to PET inputs was analyzed under energy-limited and water-limited conditions. Figure 7 presents the top 10% of parameters (based on KGE) of the model in detail for extreme cases (i.e., extremely water-limited and extremely energy-limited), when calibrated to each PET data set. Figure 8 exhibits the variation in medians of the most variable parameter along the water- to energy-limited continuum. The model parameterization of the GR4J model did not exhibit variability to changes in PET forcing data in water-limited catchments. For median values and behavioral parameter distributions of the GR4J parameters, the capacity of the production reservoir (X₁), the water exchange coefficient (X₂), non-linear routing reservoir (X₃), and the unit hydrograph time base (X₄) were homogeneous across the four different PET inputs (Figure 7).

In contrast, the model parameterization was impacted by the differences in PET forcing data in most energy-limited catchments (Figures 7 and 8). The GR4J parameter describing the water exchange coefficient (X₂) exhibited the most variability. The median values and distribution of behavioral values of parameter X₂ varied greatly between PET inputs (e.g., energy-limited catchments shown in Figure 7). Other parameters of the GR4J model describing the capacity of the production reservoir (X₁), non-linear routing reservoir (X₃), and the unit hydrograph time base (X₄) did not exhibit significant variability.

The model parameters were not impacted by the differences in the temporal signal between daily and monthly mean MOPEX PET data sets. Medians of GR4J parameter X₂ in many energy-limited sites were similar (Figure 8). Moreover, model parameters were less impacted by the difference between H-S and Hamon PET data sets than the difference of those two data sets to daily and monthly mean MOPEX PET (Figure 8).

PET plays a major role in the long-term water balance in catchments. While the hydrological literature contains a wealth of PET models, little guidance exists on the importance of PET accuracy on streamflow simulations and how/if it varies across climatic regions. To better understand the relationship between PET inputs and streamflow modeling, the variability of the GR4J model performance and parametrization were tested on a sample of 57 catchments located in the coterminous US. We compared the parameterization and performance of the model across eco-regions and energy-limited/wet and water-limited/dry catchments.

We tested 57 catchments in the USA covering eight contrasting ecological regions with distinct landform, climatic and anthropogenic land use characteristics (Figure 1) in order to understand how the impact of PET inputs on the GR4J model varies geographically. Neither model performance nor model parameterization exhibited a strong pattern across eco-regions. The GR4J model uses a simplistic approach for converting PET to AET based on energy and soil water availability for evapotranspiration and is unable to explicitly account for variability in climate, vegetation, land use, and/or soil characteristics. As a result, the inability to identify relationships between model variability and catchment attributes across eco-regions is not surprising.

The GR4J parameters and model performance were unaffected by the differences of PET inputs in water-limited catchments. Conversely, in energy-limited catchments, GR4J parameters and model performance exhibited sensitivity to PET forcing data sets when cross-validated with parameters obtained from the other PET forcing data set. In a water-limited catchment, the AET rate is less than the potential rate due to water-limitation (i.e., determined by the soil water availability). Therefore, a given change to the model forcing PET (e.g., daily MOPEX, mean monthly MOPEX, Hamon, and H-S) is unlikely to result in a significant change to the modeled AET and water-limited catchments are generally unaffected by PET inputs. In contrast, energy-limited catchments have no such water-limitation and as a result can often result in AET being equal to PET (at which point energy is limiting). When this occurs, the difference in PET inputs to an energy-limited catchment is likely to result in a potentially significant difference in the modeled AET. Thus, energy-limited catchments are expected to be largely impacted by PET forcings. Given the differences in the limiting factors that influence the changes in AET in the water- and energy-limited regions, model sensitivity to PET forcings differ between water- and energy-limited catchments. As such, the general findings of many earlier studies (Andréassian et al. 2004; Oudin et al. 2005a) that concluded hydrologic models to be insensitive to PET inputs can likely be explained by locating their test catchments along the water- to energy-limited continuum.

The GR4J model parameters were impacted by PET forcing data in energy-limited catchments, where the groundwater exchange coefficient (X₂), a water balance parameter, showed the most significant change. Variability of the water balance parameter to changes in PET amount is reasonable, as it represents a control on the partitioning of water between stores in the model. When X₂ is negative, water is lost to deep groundwater; when positive, water is gained from groundwater to the basin. Therefore, the value of X₂ varies in response to changes in PET volume (to maintain the best fit to the hydrograph). However, the other water balance parameter of the GR4J model, the capacity of the production store (X₁), did not vary significantly with PET inputs. The routing parameters of the model, the capacity of the non-linear routing reservoir (X₃), and the unit hydrograph time base (X₄) were not responsive to differences in PET inputs. Equifinality, parameter sampling/sample size, and/or the threshold used to select parameter sets in the analysis can confound our understanding of the parameters.

Under direct-validation, the predictive performance of the model is tested against the same characteristic PET data set that was used for the model calibration (i.e., assuming the PET data set is representative of the true condition). In contrast, under cross-validation, the performance of the model is tested against an independent PET data set that is not used for model calibration (i.e., assuming that the forcing PET is not necessarily representative of the true condition). Thus, while direct-validation tests the accuracy of the model, cross-validation tests the robustness of the model to variations in input data. As such, the model exhibits more variability with the PET differences when cross-validated with parameters obtained from the other PET forcing data than when direct-validated with parameters obtained from the same characteristic PET forcing data.

Most energy-limited catchments exhibited the highest model performance variability when cross-validated between daily/mean MOPEX PET and Hamon or H-S PET data sets due to increased variability in model parameters. However, cross-validation combinations of daily and mean MOPEX PET estimates and H-S and Hamon PET did not cause significant changes in the model performances or in the parameter values. Given the manner in which the daily/mean MOPEX PET data sets were derived (see section ‘Study area and data’), it is clear that both are based on long-term average values with the same PET series repeated each year. Thus, model parameter values were homogeneous between the two data sets, resulting in similar model performance. Both H-S and Hamon models were based on daily temperature data which can result in similar PET patterns. This can be attributed to the limited difference in model parameters that resulted in similar model performances.

In many energy-limited catchments, the model exhibited greater performance degradation when H-S or Hamon parameters were cross-validated to daily/mean MOPEX PET data sets than when daily/mean MOPEX parameters were cross-validated to H-S or Hamon PET data sets. H-S and Hamon PET forcing data sets represent the daily temporal dynamics of PET, while the daily/mean MOPEX PET data sets represent long-term averages (Figure 2). When a model is calibrated to a specific, complex signal (e.g., fine temporal dynamics) and tested on a much more generic signal (i.e., long-term averages), the model is less likely to reproduce the pattern than when it is calibrated to the generic signal (which underlies the complex signal) and validated on the complex signal. Over-fitting to noise in a data signal can negatively impact on the performance by decreasing the models' ability to ‘generalize’.

In this study, the daily and mean monthly MOPEX PET data sets had smaller long-term PET volumes than the H-S and Hamon PET data sets. To understand the impact of this on model parameterization and performance, the mean monthly data set was synthetically shifted such that its long-term average volume was equal to the H-S data set (without altering the dynamics of its signal). The GR4J model was re-calibrated using the shifted mean monthly data set and model performance and parameterization of the original mean monthly, the shifted mean monthly, and the H-S PET data sets were analyzed (Figure 9). Although the model performance and parameterization varied between the original mean MOPEX and the H-S data sets, the shifted mean PET data set showed a similar model performance and parameterization to H-S PET under both direct- and cross-validation scenarios. The GR4J model was more strongly impacted by the magnitude of the long-term mean of the PET than from differences in the temporal signal. Thus, in hydrologic models, the accuracy of long-term average PET is more important than the specific PET signals with respect to the performance and parameterization. The effects of this may be obscured in water-limited sites as evapotranspiration there is constrained by water availability. However, the accuracy of long-term PET dynamics is particularly important over regions where evapotranspiration is limited by the availability of energy, where this study showed increased PET-driven hydrologic model variability.

The purpose of this study was to understand the sensitivity of hydrologic model predictions to PET forcing data. It was demonstrated, across 57 test catchments (Figure 1) and four different PET products of varying sophistication (Figure 2), that hydrologic model sensitivity to PET inputs is based on the energy availability of the catchment. As a result, the disparate conclusions of previous studies can be unified under a common framework of understanding; namely, that the relationship between PET and AET is based on water and energy availability and the sensitivity of hydrologic models to PET are a result of the propagation of this threshold mechanism through the model itself. Critically, this information can be used to better allocate limited resources for performing data collection and modeling and has particular benefits in data-scarce regions.

The impact of PET inputs on the GR4J model did not exhibit a spatial pattern based on eco-region classification; however, the modeling results were sensitive to the energy availability of catchments based on a Budyko classification (Figure 3). In energy-limited catchments, the GR4J model responded to changes in PET inputs through its parameters, primarily in terms of the groundwater exchange coefficient (X₂). Consequently, the performance of the GR4J model varied with PET forcing data sets. While rainfall–runoff models are likely to be most strongly influenced by PET forcing data in wet/energy-limited catchments, in dry/water-limited catchments significant variability was not observed.

The GR4J model exhibited the highest variability when cross-validating between daily or mean MOPEX PET and daily Hamon or H-S data sets. The accuracy of long-term PET magnitude was more influential on model parameters and ultimately on model performance than the accuracy of temporal signal. The quality of PET data should not be ignored when catchments are under energy-limited conditions. More realistic PET forcing data should be used to better constrain model parameters sensitive to the PET/AET process and improve model performance, transferability, and fidelity. The results from this study can serve as a guide for data collection and resource allocation efforts in support of hydrologic modeling based on where catchments fall along the water- to energy-limited continuum (Figure 10).

The data that support the findings of this study are openly available through the National Oceanic and Atmospheric Administration's Model Parameter Estimation Experiment (MOPEX) at ftp://hydrology.nws.noaa.gov/pub/gcip/mopex/US_Data/.

Funding for this research was provided through a graduate scholarship from Clarkson University awarded to D. I. J.