PAVLIB4SWAT: a Python analysis and visualization tool and library based on Kepler.gl for SWAT models

Concerns about the availability and quality degradation of water resources, as well as extreme weather events such as floods and droughts, highlight the importance of understanding hydrological systems at the scale of large watersheds and river basins (Zamani et al. 2020; Karakoyun & Kaya 2022; Gelete et al. 2023). However, climate change and human activities such as deforestation, urban development, and unregulated sewage discharge add further complexity to predicting watershed-scale processes, creating additional uncertainty about future water resource conditions (Garrote 2017; Herman et al. 2020). To address these important questions posed by the hydrological community, numerous hydrological models have been developed to simulate the hydrological cycle, investigate cause-and-effect relationships at various spatial and temporal scales, and aid decision-making for better watershed protection and management (Arnold & Fohrer 2005; Zhu et al. 2019).

Among these hydrological models, the Soil and Water Assessment Tool (SWAT) is one of the most popular models that has been widely applied to investigate water availability, water quality, stream channel erosion, plant growth, climate change, and watershed management options, mostly due to its open-source nature, various supported model creation tools, well-prepared documents, and well-connected worldwide user community. The suitability and applicability of a SWAT model in these fields have been justified by the number of published studies related to SWAT (Gassman et al. 2007; Douglas-Mankin et al. 2010; Tuppad et al. 2011). However, even for such a prominent model, the software tools and/or systems for model input and output analysis and visualization for supporting informed decision-making are not considered fully fledged, especially those supporting participatory decision-making in a web environment, and thus inevitably hinder the full realization of SWAT models in supporting decision-making (Rajib et al. 2016; McDonald et al. 2019; Zhang et al. 2019; Deval et al. 2022).

Although not dedicated to model input and output analysis and visualization, desktop-based software tools and systems have been developed in the relatively short history of SWAT (Moriasi et al. 2019; Deval et al. 2022). These software systems and tools include ArcSWAT, QSWAT (a QGIS interface for SWAT), SWAT-CUP (a SWAT calibration and uncertainty program), SWAT Editor, and SWAT Check developed by the SWAT support team and R-SWAT and MWSWAT (a MapWindow plugin for SWAT) from the SWAT community. There are also desktop-based tools focused on SWAT model output analysis and visualization. For example, VIZSWAT (https://swat.tamu.edu/software/vizswat/) is a proprietary visualization and analysis tool that animates time series and spatial data over GIS maps. SWAT Output Viewer (https://swatviewer.com/) generates dynamic thematic maps from multiple model simulation results and provides quick performance statistics and visualization of time series for supporting model calibration. While many software systems and tools play an important role in assisting with model creation and supporting informed decision-making, most are time- and knowledge-intensive to use and sometimes serve as or rely on license-restricted software programs (e.g., ArcSWAT runs on the proprietary ArcGIS and sophisticated SWAT-CUP functionalities are available for licensed users only). These drawbacks, along with the lack of remote access, make these software systems and tools less efficient in supporting participatory decision-making among stakeholders who are not well trained with these software programs or are geographically separated.

In recent years, extensive efforts have been made to introduce SWAT model functionalities to the web, which greatly facilitate modeling result dissemination and participatory decision-making and allow SWAT models to reach a new level of impact in the SWAT model user community (Rajib et al. 2016; McDonald et al. 2019). These efforts include but are not limited to (1) establishing platform-independent and easy-to-use modeling environments that provide access to existing models, model input and output datasets and, more importantly, model simulation and data visualization functionalities (Rajib et al. 2016; Guigoz et al. 2017; Chen et al. 2020) and (2) developing open-source web applications that leverage the data sharing capabilities of existing web technologies and can be duplicated and customized to suit the unique data and modeling needs of stakeholders (Deval et al. 2022; Nguyen et al. 2022). In some instances, a project may fall into both cases.

Examples of the most relevant modeling platforms for SWAT models include SWATShare (Rajib et al. 2016), OpenGMS (Chen et al. 2020), SCOPED-W (Guigoz et al. 2017), and others (Guigoz et al. 2017; Choi et al. 2021). For example, SWATShare is a collaborative environment for SWAT model research and education building on top of HydroShare (Horsburgh et al. 2016), an online collaborative system for open sharing of hydrologic data and models. It provides access to high-performance computing resources for simulation, calibration, and visualization utilities that researchers and educators can use without relying on any local modeling or computational resources (Rajib et al. 2016). OpenGMS provides a mechanism for sharing modeling and simulation resources for geographic applications and a virtual community for collaboration among researchers from various domains to perform open geographic modeling and simulation (Zhang et al. 2020, 2021). OpenGMS presents many models, including the SWAT, storm water management model (SWMM), weather research and forecasting model, and others, which can be published as web services for sharing and reuse (Zhang et al. 2020, 2021). While these platform-based tools target providing general decision-making services to end users, there are also open-source projects that harness the potential of SWAT models for specific decision-making. For example, Babbar-Sebens et al. (2015) developed a web-based software tool for participatory optimization of conservation practices in watersheds. McDonald et al. (2019) created a set of modular web applications to fetches, extracts, and reformats climate data from National Aeronautics and Space Administration servers and online visualizes and analyses SWAT model output. Zhang et al. (2019) established an open-source framework, WOF-SWAT, which allows users to investigate individual and combined impacts of climate change, land use change, and point-source pollutants through online SWAT model simulations. More recently, Deval et al. (2022) built a Prioritization, Interactive Visualization, and Analysis Tool based on the Shiny web application framework for the R programming language to synthesize multiscenario and multiwatershed outputs from process-based geospatial models (i.e., SWAT and the Water Erosion Prediction Project (WEPP)).

These web-based platforms and open-source projects have greatly facilitated collaborative research and participatory decision-making based on SWAT models. Nevertheless, there are some issues that need to be further addressed (Rajib et al. 2016; Choi et al. 2021). First, decision-making services provided by these platforms or projects may not fit the stakeholders' demands (e.g., to animate changes in hydrological regimes over time and space and/or to compare different scenarios side-by-side within one view). Second, adapting and customizing these works to meet the needs of stakeholders cannot be regarded as an easy task. In recognizing these issues, the work presented in this article represents an effort to lower technical barriers to decision-making support by utilizing SWAT modeling in a web environment. We aim to establish a tool and library that are friendly to both end users and developers and, more importantly, require low technical expertise for developers to adopt and customize this work to their own demands (with little knowledge or even no knowledge of front-end technologies or skills). Specifically, we created a tool and library named PAVLIB4SWAT based on the Kepler.gl widget to visualize SWAT model data, including shapefiles from watershed delineation processes, model inputs, and simulated results via dynamic and interactive maps. We then evaluated the utility of PAVLIB4SWAT through a Jinjiang watershed SWAT model use case. Currently, PAVLIB4SWAT only focuses on the analysis and visualization of the input and output of a SWAT model, and model simulation is not yet incorporated.

Kepler.gl is an open-source web-based application for the visual exploration of geospatial data. It was developed by Uber's visualization team and released as an open-source project in 2018. Kepler.gl allows users to create customizable maps and visualizations of large-scale geospatial datasets with ease. The tool provides a user-friendly interface with which users can import their data in various formats, such as CSV, GeoJSON, and Shapefile. Kepler.gl offers a range of customization options, including map layers, color scales, and data filtering. Users can create interactive visualizations with different types of charts and widgets, such as histograms, heatmaps, and time series. Kepler.gl is built on top of Deck.gl, which is also developed by Uber Technologies. Deck.gl is a general purpose, flexible, and customizable framework for building interactive and performant data visualizations that are not limited to geospatial visualizations. However, this flexibility can increase the complexity, especially for users who are new to WebGL or geospatial data visualization. Kepler.gl, on the other hand, is a specific tool designed to simplify the process of creating geospatial visualizations, providing a user-friendly, drag-and-drop interface for users to render large datasets in multiple forms quickly and efficiently by utilizing WebGL technology. In Kepler.gl, all layer geometry calculations are GPU-accelerated, smoothly rendering millions of points and making Kepler.gl a much more powerful web tool than traditional mapping software and other web mapping solutions.

Kepler.gl provides a Jupyter widget that allows users to embed Kepler.gl maps and visualizations directly in a Jupyter notebook and provides a set of programming interfaces that developers can use to customize and integrate Kepler.gl functionality into their own applications. The Kepler.gl API provides a variety of methods and options for working with geospatial data, including data preprocessing, visualization configuration, and interaction. Developers can use the API to customize the appearance and behavior of Kepler.gl maps and layers, add custom data sources and formats, and build interactive dashboards and applications. One of the main advantages of the Kepler.gl API is its flexibility and scalability. The API can handle large datasets with millions of points or polygons and can be used with a variety of data formats, including CSV, JSON, and GeoJSON. It also includes numerous built-in tools for data manipulation, filtering, and analysis, allowing developers to build complex geospatial applications with ease.

SWAT (Arnold et al. 1998; Arnold & Fohrer 2005) is a semidistributed, watershed-scale hydrological model that was initially developed by the Agricultural Research Service of the United States Department of Agriculture to predict the impact of watershed management practices on water, sediment, nutrient, pesticide, and fecal bacterial yields in the agricultural landscapes of North America.

In SWAT, a watershed is delineated in two steps. First, it is subdivided into subbasins. Second, the generated subbasins are further divided into hydrological response units (HRUs), which represent the smallest unit in SWAT for each subbasin, according to unique combinations of soil, land cover, and slope range characteristics. During watershed delineation, some intermediary shapefiles are generated, which store information regarding the divided subbasin, reach segments associated with subbasins, and inlet and outlet segments associated with reach segments. To avoid generating small and negligible subbasins and HRUs, there are thresholds that users can utilize to control the number and size of the generated subbasins and HRUs. These intermediary shapefiles, driving force data regarding the temperature, precipitation, humidity, and radiation provided by modelers, and the SWAT input database are then synthesized to generate model input files that are stored in ASCII text format. The SWAT model input files are organized hierarchically. For example, files such as *.cio, *.cfg, and *.bsn are used to store watershed-level information, files such as *.pnd, *.sub, *.rte, *.swq, *.wgn, and *.wus are attributed to the subbasins, and files such as *.hru, *.mgt, *.sep, *.sol, *.chm, *.gw, and *.ops are attributed to HRUs.

A hydrological cycle simulated using SWAT is divided into two major phases: the land phase and the river routing phase. In the land phase, the water, sediment, nutrient, and pesticide load fluxes are first calculated for each HRU and then calculated by aggregating the HRU fluxes and loads in the subbasin for each subbasin. The results are used as the input for the main channel in the subbasin. The routing phase controls the movement of, for example, water and sediment through the main channel to the subbasin outlet. These two phases generate two main SWAT model files, i.e., output.sub and output.rch, which are used to store the simulated time series for each subbasin and reach, respectively.

PAVLIB4SWAT provides data visualization for the aforementioned intermediary shapefiles regarding the watershed boundary, subbasins, and reaches that were generated from watershed delineation, model inputs pertaining to time series of temperature, precipitation, etc., and model-simulated outputs for reaches and subbasins (e.g., reach segment discharge and subbasin evapotranspiration). Model visualization is a synthesis process that involves the aforementioned three kinds of data. For example, to visualize the discharge time series information, such as the number of reach segments, the start date of the simulation, and total days of the simulation need to be extracted, and the time series data need to be aligned with the shapefile geometries.

The conceptual architecture of PAVLIB4SWAT is shown in the graphical abstract and consists of graphical user interfaces (GUIs), a map generator (MG), a configuration manager (CM), and data convertors and utilities (DCUs). PAVLIB4SWAT does not include any module associated with model creation, calibration, and validation. The shape files and model input shown in the graphical abstract are the intermediate outputs of the watershed delineation process (e.g., the watershed boundary, subbasins, and HRUs in shape file format) and the swat model driver force data such as the precipitation and temperature contained in the TxtInOut folder. While PAVLIB4SWAT provides users with web-based GUIs to analyze and visualize data associated with a SWAT model, the tool itself is implemented in Python only. PAVLIB4SWAT is implemented as an open-source project, and the source code and usage examples in the Jupyter notebook are available at https://github.com/djzhang80/PAVLIB4SWAT. The implemented details of these modules are elucidated in the following paragraphs.

PAVLIB4SWAT utilizes Kepler.gl (specifically a Jupyter Kepler.gl widget) for visual exploration of large-scale SWAT-related datasets. However, Kepler.gl does not directly support these datasets, as they are usually presented in different formats and geographic coordinate systems. The DCUs (item 1 in the graphical abstract) provide various types of converters to convert SWAT-related datasets into a format supported by Kepler.gl. There are many data formats supported by Kepler.gl (e.g., CSV, GeoJSON, DataFrame, or GeoDataFrame), but GeoJSON was selected as the target data format when implementing these convertors for the sake of interoperability (e.g., allowing the converted data to be used in other GIS tools such as QGIS and ArcGIS). In addition to format conversion, converters also perform geographic coordinate system conversion, data cleaning (e.g., clearing the SWAT model output headers), and data association. The data association refers to joining the variables simulated in the SWAT model with geographical units (e.g., linking discharge to a specific river of a subbasin and evaporation to a specific HRU). Specifically, the shape file convertor utilizes the PyShp library to retrieve features from shape files and the PyProj library to achieve geographic coordinate system transformation for the geometries associated with these features. The reason for not employing a higher-level library such as the Geospatial Data Abstraction Library in Python is that we have more control over the implemented schema and thus can achieve more robust conversion than these higher-level libraries (we have tried many of these libraries, but when the input to the convertor includes Unicode encoded characters, the convertor failed). The DCUs also include many utilities that are useful for extracting information for SWAT model input/output (e.g., extracting map boundary and center point for customizing the map style).

Kepler.gl provides a wide range of configurations to create powerful and interactive maps for geospatial analysis, including data sources, layers, filters, map styles, and interactions. Users can specify configurations through configuration files in JSON format or customize the look and feel through interactive configuration operations. However, in either case, the configurations can be very complex, tedious, and time-consuming. The CM (item 2 in the graphical abstract) provides convenient functionalities to generate a preconfigured setting for users to visualize SWAT-related data in a Kepler.gl map. Currently, the CM provides options for layer configurations in Kepler.gl to specific data layer types (such as points, lines, polygons, and heatmaps), color scales for representing numerical data, filters for restricting the data displayed on the map based on certain criteria, and overall map styles for determining the overall look and feel of the map application (such as the base map type, zoom level, and view center).

PAVLIB4SWAT visualizes the SWAT model dataset through a cyclic workflow, which involves data conversion, configuration generation, map production, and map investigation and optimization. Specifically, the last two processes are cyclic. Users can alter the map states and styles in the map investigation and optimization process, save these changes as configurations to the configuration file, and reproduce the map to display the changes on the map. Figure 2 also shows an example of publishing shapefiles generated from watershed delineation. The code producing this interactive web-based map is provided in the PAVLIB4SWAT project GitHub repository. This interactive map application provides almost every functionality that mainstream mapping libraries or tools contain for users to investigate interesting information and customize the look and feel of the created map (except for data editing). Moreover, this interactive map application was produced from a few lines of code in Python and allows users to avoid the challenges of front-end (sometimes full-stack) development that require users to have many skills they may not readily possess.

Dynamic data visualization is implemented through the same static data visualization workflow. As mentioned before, the temporal information must be well organized to contain the feature properties or geometry to visualize dynamic data or time series. In both ways, the features are animated according to their associated temporal information. The first option allows dynamic data to be visualized in ordinary layer types (e.g., points, lines, polygons, clusters, heatmaps), but when temporal information is associated with feature geometries, the dynamic data are visualized through the trip layer. In addition, the dynamic data geometry types also determine the available options for data visualization. In the following paragraphs, we briefly introduce the dynamic data visualization according to the type of geometry (point, line, area) contained in the data.

In the case studies, we mainly presented the usage of PAVLIB4SWAT as a visualization tool for SWAT. Compared with other graphical interfaces for SWAT, such as ArcSWAT, QSWAT, SWAT-CUP, and SWAT Check, PAVLIB4SWAT currently does not perform extensive data preprocessing and postprocessing or manipulation of SWAT. This is because we currently focused on the analysis, visualization, and sharing of the input and output of a SWAT model to facilitate collaborative research and participatory decision-making. In fact, in terms of data visualization and sharing, PAVLIB4SWAT has many merits over other interfaces. For example, as a web-based tool, PAVLIB4SWAT is much easier for users to adopt, render, and share SWAT modeling results vividly over the Internet in comparison to desktop-based applications. Other merits of PAVLIB4SWAT over other applications are discussed in the following section.

In addition, PAVLIB4SWAT currently also has some preprocessing and postprocessing capacities. For instance, PAVLIB4SWAT can convert SWAT inputs and outputs to standard geospatial data (e.g., GeoJSON) that can be imported into a variety of GIS tools for data visualization and manipulation. Moreover, it is very easy for PAVLIB4SWAT to incorporate or integrate with other tools and libraries to manipulate model inputs and perform model simulations to achieve high-level model routines and/or applications such as model calibration and decision-making based on SWAT. To demonstrate these potentials, PAVLIB4SWAT includes a module named ‘ModelEditingService’, which is implemented by reusing the Swat_Edit.exe program that is used by ArcSWAT, SWAT-CUP, and others to perform model input editing. With this module, PAVLIB4SWAT can map climate change and watershed management scenarios to model inputs, perform scenario simulations, and then derive and compare the results of different scenarios (please see a demonstration in our Jupyter notebook).

The case study results indicate that PAVLIB4SWAT can effectively visualize SWAT model data, including shapefiles from the watershed delineation process, model inputs, and simulated results via dynamic and interactive maps. This introduces many benefits to both developers and end users and is in line with the objectives outlined in the introduction section of this article.

First, the interactive maps generated using PAVLIB4SWAT can be distributed as standalone and offline web pages and deployed to self-managed web servers or third-party online platforms. Moreover, as a Python application, PAVLIB4SWAT can also be operated in a Jupyter notebook, which allows users to showcase their code with the corresponding results. This feature allows users to access the system from anywhere at any time without installing additional software (the only needed software is a web client, which is readily available in almost every operating system). This feature can also be of great importance to easily and rapidly disseminate scientific information that modelers independently establish and foster collaboration among stakeholders with different goals and perspectives to achieve feasible and successful determinations in decision-making processes. For example, with the dual map view, users can present and compare the model results of two different management practices to assist policy formulation and decision-making.

The second merit of PAVLIB4SWAT lies in the fact that it was built on top of Kepler.gl, a powerful geospatial analysis tool for large-scale datasets. This provides PAVLIB4SWAT with great advantages in terms of mapping functionalities and performances. By utilizing PAVLIB4SWAT, users can easily create rich map applications for SWAT model data visualization out of box, which may require a large amount of effort and time using other mapping libraries, such as Leaflet and OpenLayers. In addition, Kepler.gl is built on top of MapBox GL and Deck.gl and includes numerous performance optimizations that help ensure that the tool runs smoothly, even when working with large and complex datasets. For example, it uses Deck.gl to provide high-performance rendering of data on a map, and it employs various techniques for data processing and filtering to help streamline the visualization process. These performance optimizations can be very helpful regarding visualizing a large SWAT model dataset (such as the outputs of multiple simulations from a large spatial and temporal scale model), as that would be very difficult or even impossible for other technologies without performance optimization.

Third, PAVLIB4SWAT was designed as an open-source project and implemented purely in the Python programming language. Developers or users do not have to acquire any knowledge of front-end technologies or skills (e.g., HTML, CSS, and JavaScript programming skills) to utilize the interface generated from PAVLIB4SWAT for data mapping and analysis. This gives PAVLIB4SWAT great potential to be used and improved by SWAT or other hydrological model communities. Developers, especially those in the field of hydrology, may not possess the needed skills and would need to invest significant time and effort to learn front-end development, which can be very challenging due to their diversity and variability (multiple programming languages, libraries, and frameworks are involved that are constantly evolving).

While PAVLIB4SWAT allows developers to avoid the challenges of front-end development, it also deprives developers of the opportunity to optimize the underlying code. Currently, variables in each time step of a time series are associated with their corresponding features (e.g., subbasins, reaches, gauges), even though the feature geometry does not change over time. This inevitably results in data redundancy that must be loaded to the map application and can sometimes cause performance issues when the feature geometries are extremely complex. We will discuss the solution to this issue in the next section. Another shortcoming of PAVLIB4SWAT is that feature animation is currently bound to a specific layer, which means there is no way to visualize multiple kinds of features dynamically and simultaneously. This is because a filter can currently be associated with only a specific layer, and a time window can only be linked with a trip layer.

As its name suggests, PAVLIB4SWAT was designed for SWAT model data visualization. However, it can be extended to other hydrological models. To achieve this, the only component that needs extension is the DCU module outlined in the graphical abstract, namely, incorporating data convertors for specific hydrological models. We are planning to add support for the Hydrologic Engineering Center Hydrologic Modeling System (HEC-HMS) model to PAVLIB4SWAT. The HEC-HMS model uses the Hydrologic Engineering Center Data Storage System (HEC-DSS) database for storing and retrieving model input and output. To enable HEC-HMS model data visualization in PAVLIB4SWAT, the DCU module must implement a convertor that can utilize the HEC-DSS API to extract HEC-DSS variables to GeoJSON files. Nevertheless, the HEC-DSS API does not include Python, and hence, developers are needed to use the ctypes (foreign function interface) library to communicate with HEC-DSS via its API in the C programming language. In addition, as an open-source project, we hope that more hydrological models can be added to PAVLIB4SWAT by the broader hydrological modeling community.

Other planned PAVLIB4SWAT improvements include adding decision support and fixing the deficiencies we discussed in the previous section. The data redundancy issue is planned to be addressed by dynamically generating and loading the needed datasets of the current time step. Since PAVLIB4SWAT is wrapped around a Kepler.gl Jupyter widget, which does not have dynamic data loading functionality, we have to achieve this goal through Kepler.gl widget augmentation. Specifically, we plan to modify the JavaScript of the web template (i.e., static/keplergl.html file in the Kepler.gl package) to bind the AddDataToMap function to the time window event dispatcher. The time window event dispatcher will trigger an event at every time step and invoke the AddDataToMap function, which adds the chunk of data associated with the current time step to the map. The shortcoming that PAVLIB4SWAT is unable to animate multiple kinds of features will be addressed in a similar way as the data redundancy problem, i.e., implementing a time window event listener that visualizes snapshots of multiple kinds of feature time series and binds them to the time window event dispatcher.

In this study, a Python analysis and visualization tool and library for SWAT models (named PAVLIB4SWAT) based on Kepler.gl was developed by integrating state-of-the-art methods and libraries, e.g., PyProj, PyShp, Pandas, GeoPandas, and Jupyter notebooks. The PAVLIB4SWAT utility was shown with a case study regarding the visualization of a Jinjiang watershed SWAT model dataset, which had been established in previous studies. The test results suggest that PAVLIB4SWAT can effectively visualize SWAT model data, including shapefiles from the watershed delineation process, model inputs, and simulated results via dynamic and interactive maps. In addition, as an open-source project implemented purely in the Python programming language, PAVLIB4SWAT can effectively allow developers to avoid the challenges of front-end development (otherwise, developers may be needed to acquire technical expertise regarding programming languages, including CSS, JavaScript, and HTML) and can be easily adopted and customized to suit developer demands. We believe the concept framework, tool, and library proposed and developed in this study can serve as a useful basis for the hydrological community to further develop PAVLIB4SWAT into a fully fledged decision-making tool based on SWAT models and even extend it to other hydrological models, such as WEPP, HEC-HMS, and SWMM.

This work was financially supported by the National Natural Science Foundation of China (grant number U22A20554 and 52379013), the Natural Science Foundation of Fujian Province (grant number 2021J011189) and the Science and Technology Project of Quanzhou (grant number 2021N179S).

PAVLIB4SWAT's source code is available at https://github.com/djzhang80/PAVLIB4SWAT under the MIT license.

Methodology was performed by Qiaoying Lin. Software was founded by Qiaoying Lin, Jiefeng Wu, and Yihui Fang. Funding acquisition was done by Qiaoying Lin, Dejian Zhang, and Xingwei Chen. Review for the article was written by Qiaoying Lin, Dejian Zhang, Jiefeng Wu, Yihui Fang, Xingwei Chen, and Bingqing Lin. The original draft was edited by Qiaoying Lin, Dejian Zhang, Jiefeng Wu, Yihui Fang, Xingwei Chen, and Bingqing Lin. The work was supervised by Dejian Zhang and Xingwei Chen.

All relevant data are available alongside with the source code repository, which can be accessed at https://github.com/djzhang80/PAVLIB4SWAT.

The authors declare there is no conflict.