Data analytics in control and operation of municipal wastewater treatment plants: qualitative analysis of needs and barriers

Process instrumentation, control and automation (ICA) are an integral part of the design and operation of today's municipal wastewater treatment plants (WWTPs). ICA systems allow efficient operation of the plant to meet the effluent criteria while minimizing the operational and capital expenses (Olsson 2012). Jeppsson et al. (2002) discuss that in addition to efficiency requirements, increased complexity of the plant will push the development of ICA systems. Future demands for resource recovery from wastewater will further increase the importance of understanding and controlling plant-wide and even system-wide interactions (Solon et al. 2019).

Olsson et al. (2005) state that increasing complexity of WWTPs has led to a more comprehensive and systematic monitoring of the treatment process and its environment. As a result, the availability and quality of monitoring data at WWTPs have significantly improved. However, collected data is not analyzed further to increase process knowledge and select optimal operation strategies. Ingildsen & Olsson (2016) describe measuring as only the first step of effective ICA systems, followed by the analysis and integration of information to decision-making.

Researchers have proposed several new ICA tools for WWTPs based on data analytics, i.e. deriving meaningful patterns from the raw process data. Solutions such as model-predictive control (MPC), process fault detection, and condition monitoring of hardware have been widely studied (see e.g. Åmand et al. 2013; Haimi et al. 2013; Newhart et al. 2019). Despite the promising research and piloting results, a review by Corominas et al. (2018) revealed that only 9% of studied computer-based data analysis solutions had been validated and used in full-scale applications, indicating a gap between research and practice.

Meanwhile, techniques studied for wastewater treatment are already applied in different fields of process industries. Jämsä-Jounela (2007) reviewed the future of process automation in the process industry, stating that MPC already has thousands of applications. Most of the applications are in the chemical and oil and gas industries, but MPC is increasingly adopted in the pulp and paper and food processing sectors. Online prediction of process disturbances and end-product quality are said to be successfully applied in the same sectors. Bauer et al. (2016) demonstrated that various tools for control loop performance monitoring have been adopted in the chemical, oil and gas, minerals processing, and pulp and paper sectors.

Examples from other industries seem to indicate that the barriers in applying data analytics and new ICA tools in general, are not purely technical, but rather contextual. For example, Smuts & Hussey (2011) studied why the power generation industry in the USA had not adopted advanced control applications at the same pace as other industries. Identified key barriers included perceived complexity of the technologies, lack of skilled personnel, uncertainty of cost-benefits, and lack of regulatory incentives.

Dybå et al. (2011) state that in software engineering, qualitative methods are necessary for assessing complex human behavior and organizational aspects. Whereas quantitative research aims to measure and quantify the problem, qualitative research aims to interpret and understand the underlying behaviors, beliefs, and motivation.

This study aims to review the barriers for adoption of new ICA tools and data analytics in municipal WWTPs in Finland. To gather practical views from both user and developer perspectives, the study was conducted through stakeholder interviews. To conclude general views from individual interviews, the transcripts were assessed using thematic analysis. The study complements and updates the results of state-of-the-art survey by Haimi et al. (2009) on ICA practices in Finnish WWTPs, while also discussing broader aspects of data analysis and management related to ICA.

Qualitative research methods have occasionally been applied in the water sector (see e.g. Bamberger 2000; Naman & Gibson 2015). Thematic analysis of ICA systems and advanced data analytics provides a new perspective on non-technical aspects that guide, drive, and hinder their technological development. The systematic approach enables the translation of personal views and experiences into generalized findings. We believe that these results will be of use in the design of future research and contribute to reducing the gap between research and practice.

The study was implemented through a range of stakeholder interviews. Interviewed stakeholders included personnel of WWTPs, consulting companies, automation companies and technology companies. In total, 19 interviews of approximately 2–3 hours length were held during spring 2020. From these, ten interviews were with staff and management of Finnish WWTPs. There are over 500 WWTPs in Finland, from which about 90 WWTP's population equivalent is over 10,000 (Talvitie 2018). Exact treatment requirements vary and are defined in the environmental permit. Average removal rate in 2017 for BOD₇ (biological oxygen demand) was 97%, total phosphorus 96%, and total nitrogen 56%. Various types of WWTPs organizations, with average flowrates from 2,000 to 250,000 m³/d were interviewed to obtain a representative sample. Interviewed organizations are presented in Table 1.

Nine professionals from Finnish consulting, automation, and technology companies were interviewed. The selection of companies for the interview was based on their references and experience of municipal WWTPs in Finland. Based on the share of population and market covered by these WWTPs and companies, their perceptions were presumed to provide a good overview of the municipal wastewater sector in Finland.

Interviews were held as semi-structured, meaning that the key themes and questions were defined beforehand, but responded and discussed further in a conversational manner. Questions for WWTPs dealt with process control, monitoring, data management, and maintenance, including the challenges, plans, and future views regarding the use of data analytics. Questions for companies were focused on the development opportunities and barriers of ICA and data analytics they have faced at WWTPs, including comparison to other sectors they work with. Some of the interviewees sent additional material, such as process descriptions, which were included in the analysis materials. Interviews were recorded and transcribed afterwards.

After the interviews, all collected materials were processed with thematic analysis. Thematic analysis is a transparent method of identifying, analyzing, and reporting patterns in qualitative data (Braun & Clarke 2006). Quotations from the research data, e.g. interview transcripts, are systematically coded with their meanings. A code can be considered to be a class of several quotations indicating the same idea, view, or opinion. From these codes, patterns are identified to construct generalized findings, i.e. themes.

The phases of the analysis process are presented in Figure 1. First, the overall material is reviewed to revise potential gaps or inconsistencies. After that, the initial coding framework is developed based on the research questions. When the coding process is started, all quotations in the material, considered as relevant for the research topic, are coded with their general meaning. While more codes are created, other codes can be reviewed and adjusted to a logical set of findings. Codes can be merged or split to find a balance between a desired level of detail and groundedness, i.e. the number of quotations linked to a single code.

Once all the material has been reviewed and coded, themes are explored. Codes related to each other are grouped and organized to form a complete picture of the theme and views that it encases. Individual codes and their linked quotations are reviewed again to verify their linkage to the theme. Throughout the process, the material can still be revised, and codes created and adjusted as new findings arise. Finally, structured themes are written out into results.

The analysis process was carried out with ATLAS.ti software (Scientific Software Development GmbH, v.8.4.4.). Further instructions on doing a thematic analysis with ATLAS.ti have been given by Friese et al. (2018).

Both users and providers of ICA systems were interviewed. Based on the interviews, 198 codes were created with an average groundedness of 2.6, indicating from how many quotations the code has been concluded. Further on, codes identified as significant for the research question, were organized into themes consisting of 65 codes with an average of 5.3 quotations linked to each code. This indicates that the selected codes are well grounded on the interview findings. For code lists with their respective number of quotations, the reader is referred to the Supplementary Material. Main themes and sub-themes used in thematic analysis are presented in Table 2.

Perceived barriers and future development were categorized by the source, i.e. the type of interviewee presenting these views. Topics per type of interviewee are presented in Figure 2. For WWTP staff, a variety of barriers for optimizing the process control, operation, and maintenance with data analytics originate from the limitations in the physical systems. Interviewed companies perceived most significant barriers in the digital systems, such as software integration.

Consultants shared a similar view with WWTPs but emphasized more the barriers in maintenance work and less the equipment and instrumentation. Technology providers emphasized barriers in digital tools, such as integration of software, while automation contractors highlighted the barriers in actual analysis phase, such as maintaining and updating process models. It is worth noting that the background and professional experience of interviewees clearly guided their answers. This suggests that more active interaction between different professions might fill some gaps and allow more interdisciplinary solutions.

When discussing future development, very few interviewees considered the development of equipment and instrumentation. Most expectations were focused on process control strategies developing towards more predictive and automated control. Maintenance practices were expected to develop mostly along with new digital tools.

Overall themes and their key content were compiled into figures and underlying views described in further detail. Theme figures conclude frequently mentioned views from all interviewees related to the respective theme: descriptions of current practices (now), limitations for more optimal use (barriers), identified development needs (needs), and how they expect the practices in this theme to develop (future). The first theme of physical systems, consisting of the sub-themes of instrumentation, process equipment, and maintenance, is presented in Figure 3.

Maintenance need of instrumentation appeared as the key barrier for several control applications, contributing also to a lack of personnel resources, data quality issues, and unstable control loops. Consequently, the potential benefit from data analytics is reduced. Condition monitoring of sensors and equipment is often based on alarm limits and human senses with some exceptions. For example, sludge dewatering centrifuges commonly have vibration and temperature measurements and some utilities monitor the temperature of critical bearings. Although it is recognized that sensor signals often contain relevant condition data, extracting the information from the fieldbus network is deemed difficult and laborious by the personnel.

Functionalities of instrumentation, such as self-cleaning and self-calibration, have largely developed and, according to the providers, will continue to do so. Intelligent sensors, i.e. sensors with built-in data processing capabilities, have gained some ground at WWTPs, but providers still consider that wider application is hindered by the lack of requirements in the purchasing phase.

Optimizing of control is sometimes also prevented by the physical limits of equipment, e.g. the minimum frequency of aeration compressor is too high for night-time wastewater flow or capacity of polymer dosing equipment is not sufficient for large flows. Poorly dimensioned equipment was found to be more common in smaller WWTPs, in which the actual flowrates is easily derived from the original design values. Some companies considered that re-dimensioning of the equipment based on historical data could often be economically feasible. Calculating the payback time for replacing the equipment, e.g. changing one of the three aeration compressors to a smaller one, could justify the investment.

Currently, some data-driven fault detection and fault prediction for process equipment is offered directly by the technology providers. In these cases, condition data is collected and analyzed by the individual provider. For now, this service has been utilized only for the most critical equipment, e.g. aeration compressors. Fault prediction is expected to further develop in the future, but there is no clear indication yet whether WWTPs will expand the outsourcing of equipment condition data collection and analysis to several different providers, in a distributed manner, or aim to centralize the data collection and analysis to either the automation or the maintenance software.

The second theme of digital systems, including the sub-themes of data analysis and digital tools, is presented in Figure 4. There are few tools for automatically monitoring or verifying the data quality, or for easy marking of faulty data. Lack of quality assurance makes data processing excessively laborious, further hindering its daily use for analysis. The automation contractor often designs and updates the automation and reporting dashboards and functions based directly on the client's requests. Involving more process expertise in the design of digital tools for WWTPs was considered beneficial by several interviewees. For now, the involvement of consultants as the source of process expertise is not seen as economically feasible.

Almost all interviewed utilities planned to analyze their process data more in the future, but for most utilities clear targets were not yet defined. The most frequently mentioned goal was monitoring the energy efficiency, for which the current reporting and analysis tools in the automation system were considered as insufficient. Automation companies saw that a lot more analysis could already be done with the current software, but utilities have no time or expertise to adjust the system to their needs. Definition of common key performance indicators (KPIs) for energy efficiency to enable utility benchmarking was considered beneficial by some WWTPs and companies.

An increasing interest was mentioned for identifying gradual changes in equipment performance, such as degradation of pumps or fouling of sensors. There are also development plans for basic statistical tools to distinguish trends in alarm sources and types. Some potential was seen in both predictive and descriptive analytics for overall supervision of the process state and verifying of impacts from operator actions.

In addition to current tools used in the automation software, some utilities are planning to purchase or have already purchased additional third-party software for further data analysis and visualization, not only for use at the plant but also within all units of the company. It is not yet clear how the overall data infrastructure will develop and who will provide the next generation of tools used in overall process monitoring. Automation providers probably have an advantage over competitors to provide simple add-ins to the already familiar software. However, some interviewees deem the viewpoint of automation companies to utility operations as too narrow for tools that would require a thorough understanding of process behavior.

Key findings related to process control are presented in Figure 5. Several optimized control strategies that the interviewed WWTPs had tested, e.g. ammonium-based feedforward control of aeration, were eventually disregarded due to the increased need for sensor maintenance or instability of the control loop in varying plant conditions. In some cases, instability could also be partly due to poor controller tuning, an issue that various companies considered as a major barrier for process optimization. Tuning of controllers was considered to be often inadequate in the start-up phase and decayed over the years if the plant had no dedicated staff with sufficient expertise for the task.

Most of the barriers mentioned for applying more optimal control strategies are directly or indirectly influenced by the large variation of wastewater flowrate and quality. Integration of plant operation with sewage network operation was rarely mentioned, with a few exceptions of individual development projects to improve access of plant personnel to network data. Several companies mentioned the detached network operation as a major development barrier. Most WWTPs considered that stabilizing the influent flowrate is prevented by the lack of a dedicated equalization basin, but a few had also considered equalization of the influent flowrate with predictive data analytics.

Despite the strong focus on energy efficiency, optimization of chemicals was also deemed relevant both now and in the future. Many utilities already had some development ideas and plans for reducing chemical consumption by using new parameters for dosage control, e.g. alkalinity, and were either purchasing a new instrument for the purpose or using a linear correlation value from an already measured parameter, i.e. simple soft sensor.

Highly complex or black-box control systems did not seem to attract a lot of interest among practitioners. Instead, operator support systems to help the operator to identify and prevent disturbances or decide suitable control actions, especially in the complex parts of the process like aeration, would be adopted more easily.

A lot of future expectations involved more predictive control. This could indicate some interest in integrating the network and plant operation and moving towards a system-wide control. The level of automation was expected to increase, which was seen to result in operator duties changing in the future.

Key findings related to the operating environment are presented in Figure 6. Lack of utility personnel time and resources were considered major barriers by all interviewees. Interviewed WWTP staff mostly felt that apart from their obligatory daily tasks, they simply do not have enough time for additional process optimization or major development tasks.

Several companies and some WWTP personnel also mentioned the lack of incentives from the owner, i.e. municipality, to improve the cost-efficiency of the plant. Consequently, interviewed technology and service providers saw little business potential in expert services and tools for the sole purpose of process optimization. Low quality criteria in tenders, e.g. in the purchasing of instrumentation, was considered to hinder the application of more reliable solutions. Insufficient emphasis on life-cycle costs, in terms of duration and maintenance need, in the purchasing criteria was also highlighted by the providers.

In addition to cost-efficiency, greenhouse gas emissions and overall environmental impact of WWTPs are receiving increased attention especially in larger utilities. Some WWTPs mentioned that small cost savings of a certain control strategy, which resulted in better performance but required more operator work, would not in all cases be a sufficient motive for implementation. However, the reduction of negative environmental impacts, in addition to the cost savings, was seen to justify the investment of personnel time and resources. These examples were mentioned only by WWTPs with concrete development targets and incentives defined by the company board and management.

All interviewee types shared the view that utility operations will be more centralized and outsourced in the future. A satellite model, i.e. centralizing operation of several small plants to one larger one, was expected to gradually take place and push the development towards more centralized data management, increased process automation, and higher level of outsourcing. These changes were seen to provide further motivation for using computer-based data analysis to quickly extract relevant information from the vast amount of process data, reducing the need for site visits and manual inspections. Another highlighted factor was the new generation of personnel, who are expected to bring in new skills and demand better performance and usability from the digital tools.

In addition to the themes and general views, perceptions of technology and service providers about the differences between municipal WWTPs and process industries as operating environment were analyzed. These are only preliminary findings for the planned future work of assessing and identifying potential data analytics and ICA approaches currently used in other industrial processes. These findings will provide additional perspective for development needs and opportunities of WWTPs. Key considerations are concluded in Figure 7.

Many differences in daily process control, operation, and maintenance of industrial plants compared to municipal WWTPs were seen to originate from fundamental differences between private and public sector as the operating environment. The private purchasing process appears to enable negotiation for a better outcome, such as functionalities or control logics without additional cost. Quality criteria were often considered to be set higher in the process industries with more emphasis on life-cycle costs and reliability, instead of the initial acquisition cost.

In the process industries, incentives for resource efficiency were considered to be better implemented throughout the organization. Straight-forward rewarding of each employee, e.g. for energy efficiency, was seen to push the performance towards the set targets. Performance monitoring was more systematic and continuous, e.g. real-time dashboards for energy efficiency KPIs are commonly customized and used for process supervision in the control room.

In addition to resource efficiency, many examples indicated that more effort is made to prevent process failures and the costly production downtime resulting from them. This is reflected in the daily operation in terms of comprehensive quality assurance of sensors, equipment, control loops, and the process in general. The condition of instrumentation is often monitored, e.g. with automatic mass balance checks and sensor self-diagnostics. Control loop performance is automatically monitored and condition-based maintenance, e.g. vibration and temperature monitoring, is widely applied. Advanced process control tools, e.g. fault detection with multivariate statistics, are also applied to identify process disturbances and prevent potential losses of production.

In this study, several types of stakeholders were interviewed, and a thematic analysis was conducted to assess the barriers and needs of WWTPs to apply data analytics for optimizing the process control, operation and maintenance. Many barriers discussed here are not unique to the wastewater industry, and future work will focus on reviewing solutions applied in process industries for similar challenges.

Results in this study indicate that the perceived potential of data analytics is highly dependent on the performance of underlying physical and digital systems, as well as the control strategies and operating environment of the WWTP. Currently, deficits in data collection and processing negatively impact the perceived cost-benefit of actual data analytics. Identified key barriers include laborious instrumentation maintenance, lack of process expertise in the design of current software, instability of control loops, and insufficient incentives for resource efficiency. Needs can be summarized to higher attention for self-diagnostics of instrumentation, data quality assurance, tuning of controllers, and clear development incentives on both individual and organizational levels. Despite the barriers, many potential applications for data analytics are identified, e.g. easy monitoring of energy efficiency, process disturbances, and process supervision in general. In the future, it is expected that centralization and outsourcing of operations, in addition to the next generation of staff personnel, will push the development towards a higher level of automation and thereby increase the importance of data analytics for WWTPs.

This study was conducted across a group of wastewater industry professionals in Finland. Interviews with a range of stakeholders were seen as an effective method of data collection. Expanding the interviews to a larger variety of stakeholders, e.g. policymakers, education and professional training institutes, or simply a larger variety of employees within the same plants and companies, could provide additional insights to barriers and needs in practice.

Thematic analysis provided a systematic framework for analyzing and summarizing a variety of sector expertise, experiences, and personal opinions. The method could be further utilized to its full potential by studying the user acceptance of a specific software or technology, revealing more factors affecting the perceived usefulness and ease-of-use of ICA and data analytics in practice.

We would like to acknowledge the valuable support of six Finnish water utilities for participating in the steering committee and the funding of this project. In addition, we would like to thank the Water Utilities Development Fund of Finland for funding this project.

All relevant data are included in the paper or its Supplementary Information.