Stakeholder engagement to increase the impact of water technology case studies

Stakeholder engagement is an intuitive concept, encompassing communication and relationship-building between interested parties. While there are many practical examples of engagement, efforts to harmonize and understand its tools and methods reveal a gap in scientific frameworks and workflows. Existing literature highlights that most studies emphasize the practical aspects of engagement (Kujala et al. 2022), often neglecting the need for both informal and formal structures to create meaningful impact. The purposes of stakeholder engagement vary, ranging from corporate social responsibility initiatives to collaborative planning and strategy development. A key success factor lies in uniting users toward a common goal.

This paper presents a high-level summary of findings from the ULTIMATE project, a collaborative initiative aimed at fostering water-smart industrial symbiosis through active stakeholder engagement. ULTIMATE, conducted across nine case studies (CSs) in Europe, sought to implement circular economy solutions for water, material, and energy recovery. By promoting sustainable water management practices within industrial symbiosis contexts, the project facilitated active engagement with diverse stakeholder groups, including industries and citizens.

To achieve its goals, ULTIMATE employed a multifaceted approach incorporating three core tools: co-creation exercises for designing multi-use play spaces (Voorberg et al. 2015; Mansilla & Perkis 2017), Communities of Practice (CoPs) (Wenger et al. 2002), and Water-Oriented Living Labs (WOLLs) (Bergvall-Kareborn & Stahlbrost 2009; Rubini et al. 2022a, 2022b). These tools enabled stakeholders to exchange knowledge, develop ideas, and learn together, contributing to innovative solutions for sustainable water management.

In this paper, we describe the methodologies used for these tools and provide a synthesis of insights and results. Co-creation of immersive experiences for CSs fosters shared understanding and awareness and inspires creative solutions for water reuse practices through collaborative design. CoPs foster collaborative knowledge exchange and drive innovation among stakeholders, while WOLLs serve as experimental environments for testing, refining, and scaling solutions. Together, these elements form the backbone of the ULTIMATE project's participatory approach, showcasing how multidisciplinary research can deliver actionable and impactful outcomes.

To aid reader comprehension, we define the following foundational concepts:

• Place by Design Playbook: A framework for integrating stakeholder engagement into co-creation processes, emphasizing inclusivity, adaptability, and iterative design. It guides stakeholders in identifying community challenges, building multidisciplinary teams, and empathizing with the target audience. The playbook also involves understanding the immersive environment, ideating solutions, prototyping, and reflecting on outcomes. By following these steps, it ensures that co-created solutions are tailored to community needs and grounded in collaborative input.

• CoP: A group of stakeholders – ranging from industry experts to community representatives – engaged in continuous learning and knowledge-sharing to address common challenges in water management.

• Living Labs (LLs): Real-life environments where stakeholders collaboratively develop, test, and evaluate innovative solutions. Within ULTIMATE, these LLs are termed WOLLs when they focus specifically on water reuse and management.

• IMXs: Immersive digital tools designed to engage stakeholders in understanding and solving water-related issues through visual, interactive narratives.

Co-creation is central to stakeholder engagement, enabling stakeholders to collaboratively propose, discuss, and prototype solutions for pressing water management issues (Kazadi et al. 2016). This process brings together diverse ideas and resources to create value, addressing specific needs and fostering tailored service experiences. Co-creation engages stakeholders in open dialogue and reflection, helping to identify common ground and solutions that enhance community action, social engagement, and citizen involvement (Markovic & Bagherzadeh 2018).

The ULTIMATE project employs a multi-stage methodology integrating co-creation, community participation, and innovation, supported by frameworks like the Place by Design Playbook. This playbook (ULTIMATE 2024) emphasizes inclusivity, adaptability, and iterative design, ensuring meaningful participation from community representatives, industry experts, and other stakeholders. The outputs of these co-creation activities inform the design of Immersive Media Experiences (IMXs). These IMXs combine real-world data visualizations, location-based narratives, and gamification elements to engage users in addressing water-related challenges. By merging storytelling with interactive technology, IMXs promote participation and understanding of complex concepts like water reuse and industrial symbiosis.

IMXs are central to the ULTIMATE project's engagement strategy. Designed as co-created tools, IMXs serve to translate stakeholder input into interactive and educational experiences. Three CSs – CS2 (Water-Kennis), CS3 (L'Acqua per Tutti), and CS9 (Ultimate Life of Water) – demonstrate how IMXs facilitate understanding and engagement with water management challenges. These experiences address real-world issues, using immersive narratives to foster meaningful participation and actionable insights.

The IMX L'Acqua per Tutti was created through a co-creation process structured by the Place by Design Playbook, ensuring that stakeholder insights informed every design phase. The process began with forming a multidisciplinary team comprising designers, researchers, technical experts, and community stakeholders. This team identified water management challenges specific to the CS, focusing on issues relevant to the local community.

Workshops and brainstorming sessions gathered input from stakeholders, emphasizing audience empathy. The ideation phase incorporated these insights to develop potential solutions, which were iteratively prototyped and refined based on feedback. The co-creation process ensured that the game's interactive elements and narratives authentically reflected stakeholder priorities. For example, challenges such as water reuse and management inefficiencies were translated into gamified tasks, engaging participants in problem-solving scenarios.

The Water-Kennis and Ultimate Life of Water IMXs followed similar co-creation methodologies. In Water-Kennis, stakeholder narratives and interactive elements enhanced user understanding of water reuse practices. In Ultimate Life of Water, a fictional water symbiosis scenario, explored real-world challenges in Kalundborg, such as the impact of introducing new industries on water systems. Stakeholder feedback informed the design, ensuring scenarios reflected realistic challenges.

The process of identifying potential CSs began with a systematic review of online platforms, reports, and literature, complemented by consultations with project partners. The selection process was guided by four core principles: co-creation, community sense, openness, and change-making. These principles ensured alignment with the overarching goals of the ULTIMATE project.

Out of the nine CSs in the project, three were chosen for co-creation activities: KWR Water Research Institute (KWR) in the Netherlands, Aretusa in Italy, and Kalundborg in Denmark. These were selected based on their maturity and readiness for the co-creation process. Each demonstrated sufficient organizational capacity, stakeholder engagement, and technological readiness to effectively implement and benefit from the immersive co-creation activities. Additionally, focusing on three CSs was considered optimal due to the complexity and resource-intensive nature of co-creating immersive experiences. This approach allowed the project to effectively test and exemplify the process's effectiveness across diverse contexts.

Each CS's business activities were then analyzed to uncover operational insights and areas for potential intervention. Visualizations of transactions within these activities helped to better understand their dynamics, informing the selection of appropriate tools and methodologies for engaging stakeholders.

Insights from co-creation workshops informed the development of tools such as the Onboarding Kit for participant introduction, the Facilitator's Slide Deck for leading sessions, and the ULTIMATE Playbook for guiding stakeholders in creating actionable water management solutions.

The Place by Design Playbook serves as the cornerstone of stakeholder engagement and co-creation activities. It guides stakeholders through problem identification, intervention planning, and solution prototyping using an iterative, collaborative approach. The playbook fosters meaningful collaboration and empowers stakeholders to drive actionable solutions for their communities.

By integrating co-creation processes with the development of IMXs, the playbook underpins the ULTIMATE project's methodology. This approach ensures stakeholder insights shape innovative, impactful, and sustainable water management solutions, bridging technical expertise and community-driven action.

The methodology followed distinct phases, integrating various frameworks, tools, and evaluation protocols to ensure a thorough analysis of stakeholder engagement and the impact of IMXs.

The initial phase assessed the effectiveness of co-creation processes using the Kirkpatrick Model, a structured framework evaluating four levels of impact: participants' reactions, learning outcomes, behavior change, and overall results. Feedback was gathered through targeted methods and tools, providing a comprehensive evaluation of how co-creation activities influenced stakeholder engagement and collaboration.

Building on insights from the co-creation process, further evaluations focused on assessing stakeholder tools such as co-creation activities, CoP, and LL). Feedback from the CS support team was used to evaluate these tools' effectiveness in fostering collaboration, facilitating knowledge exchange, and addressing water management challenges within the CSs.

These IMXs were developed using technologies such as virtual reality and AR, designed to foster engagement, interactivity, and experiential learning. Their purpose was to enhance stakeholder understanding, facilitate collaboration, and support informed decision-making processes.

Following the IMX development, the reflection phase of the ULTIMATE Playbook was revisited to ensure alignment with engagement goals. A standardized research protocol was developed and reviewed by CS leaders to guide consistent data collection and analysis across the case studies.

Data collection involved administering pre-treatment and post-treatment questionnaires via a secure online platform. The pre-treatment assessment focused on participants' baseline awareness and engagement with water reuse and industrial symbiosis initiatives, while the post-treatment assessment evaluated the effectiveness of the IMXs in enhancing these aspects. Participants provided feedback on various aspects of their IMX experience, including engagement, immersion, usability, emotional response, and interaction with the game elements. Open-ended questions allowed for qualitative insights into the experiences, offering suggestions for improvement.

Stakeholder evaluation was conducted through structured questionnaires during IMX sessions to engage participants in a collaborative exploration of water management challenges and opportunities. The questionnaires were strategically administered in two parts: at the beginning, to gauge initial perceptions, expectations, and baseline awareness, and at the end, to assess changes in understanding, engagement, and perceived impact. This dual-phase approach provided a comprehensive evaluation of stakeholder experiences, capturing both their initial impressions and the transformative outcomes of the immersive, co-creative process.

The evaluation framework integrated principles from established learning theories and user experience models. Key factors considered included presence, which measures the sense of being immersed in the environment; flow theory, reflecting the deep engagement participants experienced during the activity; self-efficacy, representing confidence in using the technology effectively; ease of use, assessing the technology's usability; emotional influence, exploring the affective impact of the IMX; and esthetics and awareness, which focused on the role of visual and interactive design in enhancing understanding. Data were analyzed quantitatively to identify trends and qualitatively to derive deeper insights into participants' experiences. These analyses informed the iterative refinement of the IMXs, ensuring improvements in usability, interactivity, and engagement.

CoPs have been established to discuss, demonstrate, and evaluate innovative technologies and symbiosis strategies (Fulgenzi et al. 2020). In ULTIMATE, a CoP is understood as a social learning system bringing together experts with local people, end-users and other relevant stakeholders to develop a common understanding, share best practices, and create new knowledge on a given topic, to arrive at solutions that are co-developed, supported, and accepted by the stakeholders (Wenger-Trayner & Wenger-Trayner 2015). Engaging diverse stakeholders such as water providers, technology providers, utilities, industry sectors (agriculture, energy, and water), regulators, authorities, and researchers has been crucial for good stakeholder engagement. These stakeholders regularly interacted throughout the project to exchange knowledge, develop ideas, and collaborate on sustainable water management solutions within industrial symbiosis (Cundill et al. 2015).

CoPs were established in different forms across the ULTIMATE CSs as part of activities related to business-to-business engagement. The KWR supported CS leaders in designing and implementing CoPs to engage with relevant stakeholders. A number of guiding documents have been developed to support CS in designing and implementing their CoPs and workshops are organised to initiate the process. The guidance builds on previous work conducted in a number of EU projects where CoPs were implemented, namely, BINGO, STOP-IT, and NextGen, as well as existing literature. The document is practical in application for CS owners, as well as innovative with a multitude of approaches and avenues to convene multidisciplinary CoPs. As such, the guidance has been particularly useful to support technical staff in engaging locally relevant stakeholders from various expertise and backgrounds in plenary and focus group meetings. These spaces are where stakeholders discuss, collaborate, and outline steps for implementing water-related technologies and innovations. These interactions promoted mutual learning and relationship-building among local partners.

KWR developed an eight-step guidance for preparing and implementing a CoP. The steps include: (1) defining the CoP Coordinator and Moderator, (2) setting goals and scope, (3) identifying preliminary topics, (4) mapping stakeholders, (5) reaching out to participants, (6) preparing and hosting CoP meetings, (7) maintaining engagement between meetings, and (8) evaluating and reporting.

CoP meetings in all nine CS were facilitated by CoP coordinators and moderators from project partner institutions. The number and frequency of meetings were tailored to each CS's needs and circumstances, sometimes guided by CoP roadmaps. These roadmaps helped define the scope, topics, participants, meeting types, and timelines. Templates for reporting on CoP meetings and obtaining stakeholder consent were provided to CS partners.

The CoPs were evaluated using a framework based on the work by Fulgenzi et al. (2020) implemented through an online survey (see annex 4). The evaluation assessed several indicators across six key success factors, namely organizational aspects, atmosphere, stakeholder inclusion, and representation, convergence toward shared perspectives, identification of opportunities and challenges, and knowledge generation. Insights from the evaluations were used to guide improvements and capture best practices relevant for future implementation of CoPs in ULTIMATE, but also beyond.

WOLLs are specialized LLs focused on addressing water-related challenges within real stakeholder communities, using a user-centered approach (Rubini et al. 2022a, 2022b, 2024). Unlike traditional LLs, which support innovation across various sectors, WOLLs are dedicated to developing and testing water-related solutions tailored to the needs of specific local contexts. The methodology used to assess the maturity of ULTIMATE CSs to become WOLLs is grounded in the LL literature, utilizing the Harmonisation Cube method (Mulder et al. 2008) tailored for the water sector. This method enables coordinated assessment analysis, synergistic development, harmonization, and networking of regional WOLL initiatives. To support the evolution of ULTIMATE CSs into WOLLs, a comprehensive assessment of all CSs was conducted to establish a baseline benchmark of the current situation, providing a percentage score to quantify their readiness level. Building on this foundation, we focused on the implementation of best practices and approaches in line with the WOLLs methodology, specifically engaging with those CSs identified as mature and ready for further development into WOLLs, to prioritize efforts and maximize impact.

WOLLs are recognized as a key driver for future strategic priorities in the water sector, and they play a crucial role in addressing societal challenges such as pollution and the impact of climate change. It is imperative for WOLLs to adopt a consistent approach in their establishment and practices, facilitating the generation and sharing of innovations and best practices in a coordinated manner.

This comparative analysis focuses on two case studies, L'acqua per Tutti (CS3) and Water-Kennis (CS2), which implemented IMX to address water sustainability from different perspectives. Through these cases, we evaluate methodologies, participant engagement, flow, emotion, skills, usability, and implications for future IMX development.

The evaluation of the IMX utilized constructs based on established theories, including Constructivism (Piaget 1964) for Awareness and Understanding and Perceived Impact and Social Cognitive Theory (Bandura 1997) for Perceptions and Attitudes and Changes in Perceptions. Additional constructs included Presence and Engagement (Witmer & Singer 1998), Flow (Heutte 2011), Usability (Brooke 1996), Emotion (Pekrun 2014), and Skill (Murphy et al. 1989), all assessing user interaction, engagement, and learning outcomes.

Co-creation exercises played a pivotal role in shaping the design and implementation of the IMX installations across the ULTIMATE case studies. These exercises engaged diverse stakeholders and provided critical insights that informed the IMX development. Table 1 summarizes the co-creation activities conducted in the Netherlands (CS2), Italy (CS3), and Denmark (CS9).

Using Kirkpatrick's model (1959), facilitators evaluated the success of these co-creation exercises. At KWR in the Netherlands, participants appreciated the value of engaging stakeholders and simplifying complex ideas, with lessons informing future activities to enhance stakeholder communication. Activities at the Consorzio ARETUSA in Italy focused on fostering awareness of environmental protection and natural resource safeguarding, helping to strengthen community engagement efforts. Finally, at Kalundborg Forsyning in Denmark, participants' increased willingness to adopt new ideas and their openness to innovative stakeholder engagement methods were highlighted.

The findings reported for L'Acqua per Tutti encompass both its design process and its impact on user engagement. Data on engagement were gathered through voluntary surveys conducted during the final testing phase, with feedback collected from 20 participants, including local stakeholders and community members. Metrics evaluated included perceived learning outcomes, enjoyment, and the relevance of the installation's content to real-world water sustainability challenges.

The L'acqua per Tutti IMX was deployed at the Biblioteca Bottini dell'Olio in Livorno, Italy, and targeted a general audience. Its primary aim was to make water sustainability concepts accessible through interactive elements, such as AR visualizations and a tabletop game, creating a hands-on experience that guided users through challenges and potential solutions in water symbiosis, conservation, and management.

Participant demographics consisted of an equal gender distribution, with 10 females and 10 males, aged between 20 and 28 years (average age: 24.1). Most participants had limited prior exposure to water sustainability topics or immersive technology. While 55% of participants had used immersive technologies before, 60% reported slight or no familiarity with water sustainability concepts. Despite this, 70% expressed prior involvement in community or environmental activities, highlighting a potential interest in engaging further with such topics. Pre-treatment assessments revealed a moderate baseline awareness of water sustainability, though detailed analysis indicated opportunities for significant improvement through the IMX experience. The survey also revealed a strong interest in water conservation and management, with 14 respondents indicating a very high level of interest.

Survey results demonstrated that the IMX effectively engaged participants, with an average presence and engagement score of 3.53 (SD: 1.03). Usability was rated 3.65 (SD: 0.81), indicating that most participants found the interface manageable, though some highlighted navigation difficulties. Learning outcomes were positively rated, with a skill development score of 3.62 (SD: 0.83), reflecting the IMX's capacity to enhance understanding of water sustainability issues. Additional constructs such as emotion (Mean: 3.50, SD: 0.91) and attractiveness (Mean: 3.65, SD: 0.81) underscored participants' enjoyment and appreciation of the IMX's visual design. However, the flow of interactions (Mean: 3.35, SD: 1.13) was somewhat hindered by navigation challenges, prompting feedback for clearer instructions and more diverse interactive content to better cater to a wider audience. These findings suggest that L'Acqua per Tutti successfully fostered engagement and learning while also identifying areas for refinement to improve future iterations of the IMX.

The Water-Kennis IMX was originally developed for professionals in water reuse practices and student visitors at the KWR Institute in Nieuwegein, Netherlands. During the ULTIMATE project's final meeting, it was further simulated in Livorno. Unlike L'acqua per Tutti, this IMX targeted a broader audience ranging from secondary school students to seasoned professionals, utilizing technical AR tasks and location-based physical marker stands to foster discussions around sustainable water conservation practices.

Pre-treatment data revealed a high baseline awareness of water reuse concepts among participants, reflecting their professional expertise. Since the Livorno participants were exclusively professionals, no specific data on water sustainability awareness or interest in water management were collected, as high levels were assumed given their expertise. However, prior experience with immersive technology was assessed. The participant demographics included 12 females and 12 males, with ages ranging from 20 to 65 and an average age of approximately 42.5 years. Out of the 24 respondents, 19 reported prior experience with immersive technology, while 5 did not, indicating a generally high familiarity with digital tools.

Post-treatment results indicated moderate engagement, with presence and engagement scoring 2.60 (SD: 0.60). Usability, designed for technically adept users, scored 2.35 (SD: 0.67). Skill development achieved a higher score of 2.87, demonstrating the IMX's ability to enhance participants' technical understanding of water reuse concepts.

Emotion and flow scored more modestly (2.02 and 2.38, respectively), reflecting IMX's emphasis on precise, data-driven content over emotional immersion or seamless interactivity. Participants expressed appreciation for the integration of AR technology but emphasized the need for improved navigation and additional technical depth to better align with their professional expectations. These insights suggest areas for refinement, particularly in enhancing the flow of interactions and broadening the range of technical content to meet user needs effectively.

In L'acqua per Tutti, both controlled and uncontrolled groups provided insights into user experience. The controlled group was provided with the same narrative and storyline content in a portable document format (PDF) format, along with images. This setup served as a baseline for comparing the effects of the IMX on engagement and usability.

The controlled group reported lower engagement and usability scores (presence and engagement: 2.60, SD: 0.50; usability: 2.35, SD: 0.55) compared to the uncontrolled group (presence and engagement: 3.53, SD: 1.03; usability: 3.65, SD: 0.81). This disparity may stem from environmental differences, as the controlled setup limited natural interactions, affecting flow and emotion. The uncontrolled group benefited from a freer exploration of the IMX, which better demonstrated its strengths.

No controlled group experiment was conducted for Water-kennis, limiting direct comparisons. However, the controlled setup in L'acqua per Tutti clearly demonstrated that IMX environments, which allow for dynamic interaction and immersion, provide a more engaging experience compared to static formats like PDFs.

The comparison between L'acqua per Tutti and Water-Kennis highlights the versatility of IMX as a medium for stakeholder engagement. IMXs provide dynamic, interactive experiences that surpass traditional formats like PDFs in sustaining attention and encouraging active participation.

Co-creation was instrumental in tailoring the IMXs to their respective audiences. For L'acqua per Tutti, community-focused co-creation emphasized accessibility and awareness, while Water-Kennis prioritized technical accuracy and professional usability. These adaptations ensured relevance and resonance with diverse user groups.

Common challenges across both CSs included navigation issues and content depth. Simplifying interfaces, expanding interactive elements, and diversifying content could address these barriers, enhancing usability and engagement.

Despite their differences, both IMXs demonstrated the potential of immersive technologies to engage stakeholders meaningfully, fostering deeper learning and emotional involvement. Future iterations should integrate these findings to refine IMX design and maximize impact across varied audiences.

All nine CSs established their CoPs and carried out at least one (1) CoP meeting with locally relevant stakeholders. Table 2 provides an overview of the CoP meetings held across the CSs.

A total of 25 CoP meetings were held across all CSs, with an average of 22 participating stakeholders representing a diverse range of organization types.

Overall, stakeholders found the engagement in the CoPs to be very valuable (see Figure 6), with an average rating between 4 (very valuable) and 5 (extremely valuable). Stakeholders saw the engagement in CoP meetings to be a good opportunity for open discussion, knowledge sharing, and networking with stakeholders from diverse backgrounds and expertise. The meetings helped facilitate the exchange, sharing, and learning about innovations, technologies, and best practices relevant to the respective industries or sectors.

Some CSs experienced challenges in ensuring the representation of all relevant stakeholder groups, partially attributed to the Coronavirus Disease 2019 (COVID-19) pandemic and its impact on work dynamics in many sectors. Attention to this issue was put in follow-up engagement activities. For example, organizing online and hybrid meetings to foster broader participation. Despite these efforts, stakeholders generally expressed a preference for face-to-face meetings, believing that in-person interactions enabled better engagement, interaction, and learning experiences. Where possible, this was applied.

There was also a strong emphasis on the importance of having a clear agenda and defined objectives for meetings. This helped in focusing discussions on relevant topics, addressing key challenges, and achieving meaningful outcomes.

Furthermore, stakeholders expressed the desire to define concrete actions and next steps. This was necessary to help set tangible outcomes and monitor progress toward agreed goals.

Some CSs indicated the potential for CoPs to contribute to, and even influence, policy agendas and regulatory frameworks, especially at the local and national levels. Stakeholders saw that insights and recommendations from CoP discussions could be used to inform decision-making processes and shape future policy and regulation on related topics.

Finally, the experiences in establishing the CoPs in the CSs demonstrated the need to tailor the approach considering the local context, existing stakeholder ecosystems, and industry dynamics. Flexibility and adaptability were key to ensuring the relevance and effectiveness of CoPs in addressing local challenges and identifying opportunities.

In ULTIMATE, WOLLs are envisioned as more than just experimental labs; they are seen as ecosystems for real-world experimentation and co-creation, specifically tailored to the needs of industrial symbiosis. As integral components in fostering a Water Smart Society in Europe, WOLLs provide an engaging environment for stakeholder participation, enhancing collaboration between industries and communities. This collaborative environment enhances WOLLs' ability to innovate and implement sustainable water management practices effectively. The methodology chosen for the WOLLs analysis is the Harmonisation Cube. This assessment outlines detailed evaluation criteria for six foundational elements of any LL: Governance, Service Creation, Infrastructures, Methods and Tools, User Involvement, and Innovation Outcomes. It offers insights into the maturity level of the system and identifies areas for improvement by presenting quantitative results and radar graphs. The six elements are examined from organizational, contextual, and technological perspectives through three maturity levels – Set Up, Sustainability, and Scalability – each with questions that reflect increasing levels of maturity.

During the benchmarking phase, a preliminary assessment of the ULTIMATE CSs was carried out to evaluate their development stages. In the case of CSs, development stages can also be referred to as the readiness level to become a WOLL. After this baseline analysis, the CSs that were most aligned with the WOLLs methodology in their composition, showing potential for closer fit attainable in reasonable times, were engaged in further activities to foster the evolution of the CSs into WOLLs suited for industrial symbiosis.

Four CSs with high potential to become a WOLL were provided with a Quantitative Assessment Tool and a Qualitative Documentation Tool to proceed with their mapping step. The results of the second assessment are also included in this section (see Figures 4–7 for reference).

The initial assessment of CS1's readiness to become a WOLL revealed a score of 67%, indicating a solid foundation but with room for improvement in specific areas (Figure 4). Based on these results, CS1 was advised to focus its next activities on improving the low-scoring areas of Service Creation, Method, Tools, and Governance.

Specifically, CS1 was recommended to initiate workshops aimed at refining user-centric service models and integrating advanced project management tools to enhance efficiency. Additionally, an updated governance structure to include more stakeholder input and oversight mechanisms was recommended.

Despite being initially identified as a high-potential CS, CS1 faced challenges in maintaining engagement. The non-submission of the second round assessment made it impossible to provide further support within the project timeframe. Future developments are still possible in case of interest from the CS.

The initial assessment of CS3's readiness to become a WOLL revealed a score of 30% (Figure 5), indicating significant room for improvement. Based on these results, CS3 was advised to concentrate on enhancing several key areas: Service Creation, Infrastructure, Governance, Innovation Outcomes, and Methods and Tools.

To tackle these, the recommendation was to involve local stakeholders more in service co-creation, upgrade infrastructure, develop a robust governance model, focus on measurable innovation outcomes, and adopt advanced project methods and tools. While user involvement was a strong point, contextual metrics were weak across other key areas, highlighting the need for improved management practices and environments conducive to idea generation and stakeholder engagement.

CS3 made substantial progress in addressing the gaps identified in the initial assessment. The second evaluation showed a marked improvement, with the score rising to 65% (see Annex). Significant progress made through dedicated stakeholder engagement, innovation, and long-term sustainability planning reflects the advancements. The journey from an initial score of 30% to a final score of 65% underscored the CS3's evolution into a successful WOLL.

The initial assessment of CS4's readiness to become WOLL revealed a score of 31%, indicating significant room for improvement (Figure 6). In accordance with these results, CS4 was recommended to focus its next activities on improving the low-scoring areas of governance, innovation outcomes, methods and tools, and service creation and infrastructure. Forming a governance body to streamline decision-making, setting specific innovation goals, and conducting workshops or activities to train staff on new methods and tools were key recommendations.

CS4 demonstrated significant progress in the second assessment, leading to its recognition as a WOLL. Through continuous engagement and dedicated efforts, CS4 showed marked improvement across multiple key areas such as stakeholder engagement, innovation, and sustainability. The second assessment, which resulted in a final score of 93%, highlights the advances achieved in user involvement, service creation, infrastructure, governance, innovation outcomes, and methods (see Annex).

Recommended actions included launching initiatives to boost user involvement, developing new services based on user feedback, and integrating new tools to streamline processes. Figure 7 also shows that the governance, infrastructure and innovation outcomes have already reached the level required.

Despite the promising initial score, CS5 faced challenges in maintaining engagement with the WOLL assessment framework. This lack of consistent interest and commitment in the second assessment hindered the CS5's successful evolution into a WOLL. Further actions and developments are still a possibility upon the proactive approach of the CS.

Overall, becoming a WOLL emphasized the importance of tailored approaches in nurturing WOLLs across different contexts. For each CS, a unique trajectory reflected diverse starting points and development. While specific needs varied, common areas for enhancement often included user involvement, service creation, governance, and innovation outcomes. Moving forward, systematic assessment and feedback mechanisms will ensure alignment with WOLL's goals of creating sustainable, innovative water management practices. This evolution will contribute significantly to advancing the Water Smart Society in Europe.

Through developing a stakeholder engagement toolbox, consisting of IMX, CoP, and WOLLs, we have been able to measure each of the tools' impact on stakeholder engagement in a water-smart industrial symbiosis environment offered by the CS of the ULTIMATE project. These engagement approaches have ensured that the complexities inherent in symbiotic arrangements are addressed and possible platforms to further develop, test, and validate identified solutions beyond the lifetime of the project can be explored.

The IMX served as a core element in engaging stakeholders by providing immersive, interactive platforms designed through co-creation processes. These installations were implemented in three CSs, reflecting their unique contexts and maturity levels. For instance, at CS2 (KWR in the Netherlands), the IMX offered a location-based experience, utilizing quick response (QR) codes for interaction with specific marker stands. At CS3 (Aretusa Wastewater Treatment in Rosignano, Italy), an interactive tabletop equipped with AR markers and Kinect depth sensors created a hands-on exploration environment. At CS9 (Kalundborg, Denmark), a large-scale AR-enabled vinyl foam floor facilitated the visualization and understanding of industrial symbiosis and water reuse concepts. These installations were co-created with stakeholders to ensure relevance and usability, incorporating feedback from diverse participants throughout the design process. The resulting IMXs were evaluated not only for their technical and interactive elements but also for their effectiveness in fostering awareness and engagement. However, as observed in participant feedback, the impact varied across demographics and contexts, underlining the need for iterative design improvements and tailored approaches.

Complementing the co-creation exercises were CoPs, established across all CSs. The CoPs have served as social learning systems (Wenger 2000) bringing together diverse stakeholders to share knowledge, best practices, and experiences. These communities brought together stakeholders from industry, academia, and local governments to share insights, co-develop solutions, and build a shared understanding of Water Smart Industrial Symbiosis (WSIS) challenges and opportunities. In line with previous findings in sustainability research (e.g., Lang et al. 2012; Reed et al. 2014), ULTIMATE's CoPs encountered challenges in sustaining long-term engagement, particularly as participation waned between formal meetings. To address this, project coordinators utilized a combination of personal communication methods, including emails, phone calls, and informal interactions, to maintain stakeholder connections. Additionally, efforts were made to integrate ULTIMATE activities with existing local initiatives to enhance continuity and relevance. For example, in CS9, the CoPs aligned closely with ongoing community projects, fostering trust and long-term commitment.

WOLLs, as the third pillar of the engagement framework, played a crucial role in bridging the gap between theoretical concepts and practical applications. By supporting the establishment of WOLLs in mature CSs, ULTIMATE provided platforms for experimentation, validation, and refinement of solutions in real-world settings. These LLs were structured around six foundational metrics – governance, service creation, infrastructures, methods and tools, user involvement, and innovation outcomes – assessed through organizational, contextual, and technological lenses. This approach ensured that the proposed solutions were robust, adaptable, and scalable, addressing both local and systemic challenges. For example, the WOLLs facilitated collaborative testing of water reuse technologies, aligning with regional regulatory frameworks and stakeholder needs.

Collectively, the integration of co-creation leading to IMXs, CoPs, and WOLLs created a synergistic ecosystem of engagement and innovation. This holistic framework not only accelerated the development of sustainable water management solutions but also demonstrated a paradigm shift in stakeholder collaboration. By addressing the complexities of WSIS through tailored, participatory methods, ULTIMATE exemplifies how co-creation and real-world validation can drive meaningful progress in addressing global sustainability challenges. The findings of this project, while rooted in the water sector, have broader implications for other domains where stakeholder engagement and system-level transformation are critical. As such, ULTIMATE provides a replicable model for fostering collaboration, innovation, and sustainable impact in complex socio-technical systems.

The project leading to this application has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No. 869318.

Ethical guidelines regarding informed consent, participant confidentiality, and data protection are strictly followed to ensure the well-being and privacy of participants. Measures are taken to maintain the anonymity of participants' responses and safeguard their personal information. The ethical guidelines and data management plan are in ULTIMATE Deliverables D7.5 and D7.6.

All relevant data are included in the paper or its Supplementary Information.

The authors declare there is no conflict.