Collaborative watershed management has been heavily promoted and widely implemented to address a variety of natural resource concerns, resulting in the adoption and adaptation of the approach to management by regulatory agencies. Although several characteristics or indicators of success for watershed partnerships have been identified in the literature, these often portray a direct cause and effect relationship between partnership characteristics and outcomes. However, partnerships involve dynamic processes that can be influenced by both form and function (internally and externally) throughout various stages of the partnerships' existence. Our study presents an evaluation framework from the group process and evaluation literature to highlight the importance of evaluating ‘intermediate measures of partnership effectiveness’ in watershed partnerships, using the case of Michigan's voluntary watershed-based stormwater permit. Given the increasing use of watershed partnerships in a regulatory setting that is constantly in flux and the difficulty in assessing the effects of such groups on water quality, results suggest the utility of ‘intermediate measures of partnership effectiveness’ for assessing partnership process in order to provide ongoing feedback and incentives to ensure long-term success.

Introduction

Although point source pollution has been significantly reduced in urban rivers nationwide owing to the Clean Water Act (CWA) (1972), the multi-jurisdictional and multi-faceted nature of non-point source (NPS) pollution from stormwater runoff remains more difficult to regulate. NPS pollution, in the form of pesticides, nutrients (such as phosphorus and nitrogen), oil and grease, human and animal waste, and pathogenic organisms enters lakes and streams after being washed into storm drains and discharged untreated into local waterways (US Environmental Protection Agency (EPA), 2006). The presence of many of these stressors to human and wildlife health is a consequence of urbanization and the construction of impervious surfaces (e.g., roads, rooftops, parking lots, driveways), which alter the natural flow of water by not allowing precipitation to naturally infiltrate into the soil (Wang et al., 2001; Frazer, 2005). Amendments to the CWA in 1987 mandated the Environmental Protection Agency (EPA) regulate stormwater discharges from municipal separate storm sewer systems (MS4s) and industrial and construction sites as sources of NPS pollution (US Governmental Accountability Office (GAO), 2007). As a result, NPS pollution has become a high priority for municipalities, creating a need for increased awareness of stormwater management strategies, technologies, and watershed partnerships (Rauch et al., 2005; Hardy & Koontz, 2008).

Non-structural impediments to stormwater management such as jurisdictional fragmentation, organizational administration, and lack of awareness and leadership are common concerns among natural resource managers. A watershed approach to management, which encourages cooperative collaboration across jurisdictions, has been heavily studied in academic literature (see, for example, Leach & Pelkey, 2001; Moore & Koontz, 2003; Rauch et al., 2005; Sabatier et al., 2005; Hardy & Koontz, 2008; and others) and widely implemented, leading to its adoption and adaptation by agencies at state and federal levels. Michigan's use of the watershed approach to stormwater permitting, the first statewide program of its kind, presents an interesting and unique case through which to examine the partnership process and collaboration as a means for meeting permit requirements. We offer here a qualitative assessment of the early stages of the state's collaborative watershed management process in order to provide insight on the challenges and benefits as perceived by the participants themselves, providing evidence to support the partnership process as an evaluation metric of effectiveness. In this article, we review the literature regarding evaluation of collaborative watershed approaches, assess multi-jurisdictional collaboration using the case of stormwater permitting in Michigan, and discuss the importance of formative (i.e., process) evaluation in assessing partnership effectiveness.

Evaluating collaboration in watershed partnerships

The increasing popularity of watershed partnerships across the USA, and in other countries, has sparked interest in evaluating whether or not they are effective (i.e., achieve desired outcomes) and the conditions under which success is attained. Empirical evidence points to aspects of the partnership itself and its processes, both form and function, as being important indicators for success (see, for example, Leach & Pelkey, 2001; Clark et al., 2005). Characteristics of the partnership, such as the composition of members has been shown to have an influence, with heterogeneous groups (consisting of members of the public and agency officials) proving more effective in planning and implementation than homogeneous groups (Moore & Koontz, 2003). Likewise, partnerships that cross disciplinary boundaries and bureaucratic levels have a greater chance of achieving more equal representation among stakeholders, sharing power, exchanging knowledge, and attaining collaboration (Brown, 2005). In addition to membership composition, it is widely argued that overall long-term success is dependent on organizational development and maintenance, largely the rules governing the process, and the environmental and social context in which the partnership operates (Ostrom, 1990; Imperial, 1999; Wondolleck & Yaffee, 2000; Sabatier et al., 2005; Bonnell & Koontz, 2007).

Despite these findings, Koontz & Thomas (2006) argue that too much emphasis is placed on process and not enough on environmental outcomes. Aside from the high cost of long-term monitoring and data collection, the spatial and temporal mismatch between watershed processes and management goals make using water quality as an evaluation metric unrealistic. NPS pollution accumulates over time and is subject to storm events as well as upstream effects, making it difficult for watershed groups to relate improvements in water quality to specific actions over a short time frame. As a result, stakeholder perceptions of water quality improvements or watershed partnership effectiveness are often used as proxies for water quality. Leach et al. (2002) found evidence of a cognitive bias, where participants in partnerships with a longer history (6 years or more) were more likely to perceive water quality improvements than those in younger partnerships (under 2 years). Similar results were found for performance measures related to projects that had been completed. However, there are doubts about the value (or appropriateness) of this metric because beliefs can fluctuate over time (Sáenz-Arroyo et al., 2005) and stakeholders are likely to perceive the environmental impact of the partnership differently. One would assume that partnerships that thrive would improve over time and accomplish more of their goals. If this assumption holds, it presents a challenge in evaluating partnerships that have been established for less than 6 years. Similarly, we would expect greater water quality improvements over longer time periods assuming inputs were reduced or remained unchanged, therefore sufficient time must pass before this can be a realistic measure.

The general focus on outcomes, whether water quality or activity based, also does not include assessment of all the partnership effects (intended and unintended) nor address how the partnership functions (process). The evaluation literature regarding partnerships argues that concentrating solely on results is not appropriate for long-term management, with successful partnership relationships leading to increased efficiency and effectiveness over time as the focus on activities evolves into more strategic planning (Brinkerhoff, 2002). The importance of many of the characteristics of effective partnerships as described above has been confirmed in both the watershed partnership and evaluation literature. Nonetheless, further adoption of principles found in both the group process and evaluation literature can enhance the manner in which watershed partnerships are assessed by taking into account all aspects of the partnership process and lending insight into the relationship between process and performance. We now discuss approaches to evaluation.

There are two main categories of evaluation: formative, with the purpose of providing information to improve a program, and summative, undertaken in order to judge the program performance in relation to specific goals and objectives. Process evaluation, which can be either formative or summative, represents a conceptual framework aimed at answering questions related to program operations, implementation, and service delivery (Rossi et al., 1999). Thus process evaluation provides a fitting framework for assessing partnerships, seeing as they are dynamic and ongoing, and would benefit from frequent monitoring. Currently, the watershed partnership literature primarily focuses on process evaluation, but it is unclear as to whether these evaluations are summative or formative. This can hinder attempts to utilize findings for future efforts and our understanding of the relationship between process and outcomes. Furthermore, very few studies assess all aspects of the process and the relationships among them. Although Bidwell & Ryan (2006) identified a relationship between the structure of the partnership and the resulting activities that were conducted, Mandarano & Paulsen (2011) did not find partnership process to have any significant effect on implementation.

Drawing from the evaluation literature related to partnerships and effective groups, the key findings on watershed partnerships can be organized and expanded upon as discussed here. The framework presented in Figure 1, adapted from Schulz et al. (2003), represents both the evaluation and watershed partnership literature. The framework is comprised of six sections: (a) environmental characteristics, (b) structural characteristics, (c) functional characteristics, (d) partnership programs and interventions, (e) intermediate measures of partnership effectiveness, and (f) output measures of partnership effectiveness. As illustrated earlier, assessments of watershed partnerships in the literature concentrate on the structural (partnership design and composition) and functional characteristics (process rules and operation), with occasional links to partnership activities and outcomes. This framework presents a formative approach to partnership assessment by including ‘intermediate measures of partnership effectiveness’, allowing for assessment throughout the process (Schulz et al., 2003).
Fig. 1.

Conceptual framework for assessing watershed partnerships. Adapted from Schulz et al. (2003). Italicized and bolded items were derived from Koontz (2006), Leach et al. (2002), Margerum (1999), Sabatier et al. (2005), Toupal & Johnson (1998), US GAO (2007) and not originally included in the model by Schulz et al. (2003).

Fig. 1.

Conceptual framework for assessing watershed partnerships. Adapted from Schulz et al. (2003). Italicized and bolded items were derived from Koontz (2006), Leach et al. (2002), Margerum (1999), Sabatier et al. (2005), Toupal & Johnson (1998), US GAO (2007) and not originally included in the model by Schulz et al. (2003).

These intermediate measures could potentially serve as a mechanism to improve our understanding of the relationship between structural and functional characteristics and outcomes. Intermediate measures of partnership effectiveness include: (a) perceived effectiveness to achieve goals, (b) perceived effectiveness of the partnership process, (c) perceived benefits of participation, (d) extent of member involvement, (e) shared ownership and commitment to collaborative effort, and (f) future expectations of effectiveness. In addition, because the relationship between partnership characteristics and outcomes is often viewed as a cause and effect relationship (Brinkerhoff, 2002), our analysis adds a feedback loop to emphasize the dynamic process of partnerships and the need to continuously monitor partnerships for continued longevity and success. This study defines success in watershed partnerships as the achievement of program and policy objectives as defined by the participants and the institutionalization of programs and/or partnerships.

Our study aims to contribute to the literature on collaborative watershed partnerships in two ways. First, it provides participant-based evidence to further support the importance of the identified criteria and indicators for watershed partnership success in the current literature as applied in a regulatory setting. Second, it presents and applies a conceptual framework for assessing watershed partnerships, in regulatory and non-regulatory situations, from group process and evaluation literature that includes ‘intermediate measures for partnership effectiveness’, further enhancing our understanding of process in partnership assessment.

The main objectives of this research were to conduct a participant-based assessment of collaborative stormwater management in a regulatory setting and examine the role of process evaluation in assessing watershed partnerships. Specifically, we were concerned with how participants conceptualized collaboration, what they viewed as benefits and challenges to collaboration, their perceptions of the partnership process, and perceptions of current and future effectiveness of the partnership. Although there is significant information in the literature regarding the benefits and challenges to collaboration, this study examines these issues in a regulatory setting and focuses on the importance of evaluating process. From here we provide details about our case study and methods and then turn to a discussion of our results.

Case study: Michigan's watershed-based stormwater permit

In 1990, Phase I of the EPAs stormwater program was promulgated under the CWA through National Pollutant Discharge Elimination System (NPDES) permit coverage to address polluted runoff. Under Phase I the EPA created a rule specifically targeting MS4s with populations greater than 100,000, construction sites greater than 5 acres, and 10 categories of industrial activity (EPA, 2005). Phase II, which began officially in 2003 prior to a voluntary permit period of 4 years, further augmented the program by including MS4s with populations less than 100,000 and construction activities of between 1 and 5 acres (EPA, 2005). With the increase in the number of communities responsible for stormwater, the State of Michigan piloted a watershed-based permit focused on developing cooperative partnerships across jurisdictions to address pollution prevention and mandated requirements in a more holistic and flexible manner. The watershed-based permit requires participating jurisdictions within a watershed to conduct watershed planning and management collectively (Michigan Department of Environmental Quality (MDEQ), 2003). Although the watershed-based permit is voluntary, all MS4s meeting the criteria stated above are mandated to comply with NPDES permit requirements to reduce polluted runoff from stormwater and can do so with an individual jurisdictional permit.

Under the watershed permit, the Greater Lansing Regional Committee for Stormwater Management (GLRC) was established to coordinate the creation and implementation of watershed management plans for three area rivers in mid-Michigan (see Figure 2 for map of area). The GLRC is comprised of 18 municipal government representatives, in addition to county and state government agency advisory members and local representatives of community watershed groups. GLRC members must submit a Stormwater Pollution Prevention Initiative explaining how they will evaluate and execute the action items proposed in the watershed plan within their jurisdiction and provide yearly progress reports (EPA, 2003). The permit includes six areas, or minimum control measures, for action: construction runoff, post-construction runoff, pollution prevention/good housekeeping, public education and outreach, public participation/involvement, and illicit discharge detection and elimination. While a number of the action items are related to the daily practices of the municipalities and carried out individually, others such as education and outreach are implemented collectively by the group. The permit is reissued every 5 years, with the first cycle ending April 2008 and the second beginning May 2008 (MDEQ website, n.d.). We provide the results of a qualitative study that offers preliminary assessment of the effectiveness of a collaborative watershed management process from the perspective of its participants.
Fig. 2.

Map of watersheds represented by the GLRC. Image courtesy of GLRC (www.mywatersheds.org).

Fig. 2.

Map of watersheds represented by the GLRC. Image courtesy of GLRC (www.mywatersheds.org).

Methods and data collection

The exploratory and descriptive nature of this project lends itself to semi-structured in-depth interviews to aid in understanding perception and attitudes of stakeholder groups and allow flexibility to probe deeper into issues raised by participants as well as enhance our understanding of participants’ interpretation of events, situations or actions (Maxwell, 2005; Rubin & Rubin, 2005). The interview schedule was developed and tested on three individuals representing three different categories of stakeholders – a regulatory agency, a municipality, and an engineering consultant. The knowledge gained from these interviews was used to further refine a standard set of interview questions, along with probing questions specific to each stakeholder group. A purposive sampling strategy was utilized to identify research participants, with the sample primarily representing those who are legally responsible for stormwater management, either through regulation, implementation or compliance, and those who were identified by this peer group as ‘involved’. Participants were chosen based on a stakeholder analysis through review of local reports and meeting documents, attendance at GLRC meetings, and informal discussions with key individuals involved in stormwater management at the state and regional level. Participants were also asked to suggest other individuals to interview.

Between June and December of 2007, 20 semi-structured interviews were conducted with various stakeholders involved directly with the GLRC or with the Phase-II watershed permit, including: government agency staff (n = 4), municipal leaders (n = 4), independent engineers (n = 3), non-governmental organization (NGO) leaders (n = 3), contractors/builders (n = 3), and community leaders (n = 3). Two interviews were conducted by phone, while the rest were in person. All but one of the interviews were digitally recorded and transcribed. The interviews lasted, on average, no more than 60 minutes. Research participants were asked to discuss their involvement in the watershed permit, specifically addressing the following themes: current and past involvement in collaborative projects, perception of the partnership process, the perceived benefits and challenges to collaboration (personal, organizational, and community), commitment to the collaborative effort, perceived current and future effectiveness of the watershed partnership, and how they would like to see the process proceed or be improved.

The methods for this research were approved for the duration of the project by the Michigan State University Committee on Human Subjects (IRB No. X06-471; r027412). Informed consent was used to ensure voluntary participation and confidentiality.

Data analysis

The data analysis software, Atlas-ti, was used to code interviews and organize coding structures for the transcribed interviews. We employed a grounded theory approach for the analysis of the interviews starting with in vivo or open coding as a means of identifying thematic areas based on participant responses. Grounded theory, often associated with qualitative inquiry, is an iterative and comparative method useful in developing or building upon existing theories (Lincoln & Guba, 1985; Denzin & Lincoln, 2011). Following this methodology, the transcripts of the interviews were read and the descriptive codes applied to words, phrases, and paragraphs. Within codes, the information (or words, phrases and paragraphs) was then compared for consistency by identifying similar language use within coded fragments and removing any outliers (Miles & Huberman, 1994). Further analysis consisted of sorting and grouping of codes to be analyzed for patterns among stakeholder groups and broader themes. The following themes were identified: (a) incentives and benefits to participation, (b) leadership/facilitation, (c) capacity building/awareness, (d) mutual understanding, (e) willingness to cooperate, (f) commitment, (g) communication, (h) access to resources (financial or technical), (i) flexibility/compatibility of permit, (j) challenges to collaborative partnerships, (k) mutual respect and trust, (l) current and future effectiveness, and (m) evaluation.

Findings

Overall our research participants (N = 20) conceptualized collaboration as involving good working relationships, communication, leadership, and a willingness to cooperate that can lead to increased efficiency of resource use and program success. Although the watershed partnership studied did not exhibit all of the characteristics of an effective partnership related to design/composition (structural) and process rules/operation (functional) as shown in Figure 1, participants did identify these aspects as important for collaborative partnerships. The most positive aspects of the process identified by our research participants were capacity building, networking, and increasing awareness across the watershed. The findings presented below are organized based on ‘intermediate measures for partnership effectiveness’ focusing on the collaborative process (refer to Figure 1), including perceived personal, organizational, and community benefits of participation, shared ownership and commitment to the collaborative effort, perceived effectiveness of the group's ability to achieve its goals, perceived effectiveness of the partnership process, and future expectations of effectiveness.

Perceived personal, organizational, and community benefits of participation

Stakeholders characterized the benefits of collaborative partnerships as short-term incentives such as cost sharing and avoiding duplication, and long-term benefits, including increasing awareness, building relationships and the institutionalization of policies that could have a greater effect on the entire watershed over time. In the case of the watershed-based MS4 stormwater general permit piloted in Michigan for Phase II of the NPDES, local units of government were given the option of a jurisdictional or watershed-based permit. These watersheds include areas of concentrated development and involve dozens of local government units, intensifying the challenges of regional cooperation.

Electing to work across jurisdictions was seen as beneficial by our participants because it educated group members regarding shared problems and provided support for resolving these issues individually as well as collectively. In addition, the permit partnership has allowed agencies, municipalities, and community groups to combine resources (finances, institutional knowledge, and information) for educational efforts with the intent of reaching a wider audience with a unified message – providing both organizational and community benefits. It is important to note that only the public education and post-construction controls are required to be implemented watershed-wide, while the rest of the measures are still only implemented by each individual jurisdiction. Group members perceived the required watershed-wide public education programs to be beneficial, viewing collaborative efforts as the best way to spread a consistent message and make the most of financial and human resources. The watershed planning component of the permit was time-consuming and required a larger time commitment than some participants were willing to contribute. Some felt it was part of a learning process with the long-term benefit of a better understanding of their constituents and an appreciation of the various aspects of stormwater management (e.g., engineering, environmental, economic) as illustrated in the following statement:

I think there's a lot better understanding of what this is and we've educated a number of people in terms of this area, the knowledge that the people who sit around the table have now in comparison to what they did know even three years ago is pretty amazing … and … it's people who that's not their background, they haven't had much experience in it and that's really why they weren't aware of it as much as anything … so from that perspective I think it's great’ (Municipal Official).

Many participants (n = 12) expressed how much they had learned through the process and continue to learn about stormwater, other municipalities, Best Management Practices, and related topics. They felt that they personally benefited by learning from each other and took comfort in knowing that they were not the only ones struggling with stormwater issues. Non-governmental interview participants also felt that the facilitation of communication and cooperation between local governments as part of the permit process was valuable, as it has opened avenues for future collaboration in other areas. As an NGO staff member noted:

They are saying that [they] have never talked much with [their] neighbors about certain things or about anything, and now [they] are meeting at these watershed meetings and [they] start talking about other projects and start coordinating more’ (NGO Affiliate).

Shared ownership and commitment to the collaborative effort

The main challenges identified by participants are specific to the Phase II watershed permit; however, many coincide with themes from the literature, such as time commitment, turf and ego issues, and evaluating outcomes. All of these can affect feelings of shared ownership and commitment to the collaborative effort. One aspect specifically related to the watershed permit was that participants felt the administration of the permit contradicted the collaborative nature of the watershed group by requiring individual reports by municipality, as well as individual assessments. Those interviewed who have been directly or indirectly involved in the watershed permit primarily had positive comments regarding the partnership up to this point. Even with the admittedly territorial nature of municipalities, as well as egos that can be typical among political figures, only three participants mentioned it as being a problem that needed to be addressed. Similarly, they acknowledged that at times it was a struggle to organize such a large, diverse group but saw this as an inevitable part of the process. The following statements illustrate some of the issues mentioned above.

It's a lot of people to bring together to the table you know, there's always an analogy of moving a freighter with an oar and that's what it's like, it's just a lot of people to get to go in the same direction’ (Municipal Official).

In my opinion just the fact that we were forced to come together as a group and some of us did it kicking and fighting by the way, I mean I didn't want to go to five meetings a month which I did for the first 2 and a half years, because not only did you belong to a watershed but you also had to belong to a watershed committee, so you not only went to the regional meeting, you also went to the committee meeting, so consequently there were times when I was going to 5 and 6 meetings a month and I didn't like that because I felt that I had more important things to do, better things to do, but looking back and at the end of the day I have to admit that … it's been a wholesome experience for all of us’ (Municipal Official).

Whereas the concept behind the watershed permit is to provide opportunities for cost sharing and avoid duplication, interview participants – especially those from smaller municipalities – felt that aside from cost sharing with other communities the financial burden of this unfunded mandate was excessive. Finances also played another role within the watershed groups by providing larger municipalities with more resources to contribute a slightly greater role in decision-making, as illustrated in the following:

I think [our city's] relatively fortunate that we have money in terms that we're a bigger community, we have a reasonable tax base, not that any of this is inexpensive, but because of that we have a lot of influence over what goes on and I think that has helped a lot because it doesn't feel like we're being pulled along for the ride, we get to impact decisions, influence decisions and people look to us for information’ (Municipal Official).

The US Government Accountability Office (GAO), in a report on the EPA's Stormwater program, found two main factors influencing the level of burden on communities: (a) the extent to which a community can use the flexibility of the permit in implementing less expensive measures and (b) their ability to benefit from previous stormwater management experience (GAO, 2007). This nationwide report reiterates the concerns expressed by our research participants regarding the financial burden of Phase II and acknowledges the barriers to funding stormwater activities that many communities confront. However, our findings also demonstrate the importance of taking into account how finances influence the power dynamics in the collaborative process, as this could have a negative impact on aspects of decision-making, shared ownership, commitment to collaboration and group and community empowerment – all aspects of intermediate measures of partnership effectiveness.

Perceived effectiveness of the group's ability to achieve its goals and the partnership process

Owing to the newness of the initiative, participants were hesitant to remark on the effectiveness of the group to date or in the future. Part of this is related to their conceptions of what the indicators of success or effectiveness were. There was a focus on outcomes related to water quality or the number of activities accomplished, based on the requirements of the permit. In general, respondents were concerned about how their work would be evaluated by the MDEQ. Aspects identified by stakeholders as key to current and future success of the partnership heavily focused on structural and functional characteristics of the partnership including: individual/agency leadership or outside facilitator, willingness to cooperate, open communication and transparency, mutual respect and trust, and capacity building opportunities.

The following two statements illustrate an internal and external perspective regarding perceived effectiveness:

Well, I think as far as the effect it has on the municipalities as a whole it's been good. And because it is relatively new for a lot of us I think it's [kind of] hard to say what kind of an effect it has had in the watersheds on a whole … So, as far as its effectiveness goes, you know, I think it's doing alright but I think it's still, as far as I'm concerned it's a little bit new, we're just [kind of] getting off the ground’ (Municipal Official).

You know, if you had asked that question about a year and a half, two years ago, I would have said that I had serious reservations. Quite frankly they have done a 180 degree turn, they are meeting regularly, they have leadership from the various areas they need to have leadership, they are functioning quite well and that is nice’ (Engineering Consultant).

Future expectations of effectiveness

During data collection, one significant change to the permit and the members of the partnership occurred that highly influenced the group members’ perception of the partnership's ability to be effective in the future, and even its existence. A 2007 circuit court ruling in favor of Comstock and Kalamazoo Townships exempted a majority of the townships from having to participate in Phase II based on ownership and operation of separate storm sewers (State of Michigan, 2007). As a result of the ruling, several communities left the established watershed groups under the watershed-based permit owing to the fact that they were no longer required to fulfill any permit requirements even on a jurisdictional basis. In some cases, the townships have agreed to continue working with the other communities on education-related projects, which was the case for one interview participant. However, most felt that they should not be required to pay the permit fees or meet any of the other requirements. Other participants (municipal, government agency, NGO, and engineering consultants) expressed concern over the loss of participation by the townships because they felt this would weaken the collaboration that had been built, and that elimination of the township's responsibility contradicted the purpose of the holistic watershed planning and implementation promoted through the watershed permit1. The following statements illustrate the various concerns over the loss of the townships and the effect this might have on the future of the partnership:

If you'd asked me this about nine months ago I'd say we were really effective and I think we could accomplish what we wanted to do, I think some of the decisions that have been made by the DEQ are [going to], have almost undercut the way that we're able to take care of these issues … but this whole issue with the Townships … it's really going to become a problem … well I think what's ultimately going to happen is the GLRC is going to go away, I think, but hopefully it won't’ (Municipal Official).

I think by the MDEQ not appealing that decision that kind of really made sure that we probably won't continue into the future. I just don't see that we'll be able to do it. I don't think there'll be the participation, the will, or anything like that, because people can be disinterested now you know. Before they couldn't be [disinterested], everybody had a responsibility’ (Municipal Official).

I would like to see some way to bring back the townships into the fold because the court case that just came out has fragmented them – some are in, some are out, some are in [to a lesser extent], even though they're in, and I would like to see them brought back into the fold’ (Engineering Consultant).

Although assessing the process was not something that was a part of the permit, nor something that we imagine the participants were very accustomed to, this case provides an example of where ‘intermediate measures of partnership effectiveness’ could be useful in providing guidance and adapting to changing conditions. Participants also expressed that their future effectiveness would depend on the pending changes to the permit in the next cycle. While there may be limitations based on permit requirements, it seems that some issues related to evaluation could have been addressed during the process while others remain to be worked out between the group and the agency before the next permitting cycle.

Discussion

By applying ‘intermediate measures of partnership effectiveness’ to our data, we have presented a participant-based assessment of the partnership and demonstrated the utility of the measures in a regulatory cycle such as Michigan's watershed-based stormwater permit. The critical elements of the process identified by participants in our study are important to understanding a more complete or holistic picture of what influences effectiveness and can be useful in the dissemination and replication of programs (Patton, 2002). For example, our study reinforces the importance of certain structural and functional characteristics such as having an outside facilitator and ‘learning sessions’ where outside speakers presenting on important aspects of stormwater assisted in leveling the knowledge gap among participants. However, although policy mandates may have some control over the structural and functional aspects of the watershed partnership initially, these aspects can change over time. One issue that will always be difficult to account for in partnership assessments is the turnover of members and the effect of individuals on the process and outcomes (Floress et al., 2011). The framework presented builds upon other work and fills in the gap left by summative evaluations by providing interim measures that can assist in identifying changes and problems before they affect the integrity of the partnership.

At the time of our data collection, the GLRC was completing the first cycle of the permit and was unsure about how their work up to that point would be received by the MDEQ, and also about coming changes to the second cycle and how that would change the partnership they had worked hard to create. Our findings indicate that the group perceived itself to be effective and moving toward future successful outputs pending modifications to the process based on changes to the permit, staff turnover, changes in funding, and other aspects that are highly common in watershed partnerships. Group participants also expressed a desire for more feedback throughout the permit cycle, instead of solely at the end. Learning and evaluating are important aspects of partnerships that keep people engaged and increase transparency, legitimacy and credibility, further enhancing the collaborative capacity of a partnership (Cheng & Sturtevant, 2012). Participatory evaluation, where group members are involved in establishing the desired goals and outcomes of the evaluation, may be a useful assessment tool allowing participants more control and greater investment in the overall process (Conley & Moote, 2003). As changes to the permit or membership take place, a comparative assessment of watershed permit groups throughout Michigan or elsewhere could help to further identify aspects of the process that could be improved or modified at various stages.

Brunson (1998) argues that to be successful, cross-boundary stewardship such as watershed management must accommodate ‘for both the territorial self interest and community cooperation’. Requiring individual permits for each jurisdiction (MS4) and encouraging collaborative planning and (some) implementation the Phase II watershed-based permit for stormwater management attempts to do just that. Unlike most permits, the watershed permit is not prescriptive, allowing municipalities the flexibility to decide how to implement the watershed plan on their own while encouraging collaboration throughout the watershed. In addition, the Phase II watershed-based permit in the Greater Lansing area of Michigan has helped to increase awareness of stormwater concerns, facilitate collaboration, build relationships, and initiate practices that could have important long-term implications. Because of the relative newness of the program, challenges to implementation and administration of the permit remain and will require continued education and communication on both sides.

Watershed partnerships are dynamic processes that require flexibility; however, this can be challenging in a regulatory setting. Partnerships that are formed to meet regulatory requirements would benefit from a more formal and ongoing evaluation process that is transparent and participatory. Gaining participant insight will help to improve relationships between managers/decision makers and those affected by decisions. Intermediate measures of partnership effectiveness provide a set of criteria that can be used to assess partnerships throughout the process and address issues that could affect future effectiveness. This study contributes to the knowledge base by providing participant-validated criteria for assessing partnership processes in the early stages that could benefit other similar partnerships in making the necessary adjustments throughout the permit implementation process, providing ongoing feedback and incentives to ensure long-term success. Future research should work to further validate these measures, as well as explore participant developed evaluation metrics. The watershed-based stormwater permit is an innovative approach to management that would benefit from added research to understand the dynamics of mandated intergovernmental cooperative partnerships and their effect on institutionalizing pollution prevention practices and watershed protection policies through collective action.

Acknowledgements

This research was made possible with funding from the Michigan State University (MSU) Community Vitality Partnership, MSU Environmental Science and Policy Program Environmental Research Initiative, and MSU Graduate School.

1

As of the end of the 2014 fiscal year, the GLRC was still a functioning partnership comprised of all the original members, including some of the townships that were no longer mandated to participate based on the court ruling in 2007.

References

References
Brunson
M. W.
(
1998
).
Social dimensions of boundaries: Balancing cooperation and self-interest
. In
Stewardship Across Boundaries
.
Knight
R. L.
Landres
P. B.
(eds).
Island Press
,
Washington, DC
, pp.
65
86
.
Clean Water Act [CWA]
(
1972
).
33 U.S.C. 1251–1387
.
Conley
A.
Moote
M. A.
(
2003
).
Evaluating collaborative natural resource management
.
Society & Natural Resources
16
,
371
386
.
Denzin
N. K.
Lincoln
Y. S.
(
2011
).
The Sage Handbook of Qualitative Research
.
Sage Publications
,
Thousand Oaks, CA
.
Floress
K.
Prokopy
L. S.
Ayres
J.
(
2011
).
Who's in charge: role clarity in a Midwestern watershed group
.
Environmental Management
48
(
4
),
825
834
.
Frazer
L.
(
2005
).
Paving paradise: the peril of impervious surfaces
.
Environmental Health Perspectives
113
(
7
),
A457
A462
.
Koontz
T. M.
(
2006
).
Collaboration for sustainability? A framework for analyzing government impacts in collaborative-environmental management
.
Sustainability: Science, Practice, Policy
2
(
1
),
15
24
.
Leach
W. D.
Pelkey
N. W.
(
2001
).
Making watershed partnerships work: a review of the empirical literature
.
Journal of Water Resources Planning and Management
127
(
6
),
378
385
.
Lincoln
Y.
Guba
E.
(
1985
).
Naturalistic Inquiry
.
Sage
,
New York
.
Mandarano
L.
Paulsen
K.
(
2011
).
Governance capacity in collaborative watershed partnerships: Evidence from the Philadelphia region
.
Journal of Environmental Planning and Management
54
(
10
),
1293
1313
.
Maxwell
J. A.
(
2005
).
Qualitative research design: An interactive approach
. In
Applied Social Research Methods Series
.
2nd edn
,
Vol. 41
.
Bickman
L.
Rog
D. J.
(eds).
Sage Publications
,
Thousand Oaks, CA
.
Michigan Department of Environmental Quality (MDEQ)
(
2003
).
National Pollutant Discharge Elimination System (NPDES) Wastewater Discharge General Permit. (PERMIT NO. MIG619000)
.
Miles
M. B.
Huberman
A. M.
(
1994
).
Qualitative Data Analysis: An Expanded Sourcebook
.
Sage Publications
.
Thousand Oaks, CA
.
Ostrom
E.
(
1990
).
Governing the Commons: The Evolution of Institutions for Collective Action
.
Cambridge University Press
,
Cambridge, UK
.
Patton
M. Q.
(
2002
).
Qualitative Research and Evaluation Methods
,
2nd edn
.
Sage Publications
,
Thousand Oaks, CA
.
Rauch
W.
Seggelke
K.
Brown
R.
Krebs
P.
(
2005
).
Integrated approaches in urban storm drainage: where do we stand
?
Environmental Management
35
(
4
),
396
409
.
Rossi
P. H.
Freeman
H. E.
Lipsey
M. W.
(
1999
).
Evaluation: A Systematic Approach
,
6th edn
.
Sage Publications
,
Thousand Oaks, CA
.
Rubin
H. J.
Rubin
I. S.
(
2005
).
Qualitative Interviewing: The Art of Hearing Data
,
2nd edn
.
Sage Publications
,
Thousand Oaks, CA
.
Sabatier
P. A.
Focht
W.
Lubell
M.
Trachtenberg
Z.
Vedlitz
A.
Matlock
M.
(eds) (
2005
).
Swimming Upstream: Collaborative Approaches to Watershed Management
.
MIT Press
,
Cambridge, MA
.
Sáenz-Arroyo
A.
Roberts
C. M.
Torre
J.
Cariño-Olvera
M.
Enríquez-Andrade
R. R.
(
2005
).
Rapidly shifting environmental baselines among fishers of the Gulf of California
.
Proceedings Royal Society B: Biological Sciences
272
(
1575
),
1957
1962
.
State of Michigan
(
2007
).
Declaratory judgment in the ninth circuit court for the county of Kalamazoo, January 22.
Toupal
R. S.
Johnson
M. D.
(
1998
).
Conservation partnerships: Indicators of success. Technical Report, Release 7.1 February, USDA National Conservation Service, Social Sciences Institute,Tucson, AZ.
US Environmental Protection Agency (EPA)
(
2003
).
Improving EPA's performance with program evaluation: A review of statewide watershed management approaches (EPA100-F-03-005)
.
Office of Policy, Economics and Innovation
,
Washington, DC
.
US Environmental Protection Agency
(
2005
).
Stormwater Phase II final rule [Factsheet]
.
Office of Water
,
Washington, DC
.
US Environmental Protection Agency
(
2006
).
Wadeable streams assessment: A collaborative survey of the nation's streams
.
(EPA-841-B-06-002) US EPA
,
Washington, DC
.
US Government Accountability Office (GAO)
(
2007
).
Further implementation and better cost data needed to determine impact of EPA's stormwater program on communities
.
Report to Congressional Requesters (GAO-07-479)
.
GAO
,
Washington, DC
.
Wang
L.
Lyons
J.
Kanehl
P.
Bannerman
R.
(
2001
).
Impacts of urbanization on stream habitat and fish across multiple spatial scales
.
Environmental Management
28
(
2
),
255
266
.
Wondolleck
J. M.
Yaffee
S. L.
(
2000
).
Making Collaboration Work: Lessons from Innovation in Natural Resource Management
.
Island Press
,
Washington, DC
.