What counts as ‘ results ’ in capacity development partnerships between water operators ? A multipath approach toward accountability , adaptation and learning

The on-going debate on aid effectiveness highlights that in order for capacity development interventions to remain relevant and their results sustainable, their planning, implementation, monitoring and evaluation need to be flexible and case-specific. It is not only important to account for end results, but also to adjust interventions during their implementation. Capacity development partnerships (CDP) between water operators are portrayed as a promising approach for sustained performance. However, it has been observed that these interventions require significant time to lead to the targeted water operator performance gains. Hence, managing the partnership solely through the use of key performance indicators offers the partners little insight into both the progress achieved and the effectiveness of the partnership activities in contributing to such progress. This incomplete picture is likely to limit the ability of partners to manage the project efficiently. This paper proposes a multi-path approach to monitoring and evaluating the performance of CDPs (this being understood as different from the performance of the targetedwater operator) that enables wellinformedmanagement of the partnership. The designed approach is applied to compare the progress of two partnership projects, namely those of the Lilongwe and Blantyre Water Boards in Malawi, with Vitens Evides International.


Introduction
In recent decades standard forms of public-private partnerships (PPPs) (concession, lease, management or service contracts)commonly ones associated with an investment programmehave been a widespread approach to the operational improvement of water services (Budds & McGranahan, 2003;Hall & Lobina, 2007;Hukka & Vinnari, 2007;Phumpiu, 2009).However, disappointing experiences with high-risk forms of PPPs (concessions or lease contracts) in developing countries, particularly in sub-Saharan Africa, have led many donors to instead promote shorter-term, lower-risk forms of PPPs, most often management contracts (Arroyo, 2011).These types of contract have frequently been conceived as preparation mechanisms for higher risk PPPs that do not always materialise afterwards (Mariño et al., 1998;Saghir et al., 1999;Anderson & Janssens, 2011) (e.g. the management contract in Ghana).Experience with low-risk PPP forms has shown that achieved results are difficult to sustain once the external water operator leaves.Few studies have documented post-PPP performance evolution, yet cases like the management contracts of Johannesburg (Rooyen et al., 2009) or Beira, Quelimane, Nampula and Pemba (Triche, 2009) serve to illustrate this trend.The inability of local partners to maintain performance trends once a project has ended is arguably rooted in the lack of local leadership and managerial capacity at high levels (having been replaced during the project by the external partners) and the subsequent weak capacity to prevent political interference, as well as the possible low degree of local ownership of some implemented measures (Rooyen et al., 2009).
An emerging model for water operator reform is a capacity development partnership (CDP) between water operators in which strong emphasis is on capacity development (CD) and not on capacity substitution, the partnership is associated to an investment programme and where accountability for results in both CD and key performance indicators (KPIs) is required.Two parallel projects, implemented in Blantyre andLilongwe, Malawi (2009-2013), are pioneers of this approach.These two projects are regulated under what has been called a service contract, designed by the European Investment Bank (EIB) and funded by both the EIB and the European Commission (EU).Contrary to management contracts, in which management positions are fully or partially taken over by an external partner, the focus is on developing local individual and organisational capacity with the support of an external water operator.
The key underlying assumption is that with time, the enhancement of the local water operator's capacity aligned with the supporting investments will result in sustained KPI improvements.In CDPs the external partner acts as a change agent and trainer, supporting the local partner's organisational development process.There is little empirical evidence of how this type of CDP functions; however, its strong alignment with key conditions for aid effectivenessthe use of existing local structures, support of CD towards a lasting impact, accountability for results and the strong emphasis on local ownership (OECD, 2011a)suggests its potential as a valuable mechanism for the effective development of water services.
In those cases where the external partner engaged in a CDP operates on a not-for-profit basis, the approach aligns with what is known as a Water Operator Partnership (WOP), a mechanism created by the United Nations Secretary-General's Advisory Board on Water and Sanitation (UNSGAB) in 2006 and primarily promoted by the Global Water Operators' Partnerships Alliance (GWOPA).WOPs centre on CD between water operators as the conduit to sustained performance (UNSGAB, 2006;GWOPA, 2011).Rather than a specific project type, WOPs convey a wide range of arrangements between water operators, with two key distinctive attributes: (i) CD as opposed to substitution; and (ii) a not-for-profit nature.WOPs therefore entail a great diversity of projects, ranging from short-term collaboration targeting improvement in a specific area, known as specific WOPs (see IWA, UN-HABITAT & VEI, 2009), to long-term comprehensive partnerships addressing organisational reform processes, called comprehensive WOPs (see IWA et al., 2009).Despite the strong promotion of this approach, WOPs have been criticised for being generally accepted for their nature of solidarity despite meagre empirical evidence of their effectiveness (Boag & McDonald, 2010).We argue that the inability to demonstrate concrete results in WOPs is partially rooted in the narrow vision of what the results entail and not necessarily on the low effectiveness of WOPs as CD interventions.A fundamental limitation when accounting for results in CDPs is the absence of a suitable approach for measuring CD results in water operators (Pascual Sanz & Hoffer, 2009).The complexity of measuring capacity improvements was explicitly stated by the independent auditor of the two Service Contracts in Malawi.
CDPs are likely to be slower in achieving performance improvements than partnerships with some degree of delegation to the external partner.Yet most development interventions aimed at the operational improvement of water operators have inherited from PPPs the modus operandi of merely looking at the KPI gains of the targeted water operator as the unique yardstick against which to measure the effectiveness of the intervention (Mvulirwenande et al., 2013).For most KPIs it is improbable that substantial change can occur during the first year, particularly in cases where the project needs to establish an accurate baseline scenario and procure new resources (Marin, 2009) in addition to developing knowledge transfer activities.Furthermore, changes in KPIs are likely to be the result of many contributing factors, not only the partnership project.Thus, KPIs do not necessarily show the effectiveness of the project.If other signs of progress and effectiveness more closely attributable to the partnership are not used, partners and stakeholders are likely to remain in a blind stage with respect to how well the partnership project is functioning.This situation is likely to affect the project negatively in many ways, particularly in early stages.First, it can provoke frustration between partners, who remain under the impression that no improvements are being achieved.Second, it limits the possibility to learn, adapt and account for the performance of the project, given that the KPI improvements are not yet achieved and there are no other insights into the process.Hence, we argue that the mere use of KPIs to inform about the performance of the partnership1 provides incomplete and possibly misleading information, hindering the effectiveness of the partnership.
The main question addressed in this paper is: How can monitoring and evaluation (M&E) of CDPs be conducted throughout the duration of the project such that higher accountability, learning and adaptation to emerging needs can be fostered?Grounded in different strands of literature, a multi-path approach for M&E of CDPs is proposed and empirically tested by a comparative case study of the Lilongwe Water Board (LWB) and the Blantyre Water Board (BWB), Malawi.This enables us to answer a secondary question: How did the CDPs function in LWB and BWB during the first 2.5 years of project implementation?The paper first elaborates on the main theoretical discussions on which the formulation of the proposed approach is grounded, namely, (i) capacity and its development, (ii) capacity, performance and attribution, and (iii) M&E of CD partnerships.The research model, its operationalisation and the methodology used in the two case studies are then explained.Next, the results illustrating the most salient features of the findings are presented.In the concluding section, the main contributions and limitations of the multi-path M&E approach and the analysed CDPs are discussed.

Capacity and its development
The general understanding of capacity has evolved over the past two decades.A transition can be observed, from a narrow perception of capacity as the ability of the individual or the organisation to perform effectively and efficiently (Grindle & Hilderbrand, 1995;Lusthaus et al., 2002) to a concept that emphasises its multidimensional character, expressing it as the ability of individuals, as well as their organisations and the enabling environment around the organisations, to perform and develop well together in a sustained way (Alaerts et al., 1991;Dollar & Lant, 1998;OECD, 2006OECD, , 2011b)).The more recent perception incorporates views from systems thinking, conceptualising capacity as being broader than the sum of its parts; and from complexity theory, based on which capacity is conceived as an unpredictable state or condition that dynamically emerges from a complex combination of tangible and intangible attitudes, resources, strategies and skills in a particular context that enables a human system to perform, survive and self-renew in the long term (Morgan, 2006).It is widely accepted that organisational CD is more than the acquisition of resources, knowledge and skills by the individuals in the organisation, and that it also requires strengthening the capacity of the organisation itself to act and self-renew in an unpredictable environment.Kaplan (2000) emphasises that CD is an endogenous process that external interventions can only support, not steer.The practice of CD in the development arena has moved from a narrow vision that conceived it as a mechanical process of transfer of external knowledge and/or resources from individuals in the North to the South, to one in which South-South and triangular co-operation are gradually taking a more dominant role (OECD, 2011b) as supportive forces to on-going development.Several authors caution against a narrow understanding of capacity and stress the need to consider the following aspects for effective CD: (i) Intangible or soft elements of capacitythat is, human and organisational capacities, or social capital, of the organisation, including management knowledge and skills and organisational systems and proceduresin addition to the tangible elements such as infrastructure and technological and financial resources (Kaplan, 2000;Horton, 2003).(ii) Adaptive/learning capacity or organisational talentsthe ability of the individual or organisation to learn and adapt in response to changing circumstancesin addition to the operational capacity to perform under expected conditions (Horton, 2003;Alaerts & Kaspersma, 2009;Ortiz & Taylor, 2009).(iii) The political dimension of capacitypower structures, incentives, tensions and conflicts which provide the energy that brings motion, purpose, direction and change for better or for worse (EuropeAid, 2009).
The focus on the tangible elements of capacity in development interventions has been pointed out as the main cause of the disappointing results reported in the last decade (Kaplan, 2000).Besides, Ortiz & Taylor (2009) state that adaptive capacity is not easy to develop through short-term external interventions.Instead, 'organisations must live it and learn it', which calls for the ability to learn.With regard to the political dimension of capacity, EuropeAid (2009) emphasises that although capacity resides internally and its development has to be internally driven as well, some external drivers can catalyse, initiate and help shape the internal ones.Therefore, when targeting capacity, the political dimension of the organisation also requires consideration (ibid).

Capacity, performance and attribution
The capacity and performance of a targeted organisation do not necessarily always correlate, although a strong capacity is a prerequisite for an organisation to demonstrate strong performance over time (Baser & Morgan, 2008).While the performance of a water operator informs about service delivery, capacity informs about the ability of an organisation to start delivering that service and sustain it in both expected and unexpected circumstances.The lack of clarity on how to characterise and measure capacity motivates the use of performance as a proxy for capacity.We argue against this practice, given that the use of performance as a proxy indicator provides an equivocal and potentially misleading indication of how development interventions perform in delivering CD goals.Instead we argue that performance and capacity should constitute parallel sources of evidence about the effectiveness of CD interventions.
The complexity of measuring CD and the absence of robust CD indicators are widely echoed (Morgan, 1997;Mizrahi, 2004;GWOPA, 2011).Morgan (1997) suggested that 'measures exist at the input and output end of the spectrum and many indicators can be found which address service delivery and performance outcomes.There remains, however, a 'black box' in the middle of the indicator spectrum [that has] to do with CD, which remains vague and unclear'.One and a half decades later, there are numerous methodologies that revolve around CD planning, M&E.They respond to the complex nature of capacity and therefore do not follow the evidence-based approach limited to predefined technical performance.Instead, most of them follow dynamic participatory approaches to planning, M&E, open to recognising not only intended but also unintended changes, as well as changes that are not necessarily derived from the CD intervention but from other contributing factors.Some examples are: the Most Significant Change (Davies & Dart, 2005), Outcome Mapping (Earl et al., 2001;Engel et al., 2006), the Five Capabilities Framework (Baser & Morgan, 2008;IOB, 2011) or the one specifically tailored to appraising capacity to reduce Non-Revenue Water (NRW) in water operators (see Pascual Sanz et al., 2011).
Despite the indisputable contribution of these approaches, we argue that the identification of changes that an organisation has gone through during an intervention and of plausible contributing factors to such changes, does not necessarily inform us of how effective the activities of the partnership have been.Similar to performance change, capacity change is the result of numerous factors, both endogenous and exogenous, most of which are difficult to trace and control (Earl et al., 2001;Ortiz & Taylor, 2009).Achieved capacity changes can be eroded by subsequent influences of exogenous or endogenous factors, lessening the project's impact and its apparent performance.On the contrary, the effect of an external CD intervention can be enhanced by other on-going developments, leading to KPI gains for which the external CD intervention may be a lesser contributing factor.Importantly, a significant time lag exists between the implementation of a CD intervention and its translation into stronger organisational capacity and KPI gains (Earl & Carden, 2002).Thus, the attribution of performance change or capacity change to a single intervention is rather elusive and calls for application of complementary approaches to study how CD interventions function in context and what they actually achieve.Earl et al. (2001) suggest that donors should make their recipients accountable for proving that they are progressing towards impact and improving their effectiveness, rather than being accountable for the impact itself.'[…] the threat of failing to discover 'hidden attribution' is eliminated when feedback on performance concentrates on improving rather than on proving, on understanding rather than on reporting and on creating knowledge rather than on taking credit' (Earl et al., 2001).Thus, with this shift, the connection between inputs and impact is rational rather than empirical, and the impact, instead of being the yardstick against which the performance is measured, is a guiding light with which the relevance of an intervention can be tested.This perception-based approach provides the point of departure for the discussion in the next section.Complementary M&E paths to performance and capacity changes are proposed for a better understanding of the effectiveness of the partnership activities.

M&E of the performance of CD partnerships 2
In principle, M&E requires an idea of what the means and ends are, in order to be able to determine whether and how the planned action is taking place (Ortiz & Taylor, 2009).Partnership evaluation frameworks commonly encapsulate the results of the partnership in the boxes output, outcome and impact (Asthana et al., 2002;Brinkerhoff, 2002;Kolk et al., 2008;Pfisterer, 2011;PrC, 2012).There is, however, no consensus about how to operationalise further those boxes in order to know the extent to which results are achieved.Such divergence is partially explained by the different purposes targeted via partnerships and the stage of the partnership at the time when M&E is conducted.Partnerships are dynamic relations and evolving processes, the results of which change over time (Asthana et al., 2002;Brinkerhoff, 2002;PrC, 2012).Evaluators will thus obtain different types of results, depending on the timing of the evaluation exercise.An additional complexity in the case of partnerships for CD is that the prescription of desired CD results runs against the concept of the emerging nature of CD and could negatively affect the CD process (Ortiz & Taylor, 2009).
Grounded in the literature streams of strategic alliances, evaluation of partnerships and knowledge transfer, three additional M&E paths are proposed for CDPs.
1.3.1.Degree of new knowledge consolidation.The learning perspective of partnerships points to the extent to which the performance of an alliance is measured by its ability to serve as a knowledge and learning-channelling mechanism (Argyris & Schön, 1978;Fiol & Lyles, 1985).Yet measuring knowledge presents multiple challenges.It requires measuring both explicit and tacit knowledge, which adds two difficulties: the lack of a robust system to classify knowledge (Easterby-Smith et al., 2008) and the intricate task of measuring tacit knowledge (Shin et al., 2001).Bridging these difficulties are authors who point to changes in behaviour as a measure of KT effectiveness (Davenport & Prusak, 1998;Szulanski, 2000).According to Szulanski's model (Figure 1), KT is an unfolding process composed of several stagesinitiation, initial implementation, ramp-up and integrationthat start with the identification of opportunities for knowledge transfer and end once the recipient unit (partner) is able to maintain satisfactory performance derived from the integration of knowledge into working routines (change).
Initiation comprises the process that takes place during the period from the initial idea of a transfer to the moment the decision is taken about what knowledge to transfer.Initial implementation covers the period from the time the decision is taken until the first time the recipient unit uses this new knowledge.(Szulanski, 2000).
Ramp-up encompasses the process from the first time that new knowledge is used to the moment in which new knowledge has been incorporated into working routines.Integration comprises the process of maintaining desired performance in new working routines.Szulanski claims that obstacles or what he calls 'sources of knowledge stickiness' for KT will appear in every stage and that they should be understood as inherent in the process.Szulanski's stages of KT are instrumental to unpacking the 'black box' of CD.Based on Szulanski's model, a new path for the evaluation of partnership for CD is proposed, namely the degree of consolidation of new knowledge into change of working routines, together with the sources of stickiness (obstacles) detected in the KT.The degree of consolidation of new knowledge is proposed as an indication of progress towards impact, mainly serving accountability purposes, while the sources of stickiness of KT primarily serve learning and adaptive purposes.Rahman (2006) calls the duality of alliance performance.They emphasise that while a firm's performance is gauged against the extent to which it achieves its short-and long-term goals, two significant paths simultaneously need to be pursued in order to understand partnership performance: achieving goals and maintaining a harmonious relationship.Positive inter-organisational dynamics between partners are considered an indication of the partnership's good performance (Goh, 2002;Mohr & Sengupta, 2002;Rahman & Kumaraswamy, 2008).Particularly in these types of partnership, in which the external partner acts as a change agent, the importance of relationship between partners in achieving the pursued change is critical (Schein, 1999;Block, 2011).Every stage of the process that partners go through is an opportunity for the change agent (external partner) to engage the client (local partner) in the change process.Regardless of the expertise and competence of the external partner, unless the interventions are skilfully facilitated with the commitment of the local partner, the intended changes are not likely to occur (Anderson, 2010).For example, the initial data-gathering process that the external partner needs to go through is considered a highly powerful intervention (Anderson, 2010) given that the external partner has the opportunity to use that time to invest in trust-building, through getting to know members of the organisation, promoting dialogue and mutual understanding and showing empathy towards their views of the organisation.The evidence path grounded on quality of inter-organisational dynamics is expected to serve in particular the adaptive and learning purposes.

Partners' relational quality. A complementary angle to the evaluation of partnership is brought up by authors alleging what
1.3.3.Satisfaction of partners and stakeholders.The perceived benefits and/or overall satisfaction of partners with the knowledge transfer process have been frequently proposed and used empirically as an indication of alliance performance (Becerra-Fernandez & Sabherwal, 2001;Brinkerhoff, 2002;Chini, 2004).Perceptual measures are claimed to be just as important and informative as more objective ones (Becerra-Fernandez & Sabherwal, 2001;Reuer et al., 2002;Sammarra & Biggiero, 2008).The degree of satisfaction of partners and stakeholders can provide valuable insights into the direction that the project is following and contributing and constraining factors, which provide information supportive for adaptive and learning purposes.

Approach to M&E of CDPs between water operators
The new approach (Figure 2) proposed in this paper differentiates between the progress towards impact (such as capacity developed or KPI gains, for accountability purposes) and the effectiveness of the partnership in supporting such progress (for learning and adaptation purposes).The M&E approach comprises five complementary paths.Presented in order of highest to lowest plausible attribution of the result to the partnership project, the five paths are: (i) satisfaction of partners and stakeholders; (ii) quality of inter-organisational dynamics; (iii) project inputs consolidation into change; (iv) capacity changes; and (v) KPIs.The variables time and attribution are also included in the model.Time indicates that evidence derived from all the paths will vary throughout the lifetime of the project.Attribution is included to distinguish between results with higher plausible attribution to the partnership, like satisfaction of partners and stakeholders, or quality of inter-organisational dynamics, from those to which many other factors are likely to contribute, like capacity and KPIs.The higher the plausible attribution to the partnership, the more informative the evidence will be for learning and adaptive purposes.
1.4.1.Operationalisation of variables.The first M&E path is the degree of satisfaction of both partners and stakeholders.It is operationalised in terms of degree of satisfaction with the project achievements and the aspects of the project perceived to be most valuable.The second path is the quality of inter-organisational dynamics between partners.It is operationalised through the types of formal and informal interactions between partners, the role adopted by each partner, dominant governing mechanisms and the occurrence of conflict and its impact on the partnership.The third path is the degree of consolidation of new knowledge into change.Owing to the emergent and unpredictable nature of CD, the more advanced the project is chronologically, the higher the likeliness that many factors will contribute to the perceived progress and the higher the chances that unintended effects of the partnership will occur.The model captures not only intended but also unintended effects of the project and promotes the triangulation of findings through the use of multiple constructs to inform about changes.The following constructs are used to inform each stage of KT: • STAGE 1: Progress in the delivery of project-related improvement plans is gathered as an indication of the completion of the initiation stage.
• STAGE 2: New knowledge gained from the project activities by individuals of the targeted water operators is selected as an indication that the initial implementation stage has been reached.
• STAGE3: The ramp-up stage (change) is informed by several sources: (i) detected changes in work- ing routines derived from the initially defined project activities; (ii) identification of additional effects of the project; and (iii) the perceived degree of sustainability of the improvement trend.
• STAGE 4: The integration stage is informed by the identification of well-performing working rou- tines that integrate new knowledge and are maintained over time.
The fourth proposed path refers to organisational capacity changes of the targeted water operators.In this project it was operationalised by determining specific changes in capacity to reduce NRW, given that this KPI was of highest priority in the project and a critical performance indicator for operational efficiency in water supply operations.Changes were analysed based on an adjusted version of a methodology for appraising water operator capacity to reduce NRW, through which the main organisational capacity domains affecting NRW can be traced (see Pascual Sanz et al., 2011).The fifth path reviews the KPI changes over the project duration, operationalised in this case through the three shared KPI targets set in the contract for (recipient) water operators in the case studies, namely, NRW, Working Ratio and Number of Kiosks Built.

Methods
The main criteria for the selection of the cases were: (i) the existence of a medium-to-long-term partnership between two established water operators, one from a richer and the other from a developing country, with the aim of supporting the CD of the local operator on a not-for-profit basis; (ii) the existence of an investment programme associated with the partnership; and (iii) a focus on NRW reduction.In addition, accessibility to key people able to inform about the project in retrospect (starting from the origin of the concept of the project) and minimal differences between cases to allow comparison, were also prioritised.The cases selected are partnerships where two Malawian Water Boards (WB), namely the LWB and the BWB, engaged with Vitens Evides International (VEI), a Dutch water operator, in 4year service contracts (SC) 3 .Both SCs were funded by a grant from the EU and a loan from the EIB.Each partnership project had a separate contract, but both were under the same governance structure and presented the same contractual characteristics.VEI committed to support each WB to achieve certain performance targets through continuous operational support (managerial and technical) as well as through guidance and supervision on the implementation of an investment programme.The decision to partner with an external water operator was highly driven by the EIB, which posed it as a condition for the WBs to access the loan.The selection of VEI as the external water operator followed an international competitive tender process.
This research considers the results of the projects 2.5 years after they had started.Multiple methods were applied for the collection of data.Secondary data were collected from a thorough documentary review, including legal framework documents, country development reports and statistics, water sector reports, the project contract, project proposal, project quarterly and annual reports, project KPI reports and external auditor reports.Primary data were collected during the months of May and June of 2012 in both project locations and during October and November 2012 in the Netherlands.A multi-method approach to primary data collection was applied, combining interviews (both in person and by phone), questionnaires, group discussion, workshops and participant observation (Table 1).This widespread information collection allowed checks for internal consistency, cross-referencing, significant similarities and differences between groups and the deduction of more generally applicable conclusions.The results are presented in Section 3, ordered from low to high plausibility of attribution to the partnership.

KPI achievements
Results from the first and second years of the project (Figure 3) illustrate that only the KPIs No. of Kiosks Built in BWB and Working Ratio in LWB were achieved.NRW targets were not achieved in any of the projects, although preliminary signs of decreasing NRW were observed in LWB by the end of the first year and in BWB by the end of the second year.The KPI Working Ratio showed a slight improvement in BWB in the first year but later fell back, while a steady improvement trend emerged in LWB.LWB did not build any kiosks during the first year and very few during the second year.KPI gains, despite being the decisive indicator against which the project performance was assessed, provided very little information on how effective the partnership activities were in leading to broader operational improvements.In fact, it was argued by LWB and BWB staff that the improvements in both of the KPIs that had been achieved -No. of Kiosks Built in BWB and Working Ratio in LWBwere strongly driven by the WBs' own activities and not so much as a result of VEI inputs4 .RPMs in each project were also asked to fill in the same questionnaire to be able to assess the extent to which the perceived changes coincided.The same questionnaire included a section in which the key informants were asked to indicate the degree to which internal factors, the partnership project and/or external factors had contributed to the perceived changes.

Changes in capacity to reduce NRW and contributing factors
Aggregated results from BWB show an overall positive change, perceived in most of the components, in capacity to reduce NRW, with the exception of the inter-organisational networking capability component, for which they showed no appreciation of change.Among all the positive changes, the highest scores were allocated in the components' organisational structure and availability of resources.The assessment provided by the RPM in BWB aligns well with the perception of BWB management, albeit with one exception.The collected inputs on plausible attribution point unanimously to internal organisational factors as having the strongest influence, followed by the partnership and external factors, respectively.Capacity components for which the partnership was perceived most influential (with proportions highlighted in bold) are organisational motivation, resources, capability to reduce leakages, changes in organisational structure and learning capability.
Aggregated results from members of LWB (Figure 5) demonstrate a perceived positive change in all the components of capacity to reduce NRW, with three of the 14 components showing a comparatively higher positive change than the rest: strategic management, resources and capability to reduce leakages and overflows in storage tanks.Unlike findings in BWB, the scores by VEI RPM in LWB portray a considerably different perception.Despite agreeing on some of the positive changes indicated above, VEI RPM indicated no signs of positive change in several other components such as strategic managerial level, organisational structure, responsiveness of supportive departments, inter-organisational networking capability, organisational motivation, capability to reduce customer metering inaccuracy and data handling errors, and capability to reduce leakages on service connections up to point of customer.The divergent view from VEI RPM and LWB representatives might be explained by various factors.One might be the reportedly low degree of interaction/communication between LWB management and VEI RPM, which may have prevented VEI from noticing on-going developments in LWB's capacity.Another explanation is an overly positive view by LWB members on capacity changes, arguably motivated by recent events in LWB and the presence of the CEO in the workshop.Days before the workshop took place LWB had received an EIB dissatisfaction warning.This stated that if they did not commit to the project activities, particularly with regard to the KPI No. of Kiosks Built, they would run the risk of EIB funding withdrawal.Additionally, while the CEO of BWB preferred not to participate in the workshop in order to enable a more open discussion, the CEO of LWB participated and led the discussion.Both aspects might have influenced the scores assigned by the participants to the progress in capacity change.
When comparing the findings in the BWB case with respect to the attribution of changes, internal factors in the organisation appear to have had the strongest influence on the perceived changes, followed by the influence of the partnership and external factors, respectively.In the LWB case, the components perceived to be more strongly influenced by the partnership are organisational motivation, availability of resources, capability to reduce leakage on transmission and/or distribution mains, inter-organisational networking capability and learning capability.
The workshops proved informative.Although attention was paid to maximising the similarity of conditions between the two cases in terms of participant composition and timing in relation to outside events, differences emerged between them.When comparing the dynamics of both workshops, it is notable that while BWB participants adopted a more open and self-critical position, LWB participants appeared to be take a more defensive attitude, trying to present the best image of LWB progress.This is consistent with the explanations described above, that is, the role of the presence of the CEO in the LWB workshop and the recent dispute with the EIB in the LWB case.
This path provides two types of analysis.First, it reveals how the evolution of capacity of the WBs to reduce NRW was perceived to change during the implementation of the partnership.Second, it informs about the perceived plausible attribution of such changes to different factors, one of which is the partnership project.The resulting evidence serves both accountability and learning-adaptive purposes.In addition to reporting on progress made in the different domains of change relevant for a specific KPI improvement, it proves helpful in identifying critical areas that have not been sufficiently addressed by the partnership project (i.e.illegal connections in BWB).The indication of plausible attribution can be a valuable source of information to detect not only contributing but also constraining factors (i.e.little change in organisational structure in LWB, for which the external context seems to be playing a strong limiting role), as well as to illustrate the multiplicity of contributing factors affecting changes.However, despite the added value that capacity changes and attribution confer, just like performance changes, they do not inform about the process leading to those changes.Hence, this evaluation path is limited in that it does not describe how effectively the implementation of each activity was and obstacles encountered along the process.

Project-related knowledge consolidated into changes
This M&E path focuses on the project activities as a departing point for the analysis; it informs about the extent to which the project activities resulted in change and about obstacles that emerged throughout the process.Three sources of data were used: (i) specific activities-related knowledge consolidated into change; (ii) other effects of the project; and (iii) sustainability of the perceived trend of change.
3.3.1.Specific task-related knowledge consolidated into change.The comparative degree of consolidation of the task-related knowledge into desirable performance of working routines was informed by the testimonies of both partners.First, the respective VEI RPMs and STEs (10 in total) reported their views on the progress of the knowledge related to each contractually defined task.The results are illustrated in the Knowledge Consolidation Map (Figure 6).
The Knowledge Consolidation Map shows that BWB had progressed further than LWB on the selected tasks.In LWB the progress almost always stopped at Stage 2gained individual knowledge, while BWB achieved Stage 3change in working routines in six of the project tasks.As a means of cross-checking and complementing the indication provided by VEI members, a more detailed account of achievements of Stages 2 and 3 was gathered through interviews with WB staff (17 key informants from BWB and 11 from LWB), the findings of which are described in the more detailed explanation of Stages 2 and 3, below.
3.3.2.STAGE 1 -Decision on knowledge to be created.Both projects experienced the same behaviour with respect to submission of deliverables.After 2.5 years of project implementation, all the expected deliverables had been submitted and approved with the exception of the loss reduction report.This report requires the execution of a comprehensive measurement of the network, work that was still on-going at the time of data collection.Despite this, decisions regarding what knowledge to transfer on that topic had already been taken, given that numerous training activities had been directed to that specific task.
3.3.3.STAGE 2 -Knowledge gained individually, derived from the partnership activities.The reported knowledge gains from selected key informants in both WBs are grouped into six categories: management's overall support, financial management, commercial management, physical losses reduction, GIS and O&M (see Appendix A, available online at http://www.iwaponline.com/wp/015/022.pdf).Both WBs reached Stage 2 in most of the tasks and the knowledge gains reported for each of those knowledge areas were very similar in both projects, which can be explained by the fact that the same KT approach was followed by VEI in both projects.iwaponline.com/wp/015/022.pdf)largely mirror the overall view of VEI RPMs and STEs presented in the Knowledge Consolidation Map, showing that further progress had been achieved in BWB.Among the most notable changes implemented in BWB but not in LWB are: a change in organisation structure, the creation of the position of caretakers and implementation of related working routines, and the creation and maintenance of a functioning GIS.The identification of obstacles in the KT process is also a valuable contribution of this evaluation path, predominantly serving learning and adaptive purposes.For example, the main obstacles identified for some task-related knowledge that prevented them from moving to Stage 3 were: (i) low interest from executive management (possibly influenced by external forces) in the incorporation of the tariff calculation model as a working routine; (ii) weak internal communication and coordination between departments and sections and low interest by executive management in the MIS; and (iii) insufficient training and follow-up from VEI for the tasks pumping regimes and measures for loss reduction.As anticipated by Szulanski (2000), integrating new knowledge into working routines is highly challenging and is likely to face multiple obstacles.
3.3.5.Other effects of the project.Key informants from the WBs reported other benefits from the project.At the operational level in BWB, those emphasised were: an increase in motivation, a willingness to learn and an improvement in staff attitude (staff are now confident that the targeted KPIs can be achieved)9 ; additional resources available in the WB10 ; the increased engagement of staff and a shared sense of urgency to improve 11 ; and an increased commitment of the different sections to a role in NRW reduction 12 .BWB's CEO called attention to the particularly valuable contribution of the project as a force supporting organisational change by reducing internal resistance to change, limiting political interference and increasing accountability for results.In LWB, some of the highlighted aspects were: acquisition of new equipment 13 ; stronger overall capability and commitment to reduce NRW throughout the organisation 14 ; and a higher sense of urgency to improve owing to the stronger need to prove results 15 .In contrast to BWB, the CEO of LWB emphasised the more accurate picture of the current status of LWB provided by VEI as one of the most important contributions of the project.
3.3.6.Sustainability of the change trend.Respondents from BWB offered mixed opinions when evaluating the extent to which the new trend could be maintained.On the other hand, members of LWB commonly stated that if the project ended at that time, the WB would revert to earlier attitudes, because the sense of urgency and the need to be accountable for results would disappear 16 .Similar conditions were identified by the WBs' staff for the change trend to be sustained over time (Table 2).They offered pointers for the improvement of the project's effectiveness for the remaining duration of the project.

Partners' relational capital
Interviews with members of both partners in each project provided information about the change over time in the inter-organisational dynamics between partners described by the type of interaction, the role adopted by each partner, dominant governing mechanisms and conflict management.The information collected offered valuable insights into the evolution of each project and how effectively partners worked together (see Appendix C, available online at http://www.iwaponline.com/wp/015/022.pdf).
3.4.1.Degree and type of interaction.On the side of VEI, the quality of interaction between partners was mainly determined by the RPMs of each project, who interacted with members of both WBs at the management and operational levels.Despite a similar start-up, each project had followed its own path in terms of its interactions between partners.Figure 7 illustrates how in LWB, formal (planned) interaction decreased in both projects over time at the management and operational levels.It also showed how informal interaction increased considerably in BWB throughout the project, but not in LWB.In LWB a general perception was reported to be that the VEI RPM was pushing in one direction and LWB management in another 17 .  1One specified interview/questionnaire source.
3.4.2.Evolution of partners' roles.Members of both partners were consulted about the evolution of the role of each partner throughout the project.While in both WBs, VEI was initially seen as an external consultant, 2.5 years after the project start VEI was perceived by BWB to be a supportive partner and considered part of the BWB team 18 , but in LWB the perception of VEI had evolved to one of a controlling agent, having adopted a supervisory attitude towards the project implementation.The roles of the WBs, analysed through their engagement in the project activities, had begun at a quite low level in both cases, shifting to a strong engagement only in BWB during the second year of the project.Figure 8 illustrates this pattern.
3.4.3.Dominant governing mechanisms throughout the project.In both projects, partners converged in identifying the contract as the ruling governing mechanism at the beginning.Yet only in BWB did trust, derived from interaction and from shared efforts to improve operations, gradually permeate as a strong governing mechanism at the onset of the third year of project implementation.Figure 9 illustrates these differences.
3.4.4.Conflict occurrence and impact on the partnership.Both projects suffered from considerable mistrust initially, and confusion about the role of VEI during its early stages, stoked by the long-standing privatisation rumours in the sector and the expected negative consequences for employees.The private involvement through a strong PPP form was intended to allow a private partner to (partially) take over the management of the WBs.This departure point seems to have contributed to a misperception of VEI's role in the project as the private partner, even though the contract did not entail taking over responsibilities.VEI was seen as a contractor that would implement the project activities rather than a partner that would support the WBs in implementing the project activities.This confusion triggered conflicts between partners from the start.Failure to achieve the targeted KPIs in both WBs during the first years was blamed on VEI and triggered dissatisfaction among the WBs' executive management.  1Two specified interview/questionnaire sources.
Furthermore, BWB claims about VEI, already in the first year, demanded changes in VEI resident positions so that the project would achieve the targets.In LWB, most of the conflicts derived from demands for VEI to take over engineering tasks.VEI frequently opposed such requests and suggested instead supporting LWB engineers in executing the tasks, following the CD philosophy of the project.The conflict situations in BWB were mostly managed through communication and interaction between the partners, which resulted over time in a stronger relationship and a better understanding of the project by BWB.In contrast, LWB gradually isolated the project activities from the daily routine of the WB, hence providing minimum support for the implementation of the project activities (see Figure 10).

Satisfaction of partners and stakeholders
The degree of satisfaction with the project was determined by interviews with members of both WBs at the management and operational levels, members of VEI (RPM and PD) and the primary stakeholdersdonors and national government (Appendix D, available online at http://www.iwaponline.com/wp/015/022.pdf).
3.5.1.WB members.Members of BWB at the management and operational levels expressed their satisfaction with the progress, although they claimed little visibility of progress in KPI gains.Members of LWB showed a mixed perception regarding their satisfaction with the project.The positive impressions were mostly provided by staff at operational levels, while executive management showed less appreciation of the project.The identification of the most valuable aspects of the project depended very much on the person, given that project activities affected sections and individuals differently.However, the most cited contributions in both WBs were the efforts to reduce NRW 19 and additional resources acquired throughout the project 20 .Furthermore, LWB staff mostly referred to activities contributing directly to their individual work, while BWB members pointed to the project's contribution to organisational changes, particularly in terms of soft capacity elements.LWB's interviewees mentioned efforts to update the customer database 21 , the reorganisation of the office responsible for GIS 22 and the encouragement to collaborate and share information 23 , while members Fig. 10.Conflict occurrence and its impact on the partnership. 1914 specified interview/questionnaire sources. 2018 specified interview/questionnaire sources. 21Two specified interview/questionnaire sources. 22One specified interview/questionnaire source. 23Two specified interview/questionnaire sources.
of BWB emphasised the project's impact in improving staff mind-set, their enhanced sense of pride in working at BWB, their willingness and urgency to improve24 , the increased internal communication, critical thinking and accountability between different sections25 and the improvement in management and co-ordination skills26 .Several members in both WBs at the managerial level stressed the particular value of the project by combining additional resources for investments, accompanied by the continuous and comprehensive organisational development support from an external operator27 .
3.5.2.RPMs.VEI's RPM in BWB expressed his enthusiasm about the progress made and acknowledged that without the support from the CEO it would have been very difficult.He pointed to the changes achieved in staff attitude as the most challengingand most essentialindication that things were moving in the right direction.The joint work of the VEI RPM and BWB CEO, as well as the approximately ten members of BWB firmly engaged in the project activities, seemed to be the formula that enabled the widely acknowledged positive direction of the project in BWB.In LWB, despite a challenging time for the RPM, which was arguably due to the low engagement of LWB management, the RPM acknowledged that although progress could have been broader, a good foundation had been laid for the remaining period of the project, with most of the invested programme having been implemented and the KT process started.He particularly mentioned the engagement of two people in the project with a strong learning intent, who helped to push some activities ahead.He pointed to the need for LWB management to engage in the project activities as being a critical factor for achieving progress.
3.5.3.VEI project director.The VEI PD expressed greater satisfaction with BWB than with LWB.He expects BWB and LWB to achieve 80% and 50%, respectively, of the targeted KPIs by the end of the fourth year.He pinpointed the 'good chemistry and synchronisation of efforts' between VEI and BWB as the strength that moved the project along more quickly there.
3.5.4.Stakeholders.The collected views converged in appreciating, on the one hand, the complexity of the innovative approach of an external operator working together with the WBs in strengthening their capacity, and on the other, the potential value of this type of CD in terms of the sustainability of the achieved improvements.A shared observation by donors and representatives from the Ministry was the greater progress achieved by BWB, not only in terms of changes detected, but particularly with respect to the engagement of management in the project activities and the relational quality between partners.EIB expressed its dissatisfaction with the progress achieved in LWB and its staff's low engagement in the project activities, while acknowledging the positive direction that the project had taken in BWB.Interestingly, the representative from the EU local office acknowledged the fact that although the expected results had not been achieved, there was progress in a positive direction.She claimed the need to look beyond KPIs and flexibility from donors in this kind of project, where the external partner (in this case, VEI) does not have the decision-making power but rather acts as a change agent and supporting partner.

Conclusions
This paper has proposed and tested a multi-path approach for the evaluation of CDPs that enables accountability, adaptation and learning throughout the project.Despite the fact that CDPs aim to improve both performance and capacity, M&E practices typically refer solely to KPIs, which serve to capture results for only one of the two objectives, namely performance-related results.The paper argues how this practice limits, on the one hand, the possibility to account comprehensively for results gained and on the other, the ability to learn and to adapt the CD activities of the partnership in response to emerging needs.The empirical evidence gathered from the cases supports this claim and illustrates the benefits of applying a multi-path approach to evaluate the performance of CDPs.
According to the information provided by KPIs, neither LWB nor BWB reached the targeted performance.KPIs also do not offer any other sign of how effectively the partnership project contributed (or not) to KPI gains and CD.Hence, the immediate perception was that the partnership project had failed to achieve its targets.In contrast, the empirical evidence collected from the proposed four additional complementary evaluation paths (to changes in KPIs) -NRW-related capacity changes and plausible attribution, project inputs consolidation into change, quality of inter-organisational dynamics and satisfaction with the projectshows that progress was actually achieved in both cases, BWB being the case that achieved more and worked more effectively.
The analysis and evaluation path of NRW-related capacity changes and plausible attribution provided an overview of the (perceived) degree of changes that each WB went through and the attributed role of the partnership in supporting these changes.The overall perception in both WBs was that positive changes had occurred during the project, some of which were more strongly attributed to the partnership than others.This finding further motivates the relevance of using complementary evidence to inform about the performance of this type of partnership, since it proves that although there was a lack of substantial change in the KPI NRW, positive capacity changes to reduce NRW were identified in both projects.Both partners pointed to internal factors as being the highest contributors to changes, indicating the partnership as a triggering, but secondary force.Hence, the findings reconfirm Kaplan's claim that external interventions to CD are only a supportive force to locally owned, on-going developments.
The path of project inputs consolidation into change proved highly valuable for accountability, adaptive and learning purposes.The Knowledge Consolidation Map shows how both projects made progress and illustrates how BWB had gone further in reaching changes in working routines than LWB.Insight into knowledge consolidation into change identified obstacles to an effective transfer, for which specific project tasks needed adaptation.The accounting of additional effects illustrated how, in addition to financial and technical resources, the project was perceived to contribute strongly to strengthening intangible or soft elements of capacity in both cases (i.e.improved coordination, leadership and management skills and internal communication practices).In contrast to management contracts, the managers of the WBs were responsible for the service throughout the project, which allowed them to experience the challenges and learn.This aspect is considered critical for them to be able to sustain performance under new circumstances (adaptive capacity).Also, in the case of BWB, the accounting of additional effects captured the project's contribution to the creation of a supportive environment for implementing organisational change (i.e.minimising internal resistance and political interference or proposing structural changes) and increasing incentives (i.e.increasing the sense of urgency and accountability for results).Therefore, by capturing additional effects of the project, the positive impact of the project on the political dimension of capacity was able to surface.The identified conditions to sustain the change trend offer pointers for strengthening the partnership work in the future.
The path of quality of inter-organisational dynamics signalled that LWB required effort to promote further clarification of the role of VEI, promote the engagement of the local executive management, change the communication style of VEI RPM and increase the intensity and quality of interaction between partners at both the managerial and operational levels.The path of satisfaction of partner and external stakeholders informed about the extent to which the projects were working in the right direction as well as about (perceived) contributing and limiting aspects influencing progress in each project.Overall, satisfaction of the partners and stakeholders was consistently higher in BWB than in LWB.Members of BWB expressed their satisfaction with the project at both the managerial and operational levels, while LWB's appreciation of the project was only perceived at the operational level.Commitment of executive management to the project activities was highlighted as the most critical aspect contributing to the progress achieved in BWB and which in contrast limited progress in LWB.Interestingly, the need to look at types of evidence other than KPIs to judge satisfaction with the project was claimed as necessary by a donor representative.
The M&E multi-path approach provides a more robust basis to evaluate the CDP (as opposed to the performance of the water operator) for accountability, adaptation and learning purposes.The approach adopted in this research is quite comprehensive and reliant on in-depth empirical evidence.Further work will follow towards methodological simplification and the standardisation of each of the proposed paths of evidence in order to encourage the design of incentives for CDPs contracts accordingly and to promote their use as an integral part of CDP management.
The M&E multi-path approach proposed in this paper was developed in response to a detected need for suitable evaluation approaches in CDPs between water operators, such as WOPs.However, its application can be extended to the standard forms of PPPs, where CD also plays an important role.Furthermore, the proposed approach addresses a main concern and limitation of CD, namely how to measure and trace the impact of CD interventions.Hence, the applicability of this approach can be extended to CD interventions more generally, thus providing a concrete means for improving the effectiveness of CD as envisaged by the aid effectiveness agenda.
This data were collected by means of a questionnaire whose design was guided by the methodology proposed by Pascual Sanz et al. (2011).In this work, capacity to reduce NRW is operationalised through 13 organisational capabilities, all of them considered crucial for NRW reduction, plus one open component of other organisational capabilities not contemplated within the 13 components.The questionnaire was distributed during a workshop to members of the executive and first-line management of each WB.The aggregated values of the results from the questionnaires are shown in Figures 4 and 5 .

Fig. 3 .
Fig. 3. Targeted and achieved KPIs during the first 2 years of the project in BWB and LWB (annual progress report contract Year 2) 5 .

Fig. 5 .
Fig. 5. Capacity change spider web indicating reduction of NRW in LWB.
3.3.4.STAGE 3 -Changes in working routines derived from the partnership activities.The identified changes in working routines by the WBs' staff (see Appendix B, available online at http://www.

Fig. 7 .
Fig. 7. Evolution of formal and informal interaction in BWB and LWB.

Fig. 8 .
Fig. 8. Evolution of the perceived roles of partners.

Fig. 9 .
Fig. 9. Evolution of the dominant governing mechanisms in BWB and LWB.

Table 1 .
Number and type of key informants per theme of inquiry.
b Project director (PD) from VEI, responsible for both projects residing in the Netherlands.c Short-term experts (STE) from VEI, working in each project via short-term missions.✓ indicates theme addressed.7 indicates theme not addressed.

Table 2 .
Preconditions for sustaining the change trend as identified by the WBs' staff.