Emerging outcomes from a cross-disciplinary doctoral programme on water resource systems

Interdisciplinary research and education programmes in water science are intended to produce groundbreaking research, often with an emphasis on societal relevance, and prepare future water resource experts to work across disciplines. This paper explores the emerging outcomes from an ongoing doctoral programme currently in its seventh year. Within the programme, there is both cross-disciplinary and mono-disciplinary research. Three questions are explored: (i) whether cross-disciplinary research leads to more innovative scientific findings than mono-disciplinary research, (ii) whether cross-disciplinary researchers develop professional skills that benefit their future careers, and (iii) whether cross-disciplinary research produces findings of greater societal relevance than mono-disciplinary research. Various indicators are used to measure research and education outcomes. Analysis of journal impact factors and citation rates of Institute of Scientific Information indexed publications suggests that cross-disciplinary research findings are more innovative. Comparison between graduate research profile and their career destinations suggests that researchers who learn to work across the disciplines continue to work this way in their post-doctoral positions. Analysis of media interest in research findings or their impact on policy suggests that both types of research are of societal value, but researchers often expand their understanding of a societal interest topic by bringing in new research fields.


Introduction
Some of the most exciting research progress is made at the intersection between the traditional research disciplines (National Academy of Sciences, 2004). At the same time, because real world problems rarely respect disciplinary boundaries, a research approach that works across and between the disciplines is essential to address societal challenges (Klein, 1990;Jeffrey, 2003;Carayol & Nguyen Thi, 2005;Repko, 2008). For example, societal welfare directly and indirectly depends on the quantity, quality, reliability and cost of its water supplies. When there is too much water, or too little water, or too polluted water, society suffers. Solving water problems associated with such eventsthat seem to be happening more frequently, more strongly and for longer durationsis essential for sustaining any society. But water resource systems are complex and no single discipline or research field has all the answers. An extensive collection of fields are involved in understanding, for example, the fate and transport of pollutants in a river catchment. They include hydrology, meteorology, geology, geomorphology, soil science, and biological and plant sciences, as well as economics, law and sociology. Each of these research fields provides increased understanding of how the catchment functions. Research that integrates different bodies of knowledge from different disciplines or fields is essential for understanding integrated systems such as water resources ( Jeffrey, 2003).
Despite the value and need for interdisciplinary research, there are many challenges. One relates to individual human capacity. No single individual will likely become knowledgeable in all the disciplines needed, for pollution management, for example. Water resource systems professionals, besides having depth in a particular field of expertise, therefore need to be able to understand enough of other fields of expertise to be able to work and collaborate with individuals trained in those areas. They need to know how and what each field can contribute to understanding and solving any particular water management problem. Doctoral training programmes on the topic of water are designed to prepare future water resource experts to work across disciplines (Hufschmidt, 1967;Wolman, 1977;Loucks, 2008;Blöschl et al., 2012;Uhlenbrook & de Jong, 2012;McIntosh & Taylor, 2013;Carr et al., in review).
Another challenge to interdisciplinarity is the pressure to extend the state of the art within each discipline as defined by university colleges and departments and professional journals (Saito et al., 2013). Working to extend the boundaries of particular disciplines differs from working to integrate, for example, aspects of the social and political sciences with hydrology towards identifying more effective flood or drought risk management policies. Clearly such interdisciplinary research is needed, but it may not extend the theory of either discipline, and hence may not be judged suitable for publication by reviewers of disciplinary journals. The suitability of peer reviews (that tend to be disciplinary in nature) to assess the quality and relevance of interdisciplinary research findings has been brought into question (Boix-Mansilla, 2006;Klein, 2006). Despite this concern, the quality and quantity of interdisciplinary knowledge are often measured by traditional means such as publications, grants, awards and citations, and the literature tends to find that interdisciplinary work performs more poorly when measured against disciplinary work using such criteria (Rigby & Edler, 2005;Levitt & Thelwall, 2008;van Rijnsoever & Hessels, 2011). It would be of great interest to explore whether such trends also exist in the water resource research field where integration between multiple disciplines is inherent to many of the research questions being addressed.
Several authors have drawn attention to the lack of work that questions and tests the effectiveness of interdisciplinary research and education programmes (Mitrany & Stokols, 2005;Boix-Mansilla & Dawes Duraising, 2007;Saito et al., 2013). Questions, such as whether the research they produce is novel and groundbreaking, whether the human resources and capital created through these programmes are shaping water resource research and management, or whether an interdisciplinary approach offers the best way to solve current and future water resource management challenges, remain unanswered. This paper aims to address these questions by exploring whether cross-disciplinary collaboration in an interdisciplinary water sciences programme leads to innovative science, interdisciplinary graduate skills, and research of benefit to society, using an ongoing postgraduate research and education programme as a case study.

The case study and evaluation
The Vienna Doctoral Programme on Water Resource Systems The Vienna Doctoral Programme on Water Resource Systems (www.waterresources.at), based at Vienna University of Technology, was initiated in 2009 with funding from the Austrian Science Fund (FWF) and support from the university. It is currently in its seventh year and is designed to run over a period of 12 years. An anticipated 70 students will have graduated by 2021. The goal of the Programme is to produce top graduates capable of conducting advanced, independent research of the highest international standards that integrates research from multiple disciplines in water resource management. To achieve this goal, the Programme has three overarching objectivesto develop an interdisciplinary approach, to produce cutting edge research, and to generate an international perspective (see Blöschl et al., 2012). Ten research fields are included in the Programme, reflecting the university departments or sub-departments and research focus of each of the 10 faculty membersaquatic microbiology, hydrology, hydro-climatology, hydro-geology, mathematical economics, photogrammetry, remote sensing, resource management, structural mechanics, and water quality. These are described as research fields, as they represent knowledge domains, rather than traditional academic disciplines (Huutoniemi et al., 2010). Since the start of the Programme, 38 international doctoral students with diverse academic backgrounds have enrolled. To date 19 have graduated. A further nine researchers participate as associate post-doctoral fellows.
There is some complexity to defining multidisciplinary and interdisciplinary research (Klein, 2006;Barry & Born, 2013;Siedlok & Hibbert, 2014). In this paper we use the framework of Huutoniemi et al. (2010) that conceptualises multidisciplinary and interdisciplinary as points on a continuum (Table 1). Multidisciplinary work brings researchers from different disciplines together to address a common research question by all contributing some expertise without substantially adapting their approaches or changing their own positions or understanding on the question (Borrego & Newswander, 2008). On the other hand, interdisciplinary work goes beyond the division-of-labour approach of multidisciplinary research, to actually integrate different data, methods, tools, concepts or theories and create a common understanding of the issue or question (Huutoniemi et al., 2010). In this paper, we use the term cross-disciplinary to describe all the multidisciplinary and interdisciplinary work that takes place within our Programme between researchers from more than one research field. We specifically explore research taking place between researchers from different research fields rather than also including cross-disciplinary work conducted solely by an individual or work that includes non-academic stakeholders (Pfirman & Martin, 2010).
The approach used by the Programme to facilitate interaction between the diverse researchers has been reported in detail in Blöschl et al. (2012) and evaluated in Carr et al. (in review). In brief, this includes shared offices, a study programme, seminar series, research cluster meetings, joint supervision by two faculty members, symposia held twice per year, and shared fieldwork sites such as the Hydrological Open Air Laboratory (see Blöschl et al., 2016). Students write a thesis proposal in the first semester of their doctoral studies where they are encouraged to identify cross-disciplinary research questions that they will address in collaboration with other researchers in the Programme. Evaluation of the Programme's integration mechanisms has shown that joint supervision, shared study sites, sufficient time to understand and become familiar with new material, extensive discussions and respect between collaborating researchers, and recognition of mutual research and academic benefits to all collaborating researchers are critical for achieving cross-disciplinary research findings (Carr et al., in review).
The Programme aims to make a significant contribution to water sciences through the research of the students. Students complete their PhD through publications in international peer-reviewed journals (a minimum of three papers is required but they are encouraged to write four). It is envisioned that each student's research contribution places them in a strong position for developing their careers in water resource science. The Programme also aspires to develop a culture of interdisciplinarity that extends from the students to the faculty and beyond to the entire research community. Methods or models specifically developed to address a research question that spans across more than one research field.
Brings together concepts, models or theories from more than one field to develop a new theory that extends beyond one research field.

Research questions and evaluation indicators
In this paper, the Programme case study is used to explore three questions regarding the value of cross-disciplinary research and education programmes: (i) Does cross-disciplinary collaborative research lead to more innovative scientific findings than mono-disciplinary research? (ii) Does a cross-disciplinary collaborative research training equip graduates with interdisciplinary professional skills that benefit their future careers? (iii) Does cross-disciplinary collaborative research lead to research findings of greater relevance for addressing real world problems than mono-disciplinary research?
Question 1innovative scientific findings. To investigate this research question, data were collected on the authorship of publications in Institute of Scientific Information (ISI) indexed journals produced by researchers in the Programme. New knowledge stemming from research collaboration has previously been evaluated using patents, publications, proposals and grants as criteria (Heinze & Kuhlmann, 2008;Jha & Welch, 2010;Kabo et al., 2014). Prizes and awards to researchers have been used to evaluate creativity (Heinze et al., 2009) and the number of citations and journal impact factors has been used to assess the quality of collaborative work (Rigby & Edler, 2005;Porter et al., 2006;Trochim et al., 2008;van Rijnsoever & Hessels, 2011;Wagner et al., 2011). For this evaluation, publication citation rates and journal impact factors were selected as criteria. Each member of the Programme was grouped according to the research field of their primary supervisor (the 10 research fields are listed in the first paragraph of this section). The author list for each publication was analysed and papers produced by authors from only one research field or single author papers were categorised as mono-disciplinary. Papers produced by authors from two or more research fields were categorised as cross-disciplinary. Analysis of the cross-disciplinary publications was further undertaken to categorise each paper according to its type of interdisciplinarity or multidisciplinarity based on the framework developed by Huutoniemi et al. (2010) (Table 1). Using information on the number of citations for each publication and impact factor of the journal in which it is published, the innovativeness of the cross-disciplinary publications was compared to the mono-disciplinary publications.
The assumption here is that the impact factor and citation rate are suitable proxies for innovation. The advantage of using these indicators is the availability of quantitative data on which to base the evaluation, but they have recognised limitations. For example, there is much debate over whether citation rates measure scientific quality or impact, as a widely cited paper may present a method or may be cited negatively (Cole, 2000), and some articles become recognised (and therefore more-highly cited) after considerable time delay. We are examining citations from papers published within the last six years and therefore expect the citation rates measured over this time period to be a conservative estimate of the potential impact of the research on the community. Another concern is that the journal impact factor does not necessarily reflect the impact of a specific paper (Seglen, 1997;Skoie, 1999). However, we use the journal impact factors as a broad indicator of the quality of the research, presuming that higher-quality research is published in journals with a higher impact factor. We make this assumption because each researcher in the Programme is encouraged to submit their work to one of the leading journals in their field, which is typically determined by the research community using journal impact factors. Successful publication following rigorous peer review is taken to indicate that the work is of a suitably high quality for the journal.
Question 2professional skills. It has been proposed that water science programmes should endeavour to develop graduates with a 'T-shaped profile' (Uhlenbrook & de Jong, 2012;McIntosh & Taylor, 2013). Such graduates should have both a deep disciplinary understanding of a specific field (represented by the vertical line of the T) and an ability to apply that understanding to different situations and settings (the horizontal line of the T) (McIntosh & Taylor, 2013). To evaluate interdisciplinary skills, evaluators have identified learning outcomes of whether the student's work contributes to their technical area, whether it has a broad perspective, whether it shows teamwork and interdisciplinary communication skills (Borrego & Cutler, 2010), or assessed expanded expertise and vocabularies, a widened sphere of professional reading, and changing career trajectories (Klein, 2008).
For this evaluation, we chose to explore whether working in a cross-disciplinary manner during the doctoral Programme led to working in a cross-disciplinary manner in the post-doctoral career. To do this, each graduate was categorised based on their publication records as either a cross-disciplinary researcher (if they were an author of a cross-disciplinary publication) or a mono-disciplinary researcher (if they were an author of publications produced only within their own research field). Data on graduate destinations and types of graduate work were collected by asking the graduate directly and examining the author affiliations of their post-doctoral publications. This was then compared to the nature of the doctoral research to determine how cross-disciplinary research experience at the PhD level affects the type of work conducted in the post-doctoral phase. This approach makes the assumption that taking up a cross-disciplinary post-doctoral position indicates that the graduate had developed cross-disciplinary skills. An important limitation of this assumption is the typically fairly short time span of the available data, therefore inferences can only be made about careers in the few years after graduating. However, it provides a valuable starting point for examining graduate career trajectories.
Question 3real world problems. A few authors have attempted to evaluate the impact of research programmes on addressing problems of a societal nature. For example, Mitrany & Stokols (2005) evaluated whether doctoral dissertations included discussion of how the research could be translated into community problem-solving strategies, while Trochim et al. (2008) evaluated how the findings generated by a large interdisciplinary research centre on tobacco use impacted policy and practice based on questionnaires with researchers. In this evaluation, we were interested to compare the societal impact of cross-disciplinary collaborative research findings to mono-disciplinary research findings. To do this, the authors of each publication were asked whether their work had had any impact on policy or management, such as whether the work had been commissioned by government agencies, and whether the results had been used for policy making and to show evidence of this. For example, research on foam in the aquatic environment has directly fed into policy development (see Schilling et al., 2012). Additionally, societal impact was captured using media interest in research findings as a proxy. Programme researchers report media interest or media engagement in their research findings (e.g. newspaper articles, TV or radio interviews) to the Programme coordinator. The media interest research topics were grouped according to 11 broad themes that emerged when the data were analysed: flooding, drought, groundwater quality, wastewater reuse, phosphorus recovery, socioeconomics of water management, soil moisture monitoring, water and diet, microbiology and water quality, foam, and education. A list of papers relating to each media engagement was produced and each paper was then allocated to one of the 11 themes (if a paper related to more than one theme it was allocated to the theme it addressed primarily, e.g. research that examined groundwater quality under flood conditions was allocated to the category 'groundwater quality'). The assumption here is that media attention and engagement indicate that the research is of societal value and that journalists report on topics that they believe are relevant to society. While this could be debated, media exposure undoubtedly raises societal awareness to the research being undertaken and opens up the possibility that the findings can be brought into a societal problem-solving context.

Innovative science
From October 2009 (the onset of the Programme) until the end of 2015, 86 papers where a member of the Programme is first author were published in ISI indexed journals. Analysis of the author list for these papers shows that 40 (47%) have authors from more than one research field and are therefore categorised as cross-disciplinary collaborative papers. Figure 1 shows the number of cross-disciplinary collaborative publications and mono-disciplinary publications per year. It reveals that the proportion of cross-disciplinary collaborative papers has increased since 2012.
Analysis of each cross-disciplinary paper using the framework of Huutoniemi et al. (2010) revealed that 30 of the papers could be categorised as interdisciplinary (Table 2). They were categorised as: (i) empirical interdisciplinary, e.g. remotely sensed data sets were combined with interviews and policy analysis to understand land use change in Ethiopia (see Yeshaneh et al., 2013); or (ii) methodological interdisciplinary, e.g. a new model integrating hydrology, sociology and economics was developed to explore how Fig. 1. ISI journal publications where a member of the Programme is first author (n ¼ 86) according to whether more than one research field is represented by the author list (cross-disciplinary publications) or only one research field is represented by the author list (mono-disciplinary publications). societal development responds to flooding, and how flooding subsequently responds to societal development (see Viglione et al., 2014); or (iii) theoretical interdisciplinary, e.g. the development of a new numerical solution for a specific process of fluid dynamics (see Horváth et al., 2014).
Seven of the 40 cross-disciplinary papers were categorised as multidisciplinary. They were typically categorised as contextual multidisciplinary and borrowed methods or models from one setting and applied them in a new setting without modifying them, e.g. a method for evaluating resource management of metals was applied to the new setting of nitrogen in wastewater treatment (see Sobantka et al., 2014). The low number of multidisciplinary publications could suggest that Programme researchers mainly work on interdisciplinary endeavours, or alternatively perhaps suggests that multidisciplinary work is more difficult to publish in leading ISI indexed journals. Three papers were neither interdisciplinary nor multidisciplinary despite having authors from more than one research field. Discussion with the authors revealed that the co-authors were included because in one case they had been involved in developing the data sampling plan (although this was not reported in the paper) or because they had been involved in broad discussions about the work and future collaboration is planned.
It became clear during the categorisation process that some publications fitted across two categories. For example, three publications were categorised as composite multidisciplinary combined with empirical interdisciplinary. This was because analysis of the papers showed that co-authors had written or strongly controlled a section of the paper relating to their specialisations, and the overall findings of the paper were based on a collection of empirical data from various research fields. Three papers were categorised as empirical interdisciplinary combined with methodological interdisciplinary. This means that they not only used data from several different research fields, but the data were integrated into a model specifically developed to work with data from different research fields and addressed research questions that had been jointly developed by researchers from two or more different disciplines. One paper was categorised as empirical interdisciplinary combined with theoretical interdisciplinary as it described research to explore how floods change the phosphorus concentration in large rivers, and this has led to not only a new understanding of the system but also a new theory on phosphorus behaviour in the environment (see Zoboli et al., 2015). Just over half of the cross-disciplinary publications (21) involved empirical interdisciplinarity, suggesting that combining data from different research fields may be a more common strategy for doing interdisciplinary work in the water sciences. Analysis of journal impact factors shows that the journals where cross-disciplinary collaborative research is published (n ¼ 40) have a slightly higher impact factor (average of 2.36, standard deviation of 1.22) than the journals where the mono-disciplinary research is published (n ¼ 46) (average of 2.09, standard deviation of 1.04). This difference in average impact factor is statistically significant at the 1% significance level (Mann-Whitney). Interestingly, examination of the impact factors of the journals in which the 30 interdisciplinary papers were published (n ¼ 30) revealed an even-higher average impact factor of 3.04, standard deviation of 1.33. We also attempted to explore whether these interdisciplinary publications were published in journals with a higher impact factor than the seven multidisciplinary publications, but because of the relatively small data set no meaningful differences could be detected.
The number of citations for cross-disciplinary and mono-disciplinary papers published between 2012 and 2014 where a member of the Programme is first author were analysed. This time period is selected because an almost equal number of cross-and mono-disciplinary papers were published in ISI journals between 2012 and 2014 (29 cross-disciplinary and 27 mono-disciplinary articles). Analysis of the data shows that the cross-disciplinary publications have a higher mean number of citations per paper for each of the years analysed, giving them a collective mean of eight citations per paper versus five citations per paper for the mono-disciplinary publications (Figure 2). It is also important to note that the crossdisciplinary publications have an average of five authors per paper (when one paper with 22 authors is excluded from the data set), while the mono-disciplinary publications have an average of three authors per paper. This could perhaps lead to a higher incidence of self-citation, thereby skewing the citation data. However, closer inspection of the publications with more than three citations shows no correlation to being those with a higher number of authors.
This evaluation suggests that the Programme is leading to a substantial portion of research that is crossdisciplinary in nature. Comparison of the quality of the research, using the impact factors of the journals in which the work is published and the number of citations for each piece of work, suggests that the Programme's cross-disciplinary collaborative research is more innovative than the mono-disciplinary work (based on these indicators). Cross-disciplinary collaboration therefore does seem to offer new ground for pushing the frontiers of knowledge, perhaps because of its capacity to identify science questions that exist across and between disciplines (Blöschl, 2006). Furthermore, the findings indicate that interdisciplinary work forms the majority of the research published, and yields publications in higher-quality journals. This could suggest that interdisciplinary research is more desirable to the research community than multidisciplinary work.
The literature commonly claims that it is inappropriate to measure the quality of interdisciplinary work with the same yardsticks used for mono-disciplinary work because interdisciplinary work tends to perform less well (Boix-Mansilla, 2006;Klein, 2006;Rhoten & Pfirman, 2007;Levitt & Thelwall, 2008). Our findings reveal that our cross-disciplinary work performs better than our mono-disciplinary work using indicators of impact factors and citations. This suggests that, in this setting, such quality indicators are appropriate for comparing cross-disciplinary and mono-disciplinary research outcomes.
It is, however, important to discuss the use of joint authorship as an indicator of cross-disciplinary research. Several researchers have warned that names on papers may not often accurately reflect actual input to a collaborative research project (Katz & Martin, 1997;Laudel, 2002;Wagner et al., 2011). Katz & Martin (1997) emphasise how the complex nature of human interaction cannot be easily identified. Our detailed analysis of each cross-disciplinary publication suggested that in three cases a co-author from a different discipline has been included while the work does not seem to incorporate aspects of their specific research field (Table 2). However, further discussion with the authors involved revealed that these co-authors were clearly involved in planning and shaping the work and defining the future directions of the interdisciplinary research. This supports comments by Lewis et al. (2012) that in some cases a suggestion by a colleague during a casual conversation may radically change the course and outcomes of a research project. Because there is considerable opportunity for interaction between researchers in this Programme, we feel confident that the list of joint authors reflects a minimum level of collaboration and imagine that a larger number of researchers have been engaged in each piece of work than is reflected by the author list.

Professional skills
Analysis of graduate careers shows that the disciplinary and collaborative nature of the PhD is strongly deterministic of the nature of future work (Table 3). Graduates who conducted cross-disciplinary research for their PhD seem to continue to work in this way in their future careers. Prior studies have suggested that interdisciplinary research can be detrimental for academic career progression for young scientists because it takes longer for researchers to establish themselves (Rhoten & Parker, 2004;Pfirman & Martin, 2010) but also positively correlated with continuing to work in academia after the PhD and the number of publications produced (Millar, 2013). Empirical data from this study can only comment on the immediate career opportunities for recent graduates but show that 10 out of 19 graduates have taken up academic positions. Interestingly, the majority of mono-disciplinary graduates are from the field of structural mechanics. This suggests that these researchers either see less opportunity to work with researchers from other research fields within the Programme or see less benefit from cross-disciplinary research for their future careers.
There are two exceptions to the general finding that graduates continue to work in the same way after their doctorate as they work during their doctorate. In one case (Student 7, Table 3), the graduate worked in a mono-disciplinary way during his PhD, but has since branched into cross-disciplinary research, motivated by his experiences within the Programme. In another case (Student 19, Table 3), the graduate worked in a cross-disciplinary way during her PhD, but chose to take up a post-doctoral position in a mono-disciplinary research team. This is because she was given the opportunity to work as part of a leading (mono-disciplinary) microbiological research team. She may also have discovered that she did not like cross-disciplinary work. While this is purely speculative, there are many challenges to cross-disciplinary research, which include a high demand on time (to become sufficiently familiar with the state of the art in a new research field) and extensive discussion through regular meetings with collaborators to learn to understand each other, which may lead to researchers choosing a mono-disciplinary career path (see Carr et al., in review).

Real world problems
Media engagements (newspaper articles, TV interviews, internet-based articles) relate to 33 publications in ISI listed journals where a member of the Programme is first author. Of these, 19 (58%) are cross-disciplinary collaborative publications from eight of the media interest topics (Table 4). For four research topics (drought, flooding, microbiology and water quality, and water and diet), both mono-disciplinary research teams and cross-disciplinary research teams have produced publications. For these topics, research first takes place in a specialised way by mono-disciplinary teams of researchers (leading to mono-disciplinary publications), before the research teams bring in new expertise through other researchers in the Programme to extend the research (leading to cross-disciplinary publications). For example, the impact of diet on water quality has first been investigated extensively by the aquatic chemists (Zessner et al., 2010), and has later brought in process engineering to explore nutrient dynamics (Thaler et al., 2013). Early work by the microbiology professor on Escherichia coli as a groundwater contamination indicator (Farnleitner et al., 2010) was later extended to capture aspects of hydrological dynamics by bringing in a hydrologist (Vierheilig et al., 2012) and a hydrogeologist (Vierheilig et al., 2013). These findings suggest that cross-disciplinary collaborations occur in response to the researchers' quest for deeper understanding of topics that are of high societal importance. The societal importance of a topic may perhaps motivate researchers to engage with other specialisations in order to dig deeper into the topic. Funding for eight of the 38 students comes from Austrian government departments or ministries such as the Federal Agency of Water Management, the Tyrolean State Government, Vienna Water, the Federal Government of Upper Austria or the Austrian Ministry for Agriculture. These students conduct research to answer specific questions posed by decision-makers: for example, how flood risk management can be harmonised for highland and lowland rivers; how agricultural phosphorus pollution can be better managed in Austrian catchments; how water treatment and monitoring can alleviate surface water foam formation; and the capacity of standard faecal indicators such as E. coli and intestinal enterococci to detect faecal contamination. A total of 17 first-author publications in ISI listed journals are associated with the research of these eight students, of which 10 are cross-disciplinary. This suggests that crossdisciplinary research is important for addressing questions of societal relevance, but that mono-disciplinary work also plays a major role.

Conclusions and recommendations
This evaluation has used the Vienna Doctoral Programme as a case study to explore some of the outcomes that are emerging from an established interdisciplinary doctoral programme. The evaluation indicates that innovative and novel research is produced through both mono-disciplinary work and cross-disciplinary research collaborations. As the Programme has evolved, the number of cross-disciplinary collaborative publications has increased. Not only is the amount of cross-disciplinary research increasing but it seems to be of slightly higher innovative quality (particularly the work that is categorised as interdisciplinary) than that of the mono-disciplinary research, based on the number of citations and impact factors of the journals in which the papers are published. This suggests that cross-disciplinary research teams in a water resources programme can produce very high-quality scientific findings. It also shows that such indices reveal interesting data that can be used to measure the quality of interdisciplinary research findings in the field of water resources. This could be because the water research community already recognises the necessity and added value of interdisciplinary research, as shown by the citation rates. Editors and reviewers of high-quality journals perhaps also strongly support the publication of such research findings because of their novelty. These findings are particularly interesting because they are different from findings in other fields that suggest that interdisciplinary work performs less well using bibliometric indicators than mono-disciplinary work (Boix-Mansilla, 2006;Levitt & Thelwall, 2008). The evaluation suggests that graduates who have worked in a collaborative manner on their doctoral dissertations develop the ability to work collaboratively in their future roles (whether they are in academia, industry or the public sector). This is important as it indicates that interdisciplinary programmes in water sciences are educating students to meet the needs of future employers and therefore society's needs.
Answering whether cross-disciplinary collaborative work offers the best way to address present and future water resource challenges requires further research. The evaluation described in this paper suggests that cross-disciplinary work often extends disciplinary work and allows researchers to delve deeper into understanding complex topics in integrated systems such as microbial transport in groundwater or human dietary decisions on water quality. Much high-quality research into topics of high societal interest takes place in the Programme in a mono-disciplinary setting, but the evaluation suggests that bringing in additional disciplines allows researchers to explore new and different aspects of the issues.
Importantly, the Programme reported in this paper undertakes predominantly science and engineering based research although social sciences and humanities are brought in through the involvement of economists and social scientists both internal and external to the Programme. In addition, more critical mass is being built up in this area by exposing science and engineering students to social sciences' perspectives during annual programme meetings with invited external social scientists and by sending students with a social science orientation to specialised social science departments elsewhere for extended research visits and collaborations. It would be interesting to carry out a similar evaluation for a predominantly social science programme. It would also be of value to evaluate this programme in terms of its impact on policy and practice to a greater extent than that reported in this paper. Indicators such as research engagement with non-academic stakeholders, use of research findings by non-academic stakeholders and career development beyond academia would likely provide further insight into the broader impacts of interdisciplinary programmes.
Interdisciplinary research and education programmes are essential for areas of specialisation, such as water resource systems, that span many different disciplines. Our findings suggest that such programmes, if conducted effectively with sufficient support for stimulating integration between researchers from different research fields, can lead to innovative research, produce research findings of societal interest and prepare the future generation with the collaboration skills they will need to address present and future water management challenges.