Water resource R&D efficiency in Korea – toward sustainable integrated water resources management

In the 21st century, many countries have accepted integrated water resources management (IWRM) as the fundamental concept of water management. IWRM is defined as the ‘process which promotes the coordinated development and management of water, land, and related resources, to maximize the resultant economic and social welfare equitably without compromising the sustainability of vital ecosystems’ (GWP, 2000), and originates from the ‘Integrated Water Resources Planning and Management’ initiative proposed in Chapter 18 of Agenda 21, an action plan adopted by the Rio United Nations (UN) Environmental Development Conference in 1992.

In 2012, Korea was assessed to be insufficiently prepared for IWRM from an institutional perspective. According to a UN status report on an integrated approach to water resources management, Korea had multiple restrictions related to legal frameworks in terms of segmentation (UNEP, 2012). In fact, the water management system in Korea was found to be fully segmented according to the work of each government ministry. Policies concerning water quality, water quantity, and disasters were each planned according to the purpose of different government ministries, resulting in inefficient work and overlapping investments (BAI, 2015).

Utilizing the efforts of water experts for more than 20 years since its initial 1997 draft, the ‘Framework Act on Water Management’ (hereafter, the Act) was enacted in 2018, laying the first legal framework for integrated water management. In addition, various measures for integrated water management were taken based on this Act. A framework indicating the basic ideology and principles of water management, which the government was to consider for establishing and implementing related policies, was set up; further, a National Water Management Committee was launched, having the function of formulating and amending a master plan for national water management and mediation of water disputes. In addition, tasks on water management, which had been previously divided for several government ministries, were integrated into the Ministry of Environment (MoE).

In particular, the integration of research and development (R&D) planning is noteworthy. In the past, projects related to water quality and quantity, flood control, and conservation have been managed through several ministries, most of which were sporadic and short-term projects focused on developing minimal guidelines and providing the information needed for the work of other specific ministries (KISTEP, 2000). Since the enactment of the Act, a mid- to long-term water R&D roadmap was prepared and large-scale long-term R&D has commenced, led by the MoE.

Water is an important factor in production activities, and water R&D investments increase the level of water technology, which directly and indirectly affects the nation's economic growth (Qiao & Fang, 2020). Certain aspects should be considered carefully in the integrated planning of water R&D. Since the water resource sector features both social overhead capital and public goods, it is difficult for the sector to change or expand once developed. This sector needs large budgets and employment for development. In addition, given that most R&D spending has the nature of sunken costs (Stigliz, 1987; Åstebro, 2004; Máñez et al., 2009) and that spending is proportional to the size of R&D performance (Hall, 2002; Máñez et al., 2009), the loss incurred by changing or suspending large-scale R&D programs is greater than that of general research projects. Comprehensively, investment in the development of technologies for securing and managing water resources should align with long-term goals and perspectives, and investment efficiency should be considered.

Even when the legal basis was not yet established, water experts were eager to realize integrated water management; accordingly, the Sustainable Water Resources Research Program (SWRRP) was promoted as part of the IWRM attempt in Korea. The present study aims to analyze the efficiency of the SWRRP. The efficiency of a water R&D program can be defined as the ratio of performance to input resources for water resource technology development. R&D efficiency may be improved based on the results of analyzing the causes of inefficiency associated with the manner in which the program was conducted in the past, that is, before the planning of large-scale R&D investment (Hwang et al., 2009).

Many researchers have analyzed the efficiency of R&D programs and projects in various fields such as biotechnology, information technology, and energy, and have made efforts to improve inefficient elements. However, research on the efficiency of water R&D programs and projects is lacking, as most efficiency-related research in the water resource field has focused on water use and water utilities. Therefore, this study may be considered timely with respect to this transition from non-integrated to integrated R&D planning and implementation.

The development and management of water resources in Korea was promoted in earnest along with national land development, led by the Ministry of Land, Infrastructure and Transport (MLIT, formerly the ‘Ministry of Construction’) in the 1960s. As the importance of eco-friendly river management emerged following the 1991 phenol contamination accident in the Nakdong River, certain water supply and sewage policies managed by the MLIT were transferred to the MoE. Although government ministries, including the MoE, had attempted many research investments, until then most were sporadic, short-term projects focused on developing minimal guidelines and providing work-related information needed by the ministries (KISTEP, 2000). Following the transfer of policies, a technology policy based on science and technology was established, and water-related technology development commenced.

The first of such projects was the G7 Environmental Engineering Technology Development Project (1992–2001) promoted by the MoE, an environmental R&D project aimed at improving outdated environmental technologies to match the level of more advanced countries and mainly focused on the development of post-processing technologies. After completion of this project, the Next Generation Core Environmental Technology Development Project (2001–2010) was promoted as part of proactive technology development. Subsequently, through the Next Generation Eco-innovation Technology Development Project (2010–2020), the development of aquatic ecosystem conservation and restoration technologies was promoted (Water Journal, 2018).

The MoE and various other government ministries made investments to develop water-related technologies, for example, the MLIT focused on water management research to secure water supply, and the Ministry of Agriculture, Food and Rural Affairs (formerly the ‘Ministry of Agriculture and Forestry’) invested in R&D for managing irrigation water and dams and developing agricultural groundwater. In addition, the National Emergency Management Agency promoted R&D related to natural disasters, such as floods. Further related policies and work regarding water management have been fragmented (Choi, 2013). According to Park & Lee (2019), eight government ministries promoted water-related R&D in the water management field, with approximately 80% of the investment being concentrated in water supply and sewage fields.

After the 2018 enactment of the Act, water-related R&D projects began to be integrated with the MoE, except for river maintenance-related projects, which remained as a function of the MLIT (MoE, 2018). The MoE has prepared the Water Management Technology R&D Roadmap (2030) as part of the ‘First Plan on Development of Water Management Technologies and on Promotion of the Water Industry,’ setting mid- to long-term large-scale R&D goals by 2030. Based on this roadmap, the MoE has selected 12 key technology groups in four areas, including securing the stability and efficiency of water supply, using and managing safe, clean water, securing a rehabilitated water cycle, and integrated water management based on geographic basins, and the government is planning or promoting further related projects.

There is no one-size-fits-all solution to water problems (Akhmouch & Correia, 2016). Each country has made efforts to integrate water resource management into the consideration of its geographic and economic situation. European Parliament and Council officially adopted the European Union (EU) Water Framework Directive (WFD) in 2000 to establish a framework for protecting surface water and groundwater in the EU. EU member countries are introducing this guideline to keep EU water in ‘good status’ (Jager et al., 2016). In Australia, because of severe drought in the 2000s, the need for integrated watershed management has emerged. Thus in 2007, Australia's federal government introduced the Water Act to manage entire watersheds, including local rivers, which had been previously managed by the state-level governments and formed a watershed committee to develop basin plans, which are legislative instruments for designing the environmentally sustainable diversion limits of each river (Skinner & Langford, 2013). As other examples, the US has established a watershed-level integrated water management roadmap for each state, while in Singapore, active development of water technologies and infrastructure at the national level is being carried out. Specifically, in Singapore, the Public Utilities Board manages the entire water system and R&D process (PUB, 2019). In addition, countries that lack the budget and technical knowledge are also making efforts to develop water technology in cooperation with more developed countries.

In Korea, an effort for integrated water management was undertaken differently than other countries. As mentioned earlier, before the 2018 enactment of the Act, although the legal basis was not yet established in Korea, water experts attempted to plan an R&D program based on the principles of IWRM (Kim et al., 2004). This time, the government had planned a large-scale R&D program for the development of science and technology in strategic technological fields, and the SWRRP was promoted according to the plan of the Ministry of Science and ICT (formerly the ‘Ministry of Science and Technology’).

The goal of the SWRRP was to develop technologies capable of securing three billion cubic meters of additional water resources per year to overcome the national water shortage and was implemented under a comprehensive strategy. The program was sub-divided into four major research fields covering technologies for securing surface water, groundwater, alternative water resources, and integrated management of water resources (SWRRC, 2011). Table 1 lists all research projects within each research field. The program operated for a total R&D period of 10 years, comprising three phases. The objective of the first phase (2001–2003) was to develop elementary and platform technologies. The second phase (2004–2006) objective was to utilize and systemize the elementary and platform technologies to form a basis for securing stable water resources, while the third phase (2007–2011) objective was to construct a framework establishing an integrated system and commercialization system. The program's total budget was 126 billion KRW (113 million dollars), with 7939 researchers having participated (MEST, 2011).

Upon completion, the SWRRP had achieved its initial goal to improve the level of water-related technology in Korea by more than two times compared to 2001, and produced substantial quantitative outcomes including: 1048 academic papers, 2128 conferences, 311 patents (222 application and 89 registration), 131 developments and registrations of software, and 55 contracts for the transfer of technology (SWRRC, 2011).

Many studies have been conducted on the efficiency of R&D programs and projects. Lee et al. (2009) used the Banker-Charnes-Cooper (BCC) model to measure the R&D efficiency of 548 projects of six R&D programs supported by the National Research Foundation of Korea. Park & Choi (2013) used the BCC model to analyze the efficiency of 2046 projects in biotechnology and nanotechnology from 2005 to 2010. Hsu & Hsueh (2009) applied a data envelopment analysis (DEA) based three-stage approach to evaluate the efficiency of 189 research projects within the Industrial Technology Development Program established by the Taiwanese government. Hung & Shiu (2014) used the same method to analyze the R&D efficiency of 39 academic projects within the same program. Chun et al. (2016) used the Charnes-Cooper-Rhodes (CCR) model to measure the R&D efficiency of 92 projects of a hydrogen energy R&D program supported by the Korean government from 2003 to 2012. By comparing the results of the simultaneous application of BCC and slack-based measure (SBM), the author proved that more stringent efficiency values are obtained once accounting for the slack in input and output factors.

A review of previous studies led to the following conclusions. First, the evaluation of R&D efficiency has been conducted in various academic fields such as biotechnology, information technology, telecommunication, industrial technology, and energy. However, in the field of water resources, research on the efficiency of R&D investment has not been conducted, and analyses have been limited focusing only on the use of water resources (Wang et al., 2001; Deng et al., 2016; Ren et al., 2016) or the operation and management of water utilities (Hernández-Sancho & Sala-Garrido, 2009; Romano & Guerrini, 2011; Mahmoudi et al., 2012; Ananda, 2014; Marques et al., 2014; Lorenzo-Toja et al., 2015; See, 2015). Second, most of the previous studies were analyzed using basic models. Although analysis using a basic model can derive sufficiently meaningful results, a more rigorous analysis is necessary considering that most government-led R&D investments are conducted through the operation of taxes. Finally, previous studies were also limited to only analyzing the phenomenon and thus were also limited in terms of deriving implications or suggestions to improve the efficiency of R&D investment.

Data envelopment analysis is a nonparametric method that measures relative R&D efficiency within a group of homogeneous decision-making units (DMUs) with multiple inputs and outputs. Here, the DMUs can be institutions that conduct research projects. DEA is widely used to analyze the efficiency of R&D investment because it has several advantages. First, DEA does not require prior information of a functional form between R&D inputs and outputs, nor does it require any assumptions about the functional relationship or parameters between inputs and outputs. Second, multiple inputs and outputs can be considered, and objectivity and generality can be maintained when choosing the weights of inputs and outputs. Finally, DEA is highly utilized for efficiency analysis of the public sector because it concretely suggests the degree of inefficiency by measuring relative efficiency (Cooper et al., 2007). However, there is a limitation that this method is sensitive to outliers and has difficulty distinguishing between technical efficiency and statistical errors (De Witte & Marques, 2010).

A basic DEA model, such as BCC or CCR, is a radial model based on the proportional reduction of inputs or the enlargement of outputs when measuring efficiency; thus, it is limited in that it cannot reflect slack in the input or output (Tone, 2001; Morita et al., 2005; Avkiran et al., 2008). Radial models have limitations that do not reflect the slacks, which results in leftover portions of inefficiency in the input and output. In this study, input slack refers to excessive input of research budget or researcher, and output slack refers to a lack of academic papers or patents. To overcome these limitations of radial models, non-radial models have been developed (Figure 1). Assuming two inputs (X₁, X₂) and one output (Y), Figure 1 shows four DMUs that consist of an efficiency frontier, SS’. Points A and B, which do not reach the efficiency frontier SS’, represent inefficient production units, and points C and D are the efficient production units. In radial models, the efficiency value of point A would be calculated using the origin (O) as OA′/OA based on A′, where the radiation connecting the origin with A and the efficient frontier SS’ meet. However, in this case, A reaches the incomplete frontier without touching the efficient frontier connecting C and D, which shows strong efficient units. Thus, it is difficult to say A′ is strictly efficient because A′ on the weak efficient frontier includes slack in the second input as much as A′C compared to C. That is, in the radial model, even if DMUs exist on the weak efficient frontier and contain some slack, they can be judged as efficient.

On the other hand, in non-radial models, the efficiency value of A is measured based on C, which is more efficient because it is on the same frontier as A′ but uses fewer inputs.

The SBM model is a representative non-radial model. This scalar measure deals with input excesses and output shortfalls in DMUs (Tone, 2001). In the SBM model, the value of efficiency is invariant even if the unit of measurement of each input and output changes (unit invariant), and if any slight change in input or output affects the efficiency value (monotone decreasing).

Furthermore, the model separates constant returns to scale and variable returns to scale according to the characteristics of production structure of input and output. The input-oriented SBM model with assumption of variable returns to scale is derived by adding to Equation (2).

DEA requires sufficient consideration of the purpose of the analysis, the nature of the production relationship between inputs and outputs, and the establishment of controllable factors. In particular, the purpose of the research affects the choice of orientation of the model (Cook et al., 2014). Therefore, it is necessary to redefine the goal of this study before selecting the model to apply to this study. The purpose of this study is to measure the efficiency of a water R&D program supported by the Korean government. The goal of this study was to obtain more detail not only to evaluate how efficiently the performance of the SWRRP has been conducted within given inputs, but also to prepare a basis for decision makers to use the analysis results for further R&D design.

R&D activities can result in economies of scale or diseconomies of scale (Scherer, 1983; Shefer & Frenkel, 2005). Although the production factors of R&D generally affect performance, it is difficult to say that they produce as much as the amount of inputs. When all inputs are multiplied k times, they could contribute to the calculation k times, more than this, or less than this. In other words, it is unreasonable to apply the assumption that the returns to scale are fixed.

Meanwhile, under the assumption of specific returns to scale, both the input-oriented model and output-oriented model produce the same efficient frontier, however, they establish different reference sets for inefficient units (Cook et al., 2014). Therefore, whether the purpose of the analysis is more focused on the adjustment of input or output factors can affect model choice. At the time of planning a new program, it is possible to adjust the goal of the expected performance, but it is more practical and easier to control the input of budget and labor. Thus, this study assumed the output level of the SWRRP was proper under the given budget. In addition, modifiability on the input side was examined, expecting a similar level of output. Overall, this study selected an input-oriented SBM model under the assumption of variable returns to scale to measure the relative efficiency of the projects.

Two inputs and two outputs were selected for the empirical analysis. First, the number of researchers and the amount of R&D budget were selected as input variables. Almost all previous studies on R&D efficiency have set these two variables as inputs (Hsu & Hsueh, 2009; Lee et al., 2009; Hung & Shiu, 2014). They are representative input variables of labor and capital in traditional industrial economies. Many researchers often spend more time on a research plan than on what can be deduced from the assumed participation rate; therefore, the number of researchers who participated in the project was used as a form of head count rather than the full-time equivalent to measure the efficiency in a given research environment. R&D budget can be divided into government-financed budget and enterprise-financed budget. Enterprise-financed budget is supported not only by a form of cash but also by a form of machine or equipment rental. In this study, total R&D budget included government-financed budget and enterprise-financed budget considering that some projects require companies' participation and some do not. Operating expense and labor for the Sustainable Water Resources Research Center (SWRRC), which manages the SWRRP, were excluded from the analysis, as the purpose of this study is to measure the efficiency of research projects.

A number of articles and patents were selected as output variables (Hsu & Hsueh, 2009; Lee et al., 2009; Park & Choi, 2013; Hung & Shiu, 2014; Chun et al., 2016). Articles and patent use are indicators of scientific and technological performance, respectively (Hullmann & Meyer, 2003). Scientific articles are a means for researchers to show the current state of their research and to disseminate the results; articles are also an indicator of academic excellence and productivity (Thomas et al., 2011). Patents are the most representative indicator of the performance of technological innovation activities. Furthermore, patents are used as a means of assessing the performance of key actors in the scientific and technological research fields or sub-fields (Verbeek et al., 2002). Articles and patents are highly correlated with the amount of funds and R&D personnel, which are the main R&D input factors (Comanor & Scherer, 1969; Narin & Hamilton, 1996; Kamath et al., 2017). This study included all articles published in both Korean and international journals. Further, this study included patent application, which shows the degree of activity related to R&D projects, and included all patents applied in both Korean and international patent offices.

Generally, after a research budget is allocated and research has started, there is a time lag until the achievement of the result of the article or patent (Griliches, 1979). There is no generally accepted time lag for all R&D outputs; previous studies have assumed a time lag of 1–5 years (Goto & Suzuki, 1989). Assuming no time lag for input and output data can further enhance the value of the analysis results (Chun et al., 2016). Like the data used in Chun et al. (2016), the analytical data of this study were obtained from the SWRRC, which directly collects and manages the inputs and outputs of each project. Thus, this study did not reflect a time lag.

Figure 2 shows the basic statistics of the inputs and outputs of the SWRRP in Korea. An average of 14.4 researchers per project participated and an average of 232.5 million KRW per project was invested, while an average of 2.13 papers and 0.48 patents per project were generated (see Annex 1 for yearly statistics).

This study analyzed the R&D efficiency of 585 projects of the SWRRP that were conducted from 2001 to 2010. Some projects were conducted independently, and some collaboratively. Even if several institutes conduct research collaboratively, each manages its own budget, researcher, and performance. Therefore, this study set each project as the single unit for analysis, and measured and calculated the efficiency for each unit. MaxDEA software (Cheng, 2014) was used for the analysis.

Table 2 shows the value of R&D efficiency from 2001 to 2011. Over time, the mean value of efficiency increased compared to the initial period of the program. This is a reasonable result of the accumulation of labor and capital invested in technological innovation. The mean value of efficiency for the 10-year period is 0.5215 (s.d. 0.09). It is difficult to assess the performance of water R&D programs relatively through these measures because there is no similar research in the field of water resources. However, as summarized in Table 2, considering that the proportion of projects that were efficiently carried out ranges from 7.58% to a maximum of 33.33%, it can be concluded that the SWRRP was carried out relatively inefficiently.

Table 3 shows the degree of inefficiency of inputs and outputs of inefficient DMUs for each year. The analysis shows it was necessary to reduce the number of researchers and the size of the budget for the entire research period. The excess of inputs was even more striking in the early years (2001 and 2002) of the program. The result confirms the importance of preparing countermeasures to create a stable research environment in the initial phase of the R&D program. Even though this study used the input-oriented model to minimize input size under the assumption that the performance of the SWRRP is correct, it was found that further improvement is still needed in terms of the output variables.

R&D efficiency of the projects supported by the SWRRP by type of research organization was measured. Organizations were divided into industries, universities, and research institutes, and Figure 3 shows the relative efficiency.

R&D efficiency for each organization improved from the beginning of the project. The results for each organization show the efficiency of industries is higher than that of research institutes at the beginning, but it does not improve significantly over time. The efficiency of universities is higher than that of other research organizations at the beginning, and steadily improves. In the case of research institutes, R&D efficiency is relatively low at first, but improves significantly over time.

In general, the theoretical distribution of efficiency in DEA is unknown. Thus, simple comparisons based on the mean value of efficiency cannot have statistical validity (Lee et al., 2009). Therefore, after measuring the efficiency of all projects, the Kruskal–Wallis H test (Ostertagová et al., 2014), which assesses the significant differences in the mean value obtained in the efficiency analysis among the various groups in which the sample size is different, was conducted to compare the performance of each research organization (Table 4). In addition, the Mann–Whitney U test (Nachar, 2008) was applied for the paired research organizations – industries and universities, industries and research institutes, and universities and research institutes – for the years 2001, 2002, 2003, 2004, and 2006, which showed different performance in the Kruskal–Wallis H test. As a result of the tests, the R&D efficiency of universities was relatively high in these years. However, there were no differences of performance among the organizations during other research periods as the efficiency of industries and research institutes improved.

In general, it is known that universities perform basic research, while industries and research institutes show their competence in applied and development research. The results of this analysis show that the R&D efficiency of universities is relatively high, because they are faithful to their role in the field of basic research. Meanwhile, the reason the R&D efficiency of industries does not improve is linked to the following phenomenon in Korea. According to Moon (2006), the efficiency of Korea's R&D in the industrial sector declined in the 2000s, although the phenomenon differs by size of industry that conducts R&D activities.

Next, R&D efficiency according to performance type – single and collaborative research projects – was measured. Figure 4 shows the ratio of research projects according to performance type. About three-quarters of the projects were conducted in collaboration with other research institutes, and only about one-quarter were conducted independently.

Figure 5 shows the relative efficiency of single and collaborative research projects. With regard to the annual trend in efficiency, both single and collaborative research projects show similar efficiencies at the beginning of the project. R&D efficiency of single research projects improved over time, but that of collaborative research projects did not change significantly. Overall, the relative efficiency of single research projects is higher.

The Mann–Whitney U test was performed to compare the results of the two comparison groups, and there was a difference in performance between groups over the full research period. In most research periods, it has not been confirmed which performance type is more likely to occur. However, in some research years, that is, 2002, 2007, and 2009, single research projects achieved better performance.

Two conflicting opinions exist on the effects of technological innovation and R&D efficiency on collaborative R&D activities. First, R&D costs can be reduced and research quality and efficiency can be improved by sharing the complementary resources held by industries, universities, and research institutes (Kogut & Singh, 1988; Das & Teng, 2000; Wuchty et al., 2007). In addition, there is a view that collaborative research generates economies of scale and scope (Kogut & Singh, 1988; Hagedoorn, 1993). By contrast, the second opinion argues that negative effects do exist, such as the interaction cost between collaborative research institutes or the free-riding of researchers, due to the uncertainty of R&D (Geisler & Rubenstein, 1989). In light of these opposing views, it can be judged that the negative effect caused by transaction costs has occurred more than the synergistic effects caused by collaborative research in water R&D projects.

Based on the results, the following implications are presented in terms of priority considerations for successfully promoting and managing long-term large-scale water R&D programs while maintaining continuity of the policies. First, a system should be established to maximize the advantages of collaborative research. The proportion of research projects jointly conducted in Korea's national R&D programs not only accounts for more than a certain percentage each year, but also continues to increase. Considering this reality of R&D investment, it is necessary to establish a system that maximizes the synergistic effect of collaborative research, i.e. a system that supports the active exchange of technologies and manpower between cooperative research institutes. This is because differences in perception among stakeholders leads to differences in position, which can incur conflict-induced costs. In addition, an appropriate performance distribution system should also be established. This could serve as a possible incentive for cooperative research institutes to create joint performance and increase synergistic of collaborative research.

Second, institutional support should be provided to enhance enterprises' innovative activities. Collaborative research in Korea's national R&D programs has mainly shown types of cooperation with enterprises such as industry–academia, industry–industry, industry–academia–research institutes, and industry–research institutes. Improving the R&D efficiency of a corporate research institute leads to an improvement in the efficiency of collaborative research as well as the efficiency of national R&D investment. Therefore, there exists an urgent need to develop measures that increase the R&D efficiency of corporate research institutes, which has been decreasing since the 2000s. To do this, it is necessary to make strategic investments through preliminary research on the technologies required in the industry; further, a plan that can provide monetary and non-monetary incentives to inspire researchers to increase research productivity should be discussed. In addition, because the outputs of public R&D projects are not well-linked to commercialization, institutional support should be solidified to encourage commercialization.

Finally, comprehensive, long-term planning should be implemented for integrated water management. Since long-term water planning enables strategic and organized access to the resource, it is desirable to both approach a nation's water management from a fundamentally long-term perspective, and also take short-term countermeasures for temporary issues. If policies tend to change often and inconsistently, social costs could arise out of policy transitions. It is therefore necessary to establish policy goals from a long-term perspective and to plan long-term R&D programs for contributing to the achievement of these goals.

In Korea, the legal foundation for integrated water management has only recently been established, and related measures such as the coordination of water management work between government ministries and establishment of the National Water Management Committee have been implemented. In addition, the MoE is integrating water-related R&D programs, which have typically been sporadically conducted by various government ministries, and preparing for long-term large-scale R&D.

Government-led large-scale R&D requires a large budget and pool of human resources, thus its ripple effect can be great when it is successfully promoted. However, if the program is not successfully implemented, significant sunken costs can be incurred. Therefore, to ensure successful R&D investment, a thorough strategy must be established in the pre-planning stage.

It can be effective to develop a strategy based on past experiences in order to reduce the potential for failure and ensure successful planning and implementation. Therefore, this study analyzed the performance and efficiency of large-scale R&D program, which was implemented as part of the IWRM, in terms of technology management, and has presented implications through the analysis.

Results confirmed that the R&D efficiency of collaborative research projects was relatively lower than that of single research projects. In addition, corporate research institutes conducted projects less efficiently than universities and public research institutes. Based on these results, this study recommends that: (1) a system should be established to maximize the advantages of collaborative research; (2) institutional support should be prepared to enhance the enterprise's innovative activities; and (3) comprehensive, long-term planning should be implemented for integrated water management. The results of this study can be expected to serve as reference data for developing strategies for sustainable R&D investment in the water resources field.

However, this study has some limitations, as follows. First, the objective of the SWRRP was to secure additional water resources through developing technologies. Therefore, if the amount of water resources secured through each project were used as output, then the evaluation would reflect specific characteristics of the program. Nonetheless, this issue was not analyzed, because data at the analytical unit level were not available. Second, the qualitative indicators of the R&D output (i.e. citations, impact factor, etc.) could not be used in the analysis due to language bias and data accessibility problems. In addition, this research has compared the R&D efficiency by type of research organization and performance. However, various studies take a different perspective and consider the role of internal and external environmental factors, which is deemed as important to measure the efficiency of the public sector R&D projects (Nagesh & Thomas, 2015). Future studies will need to analyze the literature on these numerous environmental factors before measuring R&D efficiency. In addition, in order to better consider the multiple factors influencing the R&D planning phase, a nonparametric-stochastic estimation method (i.e. the order-m methodology) could be employed. In the presence of numerous exogenous factors, this would allow more robust results to be obtained. Overall, the long-term accumulation of analytical results will ultimately constitute an important reference to design policies utilizing the allocated financial budget effectively.

We wish to thank the anonymous reviewers for very useful comments on this paper. This research was supported by a grant-in-aid awarded by the Basic Research Project of the Korea Institute of Geoscience and Mineral Resources (KIGAM), funded by the Ministry of Science and ICT.

Data cannot be made publicly available; readers should contact the corresponding author for details.