Abstract

In 2018, the Korean government established a legal foundation for integrated water management. Accordingly, various measures for integrated water management were taken, and water R&D is being integrated with the Ministry of Environment. Strategic planning is needed for the efficient implementation of integrated large-scale water R&D. This study aims to analyze the efficiency of large-scale R&D programs in the field of water resources conducted by the Korean government and identify matters of priority for planning future water resources R&D programs. An empirical analysis was conducted using DEA, in particular, a non-radial SBM model was applied to consider slacks in the input and output variables. The results showed that the efficiency of collaborative R&D projects was relatively lower than that of single projects. Further, corporate research institutes, which are typically considered as the leaders of technological innovation, were found to have conducted projects less efficiently than universities or public research institutes. Based on these results, this study recommends that: (1) a system to maximize the advantages of collaborative research should be established; (2) institutional support to enhance the enterprise's innovative activities should be prepared; and (3) comprehensive, long-term planning for integrated water management should be implemented.

Highlights

  • The Korean government has established a legal foundation for IWRM.

  • SBM model of DEA was used to measure efficiency of water R&D projects in Korea.

  • The efficiency of single R&D projects is better than that of collaborative projects.

  • Corporate research institutes have less efficient R&D projects than universities or public research institutes.

  • IWRM should be incorporated with comprehensive, long-term perspective.

Introduction

In the 21st century, many countries have accepted integrated water resources management (IWRM) as the fundamental concept of water management. IWRM is defined as the ‘process which promotes the coordinated development and management of water, land, and related resources, to maximize the resultant economic and social welfare equitably without compromising the sustainability of vital ecosystems’ (GWP, 2000), and originates from the ‘Integrated Water Resources Planning and Management’ initiative proposed in Chapter 18 of Agenda 21, an action plan adopted by the Rio United Nations (UN) Environmental Development Conference in 1992.

In 2012, Korea was assessed to be insufficiently prepared for IWRM from an institutional perspective. According to a UN status report on an integrated approach to water resources management, Korea had multiple restrictions related to legal frameworks in terms of segmentation (UNEP, 2012). In fact, the water management system in Korea was found to be fully segmented according to the work of each government ministry. Policies concerning water quality, water quantity, and disasters were each planned according to the purpose of different government ministries, resulting in inefficient work and overlapping investments (BAI, 2015).

Utilizing the efforts of water experts for more than 20 years since its initial 1997 draft, the ‘Framework Act on Water Management’ (hereafter, the Act) was enacted in 2018, laying the first legal framework for integrated water management. In addition, various measures for integrated water management were taken based on this Act. A framework indicating the basic ideology and principles of water management, which the government was to consider for establishing and implementing related policies, was set up; further, a National Water Management Committee was launched, having the function of formulating and amending a master plan for national water management and mediation of water disputes. In addition, tasks on water management, which had been previously divided for several government ministries, were integrated into the Ministry of Environment (MoE).

In particular, the integration of research and development (R&D) planning is noteworthy. In the past, projects related to water quality and quantity, flood control, and conservation have been managed through several ministries, most of which were sporadic and short-term projects focused on developing minimal guidelines and providing the information needed for the work of other specific ministries (KISTEP, 2000). Since the enactment of the Act, a mid- to long-term water R&D roadmap was prepared and large-scale long-term R&D has commenced, led by the MoE.

Water is an important factor in production activities, and water R&D investments increase the level of water technology, which directly and indirectly affects the nation's economic growth (Qiao & Fang, 2020). Certain aspects should be considered carefully in the integrated planning of water R&D. Since the water resource sector features both social overhead capital and public goods, it is difficult for the sector to change or expand once developed. This sector needs large budgets and employment for development. In addition, given that most R&D spending has the nature of sunken costs (Stigliz, 1987; Åstebro, 2004; Máñez et al., 2009) and that spending is proportional to the size of R&D performance (Hall, 2002; Máñez et al., 2009), the loss incurred by changing or suspending large-scale R&D programs is greater than that of general research projects. Comprehensively, investment in the development of technologies for securing and managing water resources should align with long-term goals and perspectives, and investment efficiency should be considered.

Even when the legal basis was not yet established, water experts were eager to realize integrated water management; accordingly, the Sustainable Water Resources Research Program (SWRRP) was promoted as part of the IWRM attempt in Korea. The present study aims to analyze the efficiency of the SWRRP. The efficiency of a water R&D program can be defined as the ratio of performance to input resources for water resource technology development. R&D efficiency may be improved based on the results of analyzing the causes of inefficiency associated with the manner in which the program was conducted in the past, that is, before the planning of large-scale R&D investment (Hwang et al., 2009).

Many researchers have analyzed the efficiency of R&D programs and projects in various fields such as biotechnology, information technology, and energy, and have made efforts to improve inefficient elements. However, research on the efficiency of water R&D programs and projects is lacking, as most efficiency-related research in the water resource field has focused on water use and water utilities. Therefore, this study may be considered timely with respect to this transition from non-integrated to integrated R&D planning and implementation.

Literature review

Water R&D in Korea

The development and management of water resources in Korea was promoted in earnest along with national land development, led by the Ministry of Land, Infrastructure and Transport (MLIT, formerly the ‘Ministry of Construction’) in the 1960s. As the importance of eco-friendly river management emerged following the 1991 phenol contamination accident in the Nakdong River, certain water supply and sewage policies managed by the MLIT were transferred to the MoE. Although government ministries, including the MoE, had attempted many research investments, until then most were sporadic, short-term projects focused on developing minimal guidelines and providing work-related information needed by the ministries (KISTEP, 2000). Following the transfer of policies, a technology policy based on science and technology was established, and water-related technology development commenced.

The first of such projects was the G7 Environmental Engineering Technology Development Project (1992–2001) promoted by the MoE, an environmental R&D project aimed at improving outdated environmental technologies to match the level of more advanced countries and mainly focused on the development of post-processing technologies. After completion of this project, the Next Generation Core Environmental Technology Development Project (2001–2010) was promoted as part of proactive technology development. Subsequently, through the Next Generation Eco-innovation Technology Development Project (2010–2020), the development of aquatic ecosystem conservation and restoration technologies was promoted (Water Journal, 2018).

The MoE and various other government ministries made investments to develop water-related technologies, for example, the MLIT focused on water management research to secure water supply, and the Ministry of Agriculture, Food and Rural Affairs (formerly the ‘Ministry of Agriculture and Forestry’) invested in R&D for managing irrigation water and dams and developing agricultural groundwater. In addition, the National Emergency Management Agency promoted R&D related to natural disasters, such as floods. Further related policies and work regarding water management have been fragmented (Choi, 2013). According to Park & Lee (2019), eight government ministries promoted water-related R&D in the water management field, with approximately 80% of the investment being concentrated in water supply and sewage fields.

After the 2018 enactment of the Act, water-related R&D projects began to be integrated with the MoE, except for river maintenance-related projects, which remained as a function of the MLIT (MoE, 2018). The MoE has prepared the Water Management Technology R&D Roadmap (2030) as part of the ‘First Plan on Development of Water Management Technologies and on Promotion of the Water Industry,’ setting mid- to long-term large-scale R&D goals by 2030. Based on this roadmap, the MoE has selected 12 key technology groups in four areas, including securing the stability and efficiency of water supply, using and managing safe, clean water, securing a rehabilitated water cycle, and integrated water management based on geographic basins, and the government is planning or promoting further related projects.

Sustainable water resources research program (SWRRP) as an attempt at IWRM

There is no one-size-fits-all solution to water problems (Akhmouch & Correia, 2016). Each country has made efforts to integrate water resource management into the consideration of its geographic and economic situation. European Parliament and Council officially adopted the European Union (EU) Water Framework Directive (WFD) in 2000 to establish a framework for protecting surface water and groundwater in the EU. EU member countries are introducing this guideline to keep EU water in ‘good status’ (Jager et al., 2016). In Australia, because of severe drought in the 2000s, the need for integrated watershed management has emerged. Thus in 2007, Australia's federal government introduced the Water Act to manage entire watersheds, including local rivers, which had been previously managed by the state-level governments and formed a watershed committee to develop basin plans, which are legislative instruments for designing the environmentally sustainable diversion limits of each river (Skinner & Langford, 2013). As other examples, the US has established a watershed-level integrated water management roadmap for each state, while in Singapore, active development of water technologies and infrastructure at the national level is being carried out. Specifically, in Singapore, the Public Utilities Board manages the entire water system and R&D process (PUB, 2019). In addition, countries that lack the budget and technical knowledge are also making efforts to develop water technology in cooperation with more developed countries.

In Korea, an effort for integrated water management was undertaken differently than other countries. As mentioned earlier, before the 2018 enactment of the Act, although the legal basis was not yet established in Korea, water experts attempted to plan an R&D program based on the principles of IWRM (Kim et al., 2004). This time, the government had planned a large-scale R&D program for the development of science and technology in strategic technological fields, and the SWRRP was promoted according to the plan of the Ministry of Science and ICT (formerly the ‘Ministry of Science and Technology’).

The goal of the SWRRP was to develop technologies capable of securing three billion cubic meters of additional water resources per year to overcome the national water shortage and was implemented under a comprehensive strategy. The program was sub-divided into four major research fields covering technologies for securing surface water, groundwater, alternative water resources, and integrated management of water resources (SWRRC, 2011). Table 1 lists all research projects within each research field. The program operated for a total R&D period of 10 years, comprising three phases. The objective of the first phase (2001–2003) was to develop elementary and platform technologies. The second phase (2004–2006) objective was to utilize and systemize the elementary and platform technologies to form a basis for securing stable water resources, while the third phase (2007–2011) objective was to construct a framework establishing an integrated system and commercialization system. The program's total budget was 126 billion KRW (113 million dollars), with 7939 researchers having participated (MEST, 2011).

Table 1.

Research projects of the SWRRP.

FieldsResearch projects
Integrated management of water resources Ubiquitous River Flow Monitoring System
HyGIS
Technology for Climate Change Impact Assessment on Water Resources
Integrated Water Resources Evaluation and Planning System
Basin Water Management Technology
Rehabilitation of the Hydrologic Cycle in the Anyangcheon Watershed
HydroKorea–Ecohydrologic Cycles and Nowcasting in a Complex Terrain
Integrated Water Resources Management System 
Securing surface water Surface Water Resources Investigation System
System for Surface Water Hydrological Components
RAMS (River Analysis and Modeling System)
Sustainable Surface Water Securement System 
Securing groundwater Groundwater Analysis and Evaluation
Sustainable Korean Groundwater Development System 
Securing alternative water resources Reuse System
Rainwater Storage and Utilization
Integrated Reduction Techniques for Water Leakage
High-Efficient and Energy-Saving Desalination System
Wastewater Reuse System for Agriculture 
FieldsResearch projects
Integrated management of water resources Ubiquitous River Flow Monitoring System
HyGIS
Technology for Climate Change Impact Assessment on Water Resources
Integrated Water Resources Evaluation and Planning System
Basin Water Management Technology
Rehabilitation of the Hydrologic Cycle in the Anyangcheon Watershed
HydroKorea–Ecohydrologic Cycles and Nowcasting in a Complex Terrain
Integrated Water Resources Management System 
Securing surface water Surface Water Resources Investigation System
System for Surface Water Hydrological Components
RAMS (River Analysis and Modeling System)
Sustainable Surface Water Securement System 
Securing groundwater Groundwater Analysis and Evaluation
Sustainable Korean Groundwater Development System 
Securing alternative water resources Reuse System
Rainwater Storage and Utilization
Integrated Reduction Techniques for Water Leakage
High-Efficient and Energy-Saving Desalination System
Wastewater Reuse System for Agriculture 

Upon completion, the SWRRP had achieved its initial goal to improve the level of water-related technology in Korea by more than two times compared to 2001, and produced substantial quantitative outcomes including: 1048 academic papers, 2128 conferences, 311 patents (222 application and 89 registration), 131 developments and registrations of software, and 55 contracts for the transfer of technology (SWRRC, 2011).

Previous studies on evaluating R&D efficiency

Many studies have been conducted on the efficiency of R&D programs and projects. Lee et al. (2009) used the Banker-Charnes-Cooper (BCC) model to measure the R&D efficiency of 548 projects of six R&D programs supported by the National Research Foundation of Korea. Park & Choi (2013) used the BCC model to analyze the efficiency of 2046 projects in biotechnology and nanotechnology from 2005 to 2010. Hsu & Hsueh (2009) applied a data envelopment analysis (DEA) based three-stage approach to evaluate the efficiency of 189 research projects within the Industrial Technology Development Program established by the Taiwanese government. Hung & Shiu (2014) used the same method to analyze the R&D efficiency of 39 academic projects within the same program. Chun et al. (2016) used the Charnes-Cooper-Rhodes (CCR) model to measure the R&D efficiency of 92 projects of a hydrogen energy R&D program supported by the Korean government from 2003 to 2012. By comparing the results of the simultaneous application of BCC and slack-based measure (SBM), the author proved that more stringent efficiency values are obtained once accounting for the slack in input and output factors.

A review of previous studies led to the following conclusions. First, the evaluation of R&D efficiency has been conducted in various academic fields such as biotechnology, information technology, telecommunication, industrial technology, and energy. However, in the field of water resources, research on the efficiency of R&D investment has not been conducted, and analyses have been limited focusing only on the use of water resources (Wang et al., 2001; Deng et al., 2016; Ren et al., 2016) or the operation and management of water utilities (Hernández-Sancho & Sala-Garrido, 2009; Romano & Guerrini, 2011; Mahmoudi et al., 2012; Ananda, 2014; Marques et al., 2014; Lorenzo-Toja et al., 2015; See, 2015). Second, most of the previous studies were analyzed using basic models. Although analysis using a basic model can derive sufficiently meaningful results, a more rigorous analysis is necessary considering that most government-led R&D investments are conducted through the operation of taxes. Finally, previous studies were also limited to only analyzing the phenomenon and thus were also limited in terms of deriving implications or suggestions to improve the efficiency of R&D investment.

Materials and methods

Slack-based measure (SBM) model of data envelopment analysis (DEA)

Data envelopment analysis is a nonparametric method that measures relative R&D efficiency within a group of homogeneous decision-making units (DMUs) with multiple inputs and outputs. Here, the DMUs can be institutions that conduct research projects. DEA is widely used to analyze the efficiency of R&D investment because it has several advantages. First, DEA does not require prior information of a functional form between R&D inputs and outputs, nor does it require any assumptions about the functional relationship or parameters between inputs and outputs. Second, multiple inputs and outputs can be considered, and objectivity and generality can be maintained when choosing the weights of inputs and outputs. Finally, DEA is highly utilized for efficiency analysis of the public sector because it concretely suggests the degree of inefficiency by measuring relative efficiency (Cooper et al., 2007). However, there is a limitation that this method is sensitive to outliers and has difficulty distinguishing between technical efficiency and statistical errors (De Witte & Marques, 2010).

A basic DEA model, such as BCC or CCR, is a radial model based on the proportional reduction of inputs or the enlargement of outputs when measuring efficiency; thus, it is limited in that it cannot reflect slack in the input or output (Tone, 2001; Morita et al., 2005; Avkiran et al., 2008). Radial models have limitations that do not reflect the slacks, which results in leftover portions of inefficiency in the input and output. In this study, input slack refers to excessive input of research budget or researcher, and output slack refers to a lack of academic papers or patents. To overcome these limitations of radial models, non-radial models have been developed (Figure 1). Assuming two inputs (X1, X2) and one output (Y), Figure 1 shows four DMUs that consist of an efficiency frontier, SS’. Points A and B, which do not reach the efficiency frontier SS’, represent inefficient production units, and points C and D are the efficient production units. In radial models, the efficiency value of point A would be calculated using the origin (O) as OA′/OA based on A′, where the radiation connecting the origin with A and the efficient frontier SS’ meet. However, in this case, A reaches the incomplete frontier without touching the efficient frontier connecting C and D, which shows strong efficient units. Thus, it is difficult to say A′ is strictly efficient because A′ on the weak efficient frontier includes slack in the second input as much as A′C compared to C. That is, in the radial model, even if DMUs exist on the weak efficient frontier and contain some slack, they can be judged as efficient.

Fig. 1.

Schematic of efficiency measurement. Source: Jacobs et al. (2006).

Fig. 1.

Schematic of efficiency measurement. Source: Jacobs et al. (2006).

On the other hand, in non-radial models, the efficiency value of A is measured based on C, which is more efficient because it is on the same frontier as A′ but uses fewer inputs.

The SBM model is a representative non-radial model. This scalar measure deals with input excesses and output shortfalls in DMUs (Tone, 2001). In the SBM model, the value of efficiency is invariant even if the unit of measurement of each input and output changes (unit invariant), and if any slight change in input or output affects the efficiency value (monotone decreasing).

The SBM model can be derived by inserting slack variables into the basic DEA model, and is defined as shown in Equation (1) (Cooper et al., 2007):
formula
(1)
s. t.
formula
formula
formula
where is efficiency value; is the input vector of the oth DMU; is the output vector of the oth DMU; is the weight of the jth DMU; is input slack (input excesses); and is output slack (output shortfalls). N is the number of total DMUs used for the analysis. If there is no input slack in the numerator of the objective function, the value of the numerator becomes 1. At this time, if any of the output variables have slacks, then the value of the numerator becomes smaller than 1. Conversely, if there is no output slack in the nominator, the value of the nominator becomes 1. At this time, if any input variable has slack, then the value of nominator becomes larger than 1. Therefore, in the SBM model, DMUs become efficient only if both input and output are efficient.
Discretionary inputs and outputs are considered important in the SBM model. The model is divided into input orientation and output orientation according to whether the input or output is fixed in order to find the inefficient parts for the remaining variables. While the input-oriented model focuses on minimizing input to satisfy given output levels, the output-oriented model focuses on maximizing output using given input. Equation (2) is the input-oriented SBM model, which can be derived by neglecting the nominator of the objective function (Cooper et al., 2007):
formula
(2)
s. t.
formula
formula
formula

Furthermore, the model separates constant returns to scale and variable returns to scale according to the characteristics of production structure of input and output. The input-oriented SBM model with assumption of variable returns to scale is derived by adding to Equation (2).

Assumptions

DEA requires sufficient consideration of the purpose of the analysis, the nature of the production relationship between inputs and outputs, and the establishment of controllable factors. In particular, the purpose of the research affects the choice of orientation of the model (Cook et al., 2014). Therefore, it is necessary to redefine the goal of this study before selecting the model to apply to this study. The purpose of this study is to measure the efficiency of a water R&D program supported by the Korean government. The goal of this study was to obtain more detail not only to evaluate how efficiently the performance of the SWRRP has been conducted within given inputs, but also to prepare a basis for decision makers to use the analysis results for further R&D design.

R&D activities can result in economies of scale or diseconomies of scale (Scherer, 1983; Shefer & Frenkel, 2005). Although the production factors of R&D generally affect performance, it is difficult to say that they produce as much as the amount of inputs. When all inputs are multiplied k times, they could contribute to the calculation k times, more than this, or less than this. In other words, it is unreasonable to apply the assumption that the returns to scale are fixed.

Meanwhile, under the assumption of specific returns to scale, both the input-oriented model and output-oriented model produce the same efficient frontier, however, they establish different reference sets for inefficient units (Cook et al., 2014). Therefore, whether the purpose of the analysis is more focused on the adjustment of input or output factors can affect model choice. At the time of planning a new program, it is possible to adjust the goal of the expected performance, but it is more practical and easier to control the input of budget and labor. Thus, this study assumed the output level of the SWRRP was proper under the given budget. In addition, modifiability on the input side was examined, expecting a similar level of output. Overall, this study selected an input-oriented SBM model under the assumption of variable returns to scale to measure the relative efficiency of the projects.

Data sources

Two inputs and two outputs were selected for the empirical analysis. First, the number of researchers and the amount of R&D budget were selected as input variables. Almost all previous studies on R&D efficiency have set these two variables as inputs (Hsu & Hsueh, 2009; Lee et al., 2009; Hung & Shiu, 2014). They are representative input variables of labor and capital in traditional industrial economies. Many researchers often spend more time on a research plan than on what can be deduced from the assumed participation rate; therefore, the number of researchers who participated in the project was used as a form of head count rather than the full-time equivalent to measure the efficiency in a given research environment. R&D budget can be divided into government-financed budget and enterprise-financed budget. Enterprise-financed budget is supported not only by a form of cash but also by a form of machine or equipment rental. In this study, total R&D budget included government-financed budget and enterprise-financed budget considering that some projects require companies' participation and some do not. Operating expense and labor for the Sustainable Water Resources Research Center (SWRRC), which manages the SWRRP, were excluded from the analysis, as the purpose of this study is to measure the efficiency of research projects.

A number of articles and patents were selected as output variables (Hsu & Hsueh, 2009; Lee et al., 2009; Park & Choi, 2013; Hung & Shiu, 2014; Chun et al., 2016). Articles and patent use are indicators of scientific and technological performance, respectively (Hullmann & Meyer, 2003). Scientific articles are a means for researchers to show the current state of their research and to disseminate the results; articles are also an indicator of academic excellence and productivity (Thomas et al., 2011). Patents are the most representative indicator of the performance of technological innovation activities. Furthermore, patents are used as a means of assessing the performance of key actors in the scientific and technological research fields or sub-fields (Verbeek et al., 2002). Articles and patents are highly correlated with the amount of funds and R&D personnel, which are the main R&D input factors (Comanor & Scherer, 1969; Narin & Hamilton, 1996; Kamath et al., 2017). This study included all articles published in both Korean and international journals. Further, this study included patent application, which shows the degree of activity related to R&D projects, and included all patents applied in both Korean and international patent offices.

Generally, after a research budget is allocated and research has started, there is a time lag until the achievement of the result of the article or patent (Griliches, 1979). There is no generally accepted time lag for all R&D outputs; previous studies have assumed a time lag of 1–5 years (Goto & Suzuki, 1989). Assuming no time lag for input and output data can further enhance the value of the analysis results (Chun et al., 2016). Like the data used in Chun et al. (2016), the analytical data of this study were obtained from the SWRRC, which directly collects and manages the inputs and outputs of each project. Thus, this study did not reflect a time lag.

Figure 2 shows the basic statistics of the inputs and outputs of the SWRRP in Korea. An average of 14.4 researchers per project participated and an average of 232.5 million KRW per project was invested, while an average of 2.13 papers and 0.48 patents per project were generated (see Annex 1 for yearly statistics).

Fig. 2.

Basic statistics of SWRRP in Korea.

Fig. 2.

Basic statistics of SWRRP in Korea.

Results and discussion

This study analyzed the R&D efficiency of 585 projects of the SWRRP that were conducted from 2001 to 2010. Some projects were conducted independently, and some collaboratively. Even if several institutes conduct research collaboratively, each manages its own budget, researcher, and performance. Therefore, this study set each project as the single unit for analysis, and measured and calculated the efficiency for each unit. MaxDEA software (Cheng, 2014) was used for the analysis.

General review of R&D efficiency

Table 2 shows the value of R&D efficiency from 2001 to 2011. Over time, the mean value of efficiency increased compared to the initial period of the program. This is a reasonable result of the accumulation of labor and capital invested in technological innovation. The mean value of efficiency for the 10-year period is 0.5215 (s.d. 0.09). It is difficult to assess the performance of water R&D programs relatively through these measures because there is no similar research in the field of water resources. However, as summarized in Table 2, considering that the proportion of projects that were efficiently carried out ranges from 7.58% to a maximum of 33.33%, it can be concluded that the SWRRP was carried out relatively inefficiently.

Table 2.

Value of R&D efficiency for each year.

2001200220032004200520062007200820092010
Number of projects 63 93 89 66 67 66 36 35 35 35 
Number of efficient DMUs 11 11 10 12 
% of efficient DMUs 12.70 11.83 12.36 7.58 8.96 15.15 33.33 25.71 22.86 22.86 
Score (ave.) 0.4326 0.4494 0.4483 0.4546 0.4469 0.5013 0.7002 0.6375 0.5590 0.5849 
2001200220032004200520062007200820092010
Number of projects 63 93 89 66 67 66 36 35 35 35 
Number of efficient DMUs 11 11 10 12 
% of efficient DMUs 12.70 11.83 12.36 7.58 8.96 15.15 33.33 25.71 22.86 22.86 
Score (ave.) 0.4326 0.4494 0.4483 0.4546 0.4469 0.5013 0.7002 0.6375 0.5590 0.5849 

Table 3 shows the degree of inefficiency of inputs and outputs of inefficient DMUs for each year. The analysis shows it was necessary to reduce the number of researchers and the size of the budget for the entire research period. The excess of inputs was even more striking in the early years (2001 and 2002) of the program. The result confirms the importance of preparing countermeasures to create a stable research environment in the initial phase of the R&D program. Even though this study used the input-oriented model to minimize input size under the assumption that the performance of the SWRRP is correct, it was found that further improvement is still needed in terms of the output variables.

Table 3.

Degree of inefficiency of inputs and outputs for each year (%).

2001200220032004200520062007200820092010
Researchers –64.12 –63.96 –51.22 –60.71 –62.01 –54.85 –39.76 –43.86 –40.94 –44.35 
Budget –74.35 –75.40 –60.50 –72.08 –68.66 –66.76 –41.16 –48.84 –54.40 –54.08 
Articles 0.00 8.25 18.80 2.63 124.58 0.00 0.00 4.18 5.34 17.19 
Patents 15.14 4.60 0.00 0.00 0.00 0.00 0.00 44.49 15.43 43.68 
2001200220032004200520062007200820092010
Researchers –64.12 –63.96 –51.22 –60.71 –62.01 –54.85 –39.76 –43.86 –40.94 –44.35 
Budget –74.35 –75.40 –60.50 –72.08 –68.66 –66.76 –41.16 –48.84 –54.40 –54.08 
Articles 0.00 8.25 18.80 2.63 124.58 0.00 0.00 4.18 5.34 17.19 
Patents 15.14 4.60 0.00 0.00 0.00 0.00 0.00 44.49 15.43 43.68 

Note: Positive values indicate input excess, negative values indicate output shortage.

R&D efficiency according to environmental factors

R&D efficiency of the projects supported by the SWRRP by type of research organization was measured. Organizations were divided into industries, universities, and research institutes, and Figure 3 shows the relative efficiency.

Fig. 3.

Efficiency score for industries, universities, and research institutes.

Fig. 3.

Efficiency score for industries, universities, and research institutes.

R&D efficiency for each organization improved from the beginning of the project. The results for each organization show the efficiency of industries is higher than that of research institutes at the beginning, but it does not improve significantly over time. The efficiency of universities is higher than that of other research organizations at the beginning, and steadily improves. In the case of research institutes, R&D efficiency is relatively low at first, but improves significantly over time.

In general, the theoretical distribution of efficiency in DEA is unknown. Thus, simple comparisons based on the mean value of efficiency cannot have statistical validity (Lee et al., 2009). Therefore, after measuring the efficiency of all projects, the Kruskal–Wallis H test (Ostertagová et al., 2014), which assesses the significant differences in the mean value obtained in the efficiency analysis among the various groups in which the sample size is different, was conducted to compare the performance of each research organization (Table 4). In addition, the Mann–Whitney U test (Nachar, 2008) was applied for the paired research organizations – industries and universities, industries and research institutes, and universities and research institutes – for the years 2001, 2002, 2003, 2004, and 2006, which showed different performance in the Kruskal–Wallis H test. As a result of the tests, the R&D efficiency of universities was relatively high in these years. However, there were no differences of performance among the organizations during other research periods as the efficiency of industries and research institutes improved.

Table 4.

Results of Kruskal–Wallis H test.

Industry
University
Research Institute
p-value
NaAvg. rankbNAvg. rankNAvg. rank
2001 19 33.32 30 35.17 13 20.39 0.0415c 
2002 23 42.46 51 53.59 19 34.82 0.0228c 
2003 22 43.18 47 50.78 20 33.43 0.0393c 
2004 17 24.03 34 39.59 15 30.43 0.0189c 
2005 16 26.38 35 38.87 16 30.97 0.0811 
2006 16 23.06 34 37.47 16 35.50 0.0416c 
2007 10 17.35 15 18.79 10 17.50 0.9287 
2008 10 16.90 15 19.00 15.67 0.7112 
2009 11 17.41 14 16.07 19.83 0.676 
2010 11 12.86 14 20.29 18.83 0.1619 
Industry
University
Research Institute
p-value
NaAvg. rankbNAvg. rankNAvg. rank
2001 19 33.32 30 35.17 13 20.39 0.0415c 
2002 23 42.46 51 53.59 19 34.82 0.0228c 
2003 22 43.18 47 50.78 20 33.43 0.0393c 
2004 17 24.03 34 39.59 15 30.43 0.0189c 
2005 16 26.38 35 38.87 16 30.97 0.0811 
2006 16 23.06 34 37.47 16 35.50 0.0416c 
2007 10 17.35 15 18.79 10 17.50 0.9287 
2008 10 16.90 15 19.00 15.67 0.7112 
2009 11 17.41 14 16.07 19.83 0.676 
2010 11 12.86 14 20.29 18.83 0.1619 

aAverage ranking of each group's R&D efficiency.

bNumber of projects conducted by industry, university, and research institute.

cSignificance level, p < 0.05.

In general, it is known that universities perform basic research, while industries and research institutes show their competence in applied and development research. The results of this analysis show that the R&D efficiency of universities is relatively high, because they are faithful to their role in the field of basic research. Meanwhile, the reason the R&D efficiency of industries does not improve is linked to the following phenomenon in Korea. According to Moon (2006), the efficiency of Korea's R&D in the industrial sector declined in the 2000s, although the phenomenon differs by size of industry that conducts R&D activities.

Next, R&D efficiency according to performance type – single and collaborative research projects – was measured. Figure 4 shows the ratio of research projects according to performance type. About three-quarters of the projects were conducted in collaboration with other research institutes, and only about one-quarter were conducted independently.

Fig. 4.

Ratio of research projects according to performance type (10-year mean value).

Fig. 4.

Ratio of research projects according to performance type (10-year mean value).

Figure 5 shows the relative efficiency of single and collaborative research projects. With regard to the annual trend in efficiency, both single and collaborative research projects show similar efficiencies at the beginning of the project. R&D efficiency of single research projects improved over time, but that of collaborative research projects did not change significantly. Overall, the relative efficiency of single research projects is higher.

Fig. 5.

Efficiency score for single–collaborative research conduct.

Fig. 5.

Efficiency score for single–collaborative research conduct.

The Mann–Whitney U test was performed to compare the results of the two comparison groups, and there was a difference in performance between groups over the full research period. In most research periods, it has not been confirmed which performance type is more likely to occur. However, in some research years, that is, 2002, 2007, and 2009, single research projects achieved better performance.

Two conflicting opinions exist on the effects of technological innovation and R&D efficiency on collaborative R&D activities. First, R&D costs can be reduced and research quality and efficiency can be improved by sharing the complementary resources held by industries, universities, and research institutes (Kogut & Singh, 1988; Das & Teng, 2000; Wuchty et al., 2007). In addition, there is a view that collaborative research generates economies of scale and scope (Kogut & Singh, 1988; Hagedoorn, 1993). By contrast, the second opinion argues that negative effects do exist, such as the interaction cost between collaborative research institutes or the free-riding of researchers, due to the uncertainty of R&D (Geisler & Rubenstein, 1989). In light of these opposing views, it can be judged that the negative effect caused by transaction costs has occurred more than the synergistic effects caused by collaborative research in water R&D projects.

Recommendations for sustainable long-term large-scale R&D management related to IWRM

Based on the results, the following implications are presented in terms of priority considerations for successfully promoting and managing long-term large-scale water R&D programs while maintaining continuity of the policies. First, a system should be established to maximize the advantages of collaborative research. The proportion of research projects jointly conducted in Korea's national R&D programs not only accounts for more than a certain percentage each year, but also continues to increase. Considering this reality of R&D investment, it is necessary to establish a system that maximizes the synergistic effect of collaborative research, i.e. a system that supports the active exchange of technologies and manpower between cooperative research institutes. This is because differences in perception among stakeholders leads to differences in position, which can incur conflict-induced costs. In addition, an appropriate performance distribution system should also be established. This could serve as a possible incentive for cooperative research institutes to create joint performance and increase synergistic of collaborative research.

Second, institutional support should be provided to enhance enterprises' innovative activities. Collaborative research in Korea's national R&D programs has mainly shown types of cooperation with enterprises such as industry–academia, industry–industry, industry–academia–research institutes, and industry–research institutes. Improving the R&D efficiency of a corporate research institute leads to an improvement in the efficiency of collaborative research as well as the efficiency of national R&D investment. Therefore, there exists an urgent need to develop measures that increase the R&D efficiency of corporate research institutes, which has been decreasing since the 2000s. To do this, it is necessary to make strategic investments through preliminary research on the technologies required in the industry; further, a plan that can provide monetary and non-monetary incentives to inspire researchers to increase research productivity should be discussed. In addition, because the outputs of public R&D projects are not well-linked to commercialization, institutional support should be solidified to encourage commercialization.

Finally, comprehensive, long-term planning should be implemented for integrated water management. Since long-term water planning enables strategic and organized access to the resource, it is desirable to both approach a nation's water management from a fundamentally long-term perspective, and also take short-term countermeasures for temporary issues. If policies tend to change often and inconsistently, social costs could arise out of policy transitions. It is therefore necessary to establish policy goals from a long-term perspective and to plan long-term R&D programs for contributing to the achievement of these goals.

Conclusions

In Korea, the legal foundation for integrated water management has only recently been established, and related measures such as the coordination of water management work between government ministries and establishment of the National Water Management Committee have been implemented. In addition, the MoE is integrating water-related R&D programs, which have typically been sporadically conducted by various government ministries, and preparing for long-term large-scale R&D.

Government-led large-scale R&D requires a large budget and pool of human resources, thus its ripple effect can be great when it is successfully promoted. However, if the program is not successfully implemented, significant sunken costs can be incurred. Therefore, to ensure successful R&D investment, a thorough strategy must be established in the pre-planning stage.

It can be effective to develop a strategy based on past experiences in order to reduce the potential for failure and ensure successful planning and implementation. Therefore, this study analyzed the performance and efficiency of large-scale R&D program, which was implemented as part of the IWRM, in terms of technology management, and has presented implications through the analysis.

Results confirmed that the R&D efficiency of collaborative research projects was relatively lower than that of single research projects. In addition, corporate research institutes conducted projects less efficiently than universities and public research institutes. Based on these results, this study recommends that: (1) a system should be established to maximize the advantages of collaborative research; (2) institutional support should be prepared to enhance the enterprise's innovative activities; and (3) comprehensive, long-term planning should be implemented for integrated water management. The results of this study can be expected to serve as reference data for developing strategies for sustainable R&D investment in the water resources field.

However, this study has some limitations, as follows. First, the objective of the SWRRP was to secure additional water resources through developing technologies. Therefore, if the amount of water resources secured through each project were used as output, then the evaluation would reflect specific characteristics of the program. Nonetheless, this issue was not analyzed, because data at the analytical unit level were not available. Second, the qualitative indicators of the R&D output (i.e. citations, impact factor, etc.) could not be used in the analysis due to language bias and data accessibility problems. In addition, this research has compared the R&D efficiency by type of research organization and performance. However, various studies take a different perspective and consider the role of internal and external environmental factors, which is deemed as important to measure the efficiency of the public sector R&D projects (Nagesh & Thomas, 2015). Future studies will need to analyze the literature on these numerous environmental factors before measuring R&D efficiency. In addition, in order to better consider the multiple factors influencing the R&D planning phase, a nonparametric-stochastic estimation method (i.e. the order-m methodology) could be employed. In the presence of numerous exogenous factors, this would allow more robust results to be obtained. Overall, the long-term accumulation of analytical results will ultimately constitute an important reference to design policies utilizing the allocated financial budget effectively.

Acknowledgments

We wish to thank the anonymous reviewers for very useful comments on this paper. This research was supported by a grant-in-aid awarded by the Basic Research Project of the Korea Institute of Geoscience and Mineral Resources (KIGAM), funded by the Ministry of Science and ICT.

Data availability statement

Data cannot be made publicly available; readers should contact the corresponding author for details.

References

Akhmouch
A.
&
Correia
F. N.
, (
2016
).
The 12 OECD principles on water governance – when science meets policy
.
Utilities Policy
43
,
14
20
.
doi:10.1016/j.jup.2016.06.004
.
Ananda
J.
, (
2014
).
Evaluating the performance of urban water utilities: robust nonparametric approach
.
Journal of Water Resources Planning and Management
140
,
04014021
.
doi:10.1061/(ASCE)WR.1943-5452.0000387
.
Åstebro
T.
, (
2004
).
Sunk costs and the depth and probability of technology adoption
.
The Journal of Industrial Economics
52
,
381
399
.
doi:10.1111/j.0022-1821.2004.00231.x
.
Avkiran
N. K.
,
Tone
K.
&
Tsutsui
M.
, (
2008
).
Bridging radial and non-radial measures of efficiency in DEA
.
Annals of Operations Research
164
,
127
138
.
doi:10.1007/s10479-008-0356-8
.
BAI (The Board of Audit and Inspection)
(
2015
).
The Audit Report 2014
.
The Board of Audit and Inspection of Korea
,
Seoul (in Korean)
.
Cheng
G.
, (
2014
).
Data Envelopment Analysis: Methods and MaxDEA Software
.
Intellectual Property Publishing House Co. Ltd.
,
Beijing
.
Choi
E.
, (
2013
).
Water Environment Policy and R&D Trends
.
Special Issue 108, Global Environmental Policy, Korea Environmental Industry & Technology Institute
,
Seoul. (in Korean)
.
Chun
D.
,
Hong
S.
,
Chung
Y.
,
Woo
C.
&
Seo
H.
, (
2016
).
Influencing factors on hydrogen energy R&D projects: an ex-post performance evaluation
.
Renewable and Sustainable Energy Reviews
53
,
1252
1258
.
doi:10.1016/j.rser.2015.09.074
.
Comanor
W. S.
&
Scherer
F. M.
, (
1969
).
Patent statistics as a measure of technical change
.
Journal of Political Economy
77
,
392
398
.
doi:10.1086/259522
.
Cook
W. D.
,
Tone
K.
&
Zhu
J.
, (
2014
).
Data envelopment analysis: prior to choosing a model
.
Omega
44
,
1
4
.
doi:10.1016/j.omega.2013.09.004
.
Cooper
W. W.
,
Seiford
L. M.
&
Tone
K.
, (
2007
).
Data Envelopment Analysis – A Comprehensive Text with Models, Applications, References and DEA-Solver Software
, 2nd edn.
Springer
,
New York
.
Das
T. K.
&
Teng
B. S.
, (
2000
).
A resource-based theory of strategic alliances
.
Journal of Management
26
,
31
61
.
doi:10.1016/S0149-2063(99)00037-9
.
Deng
G.
,
Li
L.
&
Song
Y.
, (
2016
).
Provincial water use efficiency measurement and factor analysis in China: based on SBM-DEA model
.
Ecological Indicators
69
,
12
18
.
doi:10.1016/j.ecolind.2016.03.052
.
De Witte
K.
&
Marques
R. C.
, (
2010
).
Influential observations in frontier model, a robust non-oriented approach to the water sector
.
Annals of Operations Research
181
,
377
392
.
doi:10.1007/s10479-010-0754-6
.
Geisler
E.
,
Rubenstein
A. H.
, (
1989
).
University–industry relations: a review of major issues
. In
Cooperative Research Development: The Industry–University–Government Relationship
.
Link
A. N.
&
Tassey
G.
(eds).
Kluwer Academic Press
,
Boston
, pp.
43
62
.
Goto
A.
&
Suzuki
K.
, (
1989
).
R&D capital, rate of return on R&D investment and spillover of R&D in Japanese manufacturing industries
.
Review of Economics and Statistics
71
,
555
564
.
doi:10.2307/1928096
.
Griliches
Z.
, (
1979
).
Issues in assessing the contribution of research and development to productivity growth
.
Bell Journal of Economics
10
,
92
116
.
doi:10.2307/3003321
.
GWP (Global Water Partnership)
(
2000
).
Integrated Water Resources Management
.
Background Paper No. 4
.
Global Water Partnership
,
Stockholm, Sweden
.
Hagedoorn
J.
, (
1993
).
Understanding the rationale of strategic technology partnering: interorganizational modes of cooperation and sectoral differences
.
Strategic Management Journal
14
,
371
385
.
doi:10.1002/smj.4250140505
.
Hall
B.
, (
2002
).
The financing of research and development
.
Oxford Review of Economic Policy
18
(
1
),
35
51
.
doi:10.1093/oxrep/18.1.35
.
Hernández-Sancho
F.
&
Sala-Garrido
R.
, (
2009
).
Technical efficiency and cost analysis in wastewater treatment processes: a DEA approach
.
Desalination
249
,
230
234
.
doi:10.1016/j.desal.2009.01.029
.
Hsu
F. M.
&
Hsueh
C. C.
, (
2009
).
Measuring relative efficiency of government-sponsored R&D projects: a three-stage approach
.
Evaluation and Program Planning
32
,
178
186
.
doi:10.1016/j.evalprogplan.2008.10.005
.
Hullmann
A.
&
Meyer
M.
, (
2003
).
Publications and patents in nanotechnology
.
Scientometrics
58
,
507
527
.
doi:10.1023/B:SCIE.0000006877.45467.a7
.
Hwang
S. W.
,
Ahn
D. H.
,
Choi
S. H.
,
Kwon
S. H.
,
Chun
D. P.
,
Kim
A. R.
&
Park
J. H.
, (
2009
).
Efficiency of National R&D Investment
.
Report 2009-24
.
Science & Technology Policy Institute
,
Seoul (in Korean)
.
Jacobs
R.
,
Smith
P. C.
&
Street
A.
, (
2006
).
Measuring Efficiency in Health Care: Analytic Techniques and Health Policy
.
Cambridge University Press
,
UK
.
Jager
N. W.
,
Challies
E.
,
Kochskämper
E.
,
Newig
J.
,
Benson
D.
,
Blackstock
K.
,
Collins
K.
,
Ernst
A.
,
Evers
M.
,
Feichtinger
J.
,
Fritsch
O.
,
Gooch
G.
,
Grund
W.
,
Hedelin
B.
,
Hernández-Mora
N.
,
Hüesker
F.
,
Huitema
D.
,
Irvine
K.
,
Klinke
A.
,
Lange
L.
,
Loupsans
D.
,
Lubell
M.
,
Maganda
C.
,
Matczak
P.
,
Parés
M.
,
Saarikoski
H.
,
Slavíková
L.
,
van der Arend
S.
&
von Korff
Y.
, (
2016
).
Transforming european water governance? participation and river basin management under the EU water framework directive in 13 member states
.
Water
8
,
156
.
doi:10.3390/w8040156
.
Kamath
R. S.
,
Kamat
R. K.
&
Pujar
S. M.
, (
2017
).
Correlating R&D expenditure and scholarly publication output using k-means clustering
.
International Journal of Information Technology, Modeling and Computing
5
,
1
7
.
doi:10.5121/ijitmc.2017.5101
.
Kim
S.
,
Shim
K. C.
,
Kim
H. T.
,
Kim
H. S.
&
Lee
S. H.
, (
2004
).
The development of technology for integrated water resources management in Republic of Korea
. In
World Water and Environmental Resources Congress 2004
,
June 27–July 1, 2004
,
Utah, US
.
KISTEP (Korea Institute of S&T Evaluation and Planning)
(
2000
).
A Study on Planning the 21st Century Frontier R&D Program (II)
.
GOVP1200102035
.
Korea Institute of S&T Evaluation and Planning
,
Seoul, Korea (in Korean)
.
Kogut
B.
&
Singh
H.
, (
1988
).
The effect of national culture on the choice of entry mode
.
Journal of International Business Studies
19
,
411
432
.
doi:10.1057/palgrave.jibs.8490394
.
Lee
H.
,
Park
Y.
&
Choi
H.
, (
2009
).
Comparative evaluation of performance of national R&D programs with heterogeneous objectives: a DEA approach
.
European Journal of Operational Research
196
,
847
855
.
doi:10.1016/j.ejor.2008.06.016
.
Lorenzo-Toja
Y.
,
Vázquez-Rowe
I.
,
Chenel
S.
,
Marín-Navarro
D.
,
Moreira
M. T.
&
Feijoo
G.
, (
2015
).
Eco-efficiency analysis of Spanish WWTPs using the LCA + DEA method
.
Water Research
68
,
651
666
.
doi:10.1016/j.watres.2014.10.040
.
Mahmoudi
M. J.
,
Fathi
B.
,
Sajadifar
H.
&
Shahsavari
A.
, (
2012
).
Measuring efficiency of water and wastewater company: a DEA approach
.
Research Journal of Applied Science, Engineering and Technology
4
,
1642
1648
.
Máñez
J. A.
,
Rochina-Barrachina
M. E.
,
Sanchis
A.
&
Sanchis
J. A.
, (
2009
).
The role of sunk costs in the decision to invest in R&D
.
The Journal of Industrial Economics
57
(
4
),
712
735
.
doi:10.1111/j.1467-6451.2009.00398.x
.
Marques
R. C.
,
Berg
S.
&
Yane
S.
, (
2014
).
Nonparametric benchmarking of Japanese water utilities: institutional and environmental factors affecting efficiency
.
Journal of Water Resources Planning and Management
140
,
562
571
.
doi:10.1061/(ASCE)WR.1943-5452.0000366
.
MEST (Ministry of Education and Science Technology)
(
2011
).
21st Century Frontier R&D: A Decade Challenge for the Future
.
Centre for R&D Performance Support
,
Seoul, Korea (in Korean)
.
MoE (Ministry of Environment)
(
2018
).
Revision of Government Organization Act for Unification of Water Management & Start of Integrated Water Management
.
Moon
H. S.
, (
2006
).
An analysis of the polarization of Korean firms' r&D investment and efficiency
.
Journal of Korean Public Policy
8
,
73
92
.
(in Korean)
.
Morita
H.
,
Hirokawa
K.
&
Zhu
J.
, (
2005
).
A slack-based measure of efficiency in context-dependent data envelopment analysis
.
Omega
33
,
357
362
.
doi:10.1016/j.omega.2004.06.001
.
Nachar
(
2008
).
The Mann-Whitney U: a test for assessing whether two independent samples come from the same distribution
.
Tutorials in Quantitative Methods for Psychology
4
,
13
20
.
Nagesh
D. S.
&
Thomas
S.
, (
2015
).
Success factors of public funded R&D projects
.
Current Science
108
,
357
363
.
Narin
F.
&
Hamilton
K. S.
, (
1996
).
Bibliometric performance measures
.
Scientometrics
36
,
293
310
.
doi:10.1007/BF02129596
.
Ostertagová
E.
,
Ostertag
O.
&
Kováč
J.
, (
2014
).
Methodology and application of the kruskal-wallis test
.
Applied Mechanics and Materials
611
,
115
120
.
doi:10.4028/www.scientific.net/AMM.611.115
.
Park
H.
&
Choi
H. G.
, (
2013
).
Measurement of R&D efficiency in NT and BT fields using DEA: a case in Korea
.
International Journal of Engineering and Technology
2
,
165
174
.
Park
N. E.
&
Lee
S. H.
, (
2019
).
Water Management Technology. 2019-7
.
Korea Institute of S&T Evaluation and Planning
,
Seoul. Korea (in Korean)
.
PUB (Public Utilities Board)
(
2019
).
Innovation in Water Singapore
.
Ren
C.
,
Guo
P.
,
Yang
G.
&
Li
R.
, (
2016
).
Spatial and temporal analyses of water resources use efficiency based on data envelope analysis and Malmquist index: Case study in Gansu province, China
.
Journal of Water Resources Planning and Management
142
,
04016066
.
doi:10.1061/(ASCE)WR.1943-5452.0000719
.
Romano
G.
&
Guerrini
A.
, (
2011
).
Measuring and comparing the efficiency of water utility companies: a data envelopment analysis approach
.
Utilities Policy
19
,
202
209
.
doi:10.1016/j.jup.2011.05.005
.
Scherer
F. M.
, (
1983
).
The propensity to patent
.
International Journal of Industrial Organization
1
,
107
128
.
doi:10.1016/0167-7187(83)90026-7
.
Shefer
D.
&
Frenkel
A.
, (
2005
).
R&D, firm size and innovation: an empirical analysis
.
Technovation
25
,
25
32
.
doi:10.1016/S0166-4972(03)00152-4
.
Skinner
D.
&
Langford
J.
, (
2013
).
Legislating for sustainable basin management: the story of Australia's Water Act (2007)
.
Water Policy
15
,
871
894
.
doi:10.2166/wp.2013.017
.
Stigliz
J. E.
, (
1987
).
Technological change, sunk costs, and competition
.
Brookings Papers on Economic Activity
3
,
883
937
.
Princeton University, Princeton
.
SWRRC (Sustainable Water Resources Research Center)
(
2011
).
Sustainable Water Resources Research Program Annual Report (2001–2011)
.
Gyeonggi-do
,
Korea (in Korean)
.
Thomas
V. J.
,
Sharma
S.
&
Jain
S. K.
, (
2011
).
Using patents and publications to assess R&D efficiency in the states of the USA
.
World Patent Information
33
,
4
10
.
doi:10.1016/j.wpi.2010.01.005
.
Tone
K.
, (
2001
).
A slacks-based measure of efficiency in data envelopment analysis
.
European Journal of Operational Research
130
,
498
509
.
doi:10.1016/S0377-2217(99)00407-5
.
UNEP (United Nations Environment Programme)
(
2012
).
Status Report on the Application of Integrated Approaches to Water Resources Management
.
DEP/1532/NA
,
Nairobi, Kenya
.
Verbeek
A.
,
Debackere
K.
,
Luwel
M.
&
Zimmermann
E.
, (
2002
).
Measuring progress and evolution in science and technology – I: the multiple uses of bibliometric indicators
.
International Journal of Management Reviews
4
,
179
211
.
doi:10.1111/1468-2370.00083
.
Wang
H.
,
Zhang
L.
,
Dawes
W. R.
&
Liu
C.
, (
2001
).
Improving water use efficiency of irrigated crops in the North China Plain – measurements and modelling
.
Agricultural Water Management
48
,
151
167
.
doi:10.1016/S0378-3774(00)00118-9
.
Water Journal
(
2018
).
Environmental Research and Development in Accordance with KEITI's Integrated Water Management
.
Available at: http://www.waterjournal.co.kr/news/articleView.html?idxno=43356 (accessed 20 July 2020) (in Korean)
.
Wuchty
S.
,
Jones
B. F.
&
Uzzi
B.
, (
2007
).
The increasing dominance of teams in production of knowledge
.
Science
316
,
1036
1039
.
doi:10.1126/science.1136099
.
This is an Open Access article distributed under the terms of the Creative Commons Attribution Licence (CC BY-NC-ND 4.0), which permits copying and redistribution for non-commercial purposes with no derivatives, provided the original work is properly cited (http://creativecommons.org/licenses/by-nc-nd/4.0/).