Performance monitoring and evaluation of water environment treatment PPP projects with multi-source heterogeneous information

Effective measurement and monitoring of the quality of services from the private sector, in the terms of performance in water environment treatment public-private partnership projects (WETPs-PPP), is one of the keys to the success of these projects. Based on the deficiencies of performance monitoring and evaluation theory and the specific characteristics of WETPs-PPP, this study developed a performance monitoring and evaluation model for WETPs-PPP. First, through a literature analysis, a performance monitoring and evaluation index system of WETPs-PPP was constructed from eight dimensions including river embankment, bridges, river water, environment, garden facilities, garden plants, special purpose vehicle, hydraulic structure, and public satisfaction. Second, by combining the adaptive weighted fusion algorithm and Intuitionistic Fuzzy (IF) average operators, multi-source, multi-dimensional, and multi-spatio-temporal data were aggregated. Third, to improve the robustness of the performance monitoring results, the MULTIMOORA evaluation method with a defuzzification procedure under the IF environment was developed to conduct performance monitoring and evaluation for WETPs-PPP. Finally, a water environment treatment and ecological restoration project was used as an example to illustrate the feasibility and effectiveness of the proposed theoretical model.

1 Introduction

The service quality of projects from the private sector in the water environment treatment public-private partnership projects (WETPs-PPP) can be represented by the effect of the operation and maintenance period. The improvement of the service quality is a basic goal for gaining a competitive advantage in the industry and improving the reputation and profitability of the private sector (Abi et al., 2019). Currently, the service quality of social and public infrastructure is a major concern of the national government and the public. In the wave of PPP projects, competition for service quality in the water environment treatment (WET) industry is becoming fierce (An et al., 2018; Li et al., 2020). The effect of operation and maintenance in WETPs-PPP and stakeholder satisfaction for service quality are two important factors that reflect the private sector’s service quality. The government has deemed it necessary to measure and monitor the quality of services provided by the private sector in terms of performance.

Through extensive prior analysis of the literature, it was determined that studies on performance evaluation are mainly concentrated on two aspects as follows. 1) The first is the construction of a performance monitoring indicator system. Many scholars have studied performance monitoring indicators in construction projects (Sanvido et al., 1992; Chua et al., 1999; Oyedele, 2013; Rashvand & Zaimi, 2014; Zou et al., 2014; Abi et al., 2019). A reasonable and complete performance monitoring indicators system in public-private partnership (PPP) projects can improve the work motivation of the private sector. Meanwhile, it can also improve the validity of results of performance monitoring and support the sustainable development of PPP projects. 2) The second aspect is the selection of an appropriate performance monitoring method, which is one of the key steps in providing a good result from performance monitoring. Some of the existing models use a subjective evaluation based on the knowledge and experience of construction experts (Lam et al., 2007; Yuan et al., 2012; Liu et al., 2015), and other models, such as Cha and Kim (2011), used simple linear summation of selected KPI values, along with the relative importance of each KPI, to obtain the performance score of the project.

The existing research provides a powerful theoretical basis for the performance monitoring and evaluation of WETPs-PPP. Essentially, the performance monitoring and evaluation of WETPs-PPP is a complex problem that involves several performance evaluation factors. Specifically, the performance monitoring of WETPs-PPP involves multi-source indicators, such as river water, landscaping, and river safety, which corresponds to different data types, for example, crisp, interval number, and linguistic value. Additionally, the performance monitoring and evaluation results of WETPs-PPP are affected by the fuzziness of humans and the complexity of projects, which need a more intuitionistic and effective performance monitoring and evaluation approach to make performance monitoring and evaluation results more robust. From the perspective of robustness research, performance evaluation methods with two different evaluation methods are better than evaluation methods with a single evaluation, and the evaluation methods with three different ways of evaluation are better than evaluation methods with two (Brauers, 2012). In combining three different ways of evaluation, the MULTIMOORA (Multi-objective Optimization by Ratio Analysis plus the Full Multiplicative Form) method (Brauers and Zavadskas, 2010), which is based on the MOORA method (Multi-objective optimization on the basis of the ratio analysis method) (Brauers and Zavadskas, 2006) with the characteristics that include being simpler and requiring a small amount of calculation, stable results, and less solution time. The comparison and selection alternatives from each of the multiple angles (Liu et al., 2014) is proposed by adding to the full multiplicative form and applied to various complex multi-criteria decision-making problems (Hafezalkotob and Hafezalkotob, 2017; Zavadskas et al., 2017; Zhao et al., 2017; Ijadi Maghsoodi et al., 2018) since it is simpler and more robust (Brauers and Zavadskas, 2012).

Upon analyzing the aforementioned observations, it is clear that some research gaps should be enriched in the WETPs-PPP. 1) There is no particular performance monitoring index system for WETPs-PPP since more research is concentrated on transportation PPP projects. 2) The existing aggregated method of performance monitoring information cannot cover the fusion of data possessing the characteristics of multi-source, multi-dimensional, and multi-spatio-temporal testing in WETPs-PPP. 3) The existing performance monitoring methods with the single evaluation have poor robustness in the performance monitoring of WETPs-PPP, which seriously affects the stability of performance monitoring results.

Based on those previously mentioned, the main work of this study is as follows. 1) A specific performance monitoring and evaluation index system for WETPs-PPP is constructed. 2) An adaptive weighted fusion algorithm and Intuitionistic Fuzzy (IF) average operators are introduced to aggregate multi-source, multi-dimensional, and multi-spatio-temporal data. 3) The IF MULTIMOORA performance monitoring method and evaluation was developed to conduct performance monitoring and evaluation for WETPs-PPP. Finally, the water environment treatment and ecological restoration project of a county was used to verify the feasibility and effectiveness of the proposed method in this study.

This study is divided into five parts as follows. The second part is the literature review, which is mainly a brief introduction to the related studies on the construction of the performance monitoring index system and performance monitoring model. The third part is the research method that includes four aspects. The fourth is the case study in which water environment treatment and the ecological restoration project of a county was used to verify the feasibility and effectiveness of the research contents. Finally, the fifth part includes conclusions through summarizing the work and future anticipation of this study.

2 Literature reviews

Performance monitoring for PPP projects refers to the issue of evaluating PPP projects using scientific and reasonable performance monitoring indicators and methods according to the performance objectives. The results of performance monitoring are not only the basis of performance-based payment but also the key to realizing the “original heart” of PPP projects by increasing the effective supply of public services. For the successful implementation process of PPP projects, effective performance evaluation and supervision are keys (Beatham et al., 2004; Brauers and Zavadskas, 2012; Liu et al., 2015). Yu et al. (2007) emphasized the key role of the index system and evaluation model in the performance evaluation of PPP projects, which directly reflects the real evaluation situation of the projects.

Many studies have addressed the need for performance evaluation index systems for PPP projects. Based on many studies, Mladenovic et al.(2013) found the performance objectives of the public sector, private sector, and users, and summarized them into three types of KPIs (key performance indexes) as follows: technical, functional, and financial. Toor and Ogunlana (2010) considered that the traditional iron triangle measures (quality, cost, and duration) are no longer suitable for evaluating the performance of large-scale infrastructure projects, and other performance indicators such as safety, effective utilization and effectiveness of resources, satisfaction of stakeholders, and reduction of conflicts and disputes have become increasingly important, which indicates that the construction industry is transformed from the traditional quantitative performance measurement standard to the evaluation standard with quantitative and qualitative performance methods. Liu et al.(2017) discussed the feasibility of implementing the whole life cycle performance evaluation of PPP projects from five aspects as follows: the satisfaction of key stakeholders, the project delivery process, the ability of public institutions, the ability of the private sector, and the contribution of key stakeholders to the project. Based on the objectives of rail transit PPP projects, a conceptual framework of KPI (key performance index) was established in which the methods of the structured questionnaire and confirmatory factor analysis were used. Then, a performance evaluation system of PPP projects was constructed with stakeholder satisfaction indicators, such as public satisfaction with the project, and government department satisfaction and physical indicators such as implementation progress and design complexity (Yuan et al., 2009; Yuan et al., 2012).

Through confirmatory factor analysis and structural equation modeling, Yuan et al.(2018) screened the operational performance indicators of public rental housing PPP projects, which mainly include housing distribution and recovery efficiency, project spatial distribution, living environment, and project financial status. Liu et al.(2016) proposed a stakeholder-oriented whole life cycle performance evaluation system, which includes key success factors of projects, the roles and responsibilities of the public sector, the selection of concessionaires, risk management, cost and time efficiency under different types, and the concept of value for the money was introduced into the system; this enables the public and private sectors to further improve performance throughout the project life cycle. Negishi et al. (2018) constructed the index system from the construction technology level, end-user level, and external system level and then present a framework for the whole life cycle performance evaluation. Through multi-case studies, Song et al.(2018) identified the influencing factors of the early termination of 11 PPP projects, such as decision-making mistakes from government and payment penalties. These previously mentioned studies provide a theoretical foundation and guiding significance for the construction of the PPP project performance evaluation index system.

Meanwhile, many scholars have conducted research on the construction of performance evaluation models for PPP projects. Mladenovic et al.(2013) discussed how the KPI of projects meets the performance objectives of stakeholders and proposed a two-level evaluation method for project performance. Based on the theories of efficiency, economy, effect, and equity, Cong and Ma (2018) constructed the performance evaluation index system and orderly weighted index model of old city reconstruction PPP projects, and carried out a performance evaluation in combination with the cloud model. Luo et al.(2018) constructed the index system of shale gas PPP projects from five aspects: economic benefits, internal process of the project, innovation and environmental protection, sustainable development, and stakeholder satisfaction.

Currently, there are some problems with performance evaluation in the operation and maintenance process of PPP projects such as insufficient legal basis for performance evaluation, inconsistent objectives of the public and private sectors, difficulties in carrying out the related-risk performance evaluation, a lack of scientific demonstration for the evaluation of medium and long-term financial affordability, and performance evaluation methods need to be improved. Through semi-structured interviews and a multi-case comparison, Lawther and Martin (2014) found that both insufficient performance supervision resources and inconsistent interpretation of projects’ output standards that may lead to unreasonable payment in PFI projects.

Theoretical studies have been provided for the performance evaluation of PPP projects. However, research on the performance evaluation of WETPs-PPP just started. Although the existing evaluation index systems and models provide strong support for the performance evaluation of WETPs-PPP, they have not met the needs of a performance evaluation of WETPs-PPP. In performance-based payment PPP projects, the success of the project depends more on the performance management system and key performance indicators (KPIs) of the project. Only when the performance level is evaluated scientifically and reasonably can it play a positive incentive role in projects to promote social capital and improve the operation quality and efficiency of public infrastructure.

A perfect performance monitoring index system and matching evaluation standards and effective performance evaluation method must be constructed as the basis of performance evaluation PPP projects. Therefore, with deepening of the construction of the WET industry, there is particular urgency to build a suitable performance index evaluation system according to the characteristics of WETPs-PPP to realize “value for money” projects.

3 Research methods

3.1 Research design

For a performance monitoring and evaluation problem of WETPs-PPP with multi-source heterogeneous information, the sets of one-level and two-level performance monitoring indicators are denoted by $G=\left\{G_1,G_2,\dots ,G_m\right\}$ and $C=\left\{C_1,C_2,\dots ,C_n\right\}$, respectively. Additionally, set $E=\left\{E_1,E_2,\dots ,E_p\right\}$ denotes experts from WET related fields, and the performance evaluation information matrix of the $k$ th expert is as follows:

$A^k={\left(\begin{array}{cccc}{a}_{11}^k& a_{12}^k& \cdots & a_{1n}^k\\ a_{21}^k& a_{22}^k& \cdots & a_{2n}^k\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}^k& a_{m2}^k& \cdots & a_{mn}^k\end{array}\right)}_{m\times n},$

in which $a_{ij}^k\left(k=\mathrm{1,2},\dots ,p;i=\mathrm{1,2},\dots ,m;j=\mathrm{1,2},\dots ,n\right)$ is the performance evaluation value described with the IF number of the $j$ th two-level performance evaluation indicator $C_j\left(j=\mathrm{1,2},\dots ,n\right)$ in $i$ th first-level performance evaluation indicator $G_i\left(i=\mathrm{1,2},\dots ,m\right)$ from the $k$ th expert $E_k\left(k=\mathrm{1,2},\dots ,p\right)$.

There are three issues that need to be addressed. The performance evaluation information matrix $A^k$ is the performance evaluation result from the $k$ th expert $E_k$, the aggregated performance evaluation result from all experts is the final performance evaluation result for WETPs-PPP. Thus, an important issue is the weights of all experts and all performance evaluation indicators, which play a key role in the aggregated process of all performance evaluation information. Another issue that needs to be solved is the performance monitoring and evaluation index system for WETPs-PPP. All data collected in the performance monitoring and evaluation process also need to be standardized since inconsistent data types cannot be executed the computing process. The main symbols in this study are summarized in Table 1.

TABLE 1

TABLE 1. Main symbols used in WETPs-PPP for the performance evaluation.

3.2 Performance monitoring and the evaluation index system for WETPs-PPP

Performance monitoring and evaluation of WETPs-PPP involves water quality safety, the municipal pipe network, landscaping, embankment safety, sewage treatment, lighting facilities, and ecological restoration. Based on the high-frequency index, a literature analysis at home and abroad, relevant policies, relevant standard quotas, and the characteristics of WETPs-PPP, the performance monitoring and evaluation indicators of WETPs-PPP are preliminarily determined and there are eight one-level and eighty-six two-level performance monitoring and evaluation indicators from five perspectives as follows: water conservancy infrastructure, landscaping engineering, municipal engineering, river water body, and public participation. The detailed explanations of eight one-level indicators are as follows.

a) The first one-level indicator is River Embankment, which is mainly the embankment facilities along the rivers. Its two-level indicators are as follows: embankment top and flood prevention road, wave (flood) wall, decompression and drainage facilities, embankment slope berm, river course protection project, auxiliary facilities of embankment, embankment body, and levee shoulder (Inazaki et al., 2010; Somanathan 2013; Song et al., 2017; Noshin et al., 2018).

b) The second one-level indicator is Bridges, whose two-level indicators are as follows: bridge deck expansion installation, protective facilities, bridge pier, bridge bearing, pedestrian crossing, and drainage facilities. The performance assessment work mainly assesses the flatness of the bridge deck and pipeline protection measures. These include whether the bridge expansion device is deformed or cracked, whether there is an absence of sundries around the bridge bearing, the pier and abutment surface is kept clean and moss, weeds, thorns, etc. are removed in time, and the wall top plate surface is free of corrosion, peeling, and water leakage (Gunal et al., 2017; Yang et al., 2017; Chaudhari et al., 2020; Song and Park 2021).

c) The third one-level indicator is River Water Environment, which mainly inspects the water environment quality such as the water surface cleanliness, water transparency, whether there is odor, ammonia nitrogen content, total phosphorus content, chemical oxygen demand (COD), and clear warning signs. Its two-level indicators are as follows: cleanliness of the water surface, signboard of the shoreline, water transparency, no peculiar smell, water quality, and pollution source control around shoreline (Li et al., 2019; Li et al., 2020).

d) The fourth one-level indicator is Garden Facilities, which mainly involves the garden buildings, landscape entertainment facilities, squares, garden roads, parking lots, and garden ancillary facilities of the project such as whether the appearance of the garden building is obviously deformed, whether the road surface is clean and tidy, and whether the lighting system is intact. Its two-level indicators are as follows: tree (flower) pool boundary stone, green fence, retaining wall and cold proof facilities, landscape plank road and hydrophilic platform parking lot, landscape gallery, landscape pavilion, landscape sculpture, pavement condition, barrier-free facilities, sign board, lighting, garden irrigation facilities, rest, entertainment and service facilities, sanitary facilities, and guardrail (Kunimoto et al., 2013; Lin et al., 2013; Nam et al., 2017; Cai et al., 2019).

e) The fifth one-level indicator is Garden Plants, and the specific assessment requirements include the appearance of intact trees, the plants are plump, the branches and leaves are healthy, the overall appearance of plants after pruning is good, no pests and diseases, regular fertilization, timely removal of weeds, loose soil, no hardening and water shortage, and timely replanting. Its two-level indicators are as follows: growth, fertilizer application, irrigation trim, pest control, loosen the soil, weeding, hole in a tree, and replanting (Kunimoto et al., 2013; Lin et al., 2013; Nam et al., 2017; Cai et al., 2019).

f) The sixth one-level indicator is Special Purpose Vehicle, which belongs to the scope of the internal assessment, and its assessment focuses on the clear division of SPV’s functions, clear process operation, and configuration of the operation team according to the plan to meet operational needs. Its two-level indicators are as follows: organization, institutional mechanism, human resource management, security management, and financial management (Carpintero & Petersen, 2015; Akomea-Frimpong et al., 2021).

g) The seventh one-level indicator is Hydraulic Structure, which mainly refers to the regulating water structures with different functions and types involved in the project, and its assessment focuses on the main works, gates, hoists, electromechanical equipment, daily management, and ancillary facilities. Its two-level indicators are as follows: concrete and rubber dam, lock chamber, expansion joints and drainage facilities, gate surface and water stop device, gate bearing and supporting device, ancillary facilities and signs, operation status of electron mechanical equipment, overall condition of the hoist, and daily maintenance records (Pajno et al., 2013; Hager & Boes, 2014).

h) The eighth one-level indicator is the Public Satisfaction, which refers to the measure of the effect of water environment governance through the public’s sense of the water environment such as vision and smell. The evaluation of the public satisfaction is usually based on the cumulative effect of their long-term experience. Its two-level indicators are as follows: water transparency, water fluidity, whether the water body has a peculiar smell, water surface cleanliness, whether the river has any obstacles, plant species diversity, plant growth trends, plant pruning status, richness of seasonal changes, improved living convenience, green plant coverage, whether the river has plenty of water, major facilities, completeness of building facilities, hierarchy of plants, comfort of rest and entertainment facilities, completeness of safety facilities, sanitation around the river, completeness of safety signs, meets the fitness needs, rationality of street lamp position and brightness, health situation, coordination degree between various facilities and the overall environment, diversity and smoothness of supervision or complaint channels, comprehensive quality of service personnel, implementation of found problems, meets the needs of life and leisure, produces a sense of pleasure when surrounding activities, and the surrounding environment brings comfort (Park & Kurosawa, 2017; Bilgin, 2018; Li et al., 2020).

To ensure the primary selected indicators have the maximum information and are less redundant, experts with rich experience in the field of WETPs-PPP are invited to score the primary performance monitoring and evaluation indicators according to their importance, which is conducted in the terms of the questionnaire. The questionnaire consists of the basic information from the experts and the judgment on the importance of the performance monitoring and evaluation indicators from the experts. The questionnaire was compiled using the 5-point Likert scale, and the 5 grades are as follows: “5-very important, 4-more important, 3-important, 2-general important, and 1-not important.” The data characteristics are shown in Table 2.

TABLE 2

TABLE 2. Characteristics of the questionnaire data.

According to the data collected through the questionnaire, the primary selected performance monitoring and evaluation indicators are the secondary screening using the consistency degree of performance evaluation information from experts, which is measured by the difference of the performance evaluation information between a certain expert and other experts. Generally, the greater the deviation, the stronger the consistency and the larger the weight of the corresponding expert. Therefore, let $a_{ij}$ be the score of importance on the $j$ th performance monitoring and evaluation indicators $x_j\left(j=\mathrm{1,2},\dots ,n\right)$ from the $i$ th expert $E_i\left(i=\mathrm{1,2},\dots ,m\right)$. Then, the importance ${\stackrel{\sim }{w}}_j$ of the $j$ th performance monitoring and evaluation indicators $x_j$ form the $i$ th expert $E_i$ can be defined as follows:

${\stackrel{\sim }{w}}_j=\frac{\sum _{i=1}^m{a}_{ij}{w}_{ij}}{\sum _{j=1}^m\sum _{i=1}^m{a}_{ij}{w}_{ij}},(1)$

where

$\left\{\begin{array}{l}\begin{array}{ll}{w}_{1j}=w_{2j}=\cdots =w_{mj}=\frac{1}{m},& V_{ij}=0\end{array}\\ \begin{array}{ll}{w}_{ij}=\frac{1/V_{ij}}{{\displaystyle \sum }_{i=1}^m\left(1/V_{ij}\right)},& V_{ij}\ne 0\end{array}\end{array}\right.(2)$

is the reliability of the performance evaluation information from the $i$ th expert $E_i$ on the $j$ th performance monitoring and evaluation indicators $x_j$, and $V_{ij}={\displaystyle \sum }_{\begin{array}{c}l=1\\ l\ne i\end{array}}^m\left|a_{ij}-a_{lj}\right|$ is the sum of all score differences of the importance on the $j$ th performance monitoring and evaluation indicator between the $i$ th expert and the other $m-1$ experts.

From Eqs 1 and 2, the importance degree of every primary selected performance monitoring and evaluation indicator can be calculated by applying the performance evaluation data values summarized in the questionnaire from the experts, and the final performance monitoring and evaluation indicators are determined by retaining the performance monitoring and evaluation indicators with an importance degree greater than or equal to 80%, whose size is taken according to the practical WETPs-PPP and the existing index selection methods (Gan et al., 2009). The calculation results and the determined final performance monitoring and evaluation index system are shown in Figure 1. From Figure 1, the final determined performance monitoring and evaluation indicators of WETPs-PPP are classified into eight one-level performance monitoring and evaluation indicators, which are as follows: special purpose vehicle, river embankment, hydraulic structure, river water environment, garden facilities, bridges, and public satisfaction. Meanwhile, the first, second, and seventh one-level performance monitoring indicators include four two-level performance monitoring and evaluation indicators. Both of the fourth and sixth one-level performance monitoring and evaluation indicators include five two-level performance monitoring and evaluation indicators. Additionally, the third and fifth one-level performance monitoring and evaluation indicators include 6, 10, and 21 two-level performance monitoring and evaluation indicators.

FIGURE 1

FIGURE 1. Performance monitoring indicators for WETPs-PPP.

3.3 Determination of the weights of both indicators and experts

The determination of performance monitoring and evaluation indicators’ weights is directly related to the reliability of performance monitoring and evaluation results, which not only represent the impact degree of performance monitoring and evaluation indicators on the previous level of indicators but also reflects their importance in the same level of indicators. Apart from the impact of performance monitoring and evaluation indicators, the weights of experts are also considered when the performance of WET is monitored and evaluated according to the differences in knowledge structure, ability, and evaluation perspective. Currently, the weighting’s methods in practical performance monitoring and evaluation problems mainly include the subjective weighting method, objective weighting method, and objective and subjective synthetic weighting methods (Wang et al., 2012; Sahoo et al., 2016). Then, each method will be considered on a case-by-case basis as follows:

(1) Determination of performance monitoring and evaluation of indicators’ weights

Fuzzy entropy can describe the fuzziness degree of the decision-maker in the process of performance monitoring and evaluation for WETPs-PPP. The greater the fuzzy entropy of the evaluation value, the greater the fuzziness degree of evaluation information from decision-makers, which are then given a smaller weight. On the contrary, it shows that the smaller the fuzziness degree of evaluation information, the larger the weight that should be given. For the evaluation information under the IF environment (Andersen et al., 2019), the deviation between membership degree and non-membership degree describes the uncertainty of evaluation information, and the hesitation degree describes the unknown degree of evaluation information. Both the uncertainty and unknown degrees reflect the fuzzy degree represented by IF entropy. The weights of indicators are measured by IF entropy (Ye, 2010; Xu & Tang, 2021), which is investigated in three aspects as follows: membership degree, non-membership degree, and hesitancy degree.

Assume that the evaluation value $a_{ij}^k$ of the $j$ th two-level indicator $C_j$ in the $i$ th one-level indicator $G_i$ from the $k$ th expert $E_k$ is described by the IF number, that is, $a_{ij}^k=\left({\mu }_{ij}^k,{\nu }_{ij}^k\right)$, then the IF entropy (Ye, 2010; Xu & Tang, 2021) of the evaluation value from the $k$ th expert is defined as follows:

$M_j^k=\frac{1}{m}{\displaystyle \sum }_{i=1}^m\cos \frac{\left({\mu }_{ij}^k-{\nu }_{ij}^k\right)\left(1-{\pi }_{ij}^k\right)\pi }{2},(3)$

and then, the weight of the $j$ th two-level indicator $C_j$ in the $i$ th one-level indicator $G_i$ from the $k$ th expert $E_k$ is denoted as follows:

$w_j^k=\frac{1-M_j^k}{n-{\displaystyle \sum }_{j=1}^n{M}_j^k}.(4)$

Additionally, let ${\overline{w}}_j$ be the average value of the maximum and minimum values, that is, ${\overline{w}}_j=\frac{\left(\underset{k=\mathrm{1,2},\dots ,p}{\min }{w}_j^k+\underset{k=\mathrm{1,2},\dots ,p}{\max }{w}_j^k\right)}{2}$, and the weight coordination coefficient of indicators’ weights is given as follows:

${\alpha }_k=\frac{S\left(w^k,\overline{w}\right)}{{\displaystyle \sum }_{k=1}^{p}S\left(w^k,\overline{w}\right),}(5)$

where $S\left(w^k,\overline{w}\right)=1-\underset{j=\mathrm{1,2},\dots ,n}{\max }\left|w_j^k-{\overline{w}}_j\right|$ is the similarity degree between $w^k$ and 和 ${\overline{w}}_j$, ${\displaystyle \sum }_{k=1}^p{\alpha }_k=1$, $0<{\alpha }_k<1$.

Therefore, the weight of the $j$ th two-level indicator $C_j$ in the $i$ th one-level indicator $G_i$ is obtained through aggregating the weights $w_j^k$ from $n$ experts as follows:

$w_j^{*}={\displaystyle \sum }_{k=1}^p{\alpha }_k{w}_j^k,(6)$

where ${\displaystyle \sum }_{j=1}^n{w}_j^{*}=1$, $0<w_j^{*}<1$.

(2) Determination of experts’ weights

The importance of experts understanding the evaluation object is described by the fuzzy and uncertain degrees of evaluation information from experts. The fuzzier and more uncertain the evaluation information given by experts, the less the experts know about the evaluation object, and the corresponding weights of the experts should be relatively small. Conversely, it shows that the expert knows more about the evaluation object, and the weights of the experts should be relatively larger. Therefore, the weights of experts can be obtained by calculating the IF entropy of evaluation information from experts. The weighting method of experts is given as follows.

Based on the principle of the entropy weight method, the weight ${\lambda }_1^k$ of the $k$ th expert $E_k$ determined by the IF entropy of the evaluation matrix is expressed as follows:

${\lambda }_1^k=\frac{1-H^k}{p-{\displaystyle \sum }_{k=1}^p{H}^k},(7)$

where $H^k={\displaystyle \sum }_{j=1}^n{w}_j^k{M}_j^k$ is the weighted IF entropy describing the fuzziness degree of evaluation information from the $k$ th expert $E_k$.

However, let $E^k=1/mnp{\displaystyle \sum }_{l=1}^p{\displaystyle \sum }_{i=1}^m{\displaystyle \sum }_{j=1}^{n}e\left({\overline{r}}_{ij}^k,{\overline{r}}_{ij}^l\right)$ be the distance measure between the evaluation information matrix $A^k$ and $A^l$, and $e\left({\overline{r}}_{ij}^k,{\overline{r}}_{ij}^l\right)$ be the Euclidean distance between IF numbers ${\overline{r}}_{ij}^k$ and ${\overline{r}}_{ij}^l$, then the weight ${\lambda }_2^k$ of the $k$ th expert $E_k$ determined by the distance measure between the evaluation information matrix is expressed as follows:

${\lambda }_2^k=\frac{1-E^k}{p-{\displaystyle \sum }_{k=1}^p{E}^k}.(8)$

Therefore, the combination weight ${\lambda }_k^{*}$ of the $k$ th expert is obtained through aggregating ${\lambda }_1^k$ and ${\lambda }_2^k$ according to the coordination coefficients $0<\alpha <1$ and $0<\beta <1$ of expert weights is as follows:

${\lambda }_k^{*}=\alpha {\lambda }_1^k+\beta {\lambda }_2^k.(9)$

It should be noted that the values of $\alpha$ and $\beta$ depend on the context, where $\alpha +\beta =1$.

3.4 Construction of an improved MULTIMOORA performance evaluation model

The performance evaluation problem of WETPs-PPP is a classical multi-indicator group evaluation problem with hybrid data information.

From the perspective of robustness, the evaluation methods obtained by composing two different evaluation methods are better than those of a single evaluation method, the evaluation methods of three different evaluation methods are better than those of two different evaluation methods, and so on (Brauers, 2012). The MOORA (Multi-objective optimization on the basis of the ratio analysis) (Brauers and Zavadskas, 2006) uses multi-criteria decision-making, which includes the ratio system and the reference point method. Compared with the existing evaluation method, the important characteristics of the MOORA method are as follows: simple, small amount of calculation, stable results, and less solution time. Additionally, it can compare and select schemes from multiple angles (Liu et al., 2014). MULTIMOORA (Brauers and Zavadskas, 2010) is proposed on the basis of MOORA by adding to the full multiplicative form, which is widely used to solve various complex multi-criteria decision-making problems. Compared with the MOORA evaluation method, the MULTIMOORA evaluation method is simpler and more robust (Brauers and Zavadskas, 2012).

The performance evaluation procedures of WETPs-PPP using the improved MULTIMOORA evaluation method are shown in Figure 2.

Step 1: Construct the evaluation matrix of experts according to the collected evaluation data.

FIGURE 2

FIGURE 2. Procedure of the performance evaluation in WETPs-PPP.

The evaluation matrix of the $k$ th expert is

$A^k={\left({\alpha }_{ij}^k\right)}_{m\times n},$

where ${\alpha }_{ij}^k$ is the evaluation value of the $j$ th two-level indicator $C_j$ in the $i$ th one-level indicator $G_i$, which may be described by a different data type, such as a crisp number, interval number, and linguistic values. Thus, the hybrid data types should be standardized and transformed into the unified type. Here, the IF number is chosen as the final type, and the operation process can be conducted using the standardized methods, such as the range standardization method and maximum value standardization method. Assume that the transformed evaluation matrix is

${\stackrel{\sim }{A}}^k={\left({\stackrel{\sim }{\alpha }}_{ij}^k\right)}_{m\times n},$

in which ${\stackrel{\sim }{\alpha }}_{ij}^k$ is IF number.

Step 2: Ratio system. Calculate the additive comprehensive evaluation values of the one-level indicators.

On the basis of the IF comprehensive evaluation matrix of every one-level indicator and its corresponding weight vectors of the two-level indicator, the comprehensive evaluation value of the $i$ th one-level indicator $G_i$ is obtained according to the IF weighted arithmetic average operator (Ouyang and Pedrycz, 2016; Teng and Liu, 2019),

${\overline{U}}_i^{*}={\mathrm{I}\mathrm{F}\mathrm{W}\mathrm{A}}_W\left({\stackrel{\sim }{\alpha }}_{i1},{\stackrel{\sim }{\alpha }}_{i2},\dots ,{\stackrel{\sim }{\alpha }}_{in}\right)=\stackrel{n}{\underset{j=1}{\oplus }}{w}_j{\overline{\alpha }}_{ij}=\left(1-{\displaystyle \prod }_{j=1}^n{\left(1-{\mu }_{ij}\right)}^{w_j},{\displaystyle \prod }_{j=1}^n{{\nu }_{ij}}^{w_j}\right)(10)$

and the more comprehensive the evaluation value ${\overline{U}}_i^{*}$, the higher the score of performance evaluation.

Step 3: Reference point method. Calculate the relative closeness degree between the evaluation values of each one-level indicator and the selected positive reference point. The positive and negative reference points can be selected using the maximum and minimum values selected in the evaluation values of two-level indicators for every one-level indicator. That is, the maximum and minimum values are regarded as the positive reference point and negative reference point, respectively.

Let $q_i^{+}$ and $q_i^{-}$ be the sum of the Hamming distances (Xu and Yager, 2006) between the $i$ th one-level indicator $G_i$ and positive and negative reference points, respectively; then, the closeness degree (Zhou and Jiang, 2010) of the $i$ th one-level indicator $G_i$ is as follows:

$q_i=\frac{q_i^{-}}{q_i^{+}+q_i^{-}}.(11)$

Thus, the greater the value of the closeness degree $q_i$, the higher the score of the corresponding performance indicator.

Step 4: The full model. Calculate the multiplication comprehensive evaluation value of every one-level indicator. The evaluation value of every two-level indicator and its weight in the $i$ th one-level indicator $G_i$, and the multiplication comprehensive evaluation value of the $i$ th one-level indicator $G_i$ can be calculated using the following formula:

${\stackrel{\sim }{U}}_i^{*}{=\mathrm{I}\mathrm{F}\mathrm{W}\mathrm{G}}_W\left({\stackrel{\sim }{\alpha }}_1,{\stackrel{\sim }{\alpha }}_2,\dots ,{\stackrel{\sim }{\alpha }}_n\right)=\langle {\displaystyle \prod }_{j=1}^n{{\mu }_j}^{w_j},1-{\displaystyle \prod }_{j=1}^n{\left(1-{\nu }_j\right)}^{w_j}\rangle .(12)$

Similarly, the more the comprehensive the value ${\stackrel{\sim }{U}}_i^{*}$ of the $i$ th one-level indicator $G_i$, the higher the score of the corresponding performance indicators.

Step 5: Aggregate performance evaluation results. Using the IF weight aggregate operator, the comprehensive evaluation values of all one-level indicators are aggregated according to the evaluation result obtained in Step 2, Step 3, and Step 4, where the weights are assumed equal in every step.

Step 6: Defuzzy and calculate the performance evaluation score of every one-level indicator. The comprehensive evaluation result obtained in Step 5 with the IF number is transformed into the performance evaluation score.

The comprehensive evaluation score of the $i$ th one-level indicator $G_i$ can be given using the following formula:

$\mathit{def}\left(G_i\right)=\frac{{\tau }_i}{2}\times G^L+\left(G^R-G^L\right),(13)$

where ${\tau }_i=\left({\mu }_i+\left(1-{\pi }_i\right)\right)/2$ is the corresponding fuzzy trend approximation factor of the $i$ th one-level indicator $G_i$, and $G=\left[G^L,G^R\right]$ 为 is the interval of the performance evaluation grade.

4 Case study

4.1 Problem description

To verify the practicability and effectiveness of the performance evaluation model constructed in Section 3, the water environment treatment and ecological restoration project of a county is used as an example. This is a new project with a proposed cooperation period of 20 years, including 2 years of construction and 18 years of operation, and the estimated total investment is 2162.6205 million Yuan. A total of nine sub-projects are included such as the water system connection project, supporting bridge project, river and lake water ecological restoration project, waterfront environment improvement project, sewage treatment plant upgrade and support project, distributed ecological sewage treatment station project, sludge and construction waste recycling project, Caohe sewage pipeline project, and the smart water ecological supervision system project.

The performance evaluation process of this PPP project is conducted once a year, and the annual evaluation group is composed of experts in water conservancy, landscaping, municipal, legal and financial industries, and the public and the evaluation results of experts and the public are obtained in the form of a questionnaire. The annual performance evaluation is divided into two parts as follows: the evaluation of daily data files and the evaluation of the project site. Specifically, for SPV, it is mainly to evaluate daily data files, and the content of the evaluation of daily data files and the project site simultaneously includes river embankments, hydraulic structures, the river water environment, garden facilities, and garden plants and bridges,. The final evaluation grades are excellent, good, medium, pass, and bad, and the corresponding score ranges are [90,100], [80,90], [70,80], [60,70], and [0,60], respectively.

4.2 Data sources

According to the performance evaluation index system of WETPs-PPP constructed in Section 3.2, indicator data are mainly divided into three types as follows. 1) Survey data from the public; fifty people near the coastal area of the project are randomly invited to fill in the questionnaire according to the public satisfaction indicators. 2) Evaluation data from experts; five experts from water conservancy, municipal gardens, finance and law, and other related industries are invited to fill in the questionnaire. 3) Real-time monitor data; the monitoring data on water quality indicators, including COD, NH₃-N, and TP, are obtained using a water quality sensor. Specifically, three water quality sensors are placed in three representative locations within the treatment area of the river water environment, and each sensor is monitored seven times a week at 1 day intervals.

4.3 Data processing

The data types obtained from the questionnaire roughly include exact values, interval values, and uncertainty linguistic values, especially linguistic values that include five-scale, seven-scale, and nine-scale linguistic values. The first and most important thing is that all data on different types should be transformed into IF numbers using the method in Section 3. For the monitoring data, such as COD, NH₃-N, and TP, the weekly monitoring data should be transformed into annual monitoring data. Specifically, the adaptive weighting fusion algorithm of WETPs-PPP is first applied to the simultaneous interpretation of the data from different sensors to obtain weekly monitoring data, and the average method is used to obtain the average value of the weekly data as the annual monitoring data.

4.4 Calculation of weights for experts and indicators

1) Calculation of experts’ weights

According to the weighting method of experts with $\alpha =\beta =0.5$ for convenience, the weights of the five experts giving the evaluation information of every two-level indicator are shown in Table 3.

2) Calculation of indicators’ weights

TABLE 3

TABLE 3. Weights of five experts in evaluating two-level indicators.

From the weighting method of indicators, the weights of every one-level and two-level indicator are given in Figure 3 and Table 4.

FIGURE 3

FIGURE 3. Weights of every one-level indicator.

TABLE 4

TABLE 4. Weights of one-level and two-level indicators.

4.5 Performance evaluation

Water environment treatment and an ecological restoration project were implemented in a county, and a performance evaluation was completed using the constructed evaluation model. The evaluation steps are as follows:

Step 1: The data obtained by questionnaires and water quality sensors were transformed into an IF number using the method of processing data in Section 3.

Step 2: Ratio system. According to the formulas of Step 2 in Section 3, all comprehensive evaluation values of one-level indicators can be calculated, which are presented in Table 5.

Step 3: According to Step 3 in Section 3, the degree of relative closeness can be calculated from Eq. 11, which is shown in Table 5.

Step 4: Based on Step 4 in Section 3, additive comprehensive evaluation values of every one-level indicators are obtained from Eq. 12, which is shown in Table 5.

Step 5: Aggregated comprehensive evaluation values. The evaluation results combining the results from Step 2 to 4 can be aggregated using ${\mathrm{I}\mathrm{F}\mathrm{W}\mathrm{A}}_W$, and the final evaluation results of every one-level indicator are also given in Table 4.

Step 6: Defuzzification. The evaluation scores of every one-level indicator can be transformed into the IF number obtained in Step 5 from Eq. 13, which is shown in Figure 4:

TABLE 5

TABLE 5. Calculated results of Step 2–Step 5.

FIGURE 4

FIGURE 4. Evaluation scores of every one-level indicator.

As shown in Figure 4, the evaluation scores of every one-level indicator C₁-C₈ are 85.37, 85.32, 89.09, 85.42, 84.54, 88.15, 85.76, and 87.24. In summary, the performance evaluation scores and grades for each level of the performance indicator are shown in Table 5. From Table 5, the performance evaluation results of the eight two-level indicators are good. Combining the corresponding weights of every one-level indicator in Table 4, the final performance evaluation score of the water environment treatment and ecological restoration projects in the county was 86.60 (the evaluation grade is good), which indicates that all evaluated items in the operation of the project were conducted smoothly. Specifically, the operation of the SPV company is normal in the organization, system and organization, financial management and safety management, the operation and maintenance of river embankments, hydraulic structures, river water environment, garden plants, bridges and garden facilities are good, and the responses from the public are also good.

From these results, the performance evaluation outcome should be rationally consulted, and scientific decisions on the project can then be made. Specifically, projects with a poor performance effect on areas will be suspended or modified, and incentive or punishment measures should also be taken to further improve project performance. However, for areas with good performance effects, the projects will continue to be carried out.

5 Conclusion

Performance monitoring is a particularly important issue in WETPs-PPP. The key to promoting successful implementation of WETPs-PPP lies in an effective performance evaluation and supervision of projects. Based on this motivation and the characteristics of WETPs-PPP, this study develops an evaluation model for performance monitoring in WETPs-PPP.

This study carried out the following aspects. 1) Using both the literature reviews and constructing the screening model with the importance of indicators, the performance evaluation index system of WETPs-PPP was constructed and included eight one-level performance monitoring indicators and fifty-nine two-level performance monitoring indicators. 2) An adaptive weighted fusion algorithm of WETPs-PPP was developed and used to process multi-source, multi-dimensional, multi-temporal, and multi-agent data information in the performance evaluation for WETPs-PPP. 3) To measure the importance of all indicators and experts, the IF entropy and distance measure methods were applied to calculate their weights. 4) Combining the actual characteristics of the performance evaluation of WETPs-PPP, an improved MULTIMOORA evaluation method was constructed, which includes five steps as follows: the ratio system, reference point method, total multiplication model, aggregation evaluation results, and defuzzification. Finally, to verify the effectiveness and rationality of the proposed method, a county water environment treatment and ecological restoration project was used as an example.

Some implications should be stated from three perspectives. 1) The performance evaluation system of WETPs-PPP is an important tool to project management for governments. According to regulations revolving around project construction, operation management, and performance evaluation, the performance status and operation and maintenance service quality of the private sector in the process of project construction or operation and maintenance should be monitored strictly and regularly evaluated. 2) The result-oriented performance evaluation can make the private sector clarify deficiencies in a certain aspect of the project to strengthen the performance improvement directly and intentionally. Under these circumstances, the private sector can obtain relatively higher performance evaluation effects at a relatively low cost by rectifying the specific area with poor performance effects. The performance evaluation will help the private sector become clearer regarding its own construction or operation and maintenance goals and allocate the project’s cost input to achieve the best performance profit. 3) WETPs-PPP plays a very important role in improving water ecology and water environment quality, ensuring urban flood control and drainage and improving the hydrophilic demand of people, and it also plays an indispensable role in the construction of the water ecology of civilized cities and the sustainable development of the economy and society. The performance evaluation results can accurately identify defects in the construction or operation and maintenance of a certain aspect in the project and then rectification or repair can be promptly conducted.

The research in this study has some limitations. First, the unchanged indicator system will affect the effectiveness of the evaluation results for WETPs-PPP, which is a limitation in practical projects. For example, some facilities may be considered very important in terms of current performance evaluation, while they may not need to occupy an excessive proportion in the performance evaluation after decades. Additionally, with improvement of human living standards and the change of the living environment, the values of things for people will also greatly change. The indicator system will also vary with the changes of people’s living standards and environment. Therefore, an iterative updating indicator system will match the limitation to the unchanged indicator system. Second, the evaluation data obtained by the expert questionnaire will more or less affect the evaluation results, which is another limitation in the evaluation process. This is because questionnaire data inevitably have a subjectivity of evaluation information from experts. Therefore, in combination with big data mining, image recognition, and other technologies, the idea of adaptive variable weight should be further introduced in the process of performance evaluation. In the future, these ideas and technologies can further improve the reliability and scientific method of performance evaluation results.

There is no doubt that additional work will be completed in future research. On one hand, the iterative updating indicator system is more in line with the actual situation of projects, which should be the future point of research. On the other hand, with the help of big data technology, data mining, and image recognition to obtain data evaluation information, it will further improve the scientific process and reliability of evaluation. Additionally, the refined management of performance evaluation can promote the improvement of the project treatment level and performance of the industry, and the management of PPP projects should be further standardized to improve the adaptability of the performance evaluation for the government.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary Material, and further inquiries can be directed to the corresponding author.

Ethics Statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent from the participants was not required to participate in this study in accordance with the national legislation and the institutional requirements.

Author contributions

Conceptualization, LS and YC; methodology, LS; software, YC; validation, LS and YC; writing—original draft preparation, LS and YC; writing—review and editing, LS and YC; funding acquisition, YC. All authors have read and agreed to the published version of the manuscript.

Acknowledgments

The authors acknowledge with gratitude the General project of National Social Science Fund (No. 21BJY219).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors, and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Abi, S. C., Andary, E., Gan, C. A., and Ahmad, I. U. (2019). Project performance rating model for water and wastewater treatment plant public projects. J. Manage. Eng. 35 (2), 4018064. doi:10.1061/(asce)me.1943-5479.0000678

CrossRef Full Text | Google Scholar

Akomea-Frimpong, I., Jin, X. H., and Osei-Kyei, R. (2021). A holistic review of research studies on financial risk management in public-private partnership projects. Eng. Constr. And Archit. Manag. 28 (9), 2549–2569. doi:10.1108/ecam-02-2020-0103

CrossRef Full Text | Google Scholar

An, X., Li, H., Wang, L., Wang, Z., Ding, J., and Cao, Y. (2018). Compensation mechanism for urban water environment treatment PPP project in China. J. Clean. Prod. 201, 246–253. doi:10.1016/j.jclepro.2018.08.003

CrossRef Full Text | Google Scholar

Andersen, S. C., Mller, K. L., Jrgensen, S. W., Jensen, L. B., and Birkved, M. (2019). Scalable and quantitative decision support for the initial building design stages of refurbishment. J. Green Build. 14 (4), 35–56. doi:10.3992/1943-4618.14.4.35

CrossRef Full Text | Google Scholar

Beatham, S., Anumba, C., Thorpe, A., and Hedges, I. (2004). KPIs: A critical appraisal of their use in construction. Benchmarking Int. J. 11, 93–117. doi:10.1108/14635770410520320

CrossRef Full Text | Google Scholar

Bilgin, A. (2018). Evaluation of surface water quality by using Canadian council of ministers of the environment water quality index (CCME WQI) method and discriminant analysis method: A case study coruh river basin. Environ. Monit. Assess. 190 (9), 554. doi:10.1007/s10661-018-6927-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Brauers, W. K. M., and Zavadskas, E. K. (2012). Robustness of MULTIMOORA: A method for multi-objective optimization. Informatica 23, 1–25. doi:10.15388/informatica.2012.346

CrossRef Full Text | Google Scholar

Brauers, W. K. M., and Zavadskas, E. K. (2006). The MOORA method and its application to privatization in a transition economy. Control Cybern. 35, 445–469.

Google Scholar

Brauers, W. (2012). Project management for a country with multiple objectives. Czech Econ. Rev. 6, 80–101.

Google Scholar

Brauers, W., and Zavadskas, E. (2010). Project management by MULTIMOORA as an instrument for transition economies/projektu vadyba SU MULTIMOORA kaip priemone pereinamojo laikotarpio ukiams. Technol. Econ. Dev. Econ. 16 (1), 5–24. doi:10.3846/tede.2010.01

CrossRef Full Text | Google Scholar

Cai, Y. F., Chen, J., Zhang, C., and Wang, B. (2019). A parking space allocation method to make a shared parking strategy for appertaining parking lots of public buildings. Sustainability 11 (1), 120. doi:10.3390/su11010120

CrossRef Full Text | Google Scholar

Carpintero, S., and Petersen, O. H. (2015). Bundling and unbundling in public-private partnerships: Implications for risk sharing in urban transport projects. Proj. Manag. J. 46 (4), 35–46. doi:10.1002/pmj.21508

CrossRef Full Text | Google Scholar

Cha, H. S., and Kim, C. K. (2011). Quantitative approach for project performance measurement on building construction in South Korea. KSCE J. Civ. Eng. 15 (8), 1319–1328. doi:10.1007/s12205-011-1323-5

CrossRef Full Text | Google Scholar

Chaudhari, A., Gore, N., Arkatkar, S., Joshi, G., and Pulugurtha, S. (2020). Pedestrian crossing warrants for urban midblock crossings under mixed-traffic environment. J. Transp. Eng. Part A Syst. 146 (5). doi:10.1061/jtepbs.0000338

CrossRef Full Text | Google Scholar

Chua, D. K. H., Kog, Y. C., and Loh, P. K. (1999). Critical success factors for different project objectives. J. Constr. Eng. Manag. 125 (3), 142–150. doi:10.1061/(asce)0733-9364(1999)125:3(142)

CrossRef Full Text | Google Scholar

Cong, X., and Ma, L. (2018). Performance evaluation of public-private partnership projects from the perspective of efficiency, economic, effectiveness, and equity: A study of residential renovation projects in China. Sustainability 10 (6), 1951. doi:10.3390/su10061951

CrossRef Full Text | Google Scholar

Gan, L., Shen, L., and Hongyuan, F. (2009). Study of the infrastructure projecte valuation factors based on sustainable development. CHINA Civ. Eng. J. 42 (11), 133–138.

Google Scholar

Gunal, M., Gelmeran, T. A., and Gunal, A. Y. (2017). Local scour around group bridge pier with different shapes. Acta Phys. Pol. A 132 (3), 632–633. doi:10.12693/aphyspola.132.632

CrossRef Full Text | Google Scholar

Hafezalkotob, A., and Hafezalkotob, A. (2017). Interval MULTIMOORA method with target values of attributes based on interval distance and preference degree: Biomaterials selection. J. Ind. Eng. Int. 13 (2), 181–198. doi:10.1007/s40092-016-0176-4

CrossRef Full Text | Google Scholar

Hager, W. H., and Boes, R. M. (2014). Hydraulic structures: A positive outlook into the future. J. Hydraulic Res. 52 (3), 299–310. doi:10.1080/00221686.2014.923050

CrossRef Full Text | Google Scholar

Ijadi Maghsoodi, A., Abouhamzeh, G., Khalilzadeh, M., and Zavadskas, E. K. (2018). Ranking and selecting the best performance appraisal method using the MULTIMOORA approach integrated Shannon’s entropy. Front. Bus. Res. China 12 (1), 2. doi:10.1186/s11782-017-0022-6

CrossRef Full Text | Google Scholar

Inazaki, T., Inoue, M., Saito, Y., Arakane, S., and Yoshida, N. (2010). “Time-lapse monitoring of the slope failure process of a model levee,” (Keystone, CO, USA, 110–117.23rd symposium on the application of geophysics to engineering and environmental problems

CrossRef Full Text | Google Scholar

Kunimoto, Y., Imura, T., and Takenaka, I. (2013). Recent study and future view of pest control in nara prefecture. Jpn. J. Appl. Entomol. Zool. 57 (2), 69–78. doi:10.1303/jjaez.2013.69

CrossRef Full Text | Google Scholar

Lam, E. W. M., Chan, A. P. C., and Chan, D. W. M. (2007). Benchmarking the performance of design-build projects. Benchmarking. 14 (5), 624–638. doi:10.1108/14635770710819290

CrossRef Full Text | Google Scholar

Lawther, W. C., and Martin, L. (2014). Availability payments and key performance indicators: Challenges for effective implementation of performance management systems in transportation public-private partnerships. Public Works Manag. Policy 19 (3), 219–234. doi:10.1177/1087724x14528476

CrossRef Full Text | Google Scholar

Li, H., Cao, Y., and Su, L. (2019). Multi-dimensional dynamic fuzzy monitoring model for the effect of water pollution treatment. Environ. Monit. Assess. 191 (6), 352. doi:10.1007/s10661-019-7502-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Li, H., Lv, L., Zuo, J., Bartsch, K., Wang, L., and Xia, Q. (2020a). Determinants of public satisfaction with an Urban Water environment treatment PPP project in Xuchang, China. Sustain. Cities Soc. 60, 102244. doi:10.1016/j.scs.2020.102244

CrossRef Full Text | Google Scholar

Li, H., Lv, L., Zuo, J., Su, L., Wang, L., and Yuan, C. (2020b). Dynamic reputation incentive mechanism for urban water environment treatment PPP projects. J. Constr. Eng. Manag. 146 (8), 4020088. doi:10.1061/(asce)co.1943-7862.0001879

CrossRef Full Text | Google Scholar

Lin, W. C., Huang, T. S., Ho, T. C., Chen, Y. T., and Chuang, J. H. (2013). Interactive lighting design with hierarchical light representation. Comput. Graph. Forum 32 (4), 133–142. doi:10.1111/cgf.12159

CrossRef Full Text | Google Scholar

Liu, H. C., Fan, X. J., Li, P., and Chen, Y. Z. (2014). Evaluating the risk of failure modes with extended MULTIMOORA method under fuzzy environment. Eng. Appl. Artif. Intell. 34 (C), 168–177. doi:10.1016/j.engappai.2014.04.011

CrossRef Full Text | Google Scholar

Liu, H., Fan, X., Li, P., and Chen, Y. (2014). Evaluating the risk of failure modes with extended MULTIMOORA method under fuzzy environment. Eng. Appl. Artif. Intell. 34, 168–177. doi:10.1016/j.engappai.2014.04.011

CrossRef Full Text | Google Scholar

Liu, J., Love, P. E. D., Davis, P. R., Smith, J., and Regan, M. (2015). Conceptual framework for the performance measurement of public-private partnerships. J. Infrastruct. Syst. 21 (1), 4014023. doi:10.1061/(asce)is.1943-555x.0000210

CrossRef Full Text | Google Scholar

Liu, J., Love, P. E. D., Smith, J., Matthews, J., and Sing, C. (2016). Praxis of performance measurement in public-private partnerships: Moving beyond the iron triangle. J. Manage. Eng. 32 (4), 4016004. doi:10.1061/(asce)me.1943-5479.0000433

CrossRef Full Text | Google Scholar

Liu, J., Love, P. E. D., Sing, M. C. P., Smith, J., and Matthews, J. (2017). PPP social infrastructure procurement: Examining the feasibility of a lifecycle performance measurement framework. J. Infrastruct. Syst. 23 (3), 4016041. doi:10.1061/(asce)is.1943-555x.0000347

CrossRef Full Text | Google Scholar

Luo, Z. H., Yang, Y. F., Pan, H. Z., and Zhong, M. F. (2018). Research on performance evaluation system of shale gas PPP project based on matter element analysis. Math. Problems Eng. 2018, 1–18. doi:10.1155/2018/4657383

CrossRef Full Text | Google Scholar

Mladenovic, G., Vajdic, N., Wündsch, B., and Temeljotov-Salaj, A. (2013). Use of key performance indicators for PPP transport projects to meet stakeholders’ performance objectives. Built Environ. Proj. Asset Manag. 3 (2), 228–249. doi:10.1108/bepam-05-2012-0026

CrossRef Full Text | Google Scholar

Nam, W. H., Kim, T., Hong, E. M., Choi, J. Y., and Kim, J. T. (2017). A wireless sensor network (WSN) application for irrigation facilities management based on information and communication technologies (ICTs). Comput. And Electron. Agric. 143, 185–192. doi:10.1016/j.compag.2017.10.007

CrossRef Full Text | Google Scholar

Negishi, K., Tiruta-Barna, L., Schiopu, N., Lebert, A., and Chevalier, J. (2018). An operational methodology for applying dynamic Life Cycle Assessment to buildings. Build. Environ. 144, 611–621. doi:10.1016/j.buildenv.2018.09.005

CrossRef Full Text | Google Scholar

Noshin, S., Alam, M., Sadiq, S., and Mandokhail, S. J. (2018). Stability analysis of flood bunds: A study on geotechnical health evaluation of embankments. IOP Conference Series: Materials Science and Engineering, 414, Baleli, Quetta, Pakistan.

CrossRef Full Text | Google Scholar

Ouyang, Y., and Pedrycz, W. (2016). A new model for intuitionistic fuzzy multi-attributes decision making. Eur. J. Operational Res. 249 (2), 677–682. doi:10.1016/j.ejor.2015.08.043

CrossRef Full Text | Google Scholar

Oyedele, L. O. (2013). Avoiding performance failure payment deductions in PFI/PPP projects: Model of critical success factors. J. Perform. Constr. Facil. 27 (3), 283–294. doi:10.1061/(asce)cf.1943-5509.0000367

CrossRef Full Text | Google Scholar

Pajno, G. B., Caminiti, L., Salzano, G., Crisafulli, G., Aversa, T., Messina, M. F., et al. (2013). Comparison between two maintenance feeding regimens after successful cow's milk oral desensitization. Pediatr. Allergy Immunol. 24 (4), 376–381. doi:10.1111/pai.12077

PubMed Abstract | CrossRef Full Text | Google Scholar

Park, S. G., and Kurosawa, K. (2017). Evaluation of regeneration potential for an artificial larch forest using light condition, plant species diversity, and plant coverage and biomass of the forest understory. J. Fac. Agric. Kyushu Univ. 62 (1), 47–52. doi:10.5109/1798140

CrossRef Full Text | Google Scholar

Rashvand, P., and Zaimi, A. M. M. (2014). Critical criteria on client and customer satisfaction for the issue of performance measurement. J. Manage. Eng. 30 (1), 10–18. doi:10.1061/(asce)me.1943-5479.0000183

CrossRef Full Text | Google Scholar

Sahoo, M., Sahoo, S., Dhar, A., and Pradhan, B. (2016). Effectiveness evaluation of objective and subjective weighting methods for aquifer vulnerability assessment in urban context. J. Hydrology 541, 1303–1315. doi:10.1016/j.jhydrol.2016.08.035

CrossRef Full Text | Google Scholar

Sanvido, V., Grobler, F., Parfitt, K., Guvenis, M., and Coyle, M. (1992). Critical success factors for construction projects. J. Constr. Eng. Manag. 118 (1), 94–111. doi:10.1061/(asce)0733-9364(1992)118:1(94)

CrossRef Full Text | Google Scholar

Somanathan, E. (2013). Are embankments a good flood control strategy? A case study on the kosi river. WATER POLICY 15, 75–88. doi:10.2166/wp.2013.002

CrossRef Full Text | Google Scholar

Song, Y., Lee, J., Joo, J., and Park, M. (2017). A study on hydraulic experiment of capture efficiency for drainage reduction facilities in mountain and urban areas. Korean Soc. Hazard Mitig. 17 (3), 339–347. doi:10.9798/kosham.2017.17.3.339

CrossRef Full Text | Google Scholar

Song, J. B., Hu, Y. B., and Feng, Z. (2018). Factors influencing early termination of PPP projects in China. J. Manage. Eng. 34 (1). doi:10.1061/(ASCE)ME.1943-5479.0000572

CrossRef Full Text | Google Scholar

Song, Y., and Park, M. (2021). A study on the development of reduction facilities' management standards for agricultural drainage for disaster reduction. Sustainability 13 (17), 9595. doi:10.3390/su13179595

CrossRef Full Text | Google Scholar

Teng, F., and Liu, P. (2019). Multiple-Attribute group decision-making method based on the linguistic intuitionistic fuzzy density hybrid weighted averaging operator. Int. J. Fuzzy Syst. 21 (1), 213–231. doi:10.1007/s40815-018-0526-z

CrossRef Full Text | Google Scholar

Toor, S., and Ogunlana, S. O. (2010). Beyond the ‘iron triangle’: Stakeholder perception of key performance indicators (KPIs) for large-scale public sector development projects. Int. J. Proj. Manag. 28 (3), 228–236. doi:10.1016/j.ijproman.2009.05.005

CrossRef Full Text | Google Scholar

Wang, W. D., Guo, J., Fang, L. G., and Chang, X. S. (2012). A subjective and objective integrated weighting method for landslides susceptibility mapping based on GIS. Environ. Earth Sci. 65 (6), 1705–1714. doi:10.1007/s12665-011-1148-z

CrossRef Full Text | Google Scholar

Xu, Z., and Yager, R. (2006). Some geometric aggregation operators based on intuitionistic fuzzy sets. Int. J. General Syst. 35 (4), 417–433. doi:10.1080/03081070600574353

CrossRef Full Text | Google Scholar

Xu, L., and Tang, Q. (2021). Cold chain vulnerability assessment through two-stage grey comprehensive measurement of intuitionistic fuzzy entropy. K. 51, 694–714. ahead-of-print. doi:10.1108/K-02-2021-0161

CrossRef Full Text | Google Scholar

Yang, Y., Li, Q., and Yan, B. (2017). Specifications and applications of the technical code for monitoring of building and bridge structures in China. Adv. Mech. Eng. 9, 168781401668427. doi:10.1177/1687814016684272

CrossRef Full Text | Google Scholar

Ye, J. (2010). Two effective measures of intuitionistic fuzzy entropy. Computing 87 (1-2), 55–62. doi:10.1007/s00607-009-0075-2

CrossRef Full Text | Google Scholar

Yu, B., Zhang, J., Wei, S., Cong, G., Zhang, D., and Chen, T. (2007). “The research on information system project performance evaluation based on fuzzy neural network,”Shanghai, China, 6170–6174.Proceedings of the 2007 international conference on wireless communications, networking and mobile computing

CrossRef Full Text | Google Scholar

Yuan, J., Wang, C., Skibniewski, M. J., and Li, Q. (2012). Developing key performance indicators for public-private partnership projects: Questionnaire Survey and analysis. J. Manage. Eng. 28 (3), 252–264. doi:10.1061/(asce)me.1943-5479.0000113

CrossRef Full Text | Google Scholar

Yuan, J., Li, W., Zheng, X., and Skibniewski, M. J. (2018). Improving operation performance of public rental housing delivery by PPPs in China. J. Manage. Eng. 34 (4), 4018015. doi:10.1061/(asce)me.1943-5479.0000615

CrossRef Full Text | Google Scholar

Yuan, J., Zeng, A. Y., Skibniewski, M. J., and Li, Q. (2009). Selection of performance objectives and key performance indicators in public–private partnership projects to achieve value for money. Constr. Manag. Econ. 27 (3), 253–270. doi:10.1080/01446190902748705

CrossRef Full Text | Google Scholar

Yuan, J., Wang, C., Skibniewski, M. J., and Li, Q. (2012). Developing key performance indicators for public-private partnership projects: Questionnaire Survey and analysis. J. Manage. Eng. 28 (3), 252–264. doi:10.1061/(asce)me.1943-5479.0000113

CrossRef Full Text | Google Scholar

Zavadskas, E. K., Bausys, R., Juodagalviene, B., and Garnyte-Sapranaviciene, I. (2017). Model for residential house element and material selection by neutrosophic MULTIMOORA method. Eng. Appl. Artif. Intell. 64, 315–324. doi:10.1016/j.engappai.2017.06.020

CrossRef Full Text | Google Scholar

Zhao, H., You, J., and Liu, H. (2017). Failure mode and effect analysis using MULTIMOORA method with continuous weighted entropy under interval-valued intuitionistic fuzzy environment. Soft Comput. 21 (18), 5355–5367. doi:10.1007/s00500-016-2118-x

CrossRef Full Text | Google Scholar

Zhou, W. K., and Jiang, W. C. (2010). Two-phase TOPSIS of uncertain multi-attribute group decision-making. J. Syst. Eng. Electron. 21 (3), 423–430. doi:10.3969/j.issn.1004-4132.2010.03.012

CrossRef Full Text | Google Scholar

Zou, W. W., Kumaraswamy, M., Chung, J., and Wong, J. (2014). Identifying the critical success factors for relationship management in PPP projects. Int. J. Proj. Manag. 32 (2), 265–274. doi:10.1016/j.ijproman.2013.05.004

CrossRef Full Text | Google Scholar