Water distribution systems (WDSs) are vital to human survival and the economic prosperity of communities across the globe. The deteriorating infrastructure issues combined with rising number of main breaks is pushing water utilities to keep up with the growing supply reliability challenges. Given the complexity associated with quantifying supply reliability, both of conceptual and computational nature, several surrogate measures which are referred to as resilience metrics were developed and used in the past. This paper presents a comparative evaluation of five such resilience metrics using the minimum cut set reliability approach supported by pressure-driven demand analyses of WDSs. Estimated reliability measures of WDS design solutions obtained using resilience metrics as co-objectives along with cost form the basis for the comparative evaluation presented in this paper. Three benchmark WDSs of different configurations and sizes are used in this study. The results suggest that the network resilience index performed best in the low cost range for smaller networks while the newly proposed probabilistic resilience index performed best in the low to moderate cost range for all networks. The identification of most competent resilience metric will support optimal design and rehabilitation decision making for water distribution systems in a computationally efficient manner.