The purpose of this study was to optimize the coagulation–flocculation effect of a wastewater treatment system using the response surface methodology (RSM) and three-step method to minimize phosphorus concentration in the distillate wastewater. In order to minimize the concentration of total phosphorus (TP), experiments were carried out using -factorial designs with three levels and three factors. A Box–Behnken design, which is the standard design of RSM, was used to evaluate the effects and interactions of three major factors (Fe:P (w/w) ratio, coagulation pH and fast mixing speed (FMS)) on the treatment efficiency. A multivariable quadratic model developed for studying the response indicated that the values for optimum conditions for Fe:P (w/w) ratio, coagulation pH and FMS were 2.40, 6.48 and 100 rev min−1, respectively. Under optimal process conditions, the TP concentration in the distillery effluent was reduced from 10 mg L−1 to 0.215 mg L−1, representing a removal efficiency of 97.85%. Based upon the statistical evaluation of results, it is inferred that RSM can be used as an appropriate approach to optimize the coag-flocculation process. Meanwhile, the study has shown that, for the equivalent dose of ferric chloride, the average three-step effect is better than that of the one-time addition.