The purpose of this study was to optimize the coagulation-flocculation effect of the wastewater treatment system using response surface method (RSM) and three-step method to minimize phosphorus concentration in the distillate wastewater. In order to minimize the concentration of total phosphorus (TP), experiments were carried out using 33-factorial designs with three levels and three factors. A Box-Behnken design, which is the standard design of RSM, was used to evaluate the effects and interactions of three major factors (Fe: P (w/w) ratio, coagulation pH and fast mixing speed (FMS)) on the treatment efficiency. Multivariable quadratic model developed for studying the response, indicated that the values for optimum conditions for Fe: P (w/w) ratio, coagulation pH and fast mixing speed (FMS) were 2.40, 6.48 and 100 r min−1, respectively. Under optimal process conditions, the total phosphorus concentration in the distillery effluent was reduced from 10 mg L−1 to 0.215 mg L−1, representing a removal efficiency of 97.85%. Based upon the statistical evaluation of results, it is inferred that RSM can be used as an appropriate approach to optimize the coag-flocculation process. Meanwhile, the study has shown that, for the equivalent dose of ferric chloride, the average three-step effect is better than that of the one-time addition.