In order to develop bioretention fillers with better phosphorus removal capacity, we built 12 bioretention columns with six kinds of modified fillers, and analyzed the operation effects of the columns under different conditions through field tests. Results show that adding water treatment residual has optimal removal rates for total phosphorus (TP) (median = 96.80%) and soluble reactive phosphorus (SRP) (median = 97.13%). The water reduction rates of the columns with improved fillers are 1.23–2.04 times that of the bioretention soil media column. The coconut chaff column has the best water storage capacity (median = 40.54%). Among the external factors affecting column operation, influent concentration of pollutants in urban surface runoff is the biggest influence factor on the removal efficiency of TP. However, there are no significant correlations between the removal efficiency of SRP and rainfall, influent concentration, and discharge ratio. The columns modified with medical stone, vermiculite, peat soil, medical stone + peat soil, green zeolite + peat soil all have good removal for phosphorus pollutant. After entering the columns, the contents of TP and SRP in most columns increased. The recommended fillers and the accumulation performance of phosphorus can help to improve purification effects in bioretention systems.