To fully investigate the effectiveness of fillers in the removal of pollutants from rainwater, gravel, zeolite, slag, volcanic rock and iron filings with a 3–5 cm particle size were applied to construct a brick paving system with a frame structure for the removal of pollutants. Total suspended solids (TSS), chemical oxygen demand (COD), ammonia nitrogen (NH3-N), total nitrogen (TN), total phosphorus (TP) and heavy metals (Cu, Zn, and Pb) in the influent and effluent were measured, and the effectiveness and mechanism of pollutant removal were further investigated. The results showed that the permeable brick system effectively reduced TSS, TP, Zn, Cu and Pb and was relatively ineffective in reducing NH3-N, TN and COD. The removal results obtained using different materials show that (1) physical interception is the main reason for TSS and TP removal, (2) the adsorption and ion exchange properties of zeolite enable it to highly absorb ammonia nitrogen, (3) iron filings can effectively reduce NO3-N, and (4) adding fillers rich in iron oxide, such as volcanic rock or slag, can contribute to COD adsorption. The study provides a feasible technical path for improving the practicability of permeable pavement.