Untreated combined sewer overflows (CSOs) cause serious water pollution problems. In this study, the effects of CSO-induced heavy metals and the remediation practice of installation of a long-term water diversion (LTWD) on the microbial environment in river sediments were analyzed in an inland river. The Zn, Cd, Cr, and Cu contents in sediments and water were analyzed. DNA extraction and polymerase chain reaction analysis were conducted based on the Illumina MiSeq platform. The results showed that CSOs have a significant adverse impact on the diversity of microbial populations in river sediments. The LTWD is helpful in improving the richness of microorganisms and the proportion of Gram −ves, but it is challenging to reduce the accumulation of heavy metals in the sediment. The correlation analysis shows a strong relationship between some metabolic pathways and Zn and Cd accumulation in river sediments. Some detoxification compound metabolisms are also promoted at these sites. Thus, chronic exposure to environmental heavy metals from CSOs decreases the river microbial community, and further affects the ecological environment of the river. Therefore, without eliminating CSOs or reducing overflow frequency, it is difficult to alleviate the accumulation of heavy metals in river sediments and improve river ecology via water diversion alone.