This paper presented the current state of our understanding of the roles of carboxylic and phenolic groups in NOM adsorption and reviewed the contradictory opinions in the literatures. Previous studies carried out by other researchers indicated that aromatic carboxylates were adsorbed onto metal (hydr)oxides via outer-sphere complexes under most conditions and phenolic groups were very crucial for formation of inner-sphere complexes between organic acids and metal (hydr)oxides. Adsorption test with in-situ ATR-FTIR spectroscopic investigation were carried out to verify the role of aromatic carboxylic and phenolic groups in the NOM adsorption onto aluminium hydroxide surfaces by using a series of aromatic carboxylic acids and dihydroxybenzoic acids as the surrogate of NOM. Our studies suggested that the formation of outer-sphere complexes dominated the adsorption of most of the aromatic carboxylates over the pH range of 5–9; inner-sphere complexes were only detected at some pH levels for some aromatic carboxylates adsorption; and the aromatic carboxylates were most likely to be adsorbed to the first surface layer of hydroxyl groups and water molecules without forming coordinative bonds with the aluminium hydroxide surfaces but strong hydrogen bonds were formed in this process. Our study also revealed that (1) the presence of phenolic groups can increase the interaction strength of carboxylate groups with aluminium hydroxide; (2) chelate formation involving a carboxylate oxygen atom and ortho-phenolic-oxygen is important for the adsorption of organic matter on aluminium hydroxide at acidic pH; and 3) the phenolic groups adjacent to each other are more important than the carboxylic groups at alkaline pH for organic matter adsorption.