Olive trees are a major source of agricultural residues. Strategies based on different management of organic amendments have been reported to increase soil fertility. The effect of sustainable organic matter input practices (application of shredded pruning residue and olive residue compost to soil) on soil properties in irrigated and rain-fed olive groves was investigated. The study took place in 40 olive groves in the region of Peza, island of Crete, Greece during a 5-year period (2012–2017). The results showed that olive trees play an important role in soil nutrient conservation under semi-arid conditions in the Mediterranean basin. The addition of olive tree residues, in combination with conservation tillage practices, improved soil fertility over the experimental period. Most of the soil properties were favored by irrigation. In olive soil parcels receiving organic materials the soil organic matter and the total nitrogen were increased in irrigated fields. The ability of surface soil to sequester carbon and nutrients beneath the tree canopy of olive groves was high. It is recommended that sustainable soil management practices should consider soil fertility variability of olive orchards.