The surface temperature and vegetation index (Ts/VI) method is a remote sensing-based quantitative approach. It is widely used for estimating the evapotranspiration, evaporative fraction, and surface soil moisture (SSM). However, this method can only be used in flat regions. In this study, we investigated the effect of altitude when using the Ts/VI method for estimating the SSM in heterogeneous regions. The results showed that the temperature vegetation dryness index (TVDI) method performed poorly at estimating the SSM in the source region of the Yangtze River; there was a weak correlation with the observed SSM, and R2 was only 0.167. However, the performance of the method improved considerably when the effects of both altitude and frozen soil were considered; the TVDI had a strong correlation with the observed SSM, and R2 improved to 0.617. In the study area, the effects of altitude on the TVDI values were greater than those of the frozen soil. In general, the Ts/VI method can obtain satisfactory results in mountainous regions if the effects of both the altitude and frozen soil are considered.