Muscle mass loss and worsening physical function are crucial issues in patients receiving hemodialysis (HD). However, few studies have investigated the association between temporal changes in muscle mass and physical function in a large number of HD patients. We examined 286 patients receiving HD (males, 58%; age, 66.8 ± 13.0 years) at a single center, and calculated the percent changes in psoas muscle mass index (%PMI) using computed tomography over two screenings, once per year (July 2011–June 2013). Physical function was evaluated using the Eastern Cooperative Oncology Group Performance Status (ECOG-PS) (range 0–4). The observation period was from July 2012 to June 2021. The median %PMI was -9.5%, and those with the lowest quartile of %PMI (< −20.5%) showed a significantly poor prognosis compared with other patients (p < 0.001). Multivariable logistic regression analysis revealed that these patients tended to have decreased physical function (ECOG-PS 2–4) [odds ratio (OR): 2.46, p < 0.001] and albumin levels (OR: 0.22, p = 0.007). Multiple-factor-adjusted Cox regression analyses showed that %PMI (hazard ratio: 0.99, p = 0.004) and each ECOG-PS stage (1–4 vs. 0) (p < 0.01) were associated with mortality. Augmenting physical activities in daily life and serum albumin levels should be considered to maintain muscle mass and improve the prognosis of patients receiving HD.