Swallowing function is associated with systemic factors. Whether trunk or appendicular skeletal muscle mass is a better indicator of swallowing-related muscle characteristics in community-dwelling older adults is not clear. Hence, we investigated the association between the characteristics of swallowing-related muscles (e.g., mass and quality) and trunk muscle mass. Community-dwelling older adults aged ≥ 65 years (n = 141; men: n = 45, women: n = 96) were recruited for this cross-sectional observational study via a health survey conducted in 2018. Trunk muscle mass index (TMI) and appendicular skeletal muscle mass index (SMI) were measured using bioelectrical impedance analysis. Cross-sectional areas (CSAs) and echo intensity (EI) of the geniohyoid muscle (GHM) and tongue were evaluated using an ultrasonic diagnostic apparatus. Multiple regression analysis was used to examine the relationship of the characteristics of swallowing-related muscle with TMI and SMI. Multiple regression analysis showed that CSA of the GHM was positively associated with both TMI (B = 24.9, p < 0.001) and SMI (B = 13.7, p = 0.002). EIs of swallowing-related muscles were not associated with TMI and SMI. Trunk muscle mass was associated with swallowing-related muscle mass and not muscle quality. The results of this study shed light on the elucidation of association of dysphagia with TMI and SMI.