This study aimed to explain factors affecting health-related quality of life among community-dwelling elderly individuals living in Japan. We conducted a questionnaire survey of 898 individuals at senior citizens clubs in a rural area near the Tokyo metropolitan area in 2015, and analyzed the responses of 715 who provided valid responses. The survey questions included basic attributes (e.g., age, economic affluence), state of health (e.g., whether s/he has heart disease or not), the Spirituality Rating Scale Related to Health in the Elderly (SP Health Scale, composed of, e.g., meaning and purpose of living, self-transcendence), the Abbreviated Lubben Social Network Scale, and the MOS 8-Item Short Form Health Survey (SF-8). Multiple regression analyses were performed using the physical component summary (PCS) and mental component summary (MCS) of the SF-8 as dependent variables and all others as independent variables. Factors associated with improved PCS were economic affluence and SP Health Scale (meaning and purpose of living), while the factors associated with reduced PCS were age and state of health (heart disease, gastrointestinal disease, osteoarthritis, lower back pain, knee pain, and eligible for long-term care). Meanwhile, factors associated with improved MCS were economic affluence, while those that were associated with reduced MCS were state of health (cancer and headache) and SP Health Scale (self-transcendence). These results suggest the need to understand spirituality in addition to the subjective economic situation, age, and medical condition of elderly individuals in order to improve their physical and mental health.