Psychometric Properties of the Depression , Anxiety , Stress Scales-21 ( DASS-21 ) in a Greek Sample

The aim of this study is to evaluate the psychometric properties of the DASS-21 in a non-clinical sample of the Greek population. The Depression Anxiety Stress Scales 21 (DASS-21) is a self-report instrument measuring anxiety, depression and stress. The validation was carried out in a sample of 12,868 Greek adults, ranging from 18 to 65 years old. Results showed that the DASS21 has satisfactory reliability and validity indexes. Moreover, the factorial structure of the scale matches the ones found in previous studies in many countries. The results of this study suggest that the Greek DASS-21 can be used as a reliable and valid instrument for the measurement of depression, anxiety and stress in the Greek population.


Introduction
The Depression Anxiety Stress Scales 21 (DASS-21) is a self-report instrument measuring anxiety, depression and stress.It is the short form of Lovibond and Lovibond's (1995) 42-item measure which was developed to encompass the full range of anxiety and depression symptoms while providing maximum differentiation between the two constructs.During scale development, a new factor consisting of symptoms of nervous tension and irritability emerged.The new factor became the third scale of the instrument, measuring stress.
Each of the three DASS-21 scales comprises seven items which were selected in order to be representative and sum as close to half of the respective full-scale scores as possible (Henry & Crawford, 2005).The depression scale assesses symptoms such as dysphoria, hopelessness, self-worthlessness, and lack of interest, the Anxiety scale comprises items evaluating somatic symptoms, situational anxiety and the subjective experience of anxious affect, while the Stress scale appraises a condition of persistent arousal and tension which consists of symptoms such as difficulty relaxing, agitation, irritability and impatience (Lovibond & Lovibond, 1995).
The psychometric properties of the DASS-21 have been studied in a number of adult populations with good results.In general, studies have reported good internal consistency for the three scales and for the total score.In particular, alpha coefficients estimates range between 0.83 and 0.94 for the Depression scale, between 0.70 and 0.87 for the Anxiety scale, and between 0.82 and 0.91 for the Stress scale in clinical (Antony, Bieling, Cox, Enns, & Swinson, 1998;Bottesi et al., 2015;Clara, Cox, & Enns, 2001) and nonclinical samples (Bados, Solanas, & Andrés, 2005;Henry & Crawford, 2005;Norton, 2007;Osman et al., 2012;Sinclair et al., 2012;Wang et al., 2016) of different cultural contexts.Internal consistency for the total scale is less often reported, but where it is reported it ranges between 0.92 and 0.96 (Daza, Novy, Stanley, & Averill, 2002;Gloster et al., 2008;Henry & Crawford, 2005;Vasconcelos-Raposo, Fernandes, & Teixeira, 2013;Wang et al., 2016).
In terms of convergent and discriminant validity, the Depression and Anxiety scales of the DASS-21 have shown acceptable correlations with other respective measures of anxiety and depression.It has been reported that the depression scale highly correlates with the Beck Depression Inventory (BDI: Beck, Rush, Shaw, & Emery, 1979) and the anxiety scale correlates highly with the Beck Anxiety Inventory (BAI; Antony et al., 1998;Bados et al., 2005;Beck & Steer, 1990;Gloster et al., 2008;Norton, 2007) and the State-Trait Anxiety Inventory (STAI; Antony et al., 1998;Spielberger, 1983;Wang et al., 2016).Although there are also high correlations between the Depression scale and anxiety measures and between the Anxiety scale and depression measures, these correlations are lower than the correlation between similar constructs (Daza et al., 2002).The Stress scale has been shown to correlate both with similar measures and with anxiety measures revealing a broader symptom pattern and an overlapping of anxiety and stress (Alfonsson, Wallin, & Maathz, 2017;Bottesi et al., 2015).
The convergent and discriminant validity of the DASS-21 have also been evaluated with scales of positive and negative affectivity and quality of life measures.In terms of affectivity, studies have mostly used the Positive and Negative Affect Schedule (PANAS; Watson, Clark, & Tellegen, 1988).These studies have reported significant negative correlations of the DASS-21 scales with positive affectivity and significant positive correlations with negative affectivity, showing an acceptable convergent and discriminant validity (Bados et al, 2005;Gloster et al., 2008;Henry & Crawford, 2005;Norton, 2007).Studies that have examined the relationship of the DASS-21 with quality of life measures have confirmed C. Pezirkianidis et al. that these constructs are significantly and inversely correlated (Gloster et al., 2008;Tonsing, 2014).Τhe Depression and Anxiety scales of the DASS-21 have been shown to discriminate adequately between clinical and nonclinical populations as well as between diagnostic groups, showing good concurrent validity (Antony et al., 1998;Bados et al., 2005;Bottesi et al., 2015;Clara et al., 2001;Gloster et al., 2008).The validity of DASS-21 as a routine clinical outcome measure has also been tested and it has been shown that the scale is responsive to improvements in clinical status after treatment (Ng et al., 2007;Ronk, Korman, Hooke, & Page, 2013).
Several studies have evaluated the factor structure of the DASS-21 using both exploratory and confirmatory approaches with inconsistent results.In general some studies have confirmed the three-factor structure of the DASS-21 (Antony et al., 1998;Clara et al., 2001;Gloster et al., 2008;Sinclair et al., 2012), although some of them show an acceptable but not good enough model fit (Bados et al., 2005).Furthermore, the 21-item version has been shown to have a cleaner factor structure and smaller inter-factor correlations than the longer version (Antony et al., 1998), mainly because some items reducing the discriminant validity of the measure have been removed (Henry & Crawford, 2005).Some researchers, however, have found that a quadripartite structure consisting of a common general factor of overall psychological distress plus orthogonal depression, anxiety and stress factors provides better fit indices (Alfonsson et al., 2017;Bottesi et al., 2015;Henry & Crawford, 2005;Osman et al., 2012;Vasconcelos-Raposo et al., 2013).Other studies report revised three-factor structures where one or more items load on other than their designated scale (Wang et al., 2016).
The above research findings show that the DASS-21 is a reliable and valid scale for the measurement of depression, anxiety, and stress in clinical and nonclinical groups across different cultures (Alfonsson et al., 2017;Antony et al., 1998;Bottesi et al., 2015;Clara et al., 2001;Daza et al., 2002;Henry & Crawford, 2005;Norton, 2007;Vasconcelos-Raposo et al., 2013).Furthermore, the DASS-21 is an instrument easy to administer and suitable both for clinical and research purposes.Therefore, examination of its psychometric properties and applicability in a large Greek community sample is needed.
To date, the 42-item version of the DASS has been translated in Greek and validated in general population and a psychiatric patient Greek sample (Lyrakos, Arvaniti, Smyrnioti, & Kostopanagiotou, 2011).Results have shown that the scale has excellent internal validity with Cronbach's alpha coefficients ranging between 0.90 and 0.97 for the three scales and the total scale.Principal components analysis confirmed the three factor model of the scale, while the scale showed good convergent and discriminant validity.However, the psychometric properties of the Greek DASS-21 have yet to be validated.
The aim of the present study was to evaluate the psychometric properties of the DASS-21 in a large non-clinical sample of the Greek population.Specifically we sought to evaluate the internal consistency and explore the factorial structure of the Greek DASS-21.Furthermore, we aimed to examine the construct validity of the scale using a number of mental health and well-being measures that have not been used in other validation studies.We also sought to investigate possible differences among separate demographic groups (gender, age groups, marital status, education level, and employment status).Finally, we wanted to examine factorial invariance of the measure and whether there would be latent mean differences across gender.Last but not least, we focused on providing normative data for the total sample and separately for men and women.

Participants
The sample of the study consisted of 12,868 Greek adults (39.3% men, 60.7% women), aging from 18 to 65 years old.The majority of our sample was employed (83.1% employed, 16.9% unemployed) and university graduates (34.3% school graduates, 10.7% university students, 41.7% university graduates, 12.8% postgraduates).Regarding the marital status of the respondents, 48.1% of them were married, 42.8% were unmarried, 6.3% were divorced and 2% were widowers.The majority of the participants had no children (51.5%; see Table 1).

Procedure
The present data are taken form a data bank of an ongoing longitudinal study, which started in 2008 in Greece, examining the effects of the economic crisis on the psychological health of the Greek population in relation to several variables including aspects of mental health.The data of the present study were collected with the help of undergraduate psychology students, who volunteered to administer the battery of tests.The volunteers were trained on the distribution, administration and collection of the questionnaires.Administration was done individually and was completed in approximately 20 -25 minutes.The data were recorded on answer sheets and scanned using the 6 th Version of Remark Office OMR.
As far as the validation is concerned we followed the next steps.
First of all, normality was tested by examining the distributional indices for each item of the DASS-21.Cohen, Cohen, West, and Aiken (2002) suggest cut-off scores of less than 2 for skewness and less than 7 for kurtosis.We then examined the Pearson r intercorrelations of the DASS-21 items.Correlations between 0.20 and 0.40 would indicate reasonable item homogeneity.Correlations less than 0.20 would be indicative of items that load at different factors and higher than 0.40 would indicate that the two items do not capture a big width of variance of the specific factor, in which they load (Piedmont, 2014).
To test the predictive model in which DASS-21 items fit, we conducted a confirmatory factor analysis (CFA) using the IBM SPSS AMOS, version 21.Thus, we created a model based on Lovibond's and Lovibond's (1995) model of three negative emotional states: depression, anxiety and stress consisting of seven items.Preliminarily, Mardia's test of multivariate normality and Mahalanobis d-squared statistic were examined (Hair, Black, Babin, & Aderson, 2010).Different fit indices were used to assess model fit: ×2 ratio (×2/degrees of freedom), the root mean-square error of approximation (RMSEA; Browne & Kudeck, 1993), the standardized root mean-square residual (SRMR; Hu & Bentler, 1995), the comparative fit index (CFI; Bentler, 1990), and the Tucker-Lewis index (TLI; Tucker & Lewis, 1973).Cut-off scores are based on the suggestions of Hu and Bentler (1999) for model fit: ×2 values less than 3, RMSEA and SRMR values less than 0.08, and CFI and TLI values higher than 0.90 indicate acceptable model fit.
We evaluated the internal consistency of the DASS-21 factors using the Cronbach's alpha and the Spearman-Brown prophecy coefficients.Values higher than 0.70 indicate good internal consistency (DeVellis, 2012;Kyriazos, 2017).To further examine scale's construct validity, we examined convergent and discriminant validity by testing the correlations between other measures and scale's factor scores.Positive correlations with similar constructs would be indicative of convergent validity and negative or non-correlations with totally different con-structs would indicate discriminant validity.
We also examined possible differences among the means of independent samples (created by the demographics) in DASS-21 factors.We tested configural, metric and scalar invariance of the DASS-21 across gender using IBM SPSS AMOS, version 21.The evaluation of the configural invariance was done by testing the model fit, when groups (males and females) do not have any cross-group constrains (Kline, 2010).We then tested metric and scalar invariance by comparing the CFI and RMSEA values of different invariance types (Cheung & Rensvold, 2002).
Using the IBM SPSS AMOS, version 21, we examined the latent mean differences of the model across males and females to specify if there is a need to compute different norms for the two samples (Byrne, 2013).Finally, we computed norms (sten scores) for the DASS-21 factors using the statistical program Stanscore 4.

Measures
Depression, Anxiety, Stress Scale.The Greek version of the Depression Anxiety and Stress Scale (DASS; Lovibond & Lovibond, 1995) consists of 21 items, which are measuring three emotional states: depression (e.g., "I couldn't seem to experience any positive feeling at all"), anxiety (e.g., "I found myself in situations that made me so anxious I was most relieved when they ended") and stress (e.g., "I found it difficult to relax").Each item was rated from 1 (Did Not Apply to Me At All) to 4 (Applied to Me Very Much or Most of the Time).In our sample (N = 12.868), the three subscales demonstrated high internal consistency (α = 0.85, 0.84 and 0.84, respectively).Modified Differential Emotional Scale.The mDES (Fredrickson et al., 2003) asks participants to recall the past 2 weeks and rate their strongest experience of each of 20 specific emotions on a 5 point Likert scale, ranging from 1 (Not At All) to 5 (Extremely).We used the Greek version of the instrument (Galanakis, Stalikas, Pezirkianidis, & Karakasidou, 2016).The Positive Emotions subscale is a composite of nine positive emotions (all but awe), with coefficient α = 0.86.The Negative Emotions subscale is a composite of 7 negative emotions (all but embarrassment), with coefficient α = 0.78.
Presence of Meaning in Life.The subscale of Meaning in Life Questionnaire (MLQ) chose to measure the presence of meaning using five items rated on a 7-point Likert-type scale ranging from 1 (Absolutely True) to 7 (Absolutely Untrue).We used the GREEK version of the instrument (Pezirkianidis, Galanakis, Karakasidou, & Stalikas, 2016).In our sample the subscale demonstrated good  1985) measures individual's tendency to believe that he/she will experience good or bad outcomes in his/her life using eight items rated on a 5-point Likert scale ranging from "Totally Disagree" to "Totally Agree".Each subscale consists of four items.The subscale of Optimism intends to capture the extent to which participants believe that good things will happen to them (e.g., "In Uncertain times, I usually expect the best"), whereas the subscale of Pessimism intends to capture the extent to which participants expect that bad outcomes will occur in their future (e.g., "If something can go wrong for me, it will").Moreover, a total score can be computed.We used the Greek version of the instrument (Moustaki & Stalikas, 2012), which demonstrated mediocre internal consistency (α = 0.60).
Gratitude Questionnaire.The Gratitude Questionnaire is a six-item, self-report questionnaire designed to assess individual differences in the proneness to experience gratitude in daily life (McCullough, Emmons, & Tsang, 2002).
Each item is rated on a 7-point Likert scale which indicate how much an individual agree with each statement (1 = strongly disagree -7 = strongly agree).We used the Greek version of the scale, which demonstrated good internal consistency (α = 0.66).
Economic Crisis Effects.We used a 10-item scale in order to assess the degree that participants were affected, at a practical and a psychological level, from the economic crisis.Each item was rated from 1 (Not At All) to 5 (Very Much).In our sample the two subscales (practical and psychological subscale) demonstrated satisfying internal consistency (Cronbach's alphas 0.71 and 0.84, respectively).

Normality Testing
We computed the distributional indices for the items of the DASS-21 (see Table 2).Item means were ranged between .037 and 1.26 in a 4-point Likert-type scale (0 to 3).Skewness and kurtosis values were less than 2 for every item except for item 21, whose skewness value was 2.05 and kurtosis 3.49.The normality indices of item 21 are marginally acceptable based on Cohen, Cohen, West, and Aiken (2003).

Inter-Item Correlations
We then examined the correlations between the DASS-21 items (see Table 3).The results showed that the correlations range from 0.23 to 0.56, which are acceptable.However, even though the correlations between 0.20 and 0.40 are indicative of high item redundancy, correlations among 0.40 and 0.60 indicate that these items capture a smaller width of the factor variance.

Confirmatory Factor Analysis
We firstly created the three-factor model using AMOS that was identifiable.We then examined the multivariate normality of the DASS-21 items using Mardia's test, whose value was 179.490, less than 483, which means that multinormality is achieved.Following the Mahalanobis d-squared cut-off score of 100, we deleted five outliers and we then tested the model fit; the extraction method was the maximum likelihood.The evaluation of the model fit indices showed that x 2 ratio was higher than 3 due to the big sample size, however, CFI and TLI values were higher than 0.90, while RMSEA and SRMR values were less than 0.08.
These values are indicative of a good model fit (see Table 4 and Figure 1).

Internal Consistency Reliability
We evaluated the Cronbach alpha and Spearman-Brown coefficients in order to examine the internal consistency of the DASS-21.The depression, anxiety and stress scales demonstrated good internal consistency.In particular, their Cronbach alphas were 0.85, 0.84, and 0.84 respectively, while their Spearman-Brown coefficients were 0.84, 0.83, and 0.85.
Table 4. Fit indices in confirmatory factor analysis for the DASS model.

Convergent and Discriminant Validity
To test convergent and discriminant validity, we examined the correlations between the DASS-21 factors and the constructs that measure other scales (see Table 5).The results show that each of the three scales, depression, anxiety, and stress correlate positively to anxiety, stress, economic crisis effects, and negative emotions and negatively to psychological resilience, optimism, hope, positive emotions, satisfaction with life, subjective happiness, presence of meaning in life, and gratitude; the correlations are on the direction expected, which is indicative of high convergent and discriminant validity of the three scales.

Mean Differences
We examined the mean differences of separate samples created by gender, age groups, marital status, education level and employment status.The results show that each of the three DASS-21 scales demonstrate statistically significant differences across every demographic information, which indicates that men and women, samples with different educational, marital or employment status and different age groups score significantly in a different way in depression, anxiety and stress; the samples' means are not equal (see Table 6).

Factorial Invariance
We examined the factorial invariance across gender of the DASS-21 three-factor model created in AMOS.Regarding the evaluation of the configural invariance existence, the results show that the data fit adequately to the model without any cross-groups constrains.More specifically, the fit indices indicate acceptable model fit, since CFI and TLI are higher than 0.90, and RMSEA and SRMR are lower than 0.08 (see Table 7).We then tested if there is metric invariance, which evaluates if the factor loadings are equivalent for men and women.The difference between CFI and RMSEA of configural and metric invariance was 0.000 and 0.001 respectively, which is less than 0.01 and indicative of metric invariance achievement.We also evaluated scale's scalar invariance, the possibility indicator    (Antony et al., 1998;Clara et al., 2001;Gloster et al., 2008;Sinclair et al., 2012).Furthermore, the examination of measurement invariance across gender showed adequate configural, metric and scalar invariance of the instrument.
These findings indicate that the Greek DASS-21 measures anxiety, depression and stress meaningfully and with the same structure, both in the total sample and across women and men.
In terms of convergent and discriminant validity we investigated the correlations between the DASS-21 scales and positive and negative emotions, satisfaction with life, meaning, subjective happiness, resilience, hope, optimism, gratitude and the effects of the economic crisis.The relationship of the depression, anxiety and stress scales with most of these constructs, to the best of our knowledge, has not been examined in other validation studies.Our results showed that the DASS-21 scales correlate positively to the economic crisis effects and negative emotions, while they demonstrate negative significant correlations to the positive constructs examined.These findings confirm that there is high convergent and discriminant validity of the three constructs and are in line with other studies that have examined correlations between the DASS-21 scales and measures of positive and negative affectivity and quality of life (Bados et al, 2005;Gloster et al., 2008;Henry & Crawford, 2005;Norton, 2007;Tonsing, 2014).
Examination of the mean differences between separate demographic groups demonstrated that people who belong to different gender, age, marital status, education level, and employment status groups demonstrate significantly different scores in depression, anxiety, and stress.Furthermore, examination of latent differences between genders showed that Greek women score significantly higher on all three scales than Greek men.Validation research in other countries has shown inconsistent results with some studies also reporting higher scores of depression, anxiety and stress in women (Apóstolo, Tanner, & Arfken, 2012), while others reveal no gender difference in any of the scale scores (Bottesi et al., 2015) or minimal influence of any demographic variables on DASS-21 scores (Henry & Crawford, 2005).Interestingly, Wang et al. (2016) in their validation study in China found that males scored significantly higher on the depression scale than females.We suggest that these inconsistencies might be explained by cultural differences.Nevertheless, the different scores between men and women in the present study led to the creation of different norms for the two genders.

C
. Pezirkianidis et al. internal consistency (α = 0.75).Subjective Happiness Scale.The Subjective Happiness Scale (SHS; Lyubomirsky & Lepper, 1999) was used to examine the subjectivity of the participants' global happiness.The scale uses four items rated on a 7-point Likert scale (1 "Not a very happy person" to 7 "A very happy person") with higher scores reflecting greater happiness (e.g., "Some people are generally very happy.They enjoy life regardless of what is going on, getting the most out of everything.To what extent does this characterization describe you?").We used the Greek version of the scale (Karakasidou, Pezirkianidis, Stalikas, & Galanakis, 2016) which demonstrated adequate internal consistency (α = 0.77).Connor-Davidson Resilience Scale.The Connor-Davidson Resilience scale (CD-RISC; Connor & Davidson, 2003) is a self-report measure, which is comprised of 25 items, each rated on a 5-point Likert scale, as follows: not true at all (0), rarely true (1), sometimes true (2), often true (3), and true nearly all of the time (4).The total score ranges from 0 -100, with higher scores reflecting greater resilience (e.g., "Can handle unpleasant feelings").This scale has sound psychometric properties and distinguishes between those with greater and lesser resilience.We used the Greek version of the scale (Dimitriadou & Stalikas, 2012), which demonstrated good internal consistency in our sample (α = 0.88).Hope Scale.The Greek version of the Hope Scale (HS; Snyder et al., 1991; Moustaki & Stalikas, 2012a) was used to measure individuals' sense of successful goal-directed determination and planning of ways to meet goals using eight items rated on a 4-point Likert scale ranging from "Definitely False" to "Definitely True".In our sample, the scale demonstrated good internal consistency (α = 0.86).Life Orientation Test.The Life Orientation Test (LOT; Scheier & Carver, **p-value < 0.001, n = sample size, p-value = value of statistical significance.df = degrees of freedom, CFI = comparative fit index, TLI = Tucker-Lewis index, RMSEA = root mean square error of approximation, SRMR = standardized root mean square residual.Extraction method: maximum likelihood.Skewness and kurtosis of the items were less than 2, except from item 21 (skewness = 2.05, kurtosis = 3.49).Mardia's test of multinormality = 179.490< p(p + 2) = 21(23) = 483.Five outliers were deleted since they had Mahalanobis d-squared > 100.

Figure 1 .
Figure 1.Standardized solution of the three-factor model of the Greek version of DASS-21.
Note: Every correlation is significant at p-value < 0.001.n = sample size, p-value = value of statistical significance.

Table 5 .
Convergent and discriminant validity: Average correlations of DASS-21 factors with other constructs.: D = depression scale, A = anxiety scale, S = stress scale.Every correlation is significant at p-value < 0.001.n = sample size, p-value = value of statistical significance.

Table 6 .
Mean differences (means and standard deviations) of DASS-21 factors by demographics.
Note: D = depression scale, A = anxiety scale, S = stress scale.Every difference is significant at p-value < 0.001.p-value = value of statistical significance, t = t-statistic in independent samples t-test, F = F-statistic in one-way analysis of variance (ANOVA), df = degrees of freedom.

Table 8 .
Latent mean differences of the DASS model across gender.

Table 9 .
Norms for the DASS factors.