Using Cohen ’ s Effect Size to Identify Distinguishing Statements in Q-Methodology

Q-methodology was introduced more than 80 years ago to study subjective topics such as attitudes, perceptions, preferences, and feelings and there has not been much change in its statistical components since then. In Q-methodology, subjective topics are studied using a combination of qualitative and quantitative techniques. It involves development of a sample of statements and rank-ordering these statements by study participants using a grid known as Q-sort table. After completion of Q-sort tables by the participants, a by-person factor analysis (i.e., the factor analysis is performed on persons, not variables or traits) is used to analyze the data. Therefore, each factor represents a group of individuals with similar views, feelings, or preferences about the topic of the study. Then, each group (factor) is usually described by a set of statements, called distinguishing statements, or statements with high or low factor scores. In this article, we review one important statistical issue, i.e. the criteria for identifying distinguishing statements and provide a review of its mathematical calculation and statistical background. We show that the current approach for identifying distinguishing statements has no sound basis, which may result in erroneous findings and seems to be appropriate only when there are repeated evaluations of Q-sample from the same subjects. However, most Q-studies include independent subjects with no repeated evaluation. Finally, a new approach is suggested for identifying distinguishing statements based on Cohen’s effect size. We demonstrate the application of this new formula by applying the current and the suggested methods on a Q-dataset and explain the differences.


Introduction
Q-methodology was introduced in 1935 by William Stephenson [1] [2] to study subjective topics such as attitudes, perceptions, preferences, and feelings.In Q-methodology, subjective topics are studied using a combination of qualitative and quantitative techniques [3].It involves development of a sample of statements, called Q-sample, related to the topic of interest and rank-ordering these statements by a group of individuals, study participants, from their points of views or preferences using a grid (known as Q-sort table).This Q-sort table usually has a quasi-normal distribution (see Figure 1) and the completed data from a Q-sort table is known as Q-sort.After completion of Q-sort tables by the participants, a by-person factor analysis (i.e., the factor analysis is performed on persons, not variables or traits) is used to analyze the Q-sorts.Because each participant completes one Q-sort table, each Q-sort represents one individual rather than one variable or trait.Using a by-person factor analysis, similar Q-sorts (individuals) are grouped together as factors.Therefore, each factor represents a group of individuals with similar views, feelings, or preferences about the topic of the study.Statistically, one individual belongs to a factor if his/her factor loading on the factor is statistically significant (p ≤ 0.05).Next, each group (factor) is usually described by a set of statements, called distinguishing statements.
Although Q-methodology was introduced more than 80 years ago, there have not been significant advances in its statistical components since then.Some of these issues have already been discussed elsewhere [4] [5].In this article, we review another important statistical issue, i.e. the criteria for identifying distinguishing statements, and suggest appropriate changes.

Methods
In the current practice of Q-methodology, a statement is identified as a distinguishing statement for a factor a compared to any other factor x, if where p is number of Q-sorts loaded on the factor.
However, this formula adapted from Spearman does not seem to be appropriate in Q-methodology analysis because, first, Spearman's formula is for correlation of sums or differences of a group of variables on the same subjects, but in Q-methodology Q-sorts on different factors are usually from different subjects and independent from each other.Second, using formula (1) for margin of error is highly dependent on the number of Q-sorts loaded on each factor.Table 1 shows the margin of error for two factors with different number of Q-sorts loaded on each factor and r values based on formula (2).As can be seen, using Stephenson's formula, factors with smaller numbers of Q-sorts have a larger margin of error which results in smaller numbers of distinguishing statements.
On the other hand, use of a proper statistical test is not applicable for identifying distinguishing statements because in Q-methodology each statement is compared separately between factors.Therefore, from a statistical point of view, sample size is equal to unity, and the margin of error will be relatively large.In addition, in most cases factors are orthogonal and factor scores are supposed to be (at least in theory) independent from each other with a standard deviation (σ) equal to unity.Hence, the correct standard error for the difference between any two factor scores will be  A margin of error of 2.76 is very large and hardly any distinguishing statement can be found using this criterion.
To overcome the abovementioned problems, we suggest using Cohen's d (or Cohen's effect size), which is very popular in psychology and clinical sciences.
Cohen's d [8] which is defined as Mean Mean d σ − = is used to compare the means of two groups based on d as effect size.Cohen suggested an effect size of 0.2 as small, 0.5 as medium, and 0.8 as large.

Results
In this section, an example is provided where the salient viewpoints of a group of individuals who participated in several Q-methodology workshops, using marijuana legalization (ML) as an example, were explored and the distinguishing statements and consensus statements were identified.
Example: 1) Marijuana Legalization First, to assemble a group of statements for Q-sample the World Wide Web was searched for statements about ML, specifically, to get a sense of supportive and opposing views.We found more than 50 statements and after a review for similarities and differences, 19 representative statements were selected for Q-sample and a Q-sort table was developed (Figure 1).Next, 38 individuals who participated in different Q-methodology workshops completed the Q-sort table.
The raw data from these Q-sort tables were entered into Stata and qconvert program [9] was used to convert raw data into usable data for analysis by qfactor program [10].

2) Factor extraction and factor rotation
To identify distinguishing statements, we used both Stephenson's formula and Cohen's d criteria.Three factors were extracted using principal axis factoring and varimax rotation.The number of Q-sorts loaded on factors 1, 2, and 3 were 12, 8, and 9, respectively.
3) Distinguishing statements and consensus statements Using Stephenson's approach, based on formulas (1) & ( 2), the margin of error for identifying distinguishing statements for Factor 1 is 0.439 (in comparison with Factor 2) and 0.425 (in comparison with Factor 3).Based on these criteria, the distinguishing statements for Factor 1 are listed in Table 2. Also, distinguishing statements for Factor 1 based on Cohen's d = 0.80 are listed in Table 3.
As we can see in these two tables, because a larger number is set to differentiate between statements' scores based on Cohen's d, a smaller number of statements emerged as distinguishing.This pattern was observed for the other two factors as well (the statements are not provided).We also listed the consensus statements Open Journal of Applied Sciences By legalizing marijuana, there will be an increase in people using the drug and therefore a need to increase rehabilitation programs which will come at the cost of taxpayers and the government −1 −1 1

16
A decrease in the use of marijuana means fewer health risks such as slowed brain function or risk of cancer −1 0 2 14 By legalizing marijuana, more people will use the drug and as a result, become addicted and more families will become dysfunctional −3 −1 1 Score ranges from −3 to +3 and negative scores indicate disagreement.using Stephenson's formula and Cohen's d in Table 4. Consensus statements are defined as the statements whose scores are not statistically different between any two factors.Because we used a larger difference to identify distinguishing statements based on Cohen's effect size, there will be a larger number of statements that will not be statistically significant between factors.Hence, it is not surprising that based this approach, there are more consensus statements than Stephenson's formula.

Discussion
Q-methodology was introduced more than 80 years ago to study subjective issues using some statistical techniques.There has not been much change or improvement in the statistical component of Q-methodology since its introduction by Stephenson [1] [2] as its founding father.Only recently there have been some suggestions for using alternative techniques [4] [5] other than what has been traditionally available in Q-methodology.In this article, we critically reviewed an important component of Q-methodology, i.e. statistical criteria for identifying

margin of error or precision of estimate
. However, the point is how to calculate se2for each factor score.The current Figure1.Q-sort table with 19 cells.Open Journal of Applied Sciences

Table 1
. Margin of error for identifying distinguishing statements based on number of Q-sorts loaded on factors (p 1 and p 2 ) using Stephenson's formula.

Table 2 .
Distinguishing statements for Factor 1 based on Stephenson's formula.

Table 3 .
Distinguishing statements for factor 1 based on Cohen's d.

Table 4 .
Consensus statements based on Stephenson's formula and Cohen's d.Certain people are able to function only with the aid of marijuana.If it is more difficult for them to access the drug, they may be unable to perform effectively in society Score ranges from −3 to +3 and negative scores indicate disagreement.