TITLE:
Testing Rating Scale Unidimensionality Using the Principal Component Analysis (PCA)/t-Test Protocol with the Rasch Model: The Primacy of Theory over Statistics
AUTHORS:
Peter Hagell
KEYWORDS:
Confidence Intervals, Dimensionality, Psychometrics, Rasch Model, Validity
JOURNAL NAME:
Open Journal of Statistics,
Vol.4 No.6,
August
27,
2014
ABSTRACT:
Psychometric
theory requires unidimensionality (i.e.,
scale items should represent a common latent variable). One advocated approach
to test unidimensionality within the Rasch model is to identify two item sets
from a Principal Component Analysis (PCA) of residuals, estimate separate
person measures based on the two item sets, compare the two estimates on a
person-by-person basis using t-tests
and determine the number of cases that differ significantly at the 0.05-level;
if ≤5% of tests are significant, or the lower bound of a binomial 95%
confidence interval (CI) of the observed proportion overlaps 5%, then it is
suggested that strict unidimensionality can be inferred; otherwise the scale is
multidimensional. Given its proposed significance and potential implications,
this procedure needs detailed scrutiny. This paper explores the impact of
sample size and method of estimating the 95% binomial CI upon conclusions
according to recommended conventions. Normal approximation, “exact”, Wilson,
Agresti-Coull, and Jeffreys binomial CIs were calculated for observed
proportions of 0.06, 0.08 and 0.10 and sample sizes from n= 100 to n= 2500.
Lower 95%CI boundaries were inspected regarding coverage of the 5% threshold.
Results showed that all binomial 95% CIs included as well as excluded 5% as an
effect of sample size for all three investigated proportions, except for the
Wilson, Agresti-Coull, and JeffreysCIs, which did not include 5% for any sample
size with a 10% observed proportion. The normal approximation CI was most
sensitive to sample size. These data illustrate that the PCA/t-test protocol should be used and
interpreted as any hypothesis testing procedure and is dependent on sample size
as well as binomial CI estimation procedure. The PCA/t-test protocol should not be viewed as a “definite” test of
unidimensionality and does not replace an integrated quantitative/qualitative
interpretation based on an explicit variable definition in view of the
perspective, context and purpose of measurement.