The Development of a Short Psychotic Symptoms Questionnaire Using Data of a Two-Wave Epidemiological Study

For use in epidemiological surveys, short self-report questionnaires measuring psychotic symptoms were not available in 1995, when we started a multi-wave epidemiological study. This paper presents such an instrument, using the epidemiological data of the 1996 and 2000 waves of a longitudinal study. Relevant items from the Dutch Abbreviated MMPI and the Symptom Checklist were combined into one questionnaire, which was completed by a regional Dutch population sample of 1528 subjects in 1996 and 525 in 2000, with a common cohort of 496. Data from a client sample were used (n = 113) for validation purposes. Principal component analyses resulted in one general psychosis factor with sufficient reliability. Validity research indicated that the instrument measured the psychotic symptoms. Positive correlations were found with schizotypal personality and experienced disability. Correlations indicating both convergent and divergent validity were found in the Rorschach test, the Dutch Abbreviated MMPI and the MMPI-2. The instrument measures the prevalence of psychotic symptoms in the general population and can be used as a short self-report screener in epidemiological surveys. Combined with a disability screener, the proportion of the population suffering disabilitating effects of psychotic symptoms can be estimated. Further research is recommended to investigate the predictive value of the instrument concerning the prognosis of experiencing psychotic symptoms or showing onset of psychotic disorder. This offers perspectives for future use in primary health care to identify persons with psychotic liability or who need support in coping with psychotic symptoms.


Introduction
This study reports on the development of a short self-report questionnaire measuring psychotic symptoms for the use in epidemiological surveys.When we started a multi-wave epidemiological study in 1995, we found no self-report questionnaires measuring psychotic symptoms in the literature.
Although in the Netherlands, some self-report questionnaires exist that include psychotic-like items, they are never used as an independent indicator for the presence of psychotic symptoms in a population survey.The Dutch version of the Symptom Checklist (SCL-90; Arrindell & Ettema, 1986) contains psychotic-like items.They do not appear as a separate psychosis factor in a factor analysis, while they do in the American version (Derogatis, Lipman, & Covi, 1973).Analyses outside the US often do not replicate the American factor structure of the SCL-90 (Holi, 2003).Others report mounting evidence that the dimensionality of the SCL-90 is questionable (Clark & Friedman, 1983).
We needed a short questionnaire measuring psychotic symptoms because a time-and cost-effective instrument was warranted for use in public mental health surveys.We developed such an instrument because we felt the need to know what proportion of the population is prone to experience psychotic symptoms.While it is known that most persons experiencing psychotic symptoms do not become psychotic (Hanssen, Bak, Bijl, Vollebergh, & van Os, 2005), it is not known to what extent they are disabled by these experiences and in need of psycho-educative or other professional support.
The aim of this study is to develop a self-report instrument used simultaneous with a disability screener (van Kemenade & Hosman, submitted) that can identify the proportion of persons with psychotic symptoms who experience these symptoms as disabling.A second aim is for the self-report instrument to identify populations commonly at risk of developing psychotic symptoms.This instrument had to meet two criteria: 1) a short self-report questionnaire; 2) sufficient reliability and validity.

Sources of Psychotic Symptom Items
We composed a 15-item list intended to measure the occurrence of psychotic symptoms.We took these items from the "OVER" section (item 2 -4, appendix) of the Dutch version of the SCL-90 and from the "Severe Psychopathology" scale from the DAMMPI (items 1, 5 -15, appendix) and made them binary.We assumed that assembling these item sources could potentially constitute a valid and reliable questionnaire.

Samples
In 1996 and 2000, these items were included in an epidemiological survey (van Kemenade, 2002) in the semirural Dutch health region Western North Brabant (WNB).The items were completed by 1528 respondents in 1996 (wave 1) and by 525 in 2000 (wave 2), of whom 496 also participated in 1996 (cohort sample).After 2004, the items were completed by 113 persons referred for a psycho-diagnostic evaluation (32 suspected of criminal behaviour and 81 suffering occupational disability for psychological reasons; referred clients sample).Table 1 offers characteristics of the respondents from wave 1 and 2, the cohort sample and the referred clients sample.
In wave 1 and 2, males and younger persons were underrepresented and older persons overrepresented in comparison with the overall population of WNB in the same age range.

Research Design
The research design has four components: 1) factor analysis on selected psychotic-like items; 2) reliability analysis on the resulting scale; 3) provision of descriptive statistics; 4) validation strategy.
For both waves and the referred clients sample, a principal component factor analysis (PCA) without rotation was computed in order to assess whether the psychotic-like items constitute a common factor.
Reliability of the resulting scales was computed with Cronbach's alpha.Descriptive data on the distribution, mean, standard deviation, mode, median, kurtosis and skewness of the psychotic symptoms questionnaire (PSQ) scores were provided.
The validation strategy aimed to demonstrate convergent and divergent validity and construct validity as well.The construct validity was assessed by measuring the actual disability in relation to the number of psychotic symptoms reported by respondents and by measuring the PSQ scores of populations commonly at risk.When populations commonly at risk obtain higher scores on the PSQ we consider this as confirming its construct validity.

Instruments for Convergent and Divergent Validation of the PSQ
In the convergent and divergent validation study, all personality disorder scales of the Personality Disorder Questionnaire-Revised (PDQ-R; Hyler & Reider, 1984;Ouwersloot, van den Brink, de Boer, & Hoogduin, 1989) were used in wave 2. The Dutch version of the MMPI-2 (clinical scales; Derksen, Mey, Sloore, & Hellenbosch, 1995;Graham, 2000), the DAMMPI (five scales; Luteijn & Kok, 1985) and the Rorschach test (main index variables according to the "Comprehensive System", i.e. perceptual thinking index, depression index and coping deficit index, measuring psychotic proneness, depressive proneness and quality of coping mechanisms; Exner, 2003) were used in the referred clients sample.

Instruments for Construct Validation of the PSQ
Van Kemenade and Hosman designed a self-report disability questionnaire (SRDQ) measuring the amount of actual disability persons experience.Respondents indicate to what extent they experience somatic, psychological and social disability in the work, family life and leisure time environments, with each domain and environment representing a separate variable.The score is in the range 0 -27 or 0 -9 for each environment or domain.A total score > 4 or an environment or domain score > 1 is considered a moderate to severe disability.A positive correlation between measures of the SRDQ and the PSQ is considered to confirm the construct validity of the PSQ.
The following instruments and questions were used to relate PSQ scores to populations commonly at risk.Scores on the PSQ were compared with the amount of help-seeking behaviour for medical and mental reasons.Help-seeking behaviour for medical reasons was measured with yes/no questions after having visited a physician during the last three months, or having visited a medical specialist or a hospital during the last year.Regarding psychological complaints, respondents were first asked whether they ever had these.If so, whether this had occurred during the past year or before this past year.If they had had these complaints during the past year, they were asked whether they had consulted a physician and whether this physician had referred them to a specialized mental health service.
The demographic factors of gender, age, marital status, origin and level of education were taken into account.Substance use was measured by inquiring after the weekly consumption of alcohol and cannabis.Psychological morbidity indicators for general health (GHQ-12; Goldberg & William, 1988), anxiety (Cheung, 2002) and depression (Zung, 1965) were completed.Mental co-morbidity was determined by the number of above cut-off scores on the psychological morbidity indicators.
Social factors consisted of SES, loneliness as measured by the scale of de Jong-Gierveld (de Jong-Gierveld & Kamphuis, 1985), and the question of ever having been a victim of physical violence and/or sexual abuse.Measures were performed in both waves, except the question concerning victimization (only wave 2).

Factor and Reliability Analysis
By using the screen plot method the PCA-analyses on the PSQ-items in each of the three samples revealed one factor with a high eigenvalue (wave 1: Eigenvalue factor 1 and 2 = 3.30 and 1.37; wave 2: Eigenvalue factor 1 and 2 = 3.02 and 1.42; referred clients sample: Eigenvalue factor 1 and 2 = 4.88 and 1.50; Table 2).Over the three samples, the mean factor loading in 10 of the 15 items reached a value of .40 or more.These included the items 1 and 7 to 15 (see appendix).A closer inspection of the factor loadings in each of the three samples revealed a rather consistent pattern of high loadings for the mentioned 10 items and rather low loadings for the remaining 5 items (items 2 -6).The alpha of the 15 items is in the range .58-.81 (mean .67),and the alpha of the 10 items with a mean loading >.40 ranged from .63 to .82(mean .71).Repeating the PCA-analyses with the 10 items only resulted in an increase of the mean explained variance by the first unrotated factor from 25.9% to 33.9%.Therefore, further analyses in this study concentrate on the 10 items with a sufficient mean factor loading.These 10 items were selected to constitute the Short Psychotic Symptoms Questionnaire (SPSQ).They all originated from the DAMMPI and included none of the SCL-90 items.There were no indications that the SCL-90 items constitute a separate factor.
Between the measures of 1996 and 2000 (cohort; n = 496), a Pearson correlation of .47 for the SPSQ is found, indicating a fairly good stability.

Descriptive Statistics
Table 3 offers descriptive statistics for the SPSQ over three samples.
According to an independent samples t-test, wave 1 and 2 have comparable means (t = 1.49, p > .05).The referred clients sample has a significantly higher mean than both waves (t > 8, p < .001).A paired samples t-test among the cohort of 496 subjects reveals an increase of the SPSQ-score between 1996 and 2000 (t = −2.77,p = .006).

Validity
Convergent and divergent validities of the SPSQ were computed, using data of wave 2 and the referred clients sample.
The schizotypal personality disorder is considered a predisposition for schizophrenia.Vollema (1999) assumes a gradual difference between schizophrenia and schizotypy.Therefore a substantial correlation is expected between the SPSQ and the schizotypy score on the PDQ-R (Table 4).
The Pearson correlation between the schizotypal personality disorder and the SPSQ-score is .52.Correlations between the SPSQ and other personality disorders were found to be lower.
In the referred clients sample, Pearson correlations were computed between the SPSQ and the main index variables of the Rorschach test, the scales of the DAMMPI and the clinical scales of the Dutch version of the MMPI-2 (Table 5).The highest Pearson correlations between the MMPI-2 clinical scales and the SPSQ are found with MMPI scales measuring psychotic symptoms and psychological turmoil (psychasthenia).
A significant Pearson correlation is found between the perceptual thinking index in the Rorschach test and the SPSQ (n = 99, r = .23,p = .022).The other correlations are non-significant (r depression index = .01and r coping deficit index = .10).
The multitrait-multimethod matrix correlation (Campbell & Fiske, 1959) results obtained by the MMPI-2, the Rorschach test and the DAMMPI indicate that the SPSQ measures fall within the psychosis spectrum.In all tests the highest correlations were found with measures sensitive for psychotic phenomena.Lower correlations were found with measures sensitive for non-psychotic phenomena.This confirms the convergent and divergent validity of the SPSQ.

The Experience of Disabling Psychotic Symptoms
The more psychotic symptoms respondents report, the more disability in daily life they are expected to experience.This was measured with the SRDQ in wave 2.
In wave 2, 108 out of 525 respondents (20.6%) report having experienced at least one psychotic symptom.Of these respondents, 46.3% score more than 4 points on the total disability scale versus 26.7% of the respondents without psychotic symptoms.Regarding disability ascribed to mental causes (score > 1), the figures are 38.0%versus 14.1%.This equates with 41 out of 108 respondents with at least one psychotic symptom experiencing moderate to severe disability.This downsizes the percentage of 20.6% to 7.8%.This group of 41 respondents was checked on help-seeking behaviour for mental reasons, mental complaints and demographics (gender, age, marital status, level of education and SES-group).Help-seeking behaviour is defined as use of psychopharmaca, visiting a physician or a mental health centre.Along these variables it was determined whether respondents could be identified as in need of help and receiving help (group 1), in need of help and not receiving it (group 2), no need for help and receiving help (group 3), and, no need for help and not receiving it (group 4).Groups 1 -4 consisted of 18, 4, 1 and 18 respondents, respectively.
Concerning the cohort sample (n = 496), in 1996 14.9% reported at least one psychotic symptom, in 2000 21.0%, and 73.8% experienced psychotic symptoms in neither year, while 8.9% experienced at least one psychotic symptom in both years.Table 6 shows the results of these groups in terms of general disability and mental disability.
F-tests reveal that the group with psychotic symptoms in both waves experience more disability than the group without psychotic symptoms in either wave.The groups with psychotic symptoms in either wave 1 or in wave 2 reach disability scores in between (general disability: F (3, 492) = 18.8, p < .001);mental disability: (F (3, 492) = 13.9, p < .001).
In wave 1 an elevated risk of psychotic symptoms was found among males (t = 2.8, p = .005)and divorced respondents (F (4, 1510) = 8.9, p < .001),and in wave 2 only among divorced respondents (F (4, 520) = 14.5, p < .001).The number of psychotic symptoms was found to be unrelated with age, origin and level of education.
Concerning substance use, alcohol use was found to be unrelated with SPSQ-scores, while soft drug use is positively related.
Regarding psychological morbidity (score on the GHQ-12, anxiety, depression, psychological co-morbidity) and social factors (loneliness, victim of violence), we found positive correlations with the SPSQ in both samples.

Discussion
Concerning the content validity, it is concluded that the SPSQ measures psychotic symptoms.PCA shows that the SPSQ contains a general factor measuring psychotic symptoms.
A strong positive correlation with the schizotypal personality disorder was found, while the correlations with other personality disorders were substantially lower.Divergent and convergent validity emerged in the correlations between the SPSQ and scores in Rorschach indices and MMPI-2 and DAMMPI scales.A positive correlation with actual disability was also found.Moreover, it was found that higher scores on the SPSQ are positively related with the amount of medical and mental health consumption and that populations commonly at risk tend to score higher on the SPSQ than other populations.These findings confirm the construct validity of the SPSQ.
We conclude that the SPSQ has generated sufficient validity figures.
The SPSQ has sufficient reliability and a fairly good stability.Studies by other researchers are welcomed to see whether comparable and even better results can be obtained.
In epidemiological surveys, utilization of the SPSQ offers the possibility of charting the population regarding the prevalence of psychotic symptoms.Prevalence figures of psychotic symptoms in this study are consistent with other studies in this respect (Eaton, Romanoski, Anthony et al., 1999;van Os, Hanssen, Bijl et al., 2001;Tien, 1991;Verdoux & van Os, 2002).
This article represents a first step in the development of a psychotic symptoms screener.If a disability screener is used simultaneously, the proportion of persons with disabling psychotic symptoms can be discerned.In 2000, 108 out of 525 respondents reported having experienced at least one psychotic symptom (20.6%).Of this group, 41 respondents experienced moderate to severe disability for mental reasons (7.8%).Analysis of their help-seeking behaviour revealed 22 respondents (4.2%) in need of help, of whom 18 actually received help (3.4%).Unmet need was established in 4 respondents (0.8%).This compares with an unmet need rate of 400/22 = 18.2%.In other epidemiological studies, rates of unmet need with respect to psychotic symptoms are found of approximately 35% (Thornicroft, Johnson, Leese, & Slade, 2000;Wiersma, Nienhuis, Geel, & Slooff, 1998).
In the cohort study it is found that more and sustainable psychotic symptoms yield higher disability figures.We conclude that the SPSQ is useful in population surveys.
A logical next step would be to investigate the predictive validity of the SPSQ.Can it contribute to the prediction of psychosis?This can be accomplished by having the SPSQ completed in repeated measurements by persons suspected to be at risk of developing a psychosis, preferably in combination with other psychosis indicators (Yung & McGorry, 2004).Once its predictive value has been established with a reliable cut-off score, the SPSQ could become part of a set of risk indicators to be used in primary health care.

Conclusion
The SPSQ has generated sufficient validity and reliability figures.In longitudinal studies, the fairly good stability of the SPSQ is able to reveal changes in a population with respect to psychotic symptoms.When a disability screener is used simultaneously, it is found that more and sustainable psychotic symptoms yield higher disability figures.
These findings implicate that the SPSQ is useful in population surveys.

Limitations of This Study
The samples were not entirely representative for the population from which they were drawn.
The SPSQ study is a spin-off product of a multi-wave epidemiological investigation.
Note: Factor loadings of .40 or more are considered relevant in this study.

Table 4 .
Pearson correlations between the personality disorders according to the PDQ-R and the scores on the SPSQ in wave 2 (n = 525).
Note: P-values < .001,except for cells where a p-value is mentioned.

Table 5 .
Pearson correlations between the MMPI-2 clinical scales and the SPSQ in the referred clients sample (n = 28).

Table 6 .
Number of symptoms in the SPSQ in 1996 and 2000 (cohort sample; n = 496; cells contain absolute numbers and percentages and general [GD] and mental disability [MD] figures consisting of means and percentage scoring > 4 points on general disability and percentage scoring > 1 point on mental disability).

Table 7 .
(a) Medical consumption during last year and SPSQ scores in wave 1 and 2; means and standard deviations; (b) Psychological complaints, mental health service consumption (MHSC) during last year and SPSQ scores in wave 1 and 2; means and standard deviations; (c) Economic position and SPSQ scores in wave 1 and 2; means and standard deviations; (d) Demographic factors and SPSQ scores in wave 1 and 2; means and standard deviations; (e) Pearson correlations between substance use, psychological and social factors and the SPSQ in wave 1 and 2.