Test-retest Reliability, Convergent Validity and Practice Effects of the Rbans in a Memory Clinic Setting: a Pilot Study

This pilot study examined the psychometric properties and clinical utility of a brief neuropsychological instrument (Repeatable Battery for the Assessment of Neuropsychological Status (RBANS). The test-retest reliability, practice effects and convergent validity of RBANS were examined in participants without objective cognitive impairment. The tests were administered at two time points at approximately a two weeks' interval, with 30 cognitively intact participants with a mean age of 63.3 ± 5.8 years. Adequate test-retest reliabilities were found for RBANS subtests, index and total scale scores with significant gain scores in immediate memory and visuospatial function. The RBANS showed good convergent validity and the RBANS supplemented with executive and language measures (Colour Trails Test and 30-item modified Boston Naming Test, respectively) demonstrated excellent convergent validity with a formal neuro-psychological battery. This pilot study has provided the preliminary evidence of reliability and convergent validity of the RBANS. Additionally, it also provides insight on the practice effects so that clinicians may assess significant changes in RBANS subtests and domain indexes for clinical practice.


Introduction
The prevalence of dementia is estimated to be 35.6 mil-lion patients in 2010 and is expected to double every 20 years [1], thereby creating increasing demand for an efficient, reliable and valid instrument for the early detection of age-related cognitive impairments.This instrument should have the following characteristics: 1) brevity-elderly people with cognitive impairment may only co-operate well with cognitive testing within a short span of time [2]; 2) test-retest reliability-repeated cognitive testing is often required to clarify diagnosis or to monitor progression of symptoms [3]; and 3) demonstrates convergent validity with a formal neuropsychological battery, which covers a range of cognitive domains required for the diagnosis of dementia.
The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) [4] has demonstrated its reliabilities and validities in the western population and may be considered for the detection of dementia in Sin-gapore.The RBANS is brief and takes approximately 30 minutes [5].It assesses a total of five cognitive domains with 12 subtests [4].RBANS has shown sensitivity of 84% and specificity of 97% for cognitive impairment in Alzheimer's disease (AD) [6].However, RBANS was limited in detecting Mild Cognitive Impairment (MCI), as the sum of index scores and six of the subtests (list learning, semantic fluency, coding, list recall, story recall and figure recall) had poor sensitivity (range: 8 -24%) in detecting MCI [7].Also, the lack of adequate executive function measures and object naming tasks [8] has been criticized as the drawback of RBANS for differential diagnosis of cognitive impairment in a clinical setting.Therefore, in the current study, we supplemented RBANS with the Color Trails Test (CTT) [9] and the 30-item modified Boston Naming Test (mBNT) [10] to address these drawbacks.
Information on test-retest reliability of the RBANS is limited.Test-retest reliability is first established using cognitively normal participants [11].A short span of time interval (under one month) minimizes the degree to which individuals will go through any physical or psychological changes and therefore is more accurate to assess the reliability of the cognitive instrument [12].However, short intervals may introduce practice effects.For the RBANS, studies with heterogeneous samples with and without cognitive impairment have reported test-retest reliability of 0.96 over one month in China [13], and of 0.72 over a three month interval in Brazil [14].Pat-ton et al. [15] reported stable scores over one-year and two-year intervals but test-retest reliability coefficients were not reported.Therefore, this study will evaluate the test-retest reliability of the RBANS and examine its practice effect over a two-week period in Singaporean older adult participants with no objective cognitive impairment (NCI).In addition, the convergent validity of the RBANS will be assessed by the correlation with a locally validated formal neuropsychological battery for Singaporean elderly [16].
The hypotheses of this pilot study were: 1) RBANS and RBANS supplemented with CTT and m-BNT would demonstrate good test-retest reliability over a two-week period; 2) There would be evident practice effects reflected by the gain scores of the individual subtest scores and domain index scores; and 3) RBANS and RBANS supplemented with CTT and m-BNT would demonstrate good convergent validity with a formal neuropsychological battery.

Participants
We recruited 30 participants with no objective cognitive impairment determined by a formal neuropsychological battery.Of these, 7 were recruited from a memory clinic at the National University Health System (NUHS) in Singapore and a further 23 were volunteers from the community.Inclusion criteria included a minimum age of 50 years and absence of major psychiatric or physical illness and/ or sensory impairment.Two participants delayed their second assessment for 114 days and were therefore excluded from the analyses.This study was approved by National Healthcare Group Domain-Specific Review Board (DSRB), and was conducted in conformity with the Declaration of Helsinki.Written informed consent was obtained from all participants.

Measures
All subtests of the RBANS were scored according to standardized criteria except for Figure Copy and Figure Recall, which were scored according to Duff et al.'s modified criteria [17].Raw scores of the 12 subtests were converted to age and education adjusted scaled scores [18] before calculating an Index score for each domain.The Sum of Index score was then converted to a Total Scale score, with a mean of 100 and standard deviation of 15.The Color Trails Test (CTT) [9,18] and 30-item modified Boston Naming Test [10] were administered as supplementary tests to the RBANS [8].
In addition, routine brief cognitive screening measures, the Abbreviated Mental Test (AMT) [30], the Mini-Mental State Examination (MMSE) [31] and the Montreal Cognitive Assessment (MoCA) [32] were ad-ministered to all participants.The 15-item Geriatric De-pression Scale (GDS [33]) was administered to screen for depression.

Procedure
The assessments and re-testing were administered by trained research psychologists in accordance with the manual over approximately two weeks.

Statistical Analysis
The descriptive statistics were used to characterize the sample's demographic and clinical profile.Pearson's correlation coefficients were calculated between Time 1 and Time 2 to examine test-retest reliability and convergent validity of the cognitive tests.Practice effects were expressed as effect sizes and calculated by the change in raw subtest scores, index scores and total scale scores between time 1 and time 2 divided by the standard deviation of time 1 scores [34].This method has been commonly used [35,36].

Sample Characteristics
Participants had a mean age of 63.3 ± 5.8 years with 12.3 ± 4.9 years of education (see Table 1).Most were Chinese (96.4%) females (67.9%).Participants' mean scores of brief cognitive screening tests (AMT, MMSE and MoCA) were within normal range.No participants reported significant symptoms of depression (GDS score: 1.39 ± 1.62).The mean test-retest interval was approximately two weeks (15.64 ± 4.72 days).

Test-retest Reliability of the RBANS
The means and standard deviations for the RBANS at each time point, along with the Pearson's correlations are presented in Table 2. Significant correlations were found between times 1 and time 2 raw scores for all subtests except list recall.Further examination of the list recall subtest revealed three outliers.Excluding these outliers led to significant correlations between list recall scores at time 1 and time 2 (r = .608,p = .001).The additional two tests used to supplement the RBANS (CTT and mBNT) also demonstrated adequate test-retest reliability (see Table 2).

Gain Scores and Practice Effects from Time 1 to Time 2
As shown in

Convergent Validity of the Tests
As shown in Table 3, at time 1 the RBANS had a strong correlation with the composite z scores of RBANS supplemented with the CTT and mBNT (r = .83,p < .001)while the strength of a significant correlation between the RBANS and a formal neuropsychological battery was relatively weaker (r = .61,p = .001).However, the correlation between the composite z scores of the RBANS supplemented with CTT and mBNT and the formal neuropsychological battery was strong (r = .80,p < .001).

Discussion
In this pilot study, we have examined the test-retest reliability, practice effects and convergent validity of a brief neuropsychological battery, the RBANS, in a sample of cognitively intact older adults in Singapore.The principal findings of this study include: 1) the RBANS has good test-retest reliability and convergent validity; and 2) the practice effects of the RBANS are evident in visuospatial function and memory domains.The first hypothesis is supported, that is, the RBANS alone has demonstrated acceptable test-retest reliability for clinical purposes.In addition, RBANS supplemented with CTT and mBNT demonstrated excellent test-retest reliability.These results are consistent with the generally sound psychometric properties of the scales as previously reported.Test-retest reliability was also significant in 11 out of 12 RBANS subtests.Interestingly, the reliability coefficient for the remaining subtest (list recall) attained significance when three outlying cases were removed from the analysis.These three cases all scored a zero at time 1, and scored highly at time 2, which may possibly reflect test anxiety affecting their performance at time 1 but not at time 2. Also of clinical interest is the finding of significant gain scores on many (but not all) of the RBANS subtests, while overall scores show little gain.
The second hypothesis was partially supported.The RBANS showed moderate to large practice effects in visuospatial function, immediate memory and delayed memory.The practice effect in immediate memory is consistent with a previous study on the practice effects of RBANS in a community sample of older adults in a western population [36].However, the practice effects demonstrated in visuospatial function differed from the previous study [36], which may be explained by different sample characteristics such as age (younger participants in the current study compared to the older participants in previous study [36]: 63.29 ± 5.84 vs 72.89 ± 5.52).In addition, the different finding may be attributed to different test-retest interval (a short interval of 2 weeks in the cur-rent study compared to a longer interval of 38 weeks in the previous study [36]).
The third hypothesis was also confirmed.The RBANS showed adequate convergent validity with the formal neuropsychological battery.Moreover, the RBANS supplemented with CTT and mBNT has dem-onstrated excellent convergent validity.This result is encouraging, in that it suggests the RBANS may be an appropriate substitute for the lengthier neuropsychologi-cal test batteries in memory disorders clinics.
The novelty of this study includes the following.First, few studies to date have investigated the clinical utility of a brief neuropsychological test battery.The findings of this study support the use of the RBANS as an assessment instrument in clinical as well as research set-tings.The inclusion of CTT and mBNT further improves the psychometric properties such as test-retest reliability and convergent validity.Consequently, these two tests will add clinical value to the RBANS.This study has also reported valuable clinical data on the expected gain scores resulting from practice effects on the RBANS.This will provide insights for clinicians to observe for significant changes in raw subtest scores and index scores of RBANS.Second, a particular strength of this study is the complete data set, free from attrition which sometimes limits the ability to interpret results of repeated measures studies.However there were a number of ceiling effects identified in the data, which may have contributed to non-significant results.Such ceiling effects are to be expected in cognitively intact participants, yet reliability and validity of tests should be first established with cognitively intact participants prior to the implementation in clinical populations.
There are several limitations in this study.First, the sample size is small; we only recruited 30 cognitively intact participants with 28 participants included for analysis, as this was a pilot study.A larger sample size is required in future studies.Second, we did not examine the test-retest reliability, convergent validity and practice effects in older adults with cognitive impairment, again due to the nature of a pilot study.Older adults with cognitive impairment may have different rate of cognitive decline or improvement compared to healthy controls.Therefore, our findings are limited to cognitively intact older adults.Future studies should include older adults with a range of cognitive status to examine the test-retest reliability, convergent validity and practice effects of the RBANS.Third, the test-retest interval was short, at a span of 2 weeks, and may not be in line with clinical practice of repeat testing at 6 months or a year.Therefore, future studies with a longer span of test-retest duration will provide more information about the RBANS for clinical utility.
In conclusion, The RBANS has good test-retest reliability and convergent validity, while RBANS supplemented with CTT and mBNT have demonstrated excellent convergent validity.In addition, the practice effects were only demonstrated in immediate memory and visuospatial function while other index and total scale scores are spared of practice effects within the short span of a 2-week interval.Therefore, RBANS or RBANS supplemented with CTT and mBNT may be adopted for routine clinical practice in memory clinics.

Table 2 ,
there were significant gain scores for five of the twelve subtests: list learning and list recall, story memory, figure copy and figure recall with moderate to large effect sizes (.45 -.87).Additionally, significant gain scores were observed in the domain index scores of visuospatial function, immediate memory and delayed memory.There were no significant gain scores in total scale scores of RBANS or the supplementary tests (CTT and mBNT).