Timeslips—comparing Agitation and Anxiety Rating Scales to Evaluate the Benefit of Non-pharmacologic Creative Sessions in Nursing Home Patients with Dementia

Timeslips™ is a group storytelling program that encourages creative expression among dementia patients without the pressure to recall the past. Analysis of the literature was conducted to determine the nine most relevant agitation and anxiety scales most appropriate for use with Timeslips in nursing home patients with dementia, who experience agitation and anxiety. Qualitative assessment of the nine scales was conducted to identify six criteria to determine the most pertinent characteristics for implementation of Timeslips within this patient population: 1) validity/relia-bility, 2) observation period, 3) training required, 4) time to administer, 5) most appropriate administrator and 6) accessibility/cost. Utilizing these six criteria, quantitative assessment was conducted using the Analytical Hierarchical Process (AHP) to identify that the Overt Agitation Severity Scale (OASS) was optimal. IRB approvals have been attained to investigate use of the OASS with Timeslips in the nursing home setting for patients with dementia, who experience agitation and anxiety.


Prevalence/Impact of Anxiety and Agitation
The prevalence of anxiety and agitation associated with early onset dementia and Alzheimer's disease (AD) is common and affects up to 70 percent of patients [1] [2].Additionally, agitation can occur in up to 90 percent of people with moderate to severe dementia [3].Both anxiety and agitation tend to increase as the disease progresses.In the case of agitation, it is most commonly characterized by repetitive mannerisms, non-loud verbal excesses, and wandering [4].These non-aggressive behavioral symptoms of aggression have been reported as the most frequent forms of agitation in dementia patients [5] although severe cognitive deficiencies can display intense aggressive and impulsive behaviors and continue to heighten with the development of dementia disorders [6].Managing agitation and difficult behavior can be the most difficult part of treating patients with AD.In anxiety, there is association with other symptoms, such as depression, behavioral disturbances and increased cognitive impairment [7].This high prevalence of agitation and anxiety in AD patients has many effects on the patient as well as on the caregiver and quality of care.Agitation and aggression behaviors are often disturbing for the individual patient, provide risk to patients and others and present challenges for clinicians [8].

Pharmacologic Treatment for Anxiety and Agitation
Current treatments for agitation and anxiety in AD patients are most commonly pharmacological in nature, with the most widely prescribed being atypical antipsychotics.In one study about nursing home prescribing rates, the antipsychotic prescribing rate for patients with dementia was 32 percent (46 of 146 patients).While drug therapy has been shown to be beneficial, adverse effects can still be seen in patients, e.g., myocardial infarction in patients taking Donepezil [9].
Other side effects for antipsychotics include worsening of confusion and lower cognitive function.Increase in risk of falls due to hypotension and syncope has also been noted [10].Switching patients from one medication to another may result in a change in mental status [11].Consequently, it is crucial to consider non-pharmacological or counseling approaches for agitation and anxiety [8].

Treating Agitation and Anxiety with Non-Pharmacologic/Counseling Approaches
Non-pharmacological or counseling approaches have been shown to be successful treatments for patients with dementia.For example, dementia patients in chronic care, who were experiencing agitated behaviors, demonstrated improvement after undergoing bright light [12] [13].Similarly, music therapy has been accepted as a safe and effective method for treating both agitation and anxiety in AD patients [14].Additionally, individuals who completed art therapy over 40 weeks showed significant improvements, e.g., sociability and mental acuity [15].Utilization of visual art to give patients an opportunity to observe the art has been shown to lower agitation and anxiety levels in patients [16].
Non-pharmacological approaches are a necessary treatment inclusion because some patients may not respond to pharmacological interventions.Caregivers may be included in these forms of treatment [17].Psychoeducative group intervention with AD patients and their caregivers was effective in decreasing caregiver burden and patient agitation and anxiety [18].Other types of non-pharmacologic interventions that have improved agitation and anxiety are creative expression in elderly patients with dementia include life story writing and sharing groups, Visible Lives Storyboards and Timeslips™ [19].

Research Question
The goal of this study is to evaluate the most commonly implemented agitation and anxiety scales and in conjunction, to determine what scale(s) are optimal for pre and post rating in concurrence with the Timeslips program.Evaluation will be accomplished by using the Analytical Hierarchy Process (AHP) to determine the most effective scale for use with Timeslips.

Review of the Literature
A comprehensive review of the literature was conducted to explain the Timeslips Program and to identify and assess the agitation and anxiety scales that are candidate for use with Timeslips.

Timeslips
Timeslips is a group storytelling program that encourages creating expression among patients with AD [20].Pressure to remember memories from the past can be frustrating for a person with dementia but the Timeslips program alleviates that pressure by relying on creativity and encouragement for imagination [19].A trained facilitator presents a group of participants with an image and assists them to create a story with the utilization of open-ended questions and each participant is asked to provide creating input in their own way.The process of asking questions and answering them is repeated until the group decides the story is finished.The completed story is read to the group [21].
Timeslips continues to be studied more extensively with findings that demonstrate its significance.Results include improved levels of expression, better communication in AD patients [19], and increased medical student interaction [22] with improved medical students' attitudes towards people with dementia when they were included in the sessions with patients [23].Furthermore, Timeslips has been shown to improve caregivers' attitudes toward dementia patients because it has improved the overall relationship between staff and residents [24].

Agitation/Anxiety Scales
Rating scales are utilized to evaluate the non-cognitive symptoms associated with Alzheimer's disease and dementia range from psychosis to mood disturbances (e.g., depression, anxiety) and agitation [25].Because these symptoms can be distressing and dangerous for patients, it is important to rate these behaviors [26] [27].
A comprehensive literature search was completed and identified numerous scales that assess and rate agitation and anxiety.These included all scales used in the assessment of dementia, including the Mini-Mental State Examination (MMSE), the Alzheimer's disease Assessment Scale (ADAS), Behavior Pathology in Alzheimer's disease Rating Scale (BEHAVE-AD), Neuropsychiatric Inventory-Questionnaire (NPI-Q) and the Psychogeriatric Dependency Rating Scale (PGDRS).While these scales are frequently used to assess dementia symptoms [27] they were not included because they were not specific enough for agitation and anxiety.
From the scales identified, nine were selected for evaluation.These nine scales include three variations of the Cohen Mansfield Agitation Inventory and two variations of the Neuropsychiatric Inventory.The scales are: A summary of each scale follows.

Overt Agitation Severity Scale (OASS) 1
The OASS includes the ability to incorporate both the intensity and frequency of observable displays of agitation, as opposed to just subjective interpretations of a wide range of symptoms and behaviors.Among the elderly in nursing facilities and AD patients, the reported incidences of agitation have been measured by the OASS [28].Because agitation also affects an individual's quality of life and is associated with depression and anxiety, an appropriate and accurate assessment of agitation is necessary [29].The OASS describes agitation as "a progressive set of objective neurobiological and anatomic/functional vocal and motor behaviors, e.g., oral/facial movements, on a continuum of behavioral expressions that extend from anxiety to aggression" [29].Kopecky et al. established its reliability and validity in a study in 1998 of agitation severity (r = 0.95, p < 0.01) [28].
The OASS is a scale used to describe and rate the severity of agitated behavior objectively.It contains 47 observable characteristics of agitation, subcategorized into 12 behaviorally related units and is completed using a multistep process.For each behavior subcategory, a frequency score from 0 (not present) to 4 (always present) is used.For exhibited behavior, a severity score is calculated by multiplying the intensity score by the frequency score.The final OASS value adds all severity scores together and summarizes the severity of agitation to include patient medications.The entire process is completed in 15 minutes usually by a physician [28].The OASS actual scale is available in Appendix A.

Pittsburgh Agitation Scale (PAS)
The Pittsburgh Agitation Scale is a rating scale that did not require assessment of specific behavior over several days.The scale takes one minute to administer and incorporates four items based on direct observations over the normal shift for a care staff member.Agitation can be classified as motor and vocal agitation, aggressive behaviors and resistance to care [30].This helps to identify the type and severity of agitation the patient experiences [31].Adequate inter-rater reliability in an acute care setting has been established [32].
The scale is a simple questionnaire for healthcare professionals consisting of four categories, aberrant vocalizations, motor agitation, aggressiveness and resisting care.Categories are scored from 0 -4, with" 0" meaning no agitation and greater than "3" demonstrating that the patient is suffering from dementia [33].
Furthermore, in a study completed by Colin Zieber et al., total PAS scores were significantly correlated to pain levels when measured by palliative care nurses (r = 0.49, p < 0.01) (2005), which determined that "resistance-to-care" agitation can be associated with uncommunicated pain [31].Another study completed by Rosen et al., found the PAS to be reliable when used by both acute care geriatric staff and trained research facilitators in nursing homes.Nurses, who administered the scale, received no further training other than the instructions written on the scale [32].The PAS actual scale is available in Appendix B.

Rating Scale for Aggressive Behavior in the Elderly (RAGE)
The RAGE scale is a 21-item scale that measures aggressive behaviors in the elderly and in psychogeriatric inpatients [34].It is often used for assessing the effectiveness of a new drug for treatment of dementia [35].The first 19 items rate the frequency of each behavior exhibited during the previous three days.The twentieth item includes the measures taken by staff to control the aggressive behaviors and the final item is a total rating of aggressive behavior [34].These last two items are not answered in terms of frequency, but rather, are yes or no answers [36].
A four-point scale is used to rate each behavior [34].The scale ranges from 0 -3 on how frequently the behavior has occurred by the patient during the past three days ranging from 0 (never) to 3 (more than once per day during the last three days).Different types of aggressive behaviors are combined into this scale, including verbal aggression, agitation, and physical aggression [37].
Training for the raters is fairly simple because it includes reading a short instruction sheet and a 10-minute healthcare staff meeting.Any member of the staff can be a part of the rating team if they observe the patient for at least two of the three days [34].In a sample size of 274 nursing home patients, the inter-rater reliability was 0.94 (p < 0.001); test-retest reliability was r = 0.91.For validity, the Pearson Correlation Coefficient was 0.86 (p < 0.001) [36].However, a significant drawback is the observation period of three days [37].Researchers helped implementation and maintained continuity by having nurses implement a checklist system to record their observations and increase reliability, especially during shift changes [36].The RAGE actual scale is available in Appendix C.

Cohen Mansfield Agitation Inventory (CMAI)
The CMAI is a widely used scale for the assessment of dementia-related symptoms of agitation [38].It consists of 29 behaviors rated on a seven-point frequency and disruptiveness scale and is a systematic agitation assessment rated by caregivers.A score of one indicates no engagement in a certain behavior, whereas a score of seven indicates engagement of several times an hour.It has been shown to be effective in characterizing longitudinal changes in patients with cognitive impairment as well as in clinical research on patients with dementia [39] as well as used to measure agitated behavior [29].Using raters who were registered nurses and certified nursing assistants (CNAs), rater reliability and validity were evaluated by Finkel et al. in a study of 232 residents [40].The internal consistency reliability (Chronbach's Alpha) was 0.86, 0.91 and 0.87 for the day, evening and night shift raters respectively [40].
This scale is fairly simple to use based upon multi patient assessment.Additionally, any healthcare professional is able to become a rater.However, the CMAI was most effective when completed by a healthcare professional, who is familiar with the patient.While the scale is relatively lengthy, it includes behaviors that are characteristic of agitation.A drawback for use with the Timeslips program is unavailability of rating by the patient's primary care provider.The CMAI long form actual scale is available in Appendix D.

Cohen Mansfield Agitation Inventory-Short Form (CMAI-SF)
The CMAI-SF is a shortened form of the longer CMAI and was developed to assess behavioral disturbances in patients with dementia in nursing homes more quickly than the CMAI.The CAMI-SF includes 14 items encompassing areas of agitation and aggression and was created to minimize the risk of incomplete data and increase convenience [41].The time span for this scale, like the CMAI, is two weeks [41].The CMAI short form actual scale is available in Appendix E.

Brief Agitation Rating Scale (BARS)-Subset of Cohen Mansfield Agitation Inventory
The BARS, derived from the CMAI scale, is a 10-item brief agitation rating scale developed to allow nurses and other caregivers in nursing homes to assess the level of agitation in patients rapidly [42].The BARS encompasses three assessment areas: physically aggressive behaviors (e.g., kicking); physically non-aggressive behaviors (e.g., intentional falling) and verbally agitated behaviors (e.g., screaming).Based on the three assessment areas, agitation has been studied in detail and reliability and validity have all been established [42].The resulting BARS includes 10 behaviors: hitting, grabbing, pushing, pacing or aimless wandering, performing repetitious mannerisms, restlessness, screaming, repetitive sentences or questions, strange noises and complaining [42], evaluated by a healthcare professional over two (2) weeks.The 10 items are scored from 1 (never) to 7 (several times an hour) and the score ranges from 10 to 70 [43].
In a study performed in the Lieberman Geriatric Health Center, assessments were completed by registered nurses and CNAs, who were the resident's main caregiver [42].Internal consistency and inter-rater reliability, which includes both correlation of the BARS to CMAI and its validity, were also determined [42].The same study concluded that the BARS is both reliable and valid and the 10-item BARS speeds the evaluative process without sacrificing reliability or validity.
These findings were corroborated in a study to examine the test-retest reliability and validity of the Norwegian version of the BARS [43].The BARS actual scale is available in Appendix F.

Neuropsychiatric Inventory-Clinical (NPI-C)
The primary purpose of the Neuropsychiatric Inventory-Clinician rating scale (NPI-C) is to provide rapid assessment of a wide variety of dementia behavior for both frequency and severity using clinical criteria and assesses the level of caregiver distress.This version of the NPI was developed because its predecessor was based on the answers by the caregiver, not the actual patient.The NPI-C also incorporates frequency and severity.Rated behaviors include delusions, agitation, depression, irritability and apathy.Both the caregiver and the patient are interviewed for this scale [44].
The clinical aspect was added because some behaviors seem similar to an untrained person, such as depression, but is distinguishable by clinicians [45].Using the NPI-C, the agitation domain was useful as a "standalone" measure in clinical studies.
The scale has also been translated into seven languages.A cross-sectional validation study was completed in eight countries in 2010 but the rater must be an experienced clinician.Training is also required, which can be completed in person or by using training notebooks and DVDs.A healthcare worker or researcher usually completes the training [44].The NPI-C actual scale is available in Appendix G.

Neuropsychiatric Inventory-Nursing Home (NPI-NH)
The NPI-NH was created for use in nursing homes by professional caregivers.The questions are rephrased from the original NPI scale to reflect that the caregiver is unlikely to have known the patient prior to their illness or whether the patient's behaviors started occurring before or during the course of the disease.An informed caregiver gives responses to the rater [46].
The patient is evaluated by 12 psychiatric symptoms including depression, agitation, and anxiety.For each category, a question is asked to determine whether they are exhibiting that behavior.If the patient says yes to the question, then a series of deeper questions are asked [47].Occurrence, severity and frequencies are measured for each behavior [48].Frequency is measured on a scale from 1 -4 and severity is measured from 1 -3 [46].
The NPI-NH is widely used and has been translated for studies in countries, such as Norway, Italy, France and Mexico [47].Training is required for the administration of the scale, such as the healthcare professional training at the Norwegian Center for Dementia Research [48].The scale was found to be extremely valid and reliable as determined by comparing patients' scores on the Cornell and BEHAVE-AD [48].However, another study conducted in the Netherlands showed that its reliability was poor because there was poor inter-observer agreement [46].The NPI-NH actual scale is available in Appendix H. 1.6.9.Behavioral Rating Scale for Dementia (BRSD) by Consortium to Establish a Registry for Alzheimer's Disease (CERAD) The Behavioral Rating Scale for Dementia (BRSD) was initially created to assess patients with dementia.The goal of the scale is to evaluate a wide range of behaviors in dementia [49].The rater depends on answers from an "informant", who is usually the primary caregiver that has daily contact with the patient over the course of the previous month [50].The scale takes less than 30 minutes to administer [51].
The scale has now been revised to 46 items.Of these, 37 pertain to frequency and are rated from 0 (not occurring) to 4 (occurring 16 or more days during the past month).Additionally, the scale contains questions about behavior before dementia began, which are rated 1 (yes) or 0 (no) [52].Six sub-scales are included: depressive symptoms, psychotic symptoms, inertia, vegetative systems, irritability/aggression and behavioral dysregulation.The final item asks whether the patient has experienced any behavior not listed under any other item [50].
The scale is only available from Duke University and requires a trained facilitator as the rater.The scale has an instruction manual and a two-page information sheet, which tells the rater how to instruct the informant about answering questions [50].In a study completed by Weiner et al., BRSD was correlated to CMAI scores (r = 0.76; p = 0.0001) and demonstrated a strong correlation between the two scales' scores (1998) [53].The BRSD-CERAD actual scale is available in Appendix I.

Analytical Hierarchy Process
The Analytical Hierarchy Process (AHP) is a multi-criteria decision support system developed by Saaty [54] [55] that allows a decision maker to structure a complex problem in the form of a hierarchy.The AHP model has been recently applied to a multitude of different corporate and non-corporate problems to improve decision making.Examples are Liberatore, Monahan, and Stout [56], Hogan and Olson [57], Ishizaka and Lusti ([58], Travana [59], Vaidya & Kumar [60], Dey, Hariharan, Kumar and Moseley [61], and Hogan, Olson & Sillup [62].The major advantage of the model is its ability to accommodate complex qualitative and quantitative information into the decision making process.Other advantages include its simplicity to use and its ability to apply consistency to the decision making process. The first consideration of the hierarchy process is the goal, which, in this study, is to determine what scale(s) are optimal for pre and post rating in concurrence with the Timeslips program.Next, there are four general steps required to implement the AHP.First, the decision maker identifies the criteria and determines their relative importance in achieving the goal and identifies the sub-criteria and determines their relative importance in achieving the related criterion.Second, the decision maker determines the relative importance of the ratings categories for each of the sub-criteria.Third, the alternatives are evaluated in the context of each of the ratings categories.Fourth, the results are synthesized to compute the relative contribution of the alternatives in achieving the goal.
Pairwise comparisons must be made to determine the relative importance of the criteria in achieving the goal.Although there are many scales that can be used to compare the criteria, Saaty [55] recommends a scale from 1 to 9 where 1 refers to "equally important", 3 "moderately more important", 5 "strongly more important", 7 "very strongly more important", and 9 "extremely more important".
w ij = the relative importance of criteria i compared to criteria j; w ij = 1 ∀ i = j; and The procedure presented here is good approximation of the weights.Saaty (1980) determined the exact relative priorities for each of the n criteria by computing the normalized eigenvector of the maximum eigenvalue of the comparison matrix [63].The normalized eigenvector is computed by raising the comparison matrix to successive powers until convergence is achieved and then normalizing the results If n = 5, W will be a 5 × 5 matrix with 1s along the main diagonal depicting comparison of the criteria with itself.Below the main diagonal are the reciprocals of the corresponding comparisons above the diagonal.Thus, in this example a total of 10 comparisons must be made.In general, if there are n criteria to be compared, a total of n(n − 1)/2 comparisons are required.

Methodology
Utilizing the nine (9) scales summarized in this paper's review of the literature, each scale was identified by its abbreviation and included for AHP assessment in the order in which they were reviewed: OASS, PAS, RAGE, CMAI, CMAI-SF, BARS, NPI-C, NPI-NH and BRSD.Each scale was then summarized and qualitatively assessed to identify the criteria that were the most significant determinants for a scale's use with the Timeslips program.These criteria would also become the basis of the pairwise comparisons for quantitative analysis the AHP.
Then using the criteria identified by the qualitative assessment, the AHP was used to determine the most effective scale for use with Timeslips.To do this, pairwise comparison decision alternatives using the AHP were established by distilling the verbal judgment of healthcare professionals into the numerical ratings previously depicted in Table 1.
Using these numerical ratings, pairwise comparisons between all the scales for each of the identified criteria were determined for pairwise comparisons of the scales' evaluation criteria.To accomplish this, criteria weights for the identified criteria were determined and used to generate an adjusted comparison matrix and criteria weights.
Note that if a comparison is made that is between two verbal ratings, then a number between the two numbers is inserted as the rating.For example, if a rating is between "extremely preferred" with a rating of "9" and "very strongly preferred" with a rating of "7", then the rating would be an "8".These were used to evaluate the competing scales to determine the optimal scale for use with Timeslips.

Qualitative Assessment to Determine Criteria for Use with AHP
Qualitative assessment of the nine agitation/anxiety scales was summarized in a comprehensive table that determined the six criteria that were most important to successful implementation of the Timeslips program.These six criteria are summarized in Table 2.

Quantitative Assessment Using the AHP
The AHP analyses consider the pairwise comparisons and computations using the six identified criteria for the six scales.So if V/R is moderately more important than OP, then a "3" is put into the matrix.Conversely, if V/R is significantly more important than TR, then an "8" is put into the matrix.These results are depicted in Table 3.  Next, an adjusted comparison matrix and criteria were determined and are shown in Table 4.This followed by determination of criteria weights, which are shown in Table 5.
Finally, evaluation of the six competing scales was completed and indicated that the OASS is the optimal scale for use with Timeslips.This is shown in Table 6.
Using the six identified criteria, the AHP quantitative analysis indicated that the Overt Agitation Severity Scale (OASS) was the optimal scale for use with Timeslips.As shown in Table 6, pair wise comparisons using the AHP determined that that OASS scale was the most suitable to be used in a Timeslips session, with a priority ranking of 0.19988.The Pittsburgh Agitation Scale was the next most desirable scale for use with Timeslips with a priority rating of 0.187367.The scale with the lowest priority rating was the CERAD, with a rating of 0.062589.

Discussion
As shown in Table 6, the OASS, which has the largest weight, was determined to be the optimal scale to use with Timeslips by the AHP methodology (0.19988).Additionally, qualitative assessment indicated that the OASS had published reliability/validity, an excellent patient population fit, is easy to score, takes about 15 minutes to implement and is easily accessible as a free download.Second by AHP analysis (0.187367), the PAS is easy to use because it requires no further training.However because it is based on an observation time from one to eight hours, the scale is too long for practical implementation with Timeslips [32].
The seven other scales had lower AHP evaluations that were too low to consider them for use with Timeslips.The RAGE scale (0.145031), which is used to assess aggressive behavior in all elderly patients, is not specific to dementia patients and is, therefore, not ideal for use with Timeslips [37].At three days, the observation period is quite long, making it quite impractical for use [34].
The three scales based on the CMAI assessment were also not optimal for use with Timeslips.The original CMAI scale (0.077658) is most effective when the rater is the primary caregiver for the patient but this person may always not be available.The second shorter form of the CMAI, the CMAI-SF (0.77354), still advises that it be completed by a healthcare professional, who knows the patient well [41].Both versions of CMAI require an observation period of two weeks, making them impractical for use with Timeslips [41].The third is an even shorter derivative; the BARS (0.1014044), which takes less than two minutes to conduct but it is not readily available online.Lastly, the seventh, eighth and ninth scales also possessed properties that precluded them from optimal use with Timeslips, the NPI-C (0.74675) and the NPI-NH (0.74403) scales require more training than the other scales studied and, consequently, would not be as easy to administer for an ideal Timeslips session.Finally, BRSD (0.062589) was not readily available online.In our analysis, it was the only scale that required payment to access it and required training to use the scale.Because the object of this study is to have a healthcare professional be a rater, a scale requiring payment for access would not enable easy application with Timeslips.

Conclusion/Implications
The AHP analysis of the nine published scales most applicable for anxiety/agitation patients has found that the OASS is the optimal scale for use with the Timeslips Creative Expressions Program in nursing homes based on administration time, ease of use and results obtained.The OASS has the ability to encompass both an intensity level and a frequency level, which form an overall severity score.
When compared to the most widely used scale, the CMAI, the OASS proved to be better in the conceptualization of agitation.The CMAI only measures behavioral symptoms and, therefore, lacks conceptualization of agitation.Consistent with the findings of this study that support the use of the OASS with the Timeslips Program, the OASS was a reliable and valid measure of agitation severity and motoric upper and lower body behaviors in adult psychiatric patients [28].

Future Study to Assess Benefit of Non-Pharmacologic Intervention
Numerous studies have been conducted in patients diagnosed with dementia and Alzheimer's disease to investigate symptoms of anxiety, which include several aberrant behaviors, e.g., anxious or worried appearance, [7].These can result in depression, behavioral disturbances or increased cognitive impairment often treated by pharmacologic intervention.
Non-pharmacologic interventions have gained interest because the current use of pharmacologic treatment has been shown to include a various range of side effects and increased treatment costs.Alternatively, creative expression activities have demonstrated improved quality of life, communication and effect on patients' emotions [11].To take the findings extracted from the literature one step further and to generate empirical data to corroborate the findings, a study is being planned to use the Timeslips Program and the OASS in a nursing home setting with a cohort of patients, with dementia, impaired by anxiety and agitation.

Table 1
depicts the criteria comparison table.

Table 1 .
Criteria comparison table.For example, if the Validity/Reliability of a scale is determined to be "strongly more important" than accessibility/Cost, a value of 5 is assigned to this comparison.The results of the comparisons are represented in a pairwise comparison matrix similar to Equation (1).

Table 2 .
Qualitative determination of criteria for use with AHP.

Table 3 .
Pairwise comparison matrix and computations for scales' evaluation criteria.

Table 4 .
Adjusted comparison matrix and criteria weights.

Table 6 .
Evaluation of competing scales.