Progress Monitoring Measures: The Interaction of Clinician Initial Motivation with Selection and Maintenance Issues


The use of Progress Monitoring (PM) measures has been shown to improve outcomes in therapy for clients who do not follow the normal trajectory of improvement. In addition to improved outcomes, there are several other documented benefits of PM that may motivate clinicians to use PM. Research has examined the broader field of selecting mental health care quality assessment tools and a review of the literature has pointed to the importance of considering motivation for assessment when selecting a measure. However, how motivation influences the selection or maintained usage of PM measures has not been studied. This study examined initial motivation as well as measure selection and continuing use of PM. Consensual Qualitative Research methodology was applied to characterize how clinicians (n = 25) started, selected, and maintained use of PM measures and how initial motivation related to measure selection and continued use. Regardless of initial motivation, convenience and effectiveness emerged as important when selecting and continuing to use a measure. Results are compared to current frameworks for selecting mental health- care quality indicators. Our results suggest that PM measures need to strike a balance, emphasizing convenience as well as efficacy in order to improve clinical uptake and adherence.

Share and Cite:

Knoll, M. , Ionita, G. , Tomaro, J. , Chen, V. and Fitzpatrick, M. (2016) Progress Monitoring Measures: The Interaction of Clinician Initial Motivation with Selection and Maintenance Issues. Psychology, 7, 444-458. doi: 10.4236/psych.2016.73046.

Received 28 January 2016; accepted 27 March 2016; published 30 March 2016

1. Introduction

Progress Monitoring (PM) measures are tools that have been developed to track client progress over the course of psychotherapy to identify clients who are not progressing. The use of these measures has been shown to improve outcomes in therapy (Anker, Dunacan, & Sparks, 2009; Reese, Norsworthy, & Rowlands, 2009; Reese, Toland, Slone, & Norsworthy, 2010; Schuman, Slone, Reese, & Duncan, 2015; Slone, Reese, Mathews-Duvall, & Kodet, 2015) , particularly for clients who do not follow the normal trajectory of improvement (Lambert, Harmon, Slade, Whipple, & Hawkins, 2005; Lambert & Shimokawa, 2011; Lambert et al., 2001) . The tools can increase efficiency, allowing not-on-track clients to receive more sessions and on-track clients to receive fewer (Harmon et al., 2007; Lambert et al., 2001; Reese, Norworthy, & Rowlands, 2009; Schuman, Slone, Reese, & Duncan, 2015; Slone, Reese, Mathews-Duvall, & Kodet, 2015) . Efficient allocation of resources has both clinical and financial advantages. In addition to improved outcomes and efficiency, clinicians have indicated that they find outcome measures useful for altering treatment, practicing ethically, determining strengths and weaknesses, meeting managed care or employer requirements, carrying our research, and practice marketing (Hatfield & Ogles, 2004) . Despite the well-documented benefits of PM, the majority of clinicians have not incorporated these tools into their practice (Hatfield & Ogles, 2004, 2007; Ionita & Fitzpatrick, 2014) .

Practical concerns have been identified as one reason that clinicians do not use outcome measures (Hatfield & Ogles, 2007) ; one practical challenge may be the burden of selecting of a measure. A variety of PM measures currently exist and vary with respect to what they assess, target populations, administration procedures, scoring/ feedback/interpretation procedures, cost, training required and privacy (Overington & Ionita, 2012) . Considering all these variations, selecting an assessment tool can be a daunting task.

1.1. Selecting Measures for Mental Health-Care Quality Assessment

Research has not yet examined how clinicians come to select a PM measure. To better understand this process, we examined the broader field of mental health-care quality assessment. In this field, an overwhelming number of assessment measures are available (Froyd, Lambert, & Froyd, 1996; Hermann et al., 2000) . The number and heterogeneity of quality measures inhibits comparisons and limits the development of broader understandings. Proposals have been made to adopt a small set of “core measures” to reduce the burden of selection, increase comparability of services, and focus resources on the most promising measures (Hermann & Palmer, 2002) . Several frameworks have been created to aid organizations, researchers and clinicians in developing such a core set (Burlingame et al., 2005; Burlingame, Lambert, Reisinger, Neff, & Mosier, 1995; Erbes et al., 2004; Green & Gracely, 1987; Hermann et al., 2006; Hermann et al., 2004; Hermann & Palmer, 2002; Lambert, Ogles, & Masters, 1992; Slade, 2002; Waraich et al., 2010) .

1.2. Existing Frameworks

We conducted a review of the existing frameworks for selecting mental health-care quality indicators and established a rank order of the most commonly identified desirable measure features. The frameworks most commonly cited factors were cost, psychometrics, suitability to the target population, and comparability as selection criteria (see Table 1). The second most commonly cited suggestions referred to the measure’s evidence-base, comprehensibility, interpretability, and domain selection or coverage. Other commonly cited features included feasibility, appropriate case-mix adjustment, and sensitivity to change. From a researcher perspective, these are key factors to consider in the mental health-care quality assessment (Burlingame et al., 2005; Burlingame et al., 1995; Erbes et al., 2004; Green & Gracely, 1987; Hermann et al., 2006; Hermann et al., 2004; Hermann & Palmer, 2002; Lambert et al., 1992; Slade, 2002; Waraich et al., 2010) .

However, these may not be the features of greatest interest to practitioners. Because of a substantial practice- research gap in psychotherapy (Stewart & Chambless, 2007) , it is equally important to explore how clinicians actually come to select a measure. Clinicians’ motivation for assessment likely influences their selection and continued usage of measures. Although several of the cited frameworks emphasize the importance of considering individual stakeholder needs (Hermann et al., 2006; Hermann et al., 2004; Hermann & Palmer, 2002; Waraich et al., 2010) , how motivation for assessment influences selection criteria or maintained usage has not yet been studied. Insight into this process may assist in the development and improvement of the measures, and ultimately increase the uptake and adherence of PM.

This study had three objectives: 1) to understand what motivated a group of clinicians to start, select, and maintain use of PM measures, 2) to compare existing frameworks for selecting measures to clinicians’ actual

Table 1. Factors discussed in published frameworks for selecting measures.

a Hermann & Palmer, 2002 ; b Hermann et al., 2004 ; c Green & Gracely, 1987 ; d Lambert, Ogles, & Masters, 1992 ; e Hermann et al., 2006 ; f Waraich et al., 2010 ; g Burlingame, Lambert, Reisinger, Neff, & Mosier, 1995 ; h Slade, 2002 ; i Erbes et al., 2004 ; j Burlingame et al., 2005 ; 1Evidence-based; 2Psychometrics; 3Valid; 4Reliable; 5Feasible; 6Affordable; 7Comprehensible; 8Interpretable; 9Comparable; 10Available Norms; 11Available Benchmarks; 12Standardized; 13Stakeholder Needs; 14Operationalized Components; 15Case Mix Adjustment; 16Under User’s Control; 17Domain Selection/ Coverage; 18Address Knowledge Gap; 19Suitability for Target Group; 20Usefulness Across Settings; 21Compatible with Various Theoretical Orientations; 22Procedural Simplicity; 23Means of Treatment Effects; 24Brevity; 25Sensitivity to Change.

selection and maintenance of use behavior, and 3) to understand how initial motivation relates to selecting a measure, and to continuing to use a PM measure in practice.

2. Method

2.1. Participants

Twenty-five clinicians (i.e., counselors, mental health workers, psychologists, social workers, and marriage and family therapists) who were currently using PM measures participated in the study. The majority of participants were male (76%), over the age of 51 (64%), and working in private practice (56%). Additionally, the majority of participants were using the Partners for Change Outcome Management System (PCOMS) (Miller, Duncan, Sorrell, & Brown, 2005) (76%). When asked to rate how much of their practice was influenced by various theoretical orientations, 56% of clinicians rated Eclectic/Integrative as the most influential (see Table 2).

2.2. Procedures

Participants were recruited in three ways. Some participants were invited to participate after completing a national survey on the usage of PM measures. Others were recruited via notices on psychotherapy related social

Table 2. Demographic data of participants.

networking webpages (e.g., Facebook, LinkedIn, etc.). Lastly, notices were distributed via PM measures’ host companies or creators to known PM measure users. Notices included a brief description of the study and a statement indicating that all participants would be entered into a draw to win a $100 Amazon gift certificate to thank them for their participation. The notices also included a link to a secure website, where participants could provide informed consent and indicate a telephone number where they could be reached for the interview.

Semi-structured interviews were conducted as part of a larger program of research investigating PM use among clinicians. The interview was piloted with six graduate students, two clinical professors, and two clinicians. Participants were provided with a brief description of PM measures and a few examples; they were then asked to indicate which PM measures they used and how often they used them. The interview then inquired about participants’ experience using PM measures. Lastly, demographic information was collected. After the interviews were completed, each participant was asked to provide feedback on the interview process. Participants all reported that they felt that the interview was able to capture their experience with these measures and no changes were suggested. All interviews were conducted via telephone and lasted between 18 and 53 minutes.

2.3. Data Analysis

Consensual Qualitative Research (CQR) was selected for its ability to characterize similarities across participants, while also providing an in-depth examination of the participant experience. Its use of consensus among multiple researchers honors multiple perspectives, thereby enhancing trustworthiness (Hill, 2012) . CQR follows a process of developing domains, where general topic areas discussed by the participants are identified; coding domains, where chunks of interview data are partitioned into the domains; constructing core ideas, where summaries of the data that capture the essence of the participant’s statement relating to the domain are created; cross analysis, where data is compared across participants by placing all participant’s core ideas into representative categories; and determining representativeness of themes, where frequency information for each category is calculated. Auditing occurs at several points: after developing the domains, coding the domains, constructing core ideas, and performing the cross analysis (Hill, 2012) .

Six female researchers (i.e., two PhD students, one MA students, two undergraduate students, and one professor) were involved in the present study. One of the doctoral students and the professor served as auditors throughout the process, while the remaining team members served as CQR coders. The other doctoral student conducted all the interviews. All researchers who were unfamiliar with CQR were trained in this research method. Prior to data collection, the research team discussed their beliefs regarding PM measures to minimize the influence of biases. The doctoral students and professor believed that PM measures are useful in clinical practice; however, only one doctoral student and the professor has experience using them in practice. The undergraduates had no prior beliefs about outcomes monitoring. These biases were acknowledged and discussed in consensus and auditor meetings to reduce their influence on the results.

To characterize what motivates clinicians to start, as well as the process of selecting and maintaining PM measures, frequencies of each category within the three domains (i.e., Initial Motivation, Selecting Measures, and Maintaining Use) were calculated. Based on the recommendations of Hill (2012) , categories were classified as general (all or all but one participant), typical (more than half of participants―general), and variant (3 or more participants―typical). Rare categories (≤ 3) were not explored further. Categories with fewer than six participants were excluded because our research questions required the examination of overlap among categories so that very small categories did not contain sufficient meaningful data.

To address the final question of how initial motivation relates to the selection and maintenance of PM, Selecting Measures and Maintaining Use categories were analyzed in reference to the Initial Motivation category. For example, nineteen of the twenty-five participants endorsed Fit as one of their important Initial Motivations. Among those 19 participants, typical and variant categories for Selecting Measures and Maintaining Use categories were then identified to understand what happens when Fit is an important reason for deciding to implement PM. The frequencies to denote general, typical, variant, and rare were calculated in reference to the n of the motivation category being explored (e.g., in reference to the 19 participants who nominated Fit as an initial motivation).

3. Results

Categories are presented within each domain explored (see Table 3). Connections between domains―specifi- cally, Initial Motivation factors as they relate to Selecting Measures and Maintaining Use―are presented in Figure 1 and Figure 2. Initial Motivation categories included the following: fit with therapist (n = 19), effectiveness and therapist development (n = 13), enhancing the role of the client (n = 11), gaining objectivity or additional data (n = 8), and accountability (n = 7). Within each Initial Motivation category, the selection criteria

Table 3. Categories, category frequencies, and exemplar core ideas for each domain explored.

Figure 1. Model demonstrating the patterns of Selecting Measures and Maintaining Use categories among the Initial Motivation categories (i.e., Fit with Therapist and Effectiveness and Therapist Development). Frequencies of participants endorsing each Initial Motivation category are presented with the associated label (T = typical; V = variant). Frequencies of participants endorsing the Selecting Measures and Maintaining Use categories among each Initial Motivation category are also presented with the associated label (T = typical; V = variant).

Figure 2. Model demonstrating the patterns of Selecting Measures and Maintaining Use categories among the Initial Motivation categories (i.e., Enhancing the Role of the Client, Gaining Objectivity and/or Additional Data, and Accountability). Frequencies of participants endorsing each Initial Motivation category are presented with the associated label (T = typical; V = variant). Frequencies of participants endorsing the Selecting Measures and Maintaining Use categories among each Initial Motivation category are also presented with the associated label (T = typical; V = variant).

and maintenance factors that were reported with general and typical frequency are described; only relevant or especially interesting categories with variant frequencies are presented.

3.1. Fit with Therapist Approach, Theory, and Values

Participants typically indicated that they were initially motivated to use a PM measure because it fit with their theory, approach, or values (n = 19). A scientific orientation, solution-focused approach, and the common factors perspective were commonly endorsed orientations that participants felt aligned well with the use of a PM measure. One participant described their initial motivation as coherence between PM and their solution-focused and client centered orientation.

I was doing a Master’s in solution-focused brief therapy and one of my colleagues in the course introduced the measure to the class [...] I felt that [it] was very coherent with my style of work. Trying to really understand what was going to be helpful to the client, the client’s point of view, so that’s how it all started.

3.1.1. Selecting Measures

Among those who cited Fit as an initial motivation, convenience factors were typically considered in selection (n = 11). Brevity, practicality, affordability, interpretability and comprehensibility of the measures were all named; brevity was discussed by all but two of these participants. Psychometrics (n = 7) and usefulness (n = 7) were also considered when selecting a measure, although with less frequency (variant). When considering psychometrics, validity, reliability, and correlations with other reputable measures were often discussed. Regarding usefulness; engaging the client, increasing efficiency, and focusing the treatment were commonly the focus of attention. One participant discussed their consideration of a measure’s brevity and usefulness in the context of their orientation.

The OQ-45 is 45 items long and even in the best implementations, it is given once or twice, maybe three times in a course of treatment. That doesn’t provide for the … type of feedback that I think is helpful in guiding patients or patients guiding the treatment in terms of what progress is being made. So a 45-item implement survey is certainly not helpful for us…

3.1.2. Maintaining Use

Among those who discussed Fit as an initial motivation, engaging clients was typically discussed with regard to maintaining use (n = 13). Client engagement encompassed ideas such as discussion in session, collaboration between the therapist and client, feedback from client, client reflection, and client-directed therapy (Duncan, Solovey, & Rusk, 1992) . This participant discussed the importance of client empowerment in implementing these measures long term.

…what we are really doing is creating an environment in which consumers or clients become the subject of their treatment instead of the object of their treatment. So this helps us [practice the techniques that research has shown to contribute] a substantial amount more to the various treatment outcomes, than does the model that we were using. And by in large our data shows it. We’re seeing patients on a more regular basis, we're having less no shows, we’re having termination sessions. Before we did this, I don't think we had more than 2% of our clients come in for a termination session […], but now, we actually have people plan discharges and it actually contributes to planning the course of their treatment […]I think that is a very healthy and positive thing.

Improved effectiveness (n = 10), guiding treatment (n = 10), and convenience (n = 10) were also typically discussed as factorsthat contributed to maintaining use. One participant was motivated by an awareness that clients tend to be more kind than honest and that having clients fill out a measure revealed more, shaping their work and making it more effective.

Well, it has saved my butt several times!… [I] have gotten some really good feedback about stuff that I had no idea was going on … so it has really helped me to shape my professional approach and the things that I pay attention to and it is just kind of the way I do therapy with clients.

Participants also typically discussed continuing to use measures because they helped to guide treatment (i.e., guide session, guide treatment, and/or guide treatment length). The tools allowed clients to guide the session; “and the biggest thing that really goes into changing a culture like that is allowing that process of empowering the voice and choice of clients in their own treatment process.” This factor was especially relevant to participants endorsing a client-directed model (Duncan, Solovey, & Rusk, 1992) .

Finally, participants also typically discussed continuing to use measures because they were convenient (i.e., fast, easy, inexpensive, and unobtrusive).

3.2. Effectiveness and Therapist Development

The second reason that typically motivated therapists to use PM measures was the benefits related to Effectiveness and therapist development (n = 13). These benefits included assessing effectiveness, improving effectiveness, assessing and improving outcomes, developing skills, developing trainee skills and developing confidence in skills. One participant highlighted his desire to remain aware of his limitations and compensate by using PM measures.

[I] really became convinced that it was very important for me to stay aware of those outcomes, to stay aware of my own limited ability to assess those outcomes in a comprehensive and valid manner, and to compensate for my own clinical deficiencies in judgment by using psychometrics. And that was the beginning of a very long journey.

3.2.1. Selecting Measures

Among those participants who were initially motivated by benefits related to Effectiveness and therapist development, convenience (n = 6) and psychometrics (n = 6) were typically involved in their selection process. Convenience factors that were named included brevity, practicality, affordability, and interpretability. Motivated by efficiency, one participant stated, “But that was my first concern, to be efficient in the work that I did. That was always an issue of mine”. This same participant also considered brevity, efficiency, and a low energy investment when selecting a measure.

When considering psychometrics, participants discussed validity, reliability and correlations with other reputable measures. One participant who discussed calculating effect sizes as an initial motivation considered psychometrics when selecting a measure.

3.2.2. Maintaining Use

Participants motivated by Effectiveness and therapist development typically (n = 8) indicated that improving effectiveness also motivated continued use. Thus, initial motivation remained the reason to continue use. Participants also typically discussed continuing to use measures because they helped identify issues in therapy (n = 8). Several types of issues were named including therapist mistakes, client problems, alliance ruptures, and lack of progress. One participant discussed how PM measures were used to identify issues that needed to be resolved, informing the participants of changes that needed to be made or work that needed to be done (i.e., therapist development).

…sometimes there have been cases that, if it weren’t for the measures, I would’ve thought were doing poorly, but they aren’t. And [there are] also cases where um I thought everything was going fine and [it] wasn’t […] It alerted me [to the fact that] I needed to change my game… to what cases do I bring to supervision and which ones do I just leave alone.

3.3. Enhancing the Role of the Client

Less frequently (variant), participants discussed being initially motivated to use a PM measure to enhance the role of the client in therapy (n = 11). Participants explained that they were motivated to use a tool that would foster collaboration in treatment, facilitate client-centered therapy, and help gain the client perspective.

I think [what] helps people to change [is] giving them a voice in their own care, respecting their input and not just respecting the input but adjusting services based on that input… [the measures] are just laid right on the table and discussed in very transparent ways and that’s how they were designed to be used. And I know that that’s what the co-developers intended for those measures.

3.3.1. Selecting Measures

Among those citing Enhancing the role of the client as an initial motivation, usefulness was the most commonly considered factor when selecting measures (n = 7). Participants typically highlighted the usefulness relative to program evaluation, session-by-session feedback, prevention of drop-out, client engagement, and gaining client perspective. Utility for client engagement and gaining client perspective were also factors that were initial motivators in this group. “The other thing that fascinated me in reviewing [the PCOMS] was that it appeared to have an ability to engage folks more, […] In fact, we changed our whole intake process, we even renamed it to the initial engagement session because engagement was still one of the most critical factors in the type of work that we do.”

Convenience was also typically considered when selecting measures (n = 6). Specifically, participants discussed considering brevity, affordability, practicality, and scoring procedures. One participant specifically discussed considering a measure because it struck a balance between satisfying their initial motivation to enhance the role of the client and convenience.

We could first allow our clients to continue to use whatever treatment they deemed appropriate and necessary […], and secondly not take time away from the session in any great degree, something that could be filled out quickly-this seemed to be something that was worthwhile.

3.3.2. Maintaining Use

Among those citing Enhancing the role of the client as an initial motivation, improving effectiveness, enhancing the role of the client, and engaging the therapist were commonly discussed as reasons for continuing use. Relative to improved effectiveness, participants typically discussed continuing to use measures because they helped improve both therapist and treatment effectiveness (n = 8). One participant specifically mentioned that they were motivated to continue by the increase in percentage of reliable change outcomes within their organization.

Participants also typically returned to their initial motivation and continued to use a PM measure because it enhanced the role of the client (n = 8). Specifically, the measures were reported to have facilitated discussion, collaboration, client feedback, client reflection, and client-directed therapy (Duncan, Solovey, & Rusk, 1992) . One participant discussed how the measures improved discussion, “[you learn] overtime how to read [them], how they answer those questions. And it provides a rich source to go back and talk to them about it.” How client-feedback improves treatment was also noted, “You get honest responses back from people that allow you to make adjustments. It really goes a long way to increasing the alliance with your clients.”

Participants who were initially motivated by Enhancing the role of the client also typically discussed continuing to use a PM measure because it helped to engage the therapist (n = 7). One participant who used a measure with trainees discussed the benefits of feedback to both the client and therapist.

I think [PM] becomes interesting, certainly for us, in terms of our own our own feedback but it’s also interesting for the clients too […] Are [things] going better, worse, about the same? And that’s the same thing that I do with supervising, is encourage them to graph the results, share them with clients, keep these results as part of their own professional accountability.

3.4. Gaining Objectivity or Additional Data

Less frequently, participants discussed being initially motivated to use PM measures to gain objectivity and/or additional data (n = 8).

I’ve always been well aware that the nature of the work that we do can be very subjective … sometimes you think someone is getting better and they’re not. And sometimes you think they’re not, and they are. I’ve been doing this for long enough that I’ve been humbled enough not to trust my opinion either way.

Some participants were more specific with what they were looking for:

There were a few criteria that we wanted [the PM measures] to have―one of them being, for example, being able to say whether someone falls within a clinical distribution or not, whether they are in the normal range, and the other one having some kind of metric for whether somebody has made a meaningful change, whether clinically significant or statistically significant change.

Maintaining Use

With typical frequency, participants who were initially motivated to gain objectivity or additional data continued using measures because they helped guide treatment (n = 5). One participant discussed how they use the additional data to know when to end treatment.

These measures are much more outcome focused … if the client comes and they’ve gotten what they need from therapy and it’s only taken them 4 sessions then that’s enough. You shouldn’t interpret that as resistance or the person is avoiding big issues in their lives. […] I think you reach a point where the response rate is so flat that adding more sessions doesn’t incrementally give you that much more. I’m much more interested at the point where they start to find out a bit. At that point for me, that is when therapy should start [to] begin winding down.

3.5. Accountability

Participants (n = 7) indicated that accountability was another initial motivation to use PM measures. Accountability was discussed by therapists both externally, in relation to third-parties, and internally, as a desire to be accountable to themselves and their clients.

We tend to blame it on the clients instead of looking at our own process and looking at what we have done or not done. And so I've always wanted to have some measure of accountability, to find out whether or not what we are doing is helpful for whom and what we can do to make our services more efficient and more useful.

Maintaining Use

Among participants who cited accountability as an initial motivation, providing information to third parties (n = 4), increasing effectiveness (n = 4), guiding therapy (n = 4), and convenience (n = 4) were discussed as reasons for maintaining use. Third parties mentioned included a hospital, physicians, the ministry and future clients (i.e., advertising). This factor was not mentioned by all participants initially motivated by accountability, as some participants were only seeking internal accountability.

Typically, participants also discussed increasing both therapist and treatment effectiveness. For this participant, the improved efficacy of treatment is attributed to accountability, as their response to negative feedback is changing techniques.

[If] there’s no progress after three sessions, I talk about it with clients … “I’m concerned that you’ve seen me three times and there’s no appreciable change, or you’re going in the wrong direction, and I wonder what we could be doing differently.” So we have that conversation and then again at six or seven sessions. And I’ve transferred clients or…tried to do something different, dramatically different.

Typically, participants also discussed continuing to use a measure because they helped to guide treatment in terms of directing sessions, the course of treatment, and the length of therapy.

Lastly, participants also typically discussed continuing to use a measure because it was convenient. Among those who were initially motivated by accountability, maintenance of use was facilitated by easy, inexpensive, and unobtrusive measures.

4. Discussion

4.1. Initial Motivation and Selecting Measures

Regardless of initial motivation, participants most often named psychometrics, usefulness, and convenience as reasons for selecting a measure. Psychometrics is also the most commonly nominated factor of influence in current research frameworks for mental health quality measurement (Table 1). Hence, researchers and clinicians agree about the importance of reliable and valid measures. Similarly, at least one of the convenience factors - brevity, practicality, affordability, simplicity, and interpretability ? was discussed in most frameworks (Table 1) and was typically a reason for selection in the clinician-participants. It seems that clinicians align well with researchers in wanting psychometrically strong and convenient measures.

Usefulness is a different story. This feature was only cited by two of the ten frameworks, both of which discussed usefulness in the context of utility across settings (Erbes et al., 2004; Green & Gracely, 1987) . Conversely, the clinician-participants framed usefulness in terms of improving client engagement, program evaluation, session-by-session feedback, prevention of drop-out, and creating focus in therapy. We can understand this as an example of the practice-research gap that is so frequently discussed in psychology (Kazdin, 2008) and other applied disciplines. The reviewed frameworks were developed by groups of researchers, revised from past frameworks, and derived from reviews of the literature. Relative to utility, the frameworks take an administrative or organizational perspective, referencing usefulness across settings. The clinicians understand utility from a clinical perspective; what is useful is what makes my work with clients better. In an effort to close the practice-research gap, frameworks need to give greater voice to clinician perspectives; in particular, utility should be framed in terms of what benefits clients.

Convenience was the most commonly cited selection factor in terms across the examined Initial Motivation categories. Convenience was also cited as a typical reason for maintaining use (Table 3). This finding may be a reflection of our sample, the majority of whom were PCOMS users. PCOMS users may value convenience, as a notable feature of the PCOMS is its brevity. In fact, the PCOMS was developed to address practical concerns regarding the Outcome Questionnaire, namely the length of the measure (Duncan, 2012) . That being noted, practical reasons have been identified as a primary barrier to using outcome measures in general (Hatfield & Ogles, 2007) , so it is not surprising that they may also figure prominently in selecting and maintaining use of a any measure, not just the PCOMS. A number of measures already have online interfaces that are easy for clients to access and use; reports can be delivered to clinicians quickly and conveniently. If we are serious about increasing the use of PM in practice, developers and distributors of measures need to prioritize communications about convenience to clinicians. Convenience needs to figure prominently in training and clinicians need opportunities to understand how to integrate rapid-use systems into their clinics.

4.2. Initial Motivation and Maintaining Use

Understanding factors that helped to maintain use of measures provides insight into the framework factors that should be emphasized to facilitate continued use. Clinicians commonly discussed the importance of improving effectiveness as a maintenance factor. Several frameworks suggest considering measures that demonstrate the means by which treatments produce effects or ones that have an acceptable sensitivity to change (Table 1). These factors would certainly contribute to increased efficacy. Emphasizing these factors or more explicitly stating efficacy as a selection factor may help to increase PM adherence.

The remaining factors that influenced continued use for these participants were not specifically mentioned in the frameworks; these include engaging clients, enhancing the role of the client, helping to identify issues in therapy, helping to guide treatment, providing information to a third party, and helping to engage therapist. The lack of acknowledgement of these factors may be because the frameworks were developed in reference to all mental health quality indicators, whereas clinicians were referring to PM measures specifically. A framework for selecting PM measures specifically should consider these important clinical parameters.

A common point of discussion within several of the maintaining use factors was the endorsement of client-directed therapy (i.e., engaging clients, enhancing the role of the client and helping to guide treatment). Client-directed therapy is a term coined by a PCOMS developer (Duncan, Solovey, & Rusk, 1992) ; it refers to therapy that uses ongoing client perceptions about the fit and progress of therapy to direct options and provide the ultimate litmus test for success. Although client-directed therapy does not preclude use of PM measures other than the PCOMS, participants who discussed their endorsement of this practice were PCOMS users. Hence, the facilitation of client-directed therapy in maintaining use may be specific to PCOMS users.

Effectiveness and therapist development, enhancing the role of the client, and accountability were all factors that initially motivated participants to consider PM and also made them want continue to using it. Effectively, the participants found that the measures delivered on what they perceived to be their promise in these areas. This is consistent with a study examining practice change in primary care settings that discussed a spiral of motivation and change: motivation was necessary for change, and practice feedback and an appreciation for the feasibility of change were necessary for motivation (Ruhe et al., 2005) . The fulfilled expectations experienced by our sample of participants acted as positive feedback and interpreted feasibility; these factors motivated continued use. As the field moves to a broader dissemination of measures, it will be important to investigate clinician perceptions of how the measures deliver on their promises. In particular, we need to study how measures contribute to therapist development, to client engagement, and to practitioner accountability to clients and to third-parties.

5. Limitations

One important limitation of this study is the large proportions of the participants who were male, over the age of 51, and using the PCOMS.In particular, because the PCOMS is notably brief, the significance of convenience may have been overstated.However, it is important to consider that the PCOMS is the most commonly used PM measure among Canadian psychologists (Ionita & Fitzpatrick, 2014) ; therefore, this sample may be somewhat representative of Canadian PM users. Regardless, the skewed sample is recognized as a potential limitation of this study; older, male, PCOMS users may differ from other PM users in their motivation, selection, and maintenance behaviors. Additionally, both the Selecting Measures and Maintaining Use domains were analyzed with respect to the Initial Motivation domain. The analysis was based on the assumption that initial motivation impacts selection and maintenance behavior. It is also possible that other, unexamined variables may be responsible for the patterns demonstrated in the data. Finally, although the researchers attempted to privilege the voice of the participants and to use multiple perspectives in the CQR analysis, researcher bias regarding the importance of PM may have influenced the findings.

6. Summary and Implications

This study compared researchers’ frameworks for selecting measures to clinicians’ actual experience of selecting and continuing to use PM measures. The results suggest that for frameworks to be more applied to PM use, they need to integrate clinician perspectives; in particular they should consider paying more attention to clinical utility factors. To improve PM adherence, frameworks should include guiding treatment, identifying issues, enhancing the role of the client, engaging the therapist, and providing information to third parties. Frameworks created in consultation with clinicians can decrease the burden of PM measure selection and increase the proportion of clinicians using and continuing to use PM measures. Of importance in these findings are the roles of convenience and effectiveness in maintaining use regardless of initial motivation. The voices of participants clearly point to the importance of striking the right balance; it is not either effectiveness or convenience, but both.


The work was supported by the Canadian Institute of Health Research under the Frederick Banting and Charles Best Canada Graduate Scholarship.

Conflicts of Interest

The authors declare no conflicts of interest.


[1] Anker, M. G., Duncan, B. L., & Sparks, J. A. (2009). Using Client Feedback to Improve Couple Therapy Outcomes: A Randomized Clinical Trial in a Naturalistic Setting. Journal of Consulting and Clinical Psychology, 77, 693-701.
[2] Burlingame, G. M., Dunn, T. W., Chen, S., Lehman, A., Axman, R., Earnshaw, D., & Rees, F. M. (2005). Selection of Outcome Assessment Instruments for Inpatients with Severe and Persistent Mental Illness. Psychiatry Services, 56, 444-451.
[3] Burlingame, G. M., Lambert, M. J., Reisinger, C. W., Neff, W. M., & Mosier, J. (1995). Pragmatics of Tracking Mental Health Outcomes in a Managed Care Setting. The Journal of Mental Health Administration, 22, 226-236.
[4] Duncan, B. L. (2012). The Partners for Change Outcome Management System (PCOMS): The Heart and Soul of Change Project. Canadian Psychology, 53, 93-104.
[5] Duncan, B. L., Solovey, A. D., & Rusk, G. S. (1992). Changing the Rules: A Client-Directed Approach to Therapy. New York: The Guildford Press.
[6] Erbes, C., Polusny, M. A., Billig, J., Mylan, M., McGuire, K., Isenhart, C., & Olson, D. (2004). Developing and Applying a Systematic Process for Evaluation of Clinical Outcome Assessment Instruments. Psychological Services, 1, 31-39.
[7] Froyd, J. E., Lambert, M. J., & Froyd, J. D. (1996). A Review of Practice of Psychotherapy Outcome Measurement. Journal of Mental Health, 5, 11-15.
[8] Green, R. S., & Gracely, E. J. (1987). Selecting a Rating Scale for Evaluating Services to the Chrinically Mentally Ill. Community Mental Health Journal, 23, 91-102.
[9] Harmon, C., Lambert, M. J., Smart, D. W., Hawkins, E., Nielsen, S. L., Slade, K., & Lutz, W. (2007). Enhancing Outcome for Potential Treatment Failures: Therapist-Client Feedback and Clinical Support Tools. Psychotherapy Research, 17, 379-392.
[10] Hatfield, D. R., & Ogles, B. M. (2004). The Use of Outcome Measures by Psychologists in Clinical Practice. Professional Psychology: Research and Practice, 35, 485-491.
[11] Hatfield, D. R., & Ogles, B. M. (2007). Why Some Clinicians Use Outcome Measures and Others Do Not. Administration and Policy in Mental Health and Mental Health Services Research, 34, 283-291.
[12] Hermann, R. C., & Palmer, R. H. (2002). Common Ground: A Framework for Selecting Core Quality Measures for Mental Health and Substance Abuse Care. Psychiatry Services, 53, 281-287.
[13] Hermann, R. C., Leff, H. S., Palmer, R. H., Yang, D., Teller, T., Provost, S., Jakubiak, C., & Chan, J. (2000). Quality Measures for Mental Health Care: Results from a National Inventory. Medical Care Research and Review, 57, 136-154.
[14] Hermann, R. C., Mattke, S., Somekh, D., Silfverhielm, H., Goldner, E., Glover, G., Pirkis, J., Mainz, J., & Chan, J. A. (2006). Quality Indicators for International Benchmarking of Mental Health Care. International Journal for Quality in Health Care, 18, 31-38.
[15] Hermann, R. C., Palmer, H., Leff, S., Shwartz, M., Provost, S., Chan, J., Chiu, W. T., & Lagodmos, G. (2004). Achieving Consensus across Diverse Stakeholders on Quality Measures for Mental Healthcare. Medical Care, 42, 1246-1253.
[16] Hill, C. E. (2012). Consensual Qualitative Research: A Practical Resource for Investigating Social Science Phenomena. Washington DC: American Psychological Association.
[17] Ionita, G., & Fitzpatrick, M. (2014). Bringing Science to Clinical Practice: A Canadian Survey of Psychologists Practice and Usage of Progress Monitoring Measures. Canadian Psychology, 55, 187-196.
[18] Kazdin, A. E. (2008). Evidence-Based Treatment and Practice: New Opportunities to Bridge Clinical Research and Practice, Enhance the Knowledge Base, and Improve Patient Care. American Psychologist, 63, 146-159.
[19] Kitson, A., Harvey, G., & McCormack, B. (1998). Enabling the Implementation of Evidence Based Practice: A Conceptual Framework. Quality in Health Care, 7, 149-158.
[20] Lambert, M. J., & Shimokawa, K. (2011). Collecting Client Feedback. Psychotherapy, 48, 72-79.
[21] Lambert, M. J., Harmon, C., Slade, K., Whipple, J. L., & Hawkins, E. J. (2005). Providing Feedback to Psychotherapists on Their Patients’ Progress Clinical Results and Practice Suggestions. Journal of Clinical Psychology, 61, 165-174.
[22] Lambert, M. J., Ogles, B. M., & Masters, K. S. (1992). Choosing Outcome Assessment Devices: An Organizational and Conceptual Scheme. Journal of Counseling and Development, 70, 527-532.
[23] Lambert, M. J., Whipple, J. L., Smart, D. W., Vermeersch, D. A., Nielsen, S. L., & Hawkins, E. J. (2001). The Effects of Providing Therapists with Feedback on Patient Progress during Psychotherapy: Are Outcomes Enhanced? Psychotherapy Research, 11, 153-166.
[24] Miller, S. D., Duncan, B. L., Sorrell, R., & Brown, G. S. (2005). The Partners for Change Outcome Management System. Journal of Clinical Psychology, 61, 199-208.
[25] Overington, L., & Ionita, G. (2012). Progress Monitoring Measures: A Brief Guide. Canadian Psychology, 53, 82-92.
[26] Reese, R. J., Norsworthy, L. A., & Rowlands, S. R. (2009). Does a Continuous Feedback System Improve Psychotherapy Outcome? Psychotherapy, 46, 418-431.
[27] Reese, R. J., Toland, M. D., Slone, N. C., & Norsworthy, L. A. (2010). Effect of Client Feedback on Couple Psychotherapy Outcomes. Psychotherapy: Theory, Research, Practice, Training, 47, 616-630.
[28] Ruhe, M. C., Weyer, S. M., Zronek, S., Wilkinson, A., Wilkinson, P. S., & Stange, K. C. (2005). Facilitating Practice Change: Lessons from the STEP-UP Clinical Trial. Preventive Medicine, 40, 729-734.
[29] Schuman, D. L., Slone, N. C., Reese, R. J., & Duncan, B. (2015). Efficacy of Client Feedback in Group Psychotherapy with Soldiers Referred for Substance Abuse Treatment. Psychotherapy Research, 25, 396-407.
[30] Slade, M. (2002). What Outcomes to Measure in Routine Mental Health Services, and How to Assess Them: A Systematic Review. Australia and New Zealand Journal of Psychiatry, 36, 743-753.
[31] Slone, N. C., Reese, R. J., Mathews-Duvall, S., & Kodet, J. (2015). Evaluating the Efficacy of Client Feedback in Group Psychotherapy. Group Dynamics: Theory, Research, and Practice, 19, 122-136.
[32] Stewart, R. E., & Chambless, D. L. (2007). Does Psychotherapy Research Inform Treatment Decisions in Private Practice? Journal of Clinical Psychology, 63, 267-281.
[33] Waraich, P., Saklikar, R. S., Aube, D., Jones, W., Haslam, D., & Hamill, K. (2010). Quality Measures for Primary Mental Healthcare: A Multistakeholder, Multijurisdictional Canadian Consensus. Quality and Safety in Health Care, 19, 519-525.

Copyright © 2024 by authors and Scientific Research Publishing Inc.

Creative Commons License

This work and the related PDF file are licensed under a Creative Commons Attribution 4.0 International License.