Attitudes to Group Assessments: Prospective Psychotherapists’ Experiences of Being Assessed by Fellow Students ()
1. Introduction
The World Health Organization has estimated that about 450 million people worldwide suffer from mental illness and that most of the mentally ill do not get the help and treatment they need [1] . In the West, an estimated 50 percent of those in need have no access to necessary assistance and in developing countries, the figure is 80 percent. Even in a welfare state such as Sweden, public health is a matter of concern due to the increase of mental diagnoses [2] . Stress, anxiety and depression are all substantially increasing and those accounted for the highest number of mental illness are people working in the field of health care, but sick leave is substantially increasing within almost all occupations. In addition, mental diagnoses are also increasing within vulnerable groups such as young women (anxiety and depression), young men (neuropsychiatric diagnoses) and migrants (anxiety, depression and post-traumatic stress).
Against this background, it becomes obvious that our societies are facing growing challenges and due to the fact that CBT treatments have strong research support and have been proven cost-effective [3] , the demand for these treatments increases [4] . The growing complexity of the mental illness in combination with a growing numbers of new treatment models in CBT [5] [6] now sets a stronger demand also on therapists and trainers. There is now a need for therapists with a deep knowledge in general CBT skills, who also have a good overview on new methods and models. Training therapists is a complicated matter. Besides, being both time and resource intensive for students as well as for educators, the scientific evidence for this task is very limited [7] . The extensive research concerning treatments has not yet provided sufficient evidence and guidelines concerning therapist training [4] . Further there is even less scientific evidence for how these skills and abilities should be operationalized, learned, evaluated and assessed [8] [9] [10] [11] . Bandura [12] argued that psychotherapy is a learning process and, therefore, greater effort should be made to develop treatment methods derived from knowledge of learning and motivation. A central component in Bandura’s social learning theory [13] illustrates the importance of observing and modeling behaviors and a recent study [14] , where 2607 US and Canadian psychotherapists participated, indicates that they largely learned from supervisors and colleagues as it seemed imitated their behavior.
Although the therapist profession requires extensive theoretical knowledge, the work is largely practical. An important part of the training is therefore to stress that theoretical knowledge must be put into practice, so that students acquire the skills and abilities needed to assess an individual patient’s needs and to select and implement appropriate treatments [4] [10] [15] [16] . Education and training in those moments occur mainly in the tutorial course, where the prospective therapists carry out treatments under the guidance of trained CBT supervisors. According to an extensive survey [4] , research on guiding and supervision of prospective CBT therapists is scarce in literature. However, psychotherapy research has gathered a lot of knowledge about “what works” and “what needs to be done” in treatment [17] [18] . This knowledge has been systematized in manuals and gradually has also instruments assessing fidelity to the manuals been constructed. A continuous effort to systematize and describe the competence, therapists needs for carrying out different treatments has been made over the years. One of the more ambitious attempts in this endeavor has been made under the IAPT project (Improving Access to Psychological Therapies) of scientists and clinicians in the United Kingdom [19] . In this project general therapist competence, CBT specific competence, and diagnosis specific competence have been defined and described. Further research has also examined the relationship between therapists’ behaviors and treatment outcomes [9] [20] [21] , but it has not yet been possible to draw any clear conclusions from those studies.
How therapeutic skills should be taught is a question that has interested several clinicians and researchers. Padesky [22] as well as Liese and Beck [23] used therapy models as a base for supervision, which is now a common practice ineducational supervision [10] . Bennett-Levy [24] has also made an ambitious attempt to define and describe the knowledge systems used for learning. His model assumes that the learning skills of the therapist are enhanced by declarative as well as procedural and reflexive processes. Declarative processes include conceptual knowledge such as theory and technology and also knowledge of interpersonal functioning, while procedural processes concern interpersonal and technical skills, frameworks, attitudes, as well as rules, plans and procedures. Reflexive processes concern the ability to relate in a curious and reflective way to oneself and to the environment. Bennett-Levy and associates [8] have also evaluated which methods and tools that are most effective for each learning process. This knowledge may now help to strengthen and streamline the learning process in terms of therapist skills, but it should be noted that so far only a few studies examine strategies leading to increased competence [25] . Other and more general models used in supervision are concerned with the role of experience [26] or interactions between learning and development [27] . For the time being there is no generally accepted learning model available [25] [28] , nor any generally accepted models for feedback and assessment of achieved skills [7] [11] [29] [30] . However, it is considered important that tutoring sessions are well prepared and have an agenda [31] .
A common practice at several educational institutions is the use of rating scales for feedback and for assessing student therapist skills. There are currently a number of tried and tested assessment tools available. One of the earliest developed assessment tools was the Cognitive Therapy Scale (CTS) developed by Beck and Young [32] . The scale measures adherence and competence and have been widely used in psychotherapy research and is considered to have acceptable psychometric properties [33] . A number of other scales reminiscent of the CTS have been developed, as Safran’s Therapy Adherence Scale (STAS) [34] , Cognitive Therapy Adherence and Competence Scale (CTACS) [35] , Assessment of Core CBT Skills (ACCS) [36] , Supervisor Rating Form (SRF) [37] , the Collaborative Case Conceptualization Rating Scale (CCCRS) [38] , and the UCL scale of Structured Observation (USO) [39] .
The above mentioned scales are often extensive and thus time-consuming for both students and supervisors to learn and to administer which is somewhat of an obstacle when time and resources for guidance are limited. An attempt to improve the efficiency of the original CTS-scale has been made by Blackburn and others, resulting in the CTS-R, Cognitive Therapy Scale, revised version [40] . In the revised version overlapping material from the original CTS has been reduced and furthermore the Dreyfuss learning model [41] was incorporated. This model suggests that adult learning develops in phases. Achieved knowledge is defined by six categories: 1) incompetent, 2) novice, 3) advanced beginner, (d) competent, 4) proficient and 5) expert. CTS-R consists of a total of twelve items tapping different CBT skills. The rating scales run0-6 where adherence and competence are weighed in together. CTS-R has been widely used in research concerning supervising. The scale is considered to have a certain bias toward cognitive dimensions, but the benefits are that it is less time consuming to learn and easier to administer, which enables repeated assessments. In addition, the manual includes [40] detailed descriptions of the scale points for each question and there is also an associated “good-practice guide” with concrete examples. CTS-R is now thoroughly explored in terms of validity and reliability. It is also translated and evaluated in Sweden [42] .
The most common model for tutoring is that students are supervised in groups of four [4] . According to Kuechler [43] there are many benefits from group tutoring for both supervisors and participants. The benefit for participants may be that a safe base is created with access to mentoring which facilitates reflection and development of a professional identity. Further, group supervision provide good conditions for teaching and learning because students can follow not only their own therapies but also those performed by the colleagues and feedback is given from both colleagues and supervisors [10] [44] [45] . Learning of therapist skills is a sensitive process that can be disturbed if the groups do not work well enough. Group processes consist of factors that are unique to the group environment, which leads to changes in the group and thus affect the group’s functioning [43] [44] . It may for example include individual events, the participants’ personal style, difficulties in therapy sessions, interactions between participants and the alliance between supervisors and participants. Examples of problematic behaviors of individual team members that may affect the group negatively are participants who constantly take up too much space, are constantly critical or silent, seems bored or find it difficult to accept help. There are also supervisor behaviors that are considered to contribute to the negative group processes [25] . Supervisor styles that are too lenient or too authoritarian are described by students as sources of dissatisfaction. In order to create good conditions for learning, the supervisor must establish and maintain a good alliance and observe, assess, support and give constructive feedback. The existence of a good alliance between supervisor and students is described as one of the most important factors that secure cooperation in the group. Students also highlight the importance a goal-oriented behavior from the supervisor as well as a non- judgmental approach towards participants [25] . However, it was not possible to find previous performed studies that analyzed the underlying factors for prospective psychotherapists’ attitudes to group assessments based on the CTS-R.
Most institutions that trains psychotherapists in CBT uses some form of rating scale of the type already discussed above as a basis for learning and evaluation [42] . The rating scale is introduced early in the training and the aim is that the students should have a scientifically evaluated instrument as the basis for the learning process. Using the rating scale, different therapist skills can be described, practiced, evaluated, and assessed during on-going tutoring. Students are continually assessed and receive frequent feedback from both supervisor and colleague students. As previously noted, occasionally difficulties occur. Sometimes the learning process is hampered when a group develops a destructive way to provide feedback or by participants who become overly cautious or idealizing in their feedbacks. Typically, these situations can be used to strengthen the process and bring about constructive, investigative and reflective discussions. Parallels may also be drawn to difficulties experienced in the treatment process. The aim of the current study was to examine prospective psychotherapists’ attitudes to group assessments based on the revised version of the Cognitive Therapy Scale (CTS-R).
2. Methods
2.1. Participants
Participants were 56 prospective therapists specializing in CBT (7 men, 48 women, and one without data) with an average age of 45.65 years (SD = 7.78, range = 31 to 64). Of the participants 19 individuals attended the basic training in CBT, and 36 individuals attended the psychotherapist program of CBT. The study took place in 2016 and the students were recruited from two educational institutions in Gothenburg, Sweden. The basic training was an undergraduate course of three semesters focused on basic CBT-knowledge in theory and practice, while the psychotherapist program consisted of six semesters which gave full jurisdiction as a licensed therapist. The participants were either in ongoing programs or had just received their diplomas. The basic professional affiliations of the participants allocated as follows: psychologists (15 persons), social workers (17 persons), and other health care staff (24 persons). In light of the purpose of the study, participants were divided in two groups after a specific procedure (see section Data Processing) based on whether or not they perceived it as a good idea to be assessed by their fellow students in the tutoring group. There were 18 participants who experienced this as less good (Negative) and 38 as very good (Positive). Analysis (Chi-Square, 5% level) showed no significant relations (ps > 0.05) between attitudes to group assessment (negative or positive) in terms of gender, level of education in CBT or profession. An Independent Samples t-test (5% level) showed no age difference regarding attitudes to group assessment (p > 0.05).
2.2. Instruments
Questions about the CTS-R and group assessments. A questionnaire was constructed to investigate students’ attitudes to the CTS-R and group assessments. The 36 questions were answered by check marks on Visual Analogue Scales, each with a length of one decimeter (about four inches) which envisioned the scale from 0 to 100. As “anchor points” clearly articulated response alternatives were used such as “not at all-in the highest degree” and “significantly reduced-significantly increased” The questions are presented in Table 1.
Table 1. The 36 items in the survey “Questions about the CTS-R and group assessments”.
There were three questions about the perception of the CTS-R as a rating scale to describe and assess therapist skills. These questions concerned the extent to which the students were familiar with the CTS-scale and whether they considered it appropriate to describe and assess therapist skills. Further questions were asked about the usefulness of the CTS-R for the students own learning processes, and how they valued the CTS-R as an instrument. Other questions aimed to investigate the students’ experiences and attitudes to group assessments. These questions concerned the extent to which they perceived group evaluations as helpful for their own learning process, and whether or not their own learning and self-esteem were affected by the assessments, as well if they felt that the group participants learning was affected. The study’s main question (To what extent do you think it is a good idea with group assessments?) was embedded as the penultimate item in the survey, followed by a control question as to whether they thought it was better to be individually assessed with only the supervisor. Thereafter an open question followed that gave students the opportunity to comment on their answers. The homogeneity of the instrument was tested by Cronbach’s Alpha which gave a high value (alpha = 0.92).
Background data. The distributed questionnaire also included demographic data on level of education in CBT, profession, gender, and age.
2.3. Procedure
In order to recruit participants for the study two institutions which train therapists in CBT was approached, namely the Department of Psychology at Gothenburg University and the Evidens University College. These accepted participation and provided addresses to students. The questionnaire, a cover letter, as well as an addressed and stamped envelope for returning were sent to a total of 100 students who attended undergraduate courses in CBT or the therapist program. The cover letter described the study’s purpose and scope, and also gave the information that participation was voluntary and anonymous. The questionnaire was thus answered anonymous and responses were sent directly to one of the authors. Of the 100 sent out questionnaires 56 responses were obtained.
2.4. Data Processing
Descriptive analysis of the distribution of participants’ responses to the question “To what extents do you think it is a good idea with group assessments?” (M = 67.09, SD = 28.00) indicated a natural division (cut point = 64), where participants could be identified who had a very positive view of group assessments (Positive) including a high mean value and a low standard deviation (n = 38, M = 83.29, SD = 9.66) compared to participants (Negative) who experienced assessments from fellow students as less good (n = 18, M = 32.88, SD = 22.55). This categorization was to form the independent variable in comparisons (Attitudes to group assessment), while the original not categorized item was used in correlation and regression analyzes.
2.5. Ethical Considerations
This study followed the ethical standards of the World Medical Association declaration of Helsinki concerning Ethical Principles of Medical Research Involving Human Subjects. The data were collected by a psychologist (first author) as a preparation for a thesis in order to become a qualified specialist in clinical psychology. In addition the questionnaire was answered anonymous and responses were sent directly to the first author. Given these conditions, in accordance with the Swedish rules on ethics, the material contained in the report can be used to compile an article.
3. Results
3.1. Comparisons between Groups
Attitudes to group assessment. Statistical testing (Levene’s Test of Equality of Error variances, 5% level) showed that conditions for multivariate analysis of variance did not exist in comparisons between the independent variable groups for about half of the included items why it was decided to instead use Independent Samples t-tests with correction for inconsistency (equal variances not assumed). Independent variable was the categorized variable Attitudes to group assessment (negative, positive) and dependent variables were the visual analog scales with the exception of item 35 which was the basis for the independent variable. The statistical analyzes yielded significant effects (ps < 0.05) for items with the following numbers: 4, 5, 6, 7, 8, 11, 12, 21, 22, 24, 28, 29, 32, 33, and 36. For means, standard deviations and statistical coefficients at the significant comparisons see Table 2.
Gender. Statistical analysis (Mann-Whitney U-test, 5% level) with all 36 items as dependent variables showed significant differences in regard to Gender for only four items: item 14 [z = −2.22, p = 0.025; Men (M = 47.86, SD = 12.42), Women (M = 35.69, SD = 9.78)], item 17 [z = −2.01, p = 0.043; Men (M = 78.29, SD = 18.16), Women (M = 65.75, SD = 17.96)], item 18 [z = −3.40, p < 0.001; Men (M = 46.57, SD = 9.03), Women (M = 64.02, SD = 11.50)], and item 34 [z = −2.06, p = 0.037; Men (M = 70.86, SD = 16.81), Women (M = 44.55, SD = 31.39)].
Level of education in CBT. Further analysis using the Mann-Whitney U test (5 % level) with all 36 questions as dependent variables and the two levels of education as independent variable showed no significant differences in terms of educational attainment (ps > 0.05).
Profession. Statistical analysis (Kruskal-Wallis, 5% level) with all 36 questions as dependent variables and the tree categories of profession as independent variable revealed significant effects concerning items 1, 9, 10, 24, 30, and 31 (ps < 0.05). Multiple comparisons according to Kruskal-Wallis indicated that the psychologists had responded with significantly lower scores on the six items as compared to the other two occupational groups, while the social workers gave significantly the highest score on four of the scales (1, 10, 30, 31) and persons belonging to the category “other health care staff” gave significantly the highest
Table 2. Means (M) and standard deviations (SD) for items in the survey in regard to attitudes to group assessment (Negative, Positive), coefficients for significant results (t, df, p), and all participants (All).
Note: Item marked with * indicate significant difference in respect to Group (negative, positive). Note: Item 35 is the basis for the independent variable and is therefore not included in the group comparison.
score on two of the scales (9, 24).
Age. Participants were divided into three age groups (Visual Binning: width = 33.33, cut points = 42 and 50), Younger (n = 21, M = 38.10, SD = 3.55), Middle (n = 16, M = 45.06, SD = 2.05) and Older (n = 18, M = 55.00, SD = 3.24). Analysis (Kruskal-Wallis, 5% level) with all 36 items as dependent variables and age group as the independent variable showed no significant effects (ps > 0.05).
3.2. Measures of Relations
Correlations. The 36 items were correlated (Pearson’s r, 5% level) with Item 35 (To what extents do you think it is a good idea with group assessments?) which is reported in Table 3.
Regression Analysis. In order to examine whether different aspects tapped in the visual analogue scales of the survey, affect the variance of the study’s main question, a step-wise linear regression analysis was conducted. Item 35 (To what extent do you think it is a good idea with group assessments) was used as the criterion variable and all the other items were used as predictor variables. The analysis generated five models as follows: item 21 (Step 1: Adj R2 = 0.64, p < 0.001); items 21 and 36 (Step 2: Adj R2 = 0.80, p < 0.001); items 21, 36 and 1 (Step 3: Adj R2 = 0.86, p < 0.001); items 21, 36, 1 and 12 (Step 4: Adj R2 = 0.88, p < 0.001); and finally items 21, 36, 1, 12 and 17 (Step 5: Adj R2 = 0.90, p < 0.001). The results showed that item 21 (To what extent do you think that you have been fairly assessed by your colleagues?) alone accounted for a full 64% of the variance in the first step. The second step included not only item 21 but also item 36 (To what extent had it been better to be individually assessed only by the supervisor?), which increased the impact with 16% to 80%. The subsequent steps indicated that the five visual analogue scales together could explain 90% of the variance of the criterion variable.
3.3. Comments on the Open Question
The questionnaire also included an open question where the students were able to justify their positions on how they had responded. Of the participants 38 individuals had commented on their answers where 22 persons (58%) preferred group assessments and 16 (42%) preferred individual assessment with only the supervisor. The students who were in favor of group assessments described the advantage of getting a more extensive feedback and not only from the supervisor. They also considered it instructive to share each other’s strengths and potential mistakes. Further it included feeling less vulnerable to the supervisor. The students who preferred individual assessments highlighted that they felt uncomfortable when evaluated by fellow students. They believed that some colleagues were uneven in their knowledge of the CTS-R and even more common that assessments were influenced by relationships between team members. They also felt that the supervisor in these respects was more knowledgeable and objective. Finally, it was argued that individual evaluation could generate a freer and more in-depth assessment, which was considered to benefit the learning process.
Table 3. Correlations (Pearson’s r) between item 35 (To what extent do you think it is a good idea with group assessments?) and the remaining items.
Note: **Significant correlations at 0.01 level (double-tailed). *Significant correlations at 0.05 level (double- tailed).
4. Discussion
The main result of the present study showed that a majority of students consisting of 38 participants (68%) had a very positive approach to group assessments, while a minority of 18 participants (32%) was more skeptical or negative. Both groups generally had a positive view of both the CTS-R and the supervisors.
The most crucial aspects of the students’ perceptions concerning attitudes to group assessments, was whether they felt that they had been fairly evaluated by their fellow students in the group (64 percent of the variance) and if they preferred an individual assessment with only the supervisor (16 percent of the variance). A full 80 percent of the variance in terms of attitude towards group assessments could thus be explained by only these two aspects! That a majority had a positive view of group assessment is in line with previous research as well as that a minority was more negative [4] [43] [44] . There is an extensive research on the characteristics of good supervision [4] [10] [16] [31] [43] [46] [47] [48] . Many highlight the strength of group supervision and there is a widespread belief that the group format provides the most effective conditions for learning. Among the benefits are, in addition to cost-effectiveness, the notion that the group can be used as a safe base for learning, a platform for developing skills and professional identity. However, there is also an established point of view that the group format is not uncomplicated, as was shown in the results of the present study where a third of the students were skeptical or negative to be assessed by their fellow students in the group. Group processes affects both learning and assessment. Such processes consist of factors that are unique to the group environment and create mechanisms that lead to changes in the group which affect the group’s functioning. It has not been possible to find previously performed studies that analyzed the underlying factors of prospective psychotherapists’ attitudes to group assessments based on the CTS-R.
The students expressed a firm belief in the CTS-R; they considered themselves familiar with the scale and regarded it as relevant for both describing and assessing therapist skills. Further, it may be worth noting that the students’ views on the CTS-R was positive regardless attitude to group assessments, occupation, education level, gender or age. However, the negative group and the positive group differed in terms of the benefits they believed the scale had on their own learning processes. Correlation analysis indicated that this difference was associated with the experience of the group assessments. Results also showed that it was not the relationship to the supervisor or his/her evaluations that had been problematic. On the contrary, the students in both groups generally had a positive opinion of their supervisor. Instead, it was the extent to which participants considered themselves as fairly judged by their student colleagues that had the greatest impact on how they regarded group assessments, indicating that the question rather concerns the experiences of the group and the way the group functioned during the evaluations.
There were significant differences in terms of gender for only four items. Concerning the question of how the perception of the CTS-R was influenced by the attitude of the supervisor, male participants experienced to a greater degree than female participants, that their attitude to the supervisor affected how they viewed the instrument. The remaining differences in regard to gender concerned questions of honesty in evaluations. The men were more likely to overestimate themselves compared to women. The men also expressed the opinion that colleagues more often had been honest in their evaluations, however, when this was not the case, the men thought that they had been somewhat underestimated by the colleagues. Contrary to the men, the women expressed that their colleagues to a lesser degree had been honest, and that they rather had been somewhat overrated by the colleagues. These results are in no way surprising, given that a very comprehensive early research has shown that women have a lower degree of self-confidence compared to men [49] [50] .
There were no differences in terms of educational attainment for any of the 36 scales that is, whether the students attended basic education or the psychotherapist program. That there were no differences with respect to the group assessments is perhaps less surprising, since these attitudes were primarily related to the issue of groups and colleagues. But a difference could have been expected concerning the extent of familiarity for CTS-R, since students at undergraduate level had received less time and experience with the instrument. However there was no such difference and one possible explanation may be that the CTS-R is considered to be relatively easy to learn and to administer in relation to other, more extensive scales.
Participants who were psychologists rated lower on six of the survey questions as compared to the other professional groups. These questions concerned the familiarity of CTS-R, and questions about the perception of the supervisor and the training. The psychologists were generally less satisfied with the supervisors and the education than the other professions. Psychologists experienced to a lesser extent that the supervisor helped them to enhance their learning and their confidence during the assessments. They also expressed less benefit from the supervisors for their learning processes than the other occupations. The social workers were the professional group that rated themselves most familiar with the CTS-R and most comfortable with the supervisors’ evaluations and comments, which they experienced, helped to strengthen their self-confidence. Social workers were also the professional group who were most satisfied with their education at large. Other health care staff, mainly nurses and physiotherapists, felt to a higher degree than the other professions that the supervisors’ comments helped to strengthen their learning during the assessments. They also rated that they had been able to express their own views, if they thought that they were unfairly underestimated. The reason to the psychologists’ response pattern may be linked to the fact that psychologists have more in-depth knowledge of psychotherapy and the highest status compared to social workers and other health care staff. Psychologists may have the least interest in mixed psychotherapy training. Reports from different programs indicate that fewer psychologists choose this path and rather choose the specialist training or psychotherapy training, only open to psychologists. It may be that social workers and other health care staff are the ones who have the most to benefit and also those with the least prestige to defend.
The students were divided into three age categories and analyzes showed that there were no significant differences in regard to age. Thus, there was no obvious reason why age would be crucial concerning perception of the CTS-R or the group assessments. Admittedly, it is possible to argue that increased age brings greater life experiences which may be relevant in this context. But recent research suggests [51] that it is primarily a good climate and effective feedback systems on the psychotherapists’ workplaces that have a positive impact on the development of skills rather than general life experience.
5. Conclusion
One limitation of the current study was that it relies on a small sample. On the other hand, there was a width with respect to professional groups, education level, gender and age. The results also showed that the attitude to group assessments was not related to these factors, but mainly focused on factors related to the evaluation process, and especially the assessments from the fellow students in the group. More studies with the same focus are needed to determine the extent to which the results are generalizable. To further improve the learning process and assessments of therapist skills, it would be of great interest to get more studies from a qualitative perspective, focusing on the students’ own experience of performance evaluations together with the supervisors and student colleagues. Also qualitative studies investigating the interactions between students and their patients [52] [53] may provide a basis for further improvements of tutoring in educational groups. Bieling and colleagues [44] are among the few who both researched and proved that process factors had a significant impact on the outcome in treatment groups. They argue that this area leaves a large potential for development and improvement concerning outcome in treatment groups. The same would apply to training and mentoring of aspiring therapists. Green [54] highlighted the irony that the most empirically studied group, therapy approach (CBT) ignores the group as a “group” and as a potential therapeutic agent in itself. The results of this study support the hypothesis that the group as a “group” is very important for the learning process in CBT tutorial.
Acknowledgements
All authors had contributed substantially to the design, performance, analysis, and reporting of the work. The authors thank the participants for taking the time to participate in the study.
Conflict of Interest
The authors confirm that this article content has no conflicts of interest.