Received 18 February 2016; accepted 10 May 2016; published 13 May 2016

ABSTRACT

The Aga Khan University (AKU) renewed its undergraduate curriculum and incorporated Problem Based Learning (PBL) as one of the main teaching strategies in years one and two. PBL requires a huge pool of facilitators. Recent AKU graduates were recruited as Teaching Assistants (TAs) to support teaching. A faculty development program was conducted to prepare TAs for their teaching role. This study aimed to evaluate the effectiveness of a faculty development program (conducted in two parts) in increasing knowledge, skills and teaching performance of the participants and assess the long term retention of knowledge by participants. A longitudinal, mixed method, evaluation study was conducted at Aga Khan University from 2006-2008. The participants included all eleven TAs recruited by AKU. Workshop evaluation forms, pre, post and delayed post test and student evaluations of TAs teaching performance were used to gather data. Evaluation of the PBL workshop was much better than the one on broader aspects of medical education since the PBL workshop was in line with the participants’ goals. TAs reported perceived improvement in PBL facilitation skills. Student evaluations of TAs facilitation skills validated this perception. The workshop helped improve knowledge (p = 0.036), which increased on immediate post-test but decayed at the end of year on delayed post-test. Short training programs lead to increase in knowledge, skills and improved teaching performance. The participants retain knowledge and skills which are applied while those that are not applied regularly decay over time. Regular, periodic refreshers should be instituted to reinforce and retain knowledge and skills.

Keywords:

Faculty Development, Evaluation, Teaching Assistants, Transfer of Training, PBL

1. Introduction

Medical teaching is a complex task for which faculty is poorly prepared (Pololi et al., 2001) . Academic medical faculty members have received little or no explicit training in how to teach or in the theories and processes of teaching (Pololi et al., 2001) .

Teaching Assistants (TAs) are increasingly being employed by the Universities to augment teaching. In the past, TAs like the majority of teaching staff were provided little or no formal training in education when they were first appointed (Wilkerson & Irby, 1998) . With the passage of time, more programs are being made available to graduate students to prepare them for teaching roles (Chism, 2002) . Several universities have instituted brief entry-level workshops on instructional techniques designed to help the academic staff become more proficient at teaching undergraduates (Smith, 1983) .

Future physicians are required to educate students, junior physicians and patients (Bulte et al., 2007) . Studies have estimated that residents spend approximately 20% of their time on teaching (Post et al., 2009) . The importance of teaching as a necessary skill for medical graduates and residents has been reaffirmed by the Liaison Committee on Medical Education (LCGME, 2016) and Accreditation Council for Graduate Medical Education (ACGME, 2016) .

Training of TAs in teaching and learning is important as it accrues several benefits. It motivates students towards an academic pathway and equips them with transferable skills (Nelson et al., 2013) . Training helps them become better communicators and to have better physician-patient interactions (Dandavino et al., 2007). A sound understanding of principles of education also improves their own teaching and learning (Dandavino et al., 2007) . Soriano et al. (2010) have identified several benefits of having such training programs including development of future physician-educators, enhancement of learning and teaching effectiveness, providing teaching assistance for faculty and contributing to curriculum development. Several researchers have also reported that skills of untrained teachers are suboptimal and formal training improves teaching skills (Lawson & Harvill, 1980; Bensinger et al., 2005) . This observation highlights the importance of instituting training programs for medical students, graduates and residents who will be future medical educators.

Aga Khan University (AKU) renewed the Bachelor of Medicine and Bachelor of Surgery (MBBS) undergraduate curriculum in 2002. In the renewed curriculum, Problem Based Learning (PBL) was used as a main teaching strategy in years one and two. With a change in teaching strategy from traditional lectures to PBL, a need was felt to increase the pool of facilitators. Therefore, recent AKU graduates were recruited as TAs to support teaching (Khan, 2006) . Eleven TAs were recruited for this purpose.

Aim

The aim of the current study was to evaluate the effectiveness of the training program held for TAs by measuring knowledge, facilitation skills and retention of the knowledge overtime.

2. Methodology

2.1. Context

2.1.1. Needs Assessment for the Workshops

The curriculum for the faculty development workshops was developed after 1) conducting key informant interviews with the Vice Chair Undergraduate Medical Education (UGME), Biological and Biomedical Sciences (BBS) and Chair Curriculum Committee to ascertain departmental needs and expectations from TAs, 2) careful analyses of the TAs job descriptions was done to understand the teaching role of TAs and the competencies required to fulfill this role. TAs job description included PBL facilitation and research. The current paper focuses on the preparation of TAs for their teaching role. TAs input in needs assessment and planning could not be obtained as the workshops were planned before the TAs were recruited and offered immediately after they joined. However, during the introductory session of the workshop, TAs were invited to articulate their expectations from the workshops. This information was used by the facilitators as a starting point and efforts were made to address those expectations as far as possible during the sessions.

2.1.2. Aim of the Workshop and Its Structure

Workshops using learner-centered instructional strategies were developed and implemented by the Department for Educational Development (DED) on the request of the BBS Department to equip the TAs with the requisite knowledge, skills and attitudes to fulfill their teaching roles.

The aim of the workshop was to:

- introduce the TAs to the educational philosophies underpinning medical education globally and at AKU,

- help TAs develop an understanding of the new trends in medical education (Curriculum, Teaching Learning, Assessment and Communication),

- impart PBL tutorial facilitation skills to the TAs,

Two workshops were held, each of two days duration.

The first workshop titled “Medical Education: From Philosophy to Practice” introduced the new trends in curriculum development, teaching and learning, assessment and the forces that drive these trends. It discussed the impact of educational psychology and adult learning theory on teaching and learning, and concluded with an introduction to principles of effective communication with a focus on feedback. This workshop was followed by a second one titled “Enhancing Problem-Based Learning (PBL) Tutorial Facilitation Skills”, which emphasized the philosophy behind PBL with special focus on developing PBL tutorial facilitation skills, handling of group dynamics and use of appropriate communication skills to provide effective feedback.

The workshop activities are given in Table 1.

2.2. Study Design and Participants

A longitudinal, mixed method, evaluation study was conducted at AKU, Karachi, Pakistan during 2006 to 2008. The participants included all TAs (n = 11) recruited by the BBS Department for teaching.

2.3. Instruments

1) Workshop evaluation form-TAs evaluated both workshops, “Medical Education from Philosophy to Practice” and “Enhancing PBL Tutorial Facilitation Skills”, separately. The workshop evaluation form was a six- statement questionnaire pertaining to workload during the workshop, content organization, encouragement of active participation, stimulation of interest in subject, preparation of faculty, and provision of clear explanations by workshop facilitators. The TAs gave their responses to the statements on a five-point Likert scale ranging from 1 = strongly disagree to 5 = strongly agree.

2) Pre, post and delayed post tests-The same written pre, post and delayed post tests were administered just before and immediately after the workshop, and again seven months later (at the end of the academic year). It consisted of MCQs, SAQs and open ended questions. These were marked according to a pre-defined key developed by two medical educationists.

3) Student ratings of TAs PBL facilitation skills-A form containing items relating to facilitation skills and management of group dynamics during PBL tutorial skills.

2.4. Data Analysis

SPSS 16 was used for quantitative data obtained through 1) Changing scores in knowledge on repeated measures

(pre, post and delayed post-test) and 2) Student ratings of TAs PBL facilitation skills during the academic year, during which time each TA facilitated in two PBL modules each.

Descriptive statistics and t-test were performed for quantitative data. In addition, thematic analysis of qualitative data was done which was obtained through 1) workshop evaluations, 2) TAs perceptions of improvement in their knowledge and facilitation skills.

3. Results

3.1. Mismatch between Expectations of the Workshop Participants and Curriculum Developers

When the participants were asked in the introductory session about their expectations, it became evident that their main objective of attending these workshops was to gain knowledge about PBL and PBL tutorial facilitation skills, and they were not much interested in current global trends in curriculum development, learning and teaching, and assessment.

It was the workshop developer’s perspective that in order to understand the principles of PBL, a basic understanding of the philosophy underpinning medical education globally and at AKU, as well as the knowledge about the relationship between curriculum, assessment and teaching and learning was essential. Hence, a deliberate decision was made to not teach PBL facilitation in isolation but it should be anchored within a framework of educational philosophy and current trends. The mismatch between the aim of workshop developers and the expectations of participants is evident in Table 2.

The first workshop “Medical Education: From Philosophy to Practice” was in line with aims of the workshop developers while the second one mirrored the participants’ expectations.

3.2. Workshop Evaluation

The results showed that rating of workshop focused on PBL was significantly higher than the one on medical education on all aspects except for “workload” and “facilitation” where the ratings were almost similar. The results are depicted in Figure 1.

3.3. Change in Scores of Knowledge on Pre, Post and Delayed Post Test

The mean of the group on pre test was M = 12.33, SD = 3.38 while on immediate post test mean score increased to M = 15.38, SD = 2.80. Hence results showed a significant increase (p = 0.036) in knowledge of the participants in the immediate post-test. However, this knowledge decayed over time as measured by the delayed post-test (M = 12.29, SD = 2.46). The results are depicted in Figure 2.

On sub-analysis of the immediate and delayed post test, it was found that knowledge related to teaching learning (tl) with focus on PBL (p = 0.077) and communication skills (cs) and feedback (fb) [p = 0.645] slightly increased on delayed post-test. However, there was very significant decrease of scores on questions related to curriculum (p = 0.001) and assessment (p = 0.001). The change in scores in assessment, curriculum, teaching learning (including PBL) and communication skills (including feedback) are shown in Figure 3.

3.4. Thematic Analysis of the Comments of Participants Regarding Their Knowledge and Skills

In the workshop evaluation form the TAs were asked about what they had learned from the workshops. Their

open ended comments were then categorized into the following themes:

Theme 1: Increase in knowledge (sample comments)

n “Very good! Encouraged thorough knowledge of PBL”,

n “A magnificent workshop! Clear, concise and effective, changed my views on PBL”,

n “It definitely broadened my perspective on this relatively new curriculum”,

n “I think it was an amazing exercise-I learnt a lot by discussion which I would not have been able to learn alone”.

Theme 2: Increased confidence about teaching and facilitation (sample comments)

n “I feel better at handling group dynamics”,

n “I feel more confident while teaching”,

n “I learned when and how to intervene in PBL session”,

n “I learnt that I should not jump in with the answer to a problem; rather guide students with relevant questions”,

n “This workshop has been extremely useful in developing confidence among the TAs”,

n “I hope to be able to apply the tips learnt during the PBL sessions”.

3.5. Corroboration of TAs Self-Perceptions of Increased Knowledge and Skills by Student Rating of TAs Facilitation Skills

TAs self-assessment regarding increased knowledge of the PBL process, confidence in teaching and handling of group dynamics (including communication and feedback) were corroborated by student ratings of TAs teaching performance throughout the year. The TAs received an overall mean rating of 7.45 ± 0.61 SD (on a rating scale of 1 - 8, where 1 is the lowest and 8 is the highest) from year one and two medical students about their facilitation skills in PBL sessions.

4. Discussion

The results of the current study demonstrated that the TAs appreciated and learned more from the PBL facilitation skills workshop rather than the generic medical education workshop. Published literature supports these findings, that faculty development is most effective when it is learner centered (Notzer & Abramovitz, 2008). Steinert et al. (2012) in a review article reported that faculty members value learning opportunities that are contextually relevant and applicable to their educational work. Another finding of the study was that although the workshop led to an immediate increase in knowledge of participants (Naeem et al., 2012; Alfaris et al., 2015) but this knowledge was not retained at the end of the year. On delayed post-test, the overall scores on knowledge decreased to a level little below the pre workshop level. This finding has been reported in other studies also that although intensive education programs enhance knowledge, it decays rapidly (Su et al., 2000) . The rate of decay is directly proportional to the degree to which the learner has mastered the knowledge or skill, hence the greater the mastery the slower the decay. The degree of mastery is increased by further practice by the learner (Farr, 1986) . Another important factor in decay is the passage of time after acquisition, during which the learned knowledge and skill is not used. It is also evident from literature that other variables such as type of task, and the conditions and strategies of instruction impact learning and retention (Farr, 1986) . Attention should be paid to all these variables in designing any training program to optimize learning, retention and transfer of both knowledge and skills. In addition, opportunity for review and retrieval of information should be provided at spaced intervals after the training is complete (Thacker & Blanchard, 2004) .

Research on the transfer of training reports that training only influences the immediate learning and retention rather than generalization and maintenance of trained skills on the job (Baldwin & Ford, 1988; De Rijdt, 2013) . According to Jelsma, van Merrienboer, & Bijlstra (1990) transfer to workplace would only occur when the tasks practiced in the training situation are the same as those performed in the workplace. Hence, the more the similarity between the two task situations, the more is the likelihood of transfer. Opportunities for application of training at the workplace and encouragement will help to transfer the training. Ford et al. (1992) reported that opportunity to apply training affected training transfer in three ways: the breadth (how many tasks on the job are covered by the training); activity level (the number of times trained tasks are actually done on the job); and the nature of task (simple or complex). Therefore, for knowledge and skills to be retained, they have to be applied and reinforced. The need for frequent reinforcement is also supported in literature (Eliot, 1999) .

Student ratings of TAs PBL facilitation skills showed strong correlation between the participants’ self- perception of increased knowledge and skills and their actual PBL facilitation ratings. Evaluation by students is considered an objective and reliable outcome measure. It has been shown that well developed evaluation instruments can give highly reliable data (Guthrie, 1954; Aleamoni, 1972) .

5. Limitation of the Study

One of the limitations of our study was small sample size and the absence of a control group. It was not possible to include a control group because the number of TAs recruited was small (n = 11) and all of them were included in the study. It is emphasized that it is unlikely that the increase in knowledge, skills and teaching performance of the TAs in the immediate post-test and the decay in components of knowledge and skills not utilized or practiced during the year on the delayed post-test, are attributable to any other factor during the study period.

6. Conclusions and Recommendations

This study builds on prior work in faculty development and adds to the literature on the effectiveness of short workshops in improving knowledge, skills and teaching performance. It adds further information to research about long term retention of knowledge and skills following training and makes recommendations about how to reinforce it.

Several lessons were learned from objective evaluation of the faculty development workshops. Aspects of the training that do not reflect participants’ objectives and expectations arouse less interest and lead to less learning and poor retention. Hence, it is important to involve the participants of the training program in the planning and development phase of faculty development activities. This will ensure alignment between the expectations of the participants and the content of the training program.

Workshops and short courses lead to short term increase in knowledge, which decays over time. Knowledge that is utilized, applied and practiced, does not decay as much or as fast. Therefore, refresher workshops should be arranged at regular intervals to reinforce knowledge and skills acquired during training. Furthermore, job opportunities should be provided to apply the newly gained knowledge and skills to promote transfer of training to the workplace. Future studies should focus on establishing the optimal time for refresher workshops and retraining.

Our study showed that short workshops increased participants’ knowledge and skills and boosted their confidence leading to improved teaching performance. We recommend that faculty development should be made mandatory for all new teaching faculty including TAs. This will increase their competence in teaching and help them be better prepared for their teaching role.

Acknowledgements

N. N. thanks Dr Rukhsana W. Zuberi for guidance during the design of study and the pre and post test and Dr Rashida Ahmed for assistance in obtaining the student’s evaluation of faculty performance data. N. N. also thanks all TAs for participating in the study.

Cite this paper

Naghma Naeem, (2016) Which Knowledge and Skills Do Participants Retain after Attending Medical Education Training Workshops?. Creative Education,07,870-877. doi: 10.4236/ce.2016.76091

References