Evaluation of Innovative Teaching Approaches : The Moderating Effect of Student Prior Experience

The success of creative or innovative teaching approaches is often measured by student perceptual ratings of the learning environment or by academic outcomes. This paper examines student perceptions of a novel human physiology laboratory format and the effect of prior experience on these perceptions. The same undergraduate human physiology course, taught at second year level, was taken by students who had previously completed a semester of human physiology (“continuing” students) and by those taking it for the first time (“new” students). The “continuing” students were significantly more positive about the novel format compared to the previous format. The class as a whole (“continuing” plus “new”) also gave a strong positive rating of the novel format. However a comparison between the “continuing” and the “new” students showed that the latter were significantly more positive in their perception of the laboratory in all areas apart from active participation. A correlational analysis indicated strong inter-rater links for the “continuing” students but weak or non-significant inter-rater correlations for the “new” students. The study suggests that, given the diversity of student backgrounds and prior experience in a given class, that perceptual ratings of the learning environment alone may not provide enough support for the effectiveness of novel teaching interventions.


Introduction
The measurement of student perceptions of the learning environment form an important part of the evaluation of learning and teaching innovations, the results of which may be used at course, program and university level (Kulic, 2001;Richardson, 2005).However, it is the students' perception of the learning context, rather than the context itself, which influences student approaches to learning (Diseth, Pallesen, Brunborg, & Larsen, 2010) and final academic outcomes.The results of the current study suggest that prior experience may affect student perceptions of the learning environment and that conclusions based on ratings alone may not allow conclusions to be reached regarding the efficacy of interventions.Given the lack of research on the agreement of student ratings of group level constructs in educational research (Lüdtke, Trautwein, Kunter, & Baumert, 2006) the results may have wider implications for the evaluation of teaching and learning innovations and the conclusions derived at all levels of the hierarchy.
This paper explores the introduction of a novel approach to improve students' human physiology laboratory experiences and the concurrent student perceptions of the approach.One of the major experiences undergraduate students have in most science-based courses is their involvement in hands-on laboratories.However, end-of-semester examination results suggested to staff teaching human physiology at RMIT University that students' had not fully understood concepts derived from their physiology laboratory experience.For this reason a novel physiology laboratory approach was developed based on the fact that students' motivation and deeper learning of concepts is increased with well-designed laboratory work, (Hoffstein & Lunetta, 1982) and that students' understanding is enhanced when they actively engage in the learning process (Jonassen, 1999;Kearney, 2004;Land & Hannafin, 2000;Mayer, 2003).We focused on improving students' engagement with the learning activity by considering involvement in and engagement with the learning activities (cognitive engagement) and also behavioral engagement (interest, enjoyment and a sense of belonging) (Fredericks, Blumenfeld, & Paris, 2004;Harper & Quaye, 2009).Students who are engaged are intrinsically motivated with self-belief in their own abilities, have a desire to improve their competency, need to develop their own learning and to be able to achieve their goals (Ainley, 2006;Yorke & Knight, 2004;Zepke & Leach, 2010).Self-perceived competence is a key motivator for engagement and when students have confidence in their own competence this is a strong motivator for ongoing active learning (Fazey & Fazey, 2001).
The initial purpose of the study was to establish, by analyzing student ratings of the learning environment, whether the novel laboratory format enhanced students' engagement, active learning and perceived understanding of key concepts.The impact of student prior experience on the validity of our approach was further investigated through inter-rater analysis.

Method
One hundred and two students taking a human physiology course at RMIT University (40% of the class) were surveyed using an end of semester questionnaire.The aim of the survey was to ascertain from the students their responses to the new laboratory format.Students were asked to give their responses on a range of questions including their active participation in the laboratories and their confidence in doing the laboratories and to how well they thought the assessment fairly reflected what they had done in the laboratories.
Two physiology courses are taught at second year undergraduate level (one in first semester and the second in second semester).However, due to program requirements, the second semester student cohort consists of second year students who have already completed first semester physiology ("continuing" students-Biomedical Science, Human Movement and Pharmaceutical Sciences programs) and first year students ("new" students-Chiropractic, Osteopathy and Chinese Medicine programs) taking their first physiology course.The latter "new" students complete their physiology sequence in the first semester of second year (in reverse order to the students in the other programs).Permission was obtained from the RMIT University Human Ethics Committee (low risk) and students were informed of the purpose of the survey and given a plain-language statement.No individual student identifiers were used, however student survey responses were divided into those who had ("continuing" students) or had not ("new" students) completed a prior semester of human physiology.
The new laboratory approach had two key components: a one-hour concept-focused human physiology laboratory and an immediate follow-up interactive discussion with feedback and associated assessment (one hour).This replaced the previous structure of a two-hour more detailed hands-on laboratory without follow-up discussion, feedback or immediate assessment.
The academic in charge of each section of the course identified material that was best suited to a short focused hands-on laboratory that would help students in understanding key concepts.Once students completed the hands-on component of the laboratory they took part in a one-hour interactive group session invigilated by a tutor.During this session problems and questions associated with the laboratory were discussed and the material covered in the laboratory was integrated into theory.The assessment task entailed a series of multiple-choice questions as well as students filling out a section giving their own reflections.The tutor marked these assessment sheets after each laboratory and feedback was provided to students in the next laboratory class.
The first part of the survey consisted of five questions (Table 1) that were only answered by the "continuing" students (Biomedical Science, Human Movement and Pharmaceutical Science Programs).These questions allowed comparison between the novel laboratory format and the original laboratory format taken by these students in the preceding semester.The second part of the survey was taken by all students (Table 2) (Biomedical Science, Chiropractic, Osteopathy, Chinese Medicine, Human Movement and Pharmaceutical Sciences programs) and consisted of eight questions designed to gather students' interpretations about their learning, the format of the laboratory and the interactive post-laboratory discussion as well as their reflections on their level of participation, confidence, enjoyment and satisfaction.
The survey used a five point Likert scale (Strongly agree (5), Agree (4), Neither Agree nor Disagree (3), Disagree (2), Strongly Disagree (1)), with a high reliability score (Cronbach's alpha = .87)for the survey items (for the items only for the "continuing" students, α = .95,and items for both "continuing" Questions given to all students (second part of survey).

Parameter Question
End-laboratory The use of end-lab discussion aided my learning

Combination
The combination of hands-on and end-lab discussion helped me to be more involved in the practical Active I was an active participant in the end-lab discussion

Assessment
The end-lab discussion gave me more confidence in doing the assessment Enjoyment I found the laboratory experience enjoyable

Feedback
The end-lab discussion gave me good feedback on my understanding of the concepts of the lab

Fairly
The assessment fairly assessed what was covered in the lab Satisfaction I was satisfied how the laboratory and assessment were carried out and "new" students, α = .81).No student identifier was recorded on the survey.The survey was administered over a two-week period during the laboratory sessions.Once the data had been collated they were analyzed using SPSS © .Initial Pearson correlations were carried out along with tests for homogeneity and normality, and ANOVA (α = .05).

Results
When comparing the original with the novel laboratory formats (first part of the survey) the "continuing" students were more satisfied with the new format than their previous one, with responses for all questions averaging 3.94 (SD = .4)(Agree = 4).
They reported more understanding of the material, were clearer in what they had to learn, enjoyed the new format better, and were more satisfied with the new laboratory structure.There were no significant differences between the responses to any of these questions.The ratings and inter-rater correlations for the responses to these questions are presented in Figure 1.
The response of the entire class to the novel format is shown in Figure 2.An initial survey of all students ("continuing" and "new" students) showed that the average response was very positive for all the parameters.However, when an examination was made of the two distinct groups a different pattern emerged.The ratings for each question in the second part of the survey are Copyright © 2012 SciRes.shown in Figure 2(a) ("new" students) and (b) ("continuing" students).The "continuing" students ratings were significantly less positive than those for the "new" students for enjoyment in doing the laboratory ("new" students 4.6; "continuing" students 3.8; F (1,101) = 15.74,p < .01).There was also a smaller significant difference between the groups for how much the combination of hands-on laboratory and discussion helped them to be more involved in the laboratory ("new" students 4.7; "continuing" students 4.2; F (1,101) = 8.12, p < .05); the end-of laboratory discussion giving them more confidence in doing the assessment ("new" students 4.5; "continuing" students 4.0; F (1,101) = 6.38, p < .05); the laboratory assessment fairly assessing what was done in the laboratory ("new" students 4.8; "continuing" students 4.3; F (1,101) = 10.70,p < .05);and students' satisfaction about how the laboratory and the assessment were carried out ("new" students 4.5; "continuing" students 4.3; F (1,101) = 11.97,p < .05).Despite these results there was no difference between the "new" and "continuing" students reported active participation in the laboratory ("new" students 4.0; "continuing" students 3.7).
Correlations between the student ratings for each question in the second part of the survey are shown in Figures 2(a) ("new" students) and (b) ("continuing" students).The "continuing" students had significantly higher correlations between feedback on concepts from the end-laboratory discussion and the laboratory assessment fairly assessing what was covered in the laboratory ("continuing" students r = .72,p < .01;"new" students r = .39,p < .05).Furthermore the "continuing" students had a significantly higher correlation between the end-laboratory discussion aiding their learning, and their satisfaction about how the laboratory and assessment were carried out ("continuing" students r = .45,p < .01;"new" students r = .02,p > .05).The "continuing" students also had a higher significant correlation between enjoying the laboratory experience more in relation to their learning being aided by the end-laboratory discussion ("continuing" students r = .44,p < .01;"new" students r = .23,p > .05); between the end-laboratory discussion and the assessment ("continuing" students r = .36,p < .05;"new" students r = .07,p > .05); between satisfaction of the laboratory and assessment, and enjoyment ("continuing" students r = .67,p < .01;"new" students r = .29,p > .05).

Discussion
Our initial evaluation of the human physiology laboratory innovation was that it was successful, based on the significant increase in positive responses from the "continuing" students when they compared the new with the previous format.In addition the group response to questions related to the new format ("new" students plus "continuing" students) was also very positive.Our initial conclusion from the study was that students' motivation and engagement was enhanced by a combination of a simpler concept focused hands on laboratory session and a post-laboratory interactive group discussion (active learning) with associated assessment.An affective component, enjoyment, was also enhanced resulting in an overall more positive learning experience than provided by the previous physiology laboratory format.In a study of a measure of student course engagement (Handelsman, Briggs, Sullivan, & Towler, 2005) the importance of participation/interaction in relation to student engagement was demonstrated.This was one of four engagement factors identified however it was the only one that was predictive of students' final exam mark.Active learning engages students and involves them in doing things and in thinking about what they are doing (Michael, 2006;Prince, 2004).
However, comparison of the sub-groups within the class demonstrated that for many areas the "new" students had a significantly greater positive rating of the novel laboratory format than the "continuing" students.It is possible that the differences are due to differences in the degree of motivation of the two groups as student motivation moderates perception of the learning environment and determines approaches to learning and outcomes (Biggs, 1985;Ramsden, 1991).Students' perceptions of the learning environment (in this case workload) have been found to be a function of individual characteristics, approaches to and perceptions of the learning context (Kember, Ng, Tse, Wong, & Pomfret, 1996).Motivation (and satisfaction) of students within a course may also depend on whether or not they intend to continue in that area of study.Diseth et al. (2010) found students, within a first semester psychology course, who did not plan to continue their psychology studies, were less satisfied with their quality of education and had a higher level of surface approach than those who did want to continue.However, in contrast, students who intended to continue in aviation were less satisfied with an online aviation physiology course compared to students who were not going to continue in aviation (Artino, 2009), even though they reported greater perceptions of task value and greater use of cognitive control strategies.In this case it was suggested that the lower satisfaction of the "aviators" resulted from the course not meeting their expectations.It was concluded that subjective perceptions of the learning environment moderated motivational and behavioral engagement (Artino, 2009).
In general, intrinsic motivation is more likely to be associated with a deep approach to learning and increased positive perceptions of the learning environment (Ramsden, 1992).In a series of studies Deci and Ryan (Deci & Ryan, 2000;Ryan & Deci, 2000a, 2000b) identified intrinsic motivation, in which students do something because it is interesting or enjoyable, as assisting self-determination.With an increase in self-determination students develop an increase in autonomy in their learning experience and students are more highly motivated for skills they value and wish to master.Significant perceptual differences have been found between students who had or had not studied science prior to entry into a new integrated anatomy practical program.Nonscience students were more positive regarding structure, organization, resources, problem-based learning and assessment fairness within the new program, possibly due to their relative lack of understanding and their desire to succeed (Tedman, Alexander, Massa, & Moses, 2011).
Our study focused on the presage components of the Biggs (1989) 3P model of teaching and learning i.e. what student prior experience contributes to the learning situation (prior knowledge, academic ability, personality etc.) and the characteristics that define the learning environment itself (e.g.quality of teaching).It is also important to establish the relationship between learning environments and students' affective experiences as the results support the conclusion that students learn better by not only becoming active participants in their own learning but also by enjoying what they are doing.As indicated above affective components may be related to intrinsic motivation and deep learning (e.g.enjoyment) or to extrinsic motivation and surface learning (e.g.unhappiness).Negative affect may develop under certain circumstances.For example some students develop negative perceptions of group work (Forrest & Miller, 2003) that may lead to continuing negative attitudes regarding the effectiveness of small groups in enhancing their learning (Forrest & Miller, 2003;Hillyard, Gillespie, & Littig, 2010).
The question remains, however, as to whether any of the factors discussed above, that affect student perception of their learning environment, provide a reasonable explanation for the differences that we observed.One possibility is that the "new" students had a higher level of intrinsic motivation due to their desire to succeed notwithstanding a relative lack of experience (as they were first year students).They could also be considered to be anticipating the fact that they would progress to an additional semester of physiology whereas the "continuing" students were completing their second year physiology studies.However, this seems unlikely as physiology is a core component of the programs in both groups (i.e. it is important for their further studies).Another possibility is that the "continuing" students had developed a negative attitude to physiology as a result of their prior semester of physiology or as a result of their general experience of tertiary study.This also seems unlikely as they preferred the new laboratory format, (compared to the previous one) and gave an overall positive response.Another option would be that the "continuing" students had become less intrinsically motivated due to a high workload at this stage of their degree.As discussed above a high workload can result in an increased surface approach to learning with a concurrent decrease in positive perceptions of the learning environment.However, workload was not measured in the current study.
A final possibility is based on the level of knowledge and understanding of the learning environment."What the learner already knows" (Marton & Booth, 1997) are elements of the student's prior experience that they reference in a given learning situation.In terms of the relationship between the student and the learning environment (relational perspective) part of the meaning that someone ascribes to learning comes from their understanding of the particular setting that they are in (Saljo, 1982).A number of studies have found that people tend to be overconfident in their judgments, particularly when those judgments are difficult to make, for example, students who were less academically competent tended to overestimate their abilities (Langendyk, 2006).However, other studies have found that surface learners provide lower evaluations of their own performance whereas deep learners accurately self-assess (Cassidy, 2007).
Thus it is possible that the "new" students are less able to "benchmark" their responses to the questionnaire due to relative inexperience regarding physiology laboratories.They are unable to make the same comparison made by the "continuing" students who previously completed a semester of physiology laboratories.The "new" students may also have more limited general knowledge of academic study as they are only in the second semester of their first year of study.In conclusion the higher ratings provided by the "new" students may reflect an overestimation related to their lack of a satisfactory benchmark.
A correlational method was used in an attempt to determine which of the various explanations discussed above would explain the discrepancy in ratings of the new laboratory format between "new" and "continuing" students.The rationale for this approach was that experienced ("continuing") students would demonstrate understanding of the links between the different aspects of the survey questionnaire at a higher level than that for the relatively less experienced "new" students.Correlational analysis has been used to support links between higher-level measurements e.g. to determine the relationship between presage, process and product, as in the Biggs 3P model, (Biggs, 1989) however it is much rarer to find this approach used to examine links within a class.The "absolute agreement" of students within a class can be measured.However, Lüdtke et al. (2006) indicate that very little educational research has been conducted on the agreement of student ratings of group-level constructs.A study describing instruments designed for evaluating clinical faculty by learners found only 9 of 21 relevant studies measured interrater reliability (Beckman, Ghosh, Cook, Erwin, & Mandrekar, 2004).Our data demonstrates a clear difference between the degree of inter-rater agreement within the "new" student group and the "continuing" student group.The high level of correlation between different elements of the survey shown by the latter group both within their comparison between old and new laboratory formats and within their perception of the new format may indicate their greater ability to benchmark their answers against prior experience.On the other hand the lack of correlation between student ratings within the "new" group may indicate their lack of experience and lack of reference to appropriate benchmarks.
If one accepts that the low correlations for inter-rater responses amongst the "new" students indicates a lack of prior relevant experience, then their higher ratings of the innovative physiology laboratory might either reflect a high level of motivation related to their lack of experience and their need to do well or that they produced an inflated response due to lack of benchmarks.Conversely the lower positive ratings of the "continuing" group may provide a more realistic measure of the innovation given the strong relationships between the student responses and the more relevant prior experience of these students.Although the correlational technique is not the only method of measuring inter-rater agreement, one is left with the conclusion that with respect to student perceptual ratings, highest is not necessarily the best indicator of innovation success.When evaluating learning and teaching innovations attention should be paid to the diversity of the student cohort and the presence of subgroups and it may be appropriate to investigate analysis methods other than simply measuring the level of student ratings.The same conclusion may also apply when considering higher-level analysis built on course level feedback.

Figure 1 .
Figure 1.Comparison of correlations (numbers on lines) and student responses (numbers in shapes) for the questions only for the "continuing" students comparing the old and new formats.NCon = New Confidence; NC = New Clear; NE = New enjoyment; NU = New Understanding; NStr = New Structure.Significance of difference is at p < .05*;p < .01**.
Figure 2. Comparison of correlations (numbers on lines) and student responses (numbers in shapes) for both "new" (a) and "continuing" (b) students.Significant difference for "new" compared with "continuing" is at p < .05*;p < .01**;NS = not significant.FB = Feedback; ASS = assessment; ELD = End-Lab Discussion; COM = Combination of end-lab and hands-on; SAT = Satisfied; LAB = Enjoyed Lab.

Table 1 .
Questions given only to the "continuing" students (first part of survey).