The Relative Effectiveness of Online Lecture Methods on Student Test Scores in a Business Course

We compared student learning outcomes (using test scores) across three modes of delivering lectures: 1) using a traditional face-to-face lecture, 2) using online instruction where the lecture was delivered using a video recording of the classroom lecture, and 3) using online instruction where the lecture was delivered using a static document created from an edited transcript of the classroom lecture embedded with charts, graphs etc. The results showed that as hypothesized, students reading the online static document performed as well as the students attending the traditional lecture and that both groups outperformed the students watching the video of the recorded lecture. The data also showed that, as hypothesized, both quickly created online instruction methods scored lower on engagement compared to the traditional face-to-face lecture. Students who were exposed to the online lecture delivered using a static document and the students attending the face-to-face lecture both reported having higher quality of notes compared to students exposed to the video recording. Finally, as hypothesized, the effect of the different instructional material on student test scores was found to be completely mediated by student engagement and perceived note quality.


Introduction
While much research has been done on the broad topic of online learning, and on the narrower subject of the effectiveness of various modes of online instruction, most of this work has been done from the perspective of fully online an easy-to-implement online lecture method that an instructor can quickly implement, and which will result in student learning comparable to what can be achieved through traditional face-to-face lectures. Online education is growing fast worldwide and this has caused it to be the focus of much research and innovative ideas [1]. This research does not seek to review the broad topic of online education but rather focuses on the ideas highlighted in the Introduction. Students enrolled in a course with traditional face-to-face lectures may have different expectations when it comes to the nature of the lectures they receive compared to students enrolled in hybrid or fully online courses. It is not unreasonable to assume that students enrolled in a mostly traditional course may be more resistant to and may have lower levels of learning when the instruction is unexpectedly provided in an online format compared to when the instruction is delivered in the traditional manner. Under these circumstances which online instructional method is most effective at promoting learning while at the same time being easy to implement? Past research has found mixed evidence for the relative effectiveness of online instruction to provide high levels of learning outcomes, especially when the learning outcomes are measured using test scores. Johnson et al. [2], Carrol and Burke [3], McLaren [4], Ni [5], McCutcheon et al. [6] and several others have found that online instruction results in student learning outcomes that are comparable to those from traditional face-to-face teaching. Means et al. [7] in a meta-analysis found that online students do modestly better on tests compared to students in traditional classes, while Pereira et al. [8] find that students in a blended learning group scored significantly higher on tests compared to those in a traditional learning group. On the other hand, Brown and Liedholm [9] find that online students fare less well.
The overwhelming majority of studies have found that online instruction results in student learning outcomes similar to those achieved using traditional face-to-face instructional methods. We can assume that the online instructional methods used in these studies were professionally designed incorporating the best practices for good online instructional design. This would include, among other factors, designing online instruction that engages students. Engagement has been identified as a significant factor in good instructional design [10] [11] and unless the online instruction is carefully designed and implemented, it runs P. Nyer Open Journal of Business and Management the risk of not engaging the online students, resulting in lower student learning for online students. Thus, a quickly put together online lecture delivered through a video recording or a static document is likely to result in lower levels of student engagement.
H1A: A quickly put together online lecture delivered through a video recording will result in lower levels of student engagement compared to student engagement achieved through a traditional face-to-face lecture.
H1B: A quickly put together online lecture delivered through a static document will result in lower levels of student engagement compared to student engagement achieved through a traditional face-to-face lecture.
The quality of notes and other study material available to students to study from will influence their performance on tests [12] [13] [14]. While students who attend traditional face-to-face lectures have but one chance to take good notes in class, students who are exposed to an online lecture have unlimited opportunities to review the online material and thus take good notes. This should then cause online students to do relatively better than students in traditional classes. However, reviewing long video recordings of lectures can be tedious and time consuming, requiring a quiet environment and a lot of bandwidth and thus it is unlikely that most students will watch the video repeatedly. Thus the likelihood that online students will watch video recordings of lectures just once combined with the lower levels of engagement for this mode of online instruction suggests that students exposed to video recordings of lectures will have worse notes compared to students in traditional classes.
H2A: A quickly put together online lecture delivered through a video recording will result in students having notes of a lower quality compared to notes taken by students in a traditional face-to-face lecture.
Students who are exposed to an online lecture delivered through a static document will have the material in a form that can easily be downloaded, printed, highlighted, annotated etc. Thus this should result in these students rating the quality of their notes highly.
H2B: A quickly put together online lecture delivered through a static document will result in students having notes of a higher quality compared to notes taken by students in a traditional face-to-face lecture. Thus students exposed to a quickly made online lecture delivered through a video recording of a lecture will experience lower levels of engagement and the logistical issues related to the difficulties in watching long video recordings repeatedly will combine to result in lower student learning outcome for these students. On the other hand, students exposed to a quickly made online lecture delivered through a static document while experiencing low levels of student engagement, will have high quality notes to study from, and this "mixed environment" should result in learning outcomes comparable to that achieved by students attending traditional face-to-face lectures.
H3A: A quickly put together online lecture delivered through a video record-P. Nyer Open Journal of Business and Management ing will result in students having lower levels of learning outcomes compared to the learning outcomes of students in a traditional face-to-face lecture.
H3B: A quickly put together online lecture delivered through a static document will result in students having learning outcomes comparable to the learning outcomes of students in a traditional face-to-face lecture.
H4: The perceived levels of engagement and quality of notes available to the students will mediate the effect of the instructional methods on student learning outcome.

Study Design and Data Collection
Data was collected over three consecutive fall semesters from 263 students A week prior to when the CLV topic was to be taught, students in semester 2 and 3 were told that the instructor had to travel to attend an academic conference and that the topic for the week would be delivered online. During the next regular face-to-face class session following the CLV lecture, and just prior to be-

P. Nyer Open Journal of Business and Management
ing tested on the CLV and PMP topics, students were administered a brief survey that measured the students' perceptions of the level of engagement they experienced when being exposed to the CLV lecture, and the quality of the notes that they had to study from-see Exhibit I. Student learning outcomes for the CLV topic (dependent variable) and the PMP topic (control variable) were measured using high value calculation questions on tests. The test questions were worded differently from how the topic was taught in the lectures (something that the students had been told to expect), and this was done to ensure that students were not merely repeating the calculations from memory but rather they were engaging in critical thinking and demonstrating a good understanding of the material.

Results
Values of the Engagement variable and the Note Quality variable for the three semesters are provided in Table 1. A score of 3.00 indicates engagement with (or note quality of) the CLV lecture equal to that for the face-to-face lecture on PMP. A lower score indicates lower engagement (or note quality). As hypothesized (H1A and H1B), engagement with the quickly prepared online lectures delivered through the video recording and through the static document were significantly lower than the engagement with the face-to-face lecture of the same topic (t 165.47 = 7.43, p = 0.00 and t 139.94 = 7.14, p = 0.00 respectively). Thus we find support for hypotheses H1A and H1B. Incidentally, there was no significant difference in the level of engagement between the two online lectures (t 160.33 = 0.63, p = 0.53).
The perceived quality of the notes for the CLV lecture relative to the PMP lecture (the latter of which was taught using a traditional face-to-face lecture in all three semesters) indicated that the students evaluated the quality of their CLV lecture notes to be poorer regardless of how the CLV lecture was administered-even for semester 1 when both lectures were taught using traditional lectures (t 91 = 3.44, p = 0.00). Perhaps this is because the CLV topic was more involved and more difficult for the students. Regardless of that, the important comparison is between the perceptions of the note quality across the three semesters.
As hypothesized (H2A), students exposed to the online lecture delivered through a video recording perceived the quality of their notes to be lower than the perceived quality of notes for the students in the traditional face-to-face class (t 176.98 = 2.50, p = 0.01). While the mean value of the perceived quality of the notes for Students in the three semesters performed equally well on the test questions on the PMP topic (see Table 2 below) which was taught in all three semesters using traditional face-to-face lectures (F 2,260 = 0.96, p = 0.38), indicating that the students enrolled in the three semesters were not significantly different in terms of their ability to perform in test questions requiring critical thinking and moderately complicated quantitative calculations. Table 2 also lists the test scores for the students in the three semesters on the test questions on the topic of CLV. The test scores on this topic taught using different techniques were significantly different from each other (F 2,260 = 5.75, p = 0.00). This was further investigated using t-tests on pair-wise combinations of the semesters. Students exposed to the traditional face-to-face lectures (Mean test score = 81.48%) and those exposed to the online lecture using the static document (Mean test score = 80.74%) did significantly better than students who were exposed to the online lecture using the video recording (Mean test score = 74.16%) (t 169.91 = 3.45, p = 0.00 and t 158.55 = 2.49, p = 0.01 respectively) while there no significant difference in the performance of the students enrolled in the face-to-face lectures and those exposed to the online lecture delivered through the static document (t 143.08 = 0.30, p = 0.77).
Does the level of engagement and note quality influence student test scores?
To test hypothesis H4 we followed Baron and Kenny's four-step approach [15] [16] a procedure that has been widely used [17] [18]. As reported earlier, the different instructional methods had a significant effect on the dependent variable-student test scores on the CLV lecture (H3A and H3B; Baron  In summary, this study found that online lectures delivered through a video recording, or through the use of a static text both scored lower on student engagement compared to a traditional lecture. Students in a traditional lecture environment and those exposed to a lecture through a static document both reported having better notes compared to students exposed to a video recording. Students in both the traditional lecture group as well as those in the static document group both outperformed the students in the video recording group. This superior performance exhibited by students in the traditional lecture and the static document groups can be attributed completely (complete mediation) to the differences in engagement and note quality across the groups.

Discussion
The scope of this research was limited to investigating effective ways of quickly offering an online lecture in a course that is otherwise taught using traditional face-to-face lectures. As discussed earlier, the motivation to offer the online lecture could be varied, including the planned absence on the part of the instructor for various reasons. Under these circumstances which easy-to-implement online instruction method results in the highest student learning outcomes? We compared student learning outcomes (using test scores on specific questions) across three modes of delivering lectures: 1) using a traditional face-to-face lecture, 2) using online instruction where the lecture was delivered using a video recording of the classroom lecture, and 3) using online instruction where the lecture was delivered using a static document. The results showed that as hypothesized, students reading the online static document performed as well as the students attending the traditional lecture and that both groups outperformed the students watching the video of the recorded lecture. The data also shows that, as hypothesized, both quickly created online instruction methods scored lower on engagement compared to the traditional face-to-face lecture. Also, as hypothesized, the perceived quality of the notes available for students was lowest for the students exposed to the video recording. Students who were exposed to the online lecture delivered using a static document and the students attending the face-to-face lecture both reported having much higher quality of notes. Finally, as hypothesized, the effect of the different instructional material on student test scores was found to be completely mediated by student engagement and perceived note quality.

Conclusions & Limitations
The conclusion one can make here is that an instructor who teaches a traditional lecture course and who wishes to offer one or two online lectures is best served by offering students a written lecture with embedded images, graphs, charts etc.

P. Nyer Open Journal of Business and Management
Not only does such a document not take much time to prepare for the instructor (compared to the time required to create complex, interactive online instructional material), these documents can be easily downloaded by students with minimal bandwidth and device memory requirements for reading offline, printing, highlighting, making annotations etc.
The scope of this study was limited as stated in the previous section, and the findings are probably not applicable to fully online courses, or even hybrid courses where a substantial amount of instruction happens online. In these cases, students are likely to have higher expectations from the online lectures, and the types of online lecture material used in this study will probably not suffice.

Conflicts of Interest
The author declares no conflicts of interest regarding the publication of this paper.