The Relative Effectiveness of Online Lecture Methods on Student Test Scores in a Business Course

Abstract

We compared student learning outcomes (using test scores) across three modes of delivering lectures: 1) using a traditional face-to-face lecture, 2) using online instruction where the lecture was delivered using a video recording of the classroom lecture, and 3) using online instruction where the lecture was delivered using a static document created from an edited transcript of the classroom lecture embedded with charts, graphs etc. The results showed that as hypothesized, students reading the online static document performed as well as the students attending the traditional lecture and that both groups outperformed the students watching the video of the recorded lecture. The data also showed that, as hypothesized, both quickly created online instruction methods scored lower on engagement compared to the traditional face-to-face lecture. Students who were exposed to the online lecture delivered using a static document and the students attending the face-to-face lecture both reported having higher quality of notes compared to students exposed to the video recording. Finally, as hypothesized, the effect of the different instructional material on student test scores was found to be completely mediated by student engagement and perceived note quality.

Share and Cite:

Nyer, P. (2019) The Relative Effectiveness of Online Lecture Methods on Student Test Scores in a Business Course. Open Journal of Business and Management, 7, 1648-1656. doi: 10.4236/ojbm.2019.74115.

1. Introduction

While much research has been done on the broad topic of online learning, and on the narrower subject of the effectiveness of various modes of online instruction, most of this work has been done from the perspective of fully online courses or at the very least hybrid courses where a significant portion of the instruction is provided online. Not as much attention has been paid to the relative effectiveness of various online instructional methods, compared to traditional face-to-face instruction, in a situation where an instructor of a course with traditional face-to-face lectures wishes to substitute one or two class sessions with online instruction. There are many reasons why this may happen: the instructor may be unable to deliver a lecture in person because of a previously scheduled travel or medical event; the instructor may want to introduce the concept of “flipping the class”; or the instructor may wish to try out offering online instruction by offering one lecture online. The objective of this research is to determine an easy-to-implement online lecture method that an instructor can quickly implement, and which will result in student learning comparable to what can be achieved through traditional face-to-face lectures.

Online education is growing fast worldwide and this has caused it to be the focus of much research and innovative ideas [1] . This research does not seek to review the broad topic of online education but rather focuses on the ideas highlighted in the Introduction. Students enrolled in a course with traditional face-to-face lectures may have different expectations when it comes to the nature of the lectures they receive compared to students enrolled in hybrid or fully online courses. It is not unreasonable to assume that students enrolled in a mostly traditional course may be more resistant to and may have lower levels of learning when the instruction is unexpectedly provided in an online format compared to when the instruction is delivered in the traditional manner. Under these circumstances which online instructional method is most effective at promoting learning while at the same time being easy to implement?

Past research has found mixed evidence for the relative effectiveness of online instruction to provide high levels of learning outcomes, especially when the learning outcomes are measured using test scores. Johnson et al. [2] , Carrol and Burke [3] , McLaren [4] , Ni [5] , McCutcheon et al. [6] and several others have found that online instruction results in student learning outcomes that are comparable to those from traditional face-to-face teaching. Means et al. [7] in a meta-analysis found that online students do modestly better on tests compared to students in traditional classes, while Pereira et al. [8] find that students in a blended learning group scored significantly higher on tests compared to those in a traditional learning group. On the other hand, Brown and Liedholm [9] find that online students fare less well.

The overwhelming majority of studies have found that online instruction results in student learning outcomes similar to those achieved using traditional face-to-face instructional methods. We can assume that the online instructional methods used in these studies were professionally designed incorporating the best practices for good online instructional design. This would include, among other factors, designing online instruction that engages students. Engagement has been identified as a significant factor in good instructional design [10] [11] and unless the online instruction is carefully designed and implemented, it runs the risk of not engaging the online students, resulting in lower student learning for online students. Thus, a quickly put together online lecture delivered through a video recording or a static document is likely to result in lower levels of student engagement.

H1A: A quickly put together online lecture delivered through a video recording will result in lower levels of student engagement compared to student engagement achieved through a traditional face-to-face lecture.

H1B: A quickly put together online lecture delivered through a static document will result in lower levels of student engagement compared to student engagement achieved through a traditional face-to-face lecture.

The quality of notes and other study material available to students to study from will influence their performance on tests [12] [13] [14] . While students who attend traditional face-to-face lectures have but one chance to take good notes in class, students who are exposed to an online lecture have unlimited opportunities to review the online material and thus take good notes. This should then cause online students to do relatively better than students in traditional classes. However, reviewing long video recordings of lectures can be tedious and time consuming, requiring a quiet environment and a lot of bandwidth and thus it is unlikely that most students will watch the video repeatedly. Thus the likelihood that online students will watch video recordings of lectures just once combined with the lower levels of engagement for this mode of online instruction suggests that students exposed to video recordings of lectures will have worse notes compared to students in traditional classes.

H2A: A quickly put together online lecture delivered through a video recording will result in students having notes of a lower quality compared to notes taken by students in a traditional face-to-face lecture.

Students who are exposed to an online lecture delivered through a static document will have the material in a form that can easily be downloaded, printed, highlighted, annotated etc. Thus this should result in these students rating the quality of their notes highly.

H2B: A quickly put together online lecture delivered through a static document will result in students having notes of a higher quality compared to notes taken by students in a traditional face-to-face lecture.

Thus students exposed to a quickly made online lecture delivered through a video recording of a lecture will experience lower levels of engagement and the logistical issues related to the difficulties in watching long video recordings repeatedly will combine to result in lower student learning outcome for these students. On the other hand, students exposed to a quickly made online lecture delivered through a static document while experiencing low levels of student engagement, will have high quality notes to study from, and this “mixed environment” should result in learning outcomes comparable to that achieved by students attending traditional face-to-face lectures.

H3A: A quickly put together online lecture delivered through a video recording will result in students having lower levels of learning outcomes compared to the learning outcomes of students in a traditional face-to-face lecture.

H3B: A quickly put together online lecture delivered through a static document will result in students having learning outcomes comparable to the learning outcomes of students in a traditional face-to-face lecture.

H4: The perceived levels of engagement and quality of notes available to the students will mediate the effect of the instructional methods on student learning outcome.

2. Study Design and Data Collection

Data was collected over three consecutive fall semesters from 263 students enrolled in the introductory marketing course. This course and all other courses in this program are taught as traditional face-to-face courses with class sessions held over a 15 week instructional period. One of the lecture modules taught in this course, towards the end of the semester, was the topic of customer lifetime value (CLV), and this topic required students to understand moderately complicated calculations, and to engage in critical thinking. This lecture module on CLV served as the basis for this study, and it was delivered using a traditional face-to-face lecture to the students in semester 1 (N = 92); using a video recording of the semester 1 class lecture in semester 2 (N = 88); and using an edited transcript (combined with images of the PowerPoint slides) in semester 3 (N = 83).

In keeping with the objectives of this research, the online lecture material used in this study was deliberately chosen to be ones that were easy to create. The video recording used in semester 2 was a recording of the face-to-face lecture delivered in semester 1 using the classroom video recording system (Panopto). Turning this video recording into an online file with time stamps and other minor editing took approximately 1.5 hours. A transcript of this video recording was created and then edited for clarity, and the resulting document was embedded with static images of PowerPoint slides, equations, spreadsheet images etc. (all from the PowerPoint file used in the classroom lecture) to create the lecture material for semester 3. This took about 3.5 hours.

All other lecture modules during these three semesters were delivered using traditional face-to-face lectures. This included a lecture on Profit Maximization Pricing (PMP) which like the lecture on CLV required the students to understand moderately complicated calculations and to engage in critical thinking. Student performance on the PMP material was used as a control to check whether the students enrolled in the three semesters were of equal academic quality.

A week prior to when the CLV topic was to be taught, students in semester 2 and 3 were told that the instructor had to travel to attend an academic conference and that the topic for the week would be delivered online. During the next regular face-to-face class session following the CLV lecture, and just prior to being tested on the CLV and PMP topics, students were administered a brief survey that measured the students’ perceptions of the level of engagement they experienced when being exposed to the CLV lecture, and the quality of the notes that they had to study from—see Exhibit I.

Student learning outcomes for the CLV topic (dependent variable) and the PMP topic (control variable) were measured using high value calculation questions on tests. The test questions were worded differently from how the topic was taught in the lectures (something that the students had been told to expect), and this was done to ensure that students were not merely repeating the calculations from memory but rather they were engaging in critical thinking and demonstrating a good understanding of the material.

3. Results

Values of the Engagement variable and the Note Quality variable for the three semesters are provided in Table 1. A score of 3.00 indicates engagement with (or note quality of) the CLV lecture equal to that for the face-to-face lecture on PMP. A lower score indicates lower engagement (or note quality). As hypothesized (H1A and H1B), engagement with the quickly prepared online lectures delivered through the video recording and through the static document were significantly lower than the engagement with the face-to-face lecture of the same topic (t165.47 = 7.43, p = 0.00 and t139.94 = 7.14, p = 0.00 respectively). Thus we find support for hypotheses H1A and H1B. Incidentally, there was no significant difference in the level of engagement between the two online lectures (t160.33 = 0.63, p = 0.53).

The perceived quality of the notes for the CLV lecture relative to the PMP lecture (the latter of which was taught using a traditional face-to-face lecture in all three semesters) indicated that the students evaluated the quality of their CLV lecture notes to be poorer regardless of how the CLV lecture was administered—even for semester 1 when both lectures were taught using traditional lectures (t91 = 3.44, p = 0.00). Perhaps this is because the CLV topic was more involved and more difficult for the students. Regardless of that, the important comparison is between the perceptions of the note quality across the three semesters. As hypothesized (H2A), students exposed to the online lecture delivered through a video recording perceived the quality of their notes to be lower than the perceived quality of notes for the students in the traditional face-to-face class (t176.98 = 2.50, p = 0.01). While the mean value of the perceived quality of the notes for

Table 1. Engagement and note quality by semester.

the students exposed to the online lecture delivered by the static document (Mean = 2.83) was higher than for the students in the traditional lecture (Mean = 2.70), this difference failed to be significant by a narrow margin (t170.55 = 1.17, p = 0.12). Thus the data did not support H2B.

Students in the three semesters performed equally well on the test questions on the PMP topic (see Table 2 below) which was taught in all three semesters using traditional face-to-face lectures (F2,260 = 0.96, p = 0.38), indicating that the students enrolled in the three semesters were not significantly different in terms of their ability to perform in test questions requiring critical thinking and moderately complicated quantitative calculations.

Table 2 also lists the test scores for the students in the three semesters on the test questions on the topic of CLV. The test scores on this topic taught using different techniques were significantly different from each other (F2,260 = 5.75, p = 0.00). This was further investigated using t-tests on pair-wise combinations of the semesters. Students exposed to the traditional face-to-face lectures (Mean test score = 81.48%) and those exposed to the online lecture using the static document (Mean test score = 80.74%) did significantly better than students who were exposed to the online lecture using the video recording (Mean test score = 74.16%) (t169.91 = 3.45, p = 0.00 and t158.55 = 2.49, p = 0.01 respectively) while there no significant difference in the performance of the students enrolled in the face-to-face lectures and those exposed to the online lecture delivered through the static document (t143.08 = 0.30, p = 0.77).

Does the level of engagement and note quality influence student test scores? To test hypothesis H4 we followed Baron and Kenny’s four-step approach [15] [16] a procedure that has been widely used [17] [18] . As reported earlier, the different instructional methods had a significant effect on the dependent variable—student test scores on the CLV lecture (H3A and H3B; Baron and Kenny’s step 1), and on the mediating variables—Engagement and Note Quality (H1A, H1B, H2A and H2B; Baron and Kenny’s step 2). The third step in Baron and Kenny’s procedure calls for an ANCOVA, this time with Engagement and Note Quality as covariates, and the student test scores on the CLV lecture as the dependent variable. As required by Baron and Kenny’s four-step test for mediation, the mediating variables (Engagement and Note Quality) were both significant (F1,258 = 4.97, p = 0.03; and F1,258 = 45.11, p = 0.00) while the effect of the different instructional methods on the student test scores became non-significant (F2,258 = 1.18, p = 0.00) indicating that the effect of the different instructional methods on

Table 2. Test scores by semester.

student test scores was completely mediated by the impact of the different instructional methods on the mediating variables of Engagement and Note Quality. Thus H4 was supported.

In summary, this study found that online lectures delivered through a video recording, or through the use of a static text both scored lower on student engagement compared to a traditional lecture. Students in a traditional lecture environment and those exposed to a lecture through a static document both reported having better notes compared to students exposed to a video recording. Students in both the traditional lecture group as well as those in the static document group both outperformed the students in the video recording group. This superior performance exhibited by students in the traditional lecture and the static document groups can be attributed completely (complete mediation) to the differences in engagement and note quality across the groups.

4. Discussion

The scope of this research was limited to investigating effective ways of quickly offering an online lecture in a course that is otherwise taught using traditional face-to-face lectures. As discussed earlier, the motivation to offer the online lecture could be varied, including the planned absence on the part of the instructor for various reasons. Under these circumstances which easy-to-implement online instruction method results in the highest student learning outcomes? We compared student learning outcomes (using test scores on specific questions) across three modes of delivering lectures: 1) using a traditional face-to-face lecture, 2) using online instruction where the lecture was delivered using a video recording of the classroom lecture, and 3) using online instruction where the lecture was delivered using a static document. The results showed that as hypothesized, students reading the online static document performed as well as the students attending the traditional lecture and that both groups outperformed the students watching the video of the recorded lecture. The data also shows that, as hypothesized, both quickly created online instruction methods scored lower on engagement compared to the traditional face-to-face lecture. Also, as hypothesized, the perceived quality of the notes available for students was lowest for the students exposed to the video recording. Students who were exposed to the online lecture delivered using a static document and the students attending the face-to-face lecture both reported having much higher quality of notes. Finally, as hypothesized, the effect of the different instructional material on student test scores was found to be completely mediated by student engagement and perceived note quality.

5. Conclusions & Limitations

The conclusion one can make here is that an instructor who teaches a traditional lecture course and who wishes to offer one or two online lectures is best served by offering students a written lecture with embedded images, graphs, charts etc. Not only does such a document not take much time to prepare for the instructor (compared to the time required to create complex, interactive online instructional material), these documents can be easily downloaded by students with minimal bandwidth and device memory requirements for reading offline, printing, highlighting, making annotations etc.

The scope of this study was limited as stated in the previous section, and the findings are probably not applicable to fully online courses, or even hybrid courses where a substantial amount of instruction happens online. In these cases, students are likely to have higher expectations from the online lectures, and the types of online lecture material used in this study will probably not suffice.

Exhibit I

Pre-test Survey

Last week you had a lecture on Customer Lifetime Value (CLV) and in a few minutes you are going to be tested on that material. Kindly answer the two following questions, comparing your perceptions of the CLV lecture with the lecture on profit maximization pricing (which you studied two weeks previously).

To what extent did you feel engaged with the CLV lecture compared to the lecture on profit maximization pricing?

To what extent was the quality of notes you have for the CLV lecture better than or worse than the notes for the lecture on profit maximization pricing?

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1] Nistor, C. and Nyer, P. (2018) A New Framework for Online Business Teaching. Open Journal of Business and Management, 6, 576-584.
https://doi.org/10.4236/ojbm.2018.63044
[2] Johnson, S.D., Aragon, S.R., Shaik, N. and Palma-Rivas, N. (2000) Comparative Analysis of Learner Satisfaction and Learning Outcomes in Online and Fact-to-Face Learning Environments. Journal of Interactive Learning Research, 11, 29-49.
[3] Norman, C. and Burke, M. (2010) Learning Effectiveness Using Different Teaching Modalities. American Journal of Business Education, 3, 65-74.
[4] McLaren, C.H. (2004) A Comparison of Student Persistence and Performance in Online and Classroom Business Statistics Experiences. Decision Sciences Journal of Innovative Education, 2, 1-10.
[5] Ni, A.Y. (2017) Comparing the Effectiveness of Classroom and Online Learning: Teaching Research Methods. Journal of Public Affairs Education, 19, 199-215.
https://doi.org/10.1080/15236803.2013.12001730
[6] McCutcheon, K., Lohan, M., Traynor, M. and Martin, D. (2015) A Systematic Review Evaluating the Impact of Online or Blended Learning vs. Face-to-Face Learning of Clinical Skills in Undergraduate Nurse Education. Journal of Advanced Nursing, 71, 255-270.
[7] Means, B., Toyama, Y., Murphy, R., Bakia, M. and Jones, K. (2010) Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies. Monograph.
http://www.ed.gov/about/offices/list/opepd/ppss/reports.html
[8] Pereira, J., Pleguezuelos E., Merí, A., Molina-Ros, A., Molina-Tomas, C. and Masdeu, C. (2007) Effectiveness of Using Blended Learning Strategies for Teaching and Learning Human Anatomy. Medical Education, 41, 189-195.
[9] Brown, B.W. and Liedholm, C.E. (2002) Can Web Courses Replace the Classroom in Principles of Microeconomics? The American Economic Review, 92, 444-448.
[10] Dixson, M.D. (2010) Creating Effective Student Engagement in Online Courses: What Do Students Find Engaging? Journal of the Scholarship of Teaching and Learning, 10, 1-13.
[11] Kahn, P., Everington, L., Kelm, K., Reid, I. and Watkins, F. (2017) Understanding Student Engagement in Online Learning Environments: The Role of Reflexivity. Educational Technology Research and Development, 65, 203-218.
https://doi.org/10.1007/s11423-016-9484-z
[12] Kiewra, K.A. (2002) How Classroom Teachers Can Help Students Learn and Teach Them How to Learn. Theory into Practice, 41, 71-80.
https://doi.org/10.1207/s15430421tip4102_3
[13] Rahmani, M. and Sadeghi, K. (2011) Effects of Note-Taking Training on Reading Comprehension and Recall. The Reading Matrix, 11, 116-128.
[14] Chang, W. and Ku, Y. (2014) The Effects of Note-Taking Skills Instruction on Elementary Students’ Reading. The Journal of Educational Research, 108, 278-291.
https://doi.org/10.1080/00220671.2014.886175
[15] Baron, R.M. and Kenny, D.A. (1986) The Moderator-Mediator Variable Distinction in Social Psychological Research: Conceptual, Strategic, and Statistical Considerations. Journal of Personality and Social Psychology, 51, 1173-1182.
[16] Bagozzi, R.P. and Yi, Y. (1989) On the Use of Structural Equation Models in Experimental Designs. Journal of Marketing Research, 26, 271-284.
https://doi.org/10.1177/002224378902600302
[17] Gopinath, M., Glassman, M. and Nyer, P. (2018) The Impact of Multilingual Product Packaging on Readability and Product Evaluation. Journal of Applied Marketing Theory, 8, 1-20.
[18] Nyer, P. (2000) An Investigation into Whether Complaining Can Cause Increased Consumer Satisfaction. Journal of Consumer Marketing, 17, 9-17.
https://doi.org/10.1108/07363760010309500

Copyright © 2024 by authors and Scientific Research Publishing Inc.

Creative Commons License

This work and the related PDF file are licensed under a Creative Commons Attribution 4.0 International License.