The Influence of Instructional Design and Instructional Material on Learners’ Motivation and Completion Rates of a MOOC Course

Despite the formal, non-formal and informal training opportunities offered by MOOCs, they show low completion rates due to both personal factors of the participants and the instructional design of the courses. Empirical research has shown that good instructional design can help increase learners’ motivation, helping them achieve high performance by completing the program. To determine to which extent the quality instructional material, which is the result of a good instructional design, contributes to their motivation and high rates of completion of the courses, we created an eight (8) week MOOC program on “Violence and bullying in schools”, following the instructional design model “Systems Approach Model”. The motivation of the learners from the instructional material was measured using the Instructional Material Motivation Survey (IMMS) tool and showed that all the learners were highly motivated both as a whole and in every factor of the ARCS model (Attention, Relevance, Confidence, Satisfaction) contributing to very high completion rates, as from the 1309 learners who started the program, 1050 completed it (80.2%). Considering the results of our research, we, finally, suggest some points that can be taken into account when designing similar programs.

Participants in MOOCs do not pay tuition fees nor do they have to meet certain criteria to enroll in them, even if their creator suggests having specific knowledge and skills to be able to understand their content. Their learning material is offered through short videos, slides or other digital files (Hoy, 2014) and is hosted on online platforms such as Coursera and Edx. For the evaluation of the learners, assignments are assigned that are graded by graduates, teachers or other learners. Small, closed-ended quizzes that are automatically graded by computers are also used. Upon successful completion of the program, a free charge non-formal electronic certificate of completion or a formal certificate of payment and participation in formal examinations is provided (Karnouskos & Holmlund, 2014).
Despite the ease of access and training opportunities they offer, many participants appear disappointed with their participation due to their form, their instructional design, the lack of frequent contact with the instructors and the vague instructions given to them (Yuan & Powell, 2013;Hew & Cheung, 2014).
Eventually, a very small percentage manages to complete them. In general, their completion rates range between 5% -15% (Jordan, 2013) or less than 10% as concluded by Alraimi, Zo, & Ciganek (2015) considering the completion rates of some other surveys.
The motivation of the learners plays an important role in the participation and the completion of the online programs. According to the research, the motivation of the learners can be achieved through a good instructional material, which is the result of an equally good instructional design, contributing significantly to the increase of the completion rates of the online programs. In the present study, we examine the degree of motivation of learners from the instructional design and instructional material of the first MOOC of the University of the Aegean (Greece), lasting eight (8) weeks, on "Violence and bullying in schools" and whether the degree of motivation contributed to increased program completion rates.

Instructional Design
In the modern concept of teaching, all its parts (instructor, students, learning material, learning environment) have a critical role and any change in one of them can affect the rest, but also the final learning outcome. That is, they function as a system and a way to improve the learning outcome is through instructional design (Dick, Carey, & Carey, 2015).
In online learning environments, where lessons are conducted via the Internet, instructional design is considered necessary, as it systematizes the development process of these programs and contributes to achieving the learning goals  (Sofos, Kostas, & Paraschou, 2015) ensuring that the educational material created is effective and suitable for the educational needs of the learners. One of the instructional design models we relied on to develop our program is Dick, Carey, & Carey's "Systems Approach Model". This model is one of the best known (Gagne, Briggs, & Wager, 1992;Sofos et al., 2015), most popular and most influential, and it is comparison model for all other instructional design models (Gustafson & Branch, 2002).
The model is completed in ten different steps that can be followed linearly, cyclically, or in parallel (Dick et al., 2015). These steps are: 1) Identification of instructional goals. Instructional goals are more generally articulated to performance goals. Therefore, an educational goal may equate to a set of performance goals (Oosterhof, 2010) achieved through the achievement of the performance goals associated with (Sofos et al., 2015).
2) Conducting an instructional analysis, during which the educational goals of the previous step are analyzed and the steps for their achievement are determined, as well as the skills, knowledge and attitudes that the learners must possess to achieve them to the maximum extent.
3) Analysis of the learners and the context, during which the learning characteristics of the learners and the educational context in which they will learn and apply their new knowledge/skills are clarified. 4) Setting performance objectives, that is, what learners will be able to do, as well as how it will be demonstrated that they can do it. 5) Development of assessment instruments, which will examine the degree of achievement of the performance objectives of the previous stage. 6) Development of the instructional strategy that will lead to the achievement of the performance objectives. The instructional strategy may include pre-learning activities to motivate learners and increase their interest, activities of presenting new learning material, activities of active participation in the learning process, practice and reflection and activities of evaluating new knowledge and applying it in real conditions. 7) Development and/or selection of the instructional material based on which the instructional strategy of the previous stage will be implemented. 8) Development and construction of formative evaluation that will identify potential problems in instructional planning and possibilities for further improvement. 9) Review of the instructional intervention, based on the results of the formative evaluation, which will allow its improvement. 10) Developing and conducting a summative evaluation, which as a step does not belong to the design process, however, it is necessary conclude the success or not of the teaching. the reasons why one will decide and act with certain behavior and the reasons that determine the intensity of the effort that one will make (Keller, 2010).
Motivations can be categorized as intrinsic and extrinsic. Intrinsic motivation stems from the learner himself and is related, for example, to his need for learning, interest, curiosity, and inner satisfaction. On the other hand, extrinsic motivations stem from learner's external environment and are related to rewards and applause (Davidson, Sternberg, & Sternberg, 2003;Dembo & Seli, 2020).
Motivation becomes even more important in autonomous learning environments, such as MOOCs than in traditional learning environments, due to the instructor's lack of control over learners' activity and the lack of communication between learners (Semenova, 2020).
One way to increase the motivation of learners to continue and complete the program in which they participate, is its instructional design, as various studies have reported that poor instructional design is an obstacle that leads to dropping out of courses (Gütl, Rizzardini, Chang, & Morales, 2014;Nawrot & Doucet, 2014;Loizzo, Ertmer, Watson, & Watson, 2017), while on the contrary, a good instructional design can promote learning (Yousef, Chatti, Schroeder, & Wosnitza, 2014;Jung, Kim, Yoon, Park, & Oakley, 2019). Similarly, the instructional material of a MOOC program that is created or selected during the instructional design, is an important factor that influences the participation of the learners. In the research of Wang & Baker (2015) the participants who completed the program "Big Data in Education" considered the interest of the learners in the instructional material more important for the completion of the program than their interest in the MOOCs, while in the research of Hone & El Said (2016) found that learning material significantly affects participation in the program.
More recently, in the Hew (2018) study, in which ten highly-rated MOOCs were examined taking into account the comments on the CourseTalk website of the learners who attended them, one of the factors that contributed to the active participation of the learners was the instructional material that meets needs and preferences of the learners.

Current Study
The purpose of this study is to evaluate the instructional design and instructional material of the MOOC program that we created, in terms of the degree of motivation that caused to those who attended it and to investigate the degree that their motivation from the instructional material contributed to increased completion rates.
Therefore, the following research questions were posed: • Did the instructional material influence the motivations of the learners, both overall and in terms of the four factors of the ARCS (Attention, Relevance, Confidence, Satisfaction) model?
• Did the instructional design influence the course completion rates?

Research Model and Procedure
For the present study, which is part of the first researcher's doctoral research, we adopted the quantitative research method. The IMMS instrument was integrated into the program hosting platform and was answered once at the end of the program by the learners who completed it.

Research Context
The program, which was the first attempt at the University of the Aegean in the field of MOOCs, was conducted from 3/2 to 29/3/2020 and was hosted on an OpenEdx platform that we installed on a server of the University. 2) Short introductory video (up to 2 minutes) that summarized the highlights of the previous week and informed about the topic and goals of the week that was starting.
3) Motivational activities that motivated the learners to submit their previous views, knowledge, attitudes, experiences and to develop a dialogue between them.
4) The main instructional material with short videos of up to 6 minutes with built-in slides that highlighted the main points that were heard or presented other explanatory elements (graphs, sketches, etc.). Videos with facts, testimonies, simulations, and analogies were also used as examples to explain the concepts presented in the main instructional material. 8) Additional educational material to deepen the knowledge presented.

Sample
Initial interest in attending the program expressed 1952 active teachers, pedagogical students, and individuals, but the majority were active teachers. Some participants did not activate their account or never showed up when the program started. In total, 1309 people participated in at least one of the activities of the program. Of these, another 259 dropped out of the program at some point, most in the first week of the course. Finally, 1050 people completed the program and were asked to answer the IMMS instrument stating the degree to which they agree with each of its statements. Two people did not answer, so the final sample is the 1048 learners whose answers were recorded.

Instrument
The IMMS instrument was used to measure the motivation of the learners from the educational material.
The IMMS instrument is designed based on the principles of the ARCS model, which provides strategies for developing instructional material that will create  (Keller, 2010).
The questionnaire contains 36 statements (some of which are inverse), each of which corresponds to one of the four factors of the ARCS model (Keller, 2010): • the Attention that refers to the curiosity, activation, and interest caused to the learners by the course material.
• the Relevance that learners recognize in the course material to their needs, motivations, experiences, and interests.
• the Confidence inspired by the course material, i.e. the expectation for positive learning outcomes.
• the Satisfaction provided to students during or after the course.
The instrument was translated from English to Greek following the for-

Data Analysis
For the analysis of the answers of the IMMS questionnaire the parametric One sample t-test was used, as the distribution of the sample was close to normal.
This test checks whether the average of the population from which the sample was taken differs from a control value. The control value was set to 3, which on the Likert scale of the IMMS instrument corresponds to the average of its scale, as proposed by its manufacturer (Keller, 2010).
No inferential statistics were used to answer the second research question, only descriptive ones.

Research Question 1
In all factors but also in general, the learners express that they were greatly mo- In the factor with the lowest overall average (Confidence), the lowest averages are collected by all inverse statements with averages well below the scale average (3). Specifically, the lower averages are collected by the statements "I could not really understand quite a bit of the material in this lesson" (M = 1.45, SD = 0.897), "This material was more difficult to understand than I would like for it to be" (M = 2.10, SD = 1.105), "The exercises in this lesson were too difficult" (M = 2.13, SD = 1.030) and the statement "Many of the pages had so much information that it was hard to pick out and remember the important points" (M = 2.71, SD = 1.186). On the contrary, the highest averages are gathered by the statements with which they largely agree, with averages well above the average of the scale. Specifically, they agree with the statements, "The good organization of the content helped me be confident that I would learn this material" (M = 4.43, SD = 0.725), "As I worked on this lesson, I was confident that I could learn the content" (M = 4.24, SD = 0.731), "After working on this lesson for a while, I was confident that I would be able to pass a test on it" (M = 4.03, SD = 0.908), "After reading the introductory information, I felt confident that I knew what I was supposed to learn from this lesson" (M = 3, 97, SD = 0.884) and the statement "When I first looked at this lesson, I had the impression that it would be easy for me" (M = 3.58, SD = 0.957).
Subsequently, the One-Sample t-test was performed on all the factors of the ARCS model, but also as a whole, as the distribution of the sample was close to normal. This test checks whether the average of the population from which the sample was taken differs from a control value. In the test, the control value was set to 3, which on the Likert scale of the IMMS instrument corresponds to the average of its scale, as suggested by its manufacturer (Keller, 2010).
The test showed that in all factors, but also in general there were statistically significant differences (p = 0.00 < 0.05). Also, the value of it in all variables was found outside the acceptance range, which sets the 95% Confidence Interval of the Difference. Therefore, we conclude that the training material had a statistically significant effect on the motivations of the learners, both as a whole and in all the factors of the ARCS model (Table 1).

Research Question 2
To calculate the completion rates, a different way is followed by each researcher (Grainger, 2013). One of them is to consider the initial number of people enrolled in the program, resulting in small completion rates. Another one, which we also adopted, is to consider the number of people who completed the program in relation to those who participated in, at least, one activity of it. Based on this method of calculation, 1050 people completed the program from the initial 1309 who participated in at least one activity of the program (f = 80.2%).

Research Question 1
Overall, the learners disagree, almost completely, that the program was boring and unattractive, difficult to learn and abstract to the extent that it did not arouse their interest and attention. On the contrary, they almost completely agree that they felt good about completing the program because they learned something useful and successfully completed something important to them. The positive emotions of those who successfully complete a program were also highlighted in the research of Milligan & Littlejohn (2014) and Kleiman, Wolf, & Frye (2015), due to the acquisition of knowledge, experience and professional development opportunities (Zutshi, O'Hare, & Rodafinos, 2013;Milligan & Littlejohn, 2014;Van Hentenryck & Coffrin, 2014;Kizilcec & Halawa, 2015;Park, Jung, & Reeves, 2015;Huang & Hew, 2016;Koutsodimou & Tzimogiannis, 2016) or simply because they achieved their goals (Beaven, Hauck, Comas-Quinn, Lewis, & de los Arcos, 2014;Wilkowski, Deutsch, & Russell, 2014;Kleiman et al., 2015;Li, 2015). In fact, the better the course design and the quality of the instructors and learning material (Oakley, Poole, & Nestor, 2016), the more satisfaction they feel and declare they are willing to attend other courses in the future (Belanger & Thornton, 2013;Tomkin & Charlevoix, 2014;Koutsodimou & Tzimogiannis, 2016), something that seems to happen in our research, too.  Keller (2010: p. 46) states that if there is an improvement in the factors Attention, Relevance and Confidence, then learners will be motivated to learn, and then they should be satisfied to continue wanting to learn. We believe, therefore, that all the learners were motivated by the instructional material and the wider instructional design that was followed and were satisfied until the end of the program.
Among the four factors, the Satisfaction factor improved the most. This finding confirms Gagné and Driscoll (1988) who characterize Satisfaction as the easiest factor to achieve and is usually achieved with constructive and timely feedback, as was the case in our program. On the other hand, while it is easy to provoke attention, it is difficult to maintain it (Keller, 2010). From the second-best average that the Attention factor had, it seems that both the challenge and its maintenance in the program we implemented were achieved. The Relevance factor refers to how relevant the material is to the goals, needs, experiences and interests of the learners. This factor had the third-best average because of the statement "It is clear to me how the content of this material is related to things I already know" which received the lowest average among the other statements in both groups. Given that the content of the program was entirely related to the issues of violence and bullying in schools that most teachers face daily, there may have been a misunderstanding in the wording of the statement and specifically in the section "related to things I already know" which they perceived to refer to the knowledge they already possessed and not to issues related to their daily lives. The same question, with the same wording, received the lowest average among the other questions of the factor and in Gnostopoulou's research (2018).
Τhe Confidence factor shows the lowest average, although it is much higher than the average of the scale (M > 3), because, at the beginning, the program seemed demanding and difficult, reducing their beliefs that they could obtain the knowledge they would like and complete it successfully, while then the tight schedule and the several evaluations (quizzes, final assignments) that existed played a role.

Research Question 2
The program was completed by 80.2% of the learners who started it. This percentage is very high in relation to the percentage of people who complete MOOCs

Conclusion, Limitations and Future Research
The present study investigated to which extent the instructional material, which is the result of a good instructional design, of the first MOOC program that we created helped to increase the motivation of those who attended it to higher rates of completion of the program. The results showed that the learners who participated in the program were largely motivated by the instructional material, both as a whole and in each factor of the ARCS model. Their degree of motivation is much higher than the average of the scale, consistent with the findings of Huang & Hew's research (2016, but surpassing the results of other research designed following Keller's instructional design model (Li, 2015;Huang & Hew, 2016;Gnostopoulou, 2018). This fact shows that it does not matter which model of instructional design will be used, if it is used correctly, considering the requirements and needs of the learners. Of course, this is difficult to do in MOOCs involving a heterogeneous set of learners. However, the requirements of participating in the program out of personal or professional interest may be considered, as this is the most common motivation for enrolling in MOOCs.
The high degree of motivation of the learners contributed to very small dropout rates, only 19.8%, confirming empirical research that the instructional material, which is the result of an equally good educational planning, affects the participation and completion of the programs.
In order to design future successful programs in terms of student performance and low dropout rates, we suggest: • following a careful instructional design according to a specific model, such as that of Keller (2010) or Dick et al. (2015). • use of illustrative examples, case studies, facts or events that contradict students' perceptions and knowledge, use of analogies and allegories, alternation of methods and means of instructional material and frequent change of interaction control between the teacher and the learner. • introductory videos that will guide, inform and arouse the interest of the learners to watch the section. • short videos of up to 6 minutes with integrated slides or other multimedia material that will highlight to students the most important points from what they hear and see.
• closed-ended quizzes of various types of questions and not only cognitive ones in which explanatory comments will be provided in each answer.
• interesting start-up activities and optional tasks that will encourage dialogue.
• small final weekly projects to apply the new knowledge and the provision of exemplar answers either in the forum or automatically after grading a task.
• equal distribution of material in weekly units.
• provision of additional instructional material and bibliography.
Despite the large sample of our research, some limitations do not allow the results to be generalized. In particular, our research examined only one course in which a relatively homogeneous sample participated, as mainly teachers with at least a higher education degree, as well as knowledge and experience in the subject of the program participated. Nevertheless, the findings show that through the general instructional design of the program and the quality instructional material that meets the needs and requirements of the learners, enabling them to apply their knowledge to real problems through various activities, the motivation and active participation of the learners in the program can be achieved, accomplishing, in the end, high completion rates.
Our research extends the existing literature examining the pedagogical quality of MOOCs. These findings, combined with other researchers' suggestions that MOOCs have room for improvement in their instructional design, can be considered by future program designers to design quality programs that will help learners meet the goals for which they attend them.