Assessing the Impact of an Instructional Design Course on Arabian Gulf University Distance Learning Students’ Instructional Design Competencies

This study investigated the impact of an instructional design graduate course (module) on the Arabian Gulf University graduate students’ learning outcomes represented in the instructional design general competencies generated by the International Board of Standards for Training, Performance and Instruction (IBSTPI) 1986. The study also sought to provide an analysis of the general instructional design competencies and performance statements including Knowledge, Skills, and Affective competencies that are necessary to achieve effective instructional design and development outcomes with a specific focus on distance teaching and training applications. The study used a developmental research method guided by ADDIE instructional design model on a sample of 22 graduate diploma and master students in the department of distance learning. Results of data analysis revealed statistically significant differences (P ≤ 0.05) between the students’ pre and post mean scores on instructional design general competencies scale in favor of the post administration. Data analysis results also indicated that students believed that the course content was realistic, clear, and useful. It links theory to its real and practical applications in the field of developing distance teaching and training materials. The huge content, the big effort required for successfully passing the course compared to other courses; as well as the short period of time allocated for studying the course materials represented the challenging part of the course. To improve the course outcomes in the future students recommended the need for an Arabic version of the workbook and renovation of the broken internet links related to course topics or replacing them with active ones. How to cite this paper: Ismail, A.M.A. and Alkhazali, T.M.N. (2019) Assessing the Impact of an Instructional Design Course on Arabian Gulf University Distance Learning Students’ Instructional Design Competencies. Open Journal of Social Sciences, 7, 106-129. https://doi.org/10.4236/jss.2019.79009 Received: October 12, 2018 Accepted: September 13, 2019 Published: September 16, 2019 Copyright © 2019 by author(s) and Scientific Research Publishing Inc. This work is licensed under the Creative Commons Attribution International License (CC BY 4.0). http://creativecommons.org/licenses/by/4.0/

general instructional design competencies and performance statements including Knowledge, Skills, and Affective competencies that are necessary to achieve effective instructional design and development outcomes with a specific focus on distance teaching and training applications. The study used a developmental research method guided by ADDIE instructional design model on a sample of 22 graduate diploma and master students in the department of distance learning. Results of data analysis revealed statistically significant differences (P ≤ 0.05) between the students' pre and post mean scores on instructional design general competencies scale in favor of the post administration. Data analysis results also indicated that students believed that the course content was realistic, clear, and useful. It links theory to its real and practical applications in the field of developing distance teaching and training materials. The huge content, the big effort required for successfully passing the course compared to other courses; as well as the short period of time allocated for studying the course materials represented the challenging part of the course. To improve the course outcomes in the future students recommended the need for an Arabic version of the workbook and renovation of the broken internet links related to course topics or replacing them with active ones.

Introduction
Instructional design is the practice of creating instructional experiences which make the acquisition of knowledge and skills more efficient, effective, and appealing [1]. The process consists broadly of determining the state and needs of the learner, defining the end goal of instruction, and creating some intervention to assist in the transition. It is a systematic process by which the instructional materials are designed, developed, and delivered. The terms instructional design, instructional technology, learning experience design, educational technology, curriculum design, and instructional systems design (ISD) are often used interchangeably. It is a way of planning instruction considering the learner, end goal or product, and evaluation. Many times, multimedia tools are used to improve instruction and increase student engagement [2].
When designing instructional material for open and distance learning the concept of "deep design" is highly recommended. The term "deep design" is used intentionally to distinguish student and learning centered lesson planning from the classroom-centred, activity-oriented planning that is common among beginning teachers [3]. Deep design work is not directly visible to students or to anyone else who is not part of it. The teacher and classroom-centered instructional design models focus on activities, teacher performance, classroom events and experiences burning question: "What will students be doing today?" and planning addresses only the teacher's time with students. On the other hand; the student and learning-centered instructional design models focus on what kinds of thinking students do, the intellectual skills students develop burning question: "What will students be learning today?", planning addresses long-term outcomes and what students take away from the classroom events and experiences.
To achieve effective learning outcomes, the science of instruction and instructional design models are used to guide the development of instructional design strategies that elicit appropriate cognitive processes. [4] explored the competencies required for an instructional design manager to be effective in higher education settings; they used a Delphi study surveying managers and leaders of instructional design through an anonymous consensus-building process consisting of two rounds of surveys. Results identified eight major categories with 64 competencies as relevant for leading and managing instructional design in higher education. Competencies specific and beneficial to online high school teachers who are modifying their own courses were identified by [5]. He stressed out that existing instructional design standards available to guide online teachers are not only too numerous, they are also inconsistent. Moreover, a lack of clarity exists The instructional designer's competencies essential for the context of online higher education were investigated by [6] who have selected an instructional design unit in a university research course as a case of investigation. To identify and compare competencies at organizational and individual levels, their study employed a mixed method to collect and analyze data based on a validated instructional design (ID) competency model by the IBSTPI as a framework.
Throughout the study Instructional design competencies expected jobs/tasks and currently performed jobs/tasks were systematically analyzed, and the applicability of the IBSTPI model in this specific context of online higher education was verified. Based on the empirical findings, the study proposed a refined competency model to improve the instructional design competencies performance in human resources development and management practice.
Critical discussions within the field of instructional design have addressed the roles and competencies of designers, as well as the nature of design work per se.
In this concern, [7] presented an overarching metaphor-namely, instructional design as a journey into the unfamiliar-that views design as a two-fold learning enterprise (i.e., innovative and maintenance learning) and characterizes designers as sojourner learners. The metaphor placed instructional design in a narrative context and considered designers, rather than formalisms, as the primary drivers of the design process. The study presented several implications of his metaphor related to the identity and innovative practices of designers in the field and finally, suggested that this metaphor could serve as a framework for inquiries into everyday instructional design work, examinations of innovative design practices, and further discussion of the respective roles of instructional designers and design formalisms.
In their study [8] investigated developing competency of teachers in basic education schools. The research instruments included the semi-structured in-depth interview form, questionnaire, program developing competency, and evaluation competency form. Data analysis revealed that factors of competency were: 1) the persistence in learning management and work practice, 2) competency in planning, goal setting, learning management and work practice, 3) competency in Information and Communications Technology (ICT) use in learning management and work practice, 4) creativity of learning management, 5) competency in following up and evaluation in knowledge management and work practice, and 6) competency in improving and developing the learning management and work practice. Program for developing competency included 9 factors and 7 learning units; total of 200 hours.
The original set of instructional design competencies was developed in 1986 as a result of more than a year of research, discussion, and validation by a group of instructional design professionals and academicians [9]. The level of proficiency described in the 1986 competencies was taken to represent an instructional de- In response to this changing environment, the IBSTPI set out to review and revise the ID competencies in the year 2000 and, more recently, in 2012. With every revision, more elements have been added to the process including: 1) The influence of advanced technologies, team-based design, and business management skills; 2) The professional foundations of design, as well as planning and analysis, design and development, and implementation and management skills; 3) A categorization of competencies as essential or advanced;

Aim of the Study
The present study aimed to investigate the impact of an instructional design graduate course (module) on the Arabian Gulf University graduate students' learning outcomes represented in the instructional design general competencies generated by the IBSTPI 1986, and to explore satisfaction of the students with the learning experience.
ganization to understand, talk about, and apply the competencies in an organizing scheme". With the advent of performance-based educational techquices, competencies have served as the nucleus of program design and the development efforts [9]. According to [12] the basis of this approach was the demand for clearly definable measures of program effectiveness in teacher education programs. Another origin was that competency-based education applied the innovative systems design techniques and elements of mastery learning [13].
Competency based education program design is not a new approach; it was widely used in both teacher education and K-12 education during the 1970s of the past century. Today, competencies continue to be used in many of these same activities in higher education and business and training environments. As such, competency based education is applied in the distance teaching and training program at the Arabian Gulf University. In institutions which utilize and make use of distance education, teaching responsibilities are usually divided into two phases; that of course development in which course materials are prepared in advance, and tutoring in which instructional support is provided to learners as they are using the materials. Course development tends to be subdivided further into two aspects; that of providing subject matter expertise, and that of providing expertise in the area of instructional techniques appropriate to distance education. Similar competencies are required for both aspects of course development, with some specific competencies required for subject matter specialists and instructional design specialists. The generic IBSTPI competency development model is shown in Figure 1.
The effectiveness of courses delivered over a distance, like face-to-face instruction, depends on the planning of the course, class activities and the instructional materials used. The use of systematic instructional design in course planning can help to make any instruction more successful in promoting learning.
However, there are significant differences between face-to-face instruction and distance learning. [14] reviewed applicable instructional design models for distance education, the unique characteristics of distance education in higher education via interactive television (ITV) are discussed, and a nine-step instructional design model, designed specifically for college course delivery via interactive television, is presented. Their model is an adaptation of the Dick and Carey model which is discussed in the article. Implementation of this model in situations which do not have large support staff and budgets for ITV is discussed.
In the distance teaching and training program at the Arabian Gulf University, these competencies are addressed in many courses [15] taught in the first se-

Statement of the Problem
Researchers in the field of educational technology and distance learning and training assert that instructional design competencies play a critical role in the success in developing effective learning materials. Therefore, the present study is conducted to investigate the impact of an instructional design graduate course (module) on the Arabian Gulf University graduate students' learning outcomes represented in the instructional design general competencies generated by the IBSTPI 1986 and to explore satisfaction of the students with the learning experience.

Research Questions
The main question of the present study was "what was the impact of an instruc-

Research Hypotheses
The present study sought to test the following research hypotheses: 1) Instructional design course learning materials and teaching strategy would help students to master the instructional design course knowledge competencies to a level of 85%.
2) Instructional design course would significantly develop Arabian Gulf University Diploma and Master Students' instructional design competencies.
3) Instructional design course materials and teaching strategy would lead to a high level of satisfaction with learning among Arabian Gulf University students.

Importance of the Study
Studying the development of instructional design competencies by distance learning specialists is very important for many reasons: Firstly, most of the instructional design competencies are needed by all instructional designers and distance learning writers; Secondly, based on their learning and training, distance teaching and training program candidates must demonstrate that they have acquired the basic and the general instructional design competencies in order to perform all the specialized tasks related to distance learning design and learning material development; Thirdly, a distance learning designer and learning material writer needs to master how to create learning contents with information that can be used and applied in solving a real distance learning problem;

Limitations
The sample of the present study was selected from the Arabian Gulf University distance teaching and training master and postgraduate diploma students who were studying the instructional design course. Instructional design competencies investigated were adapted from the general instructional competencies published by the IBSTPI in 1986, in addition to the knowledge component of these competencies generated from conducting content analysis of the instructional design course. Therefore, generalization of the results would be limited to instructional design courses and training contexts and learning communities which possess similar characteristics and study the same course contents.

Method and Procedures
This is a developmental research study in which a mixed research method was utilized. The developmental research can be defined as the systematic study of designing, developing, and evaluating instructional programs, processes, and products that must meet criteria of internal consistency and effectiveness [15].
In the developmental research mixed research methods may be applied. Furthermore, Richey (1994) classified three approaches of developmental research; the first type involves situations in which the product-development process is analyzed and described, and the final product is evaluated. The second type of developmental research focuses more on the impact of the product on the learner or the organization. The third type is oriented toward a general analysis of design, development or evaluation processes as a whole or as components. In the present study a mixed research method combined three research approaches: 1) A descriptive research approach was implemented for analyzing the students' characteristics, instructional design module content analysis, previous research and resources analysis to establish a list of instructional design general competencies and develop the course online component.
2) A systems development approach by utilizing the ADDIE instructional design model in guiding the course development starting by the analysis phase and ending with the evaluation phase which intended to assess the impact of the proposed course on developing distance teaching and training program master students' instructional design competencies.
3) An experimental research approach with quasi research design (one group design) to assess the impact of the developed course on the students' instructional design competencies which were assessed at the beginning and towards the end of the course after successfully completing the course materials. Center.

Instructional Design Mastery Learning Assessment Methods
These methods included the following: 1) Instructional design reading assignment which aimed to provide students with basic knowledge of various instructional design models and their applications in designing and developing distance learning materials. 2) A course main assignment. The purpose of this assignment was to help students apply the instructional design standards in developing distance learning materials. 3) Six formative objective tests in instructional design aimed to help students master the cognitive competencies of each of the six learning units. 4) A closed book final exam in instructional design used to assess the students' mastery level of the cognitive instructional design competencies.

Instructional Design General Competencies List (Scale)
The instructional design general competencies scale was adapted from the IBSTPI 1986 list of competencies. The list of the instructional design general competencies was composed of 16 basic competencies; each main competency has its own performance statements (indicators). The rating used for the assessment is a 5-point Likert-type scale, where 5 meant "a very high level of mastery" and 1 referred to a weak level of mastery of the nominated competency. The students were asked to judge their mastery of a competency before and after studying the instructional design course. Reliability coefficient of the instrument (Cronbach's Alpha) was calculated and found to be around 0.950 which meant that the instrument was appropriate to be used ( Table 1). The above scale was administrated at the beginning and at the end of studying the instructional design course material. The study asked the student to rate her/his level of mastery of each of the 16 competencies, then compare the means and the gain in these competencies to test the impact of studying the course materials on developing the students' ID general competencies.

The Satisfaction with Learning Scale
The satisfaction with learning scale was prepared by the AGU quality and excellence center which was developed and validated by the National Commission for Academic Accreditation and Assessment-Saudi Arabia. The scale was composed of three parts; part one aimed to collect information about the course (title and code, program/department information and the academic year, in addition to the instructions on how to respond to the questions; part two aimed to assess students' degree of satisfaction with the course and its contents. This part consisted of 26 items (questions) which addressed formed students' satisfaction with the course. These questions were distributed among 4 dimensions: the study plan of the course, the teaching activities, judging the course value and the overall evaluation of the course. Table 2     2) The teaching activities 17 4, 5, 6, …, 20 3) The scientific value of the course. which enabled the student to report on her/his observations related to her/his satisfaction with the course as well as the learning experience and included the following questions: 1) What was the best part of the course you liked most?
2) What was the part of the course that you did not like?
3) What suggestions did you have for improving the course?

Results
The main purpose of the study aimed to answer the main question of the study

Results Related to Instructional Design General Competencies
Results related to instructional design general competencies were generated

Results Related to Topics and Contents of the Instructional Design Course
Instructional Design course (DLRDT536) is a three (3 Cr

Results Related to the Proposed Mastery Learning Strategy Used for Teaching the Instructional Design Course
Mastery learning is an instructional strategy and educational philosophy, first formally proposed by Benjamin Bloom in 1968. Mastery learning maintains that students must achieve a level of mastery (e.g., 90% on a knowledge test) in prerequisite knowledge before moving forward to learn subsequent information. If a student does not achieve mastery on the test, they are given additional support in learning and reviewing the information and then tested again. This cycle continues until the learner accomplishes mastery, and she/he may then move on to the next stage. Mastery learning is a method of instruction where the focus is on the role of feedback in learning. Furthermore, mastery learning refers to a category of instructional methods which establishes a level of performance that all students must master before moving on to the next unit [16].
Adeniji and his colleagues [17] examined the effect of mastery learning on senior secondary school students' achievement and retention in circle geometry.
The study was conducted in Ilorin, Kwara state, Nigeria using a qua- showed that senior school students' achievement in Geometry improved significantly when taught circle geometry using mastery learning approach. There was no gender difference found as well as no difference in the achievement of low, medium and high scoring students when taught with mastery learning approach.
There was also a statistically significant difference in the post test mean score and retention score of students taught circle geometry using mastery learning approach.
In 2017, [18] reported on study aimed to know the improvement of the students' achievement in poetry mastery and their perception regarding to the se- The data were collected by test in order to know the students' achievement in poetry mastery. The students' perception toward the implementation of the semiotic method for teaching and learning poetry was collected by questionnaire.
The data were analyzed by descriptive analysis. The result of the study showed that: 1) the average score of the students' achievement of poetry mastery was (71.75) which increased (11.59) point from pre-test (60.16), and 2) in general, the students' perception toward the semiotic method for teaching and learning poetry were good and positive.
Hussain and Suleiman [19] study investigated the effect of Bloom's mastery In the present study the authors attempted to assess the impact of the course proposed learning materials and teaching strategy by comparing the students' mean in the final achievement exam covering the cognitive components of the instructional design course competencies to a mastery level of 85% and the shape of the grades' distribution to the normal achievement curve and report the final results.
Bloom [20] when first proposed his mastery learning strategy in 1968, was convinced that most students could attain a high level of learning capability if the following conditions were available: 1) Instruction is approached sensitively and systematically 2) Students are helped when and where they have learning difficulties 3) Students are given sufficient time to achieve mastery 4) There is some clear criterion of what constitutes mastery.
The proposed mastery learning approach for studying a learning unit used with students enrolled in the instructional design course is shown in Figure 2.

Results Related to the Impact of the Instructional Design Teaching Strategy on Arabian Gulf University Diploma and Master Students' Mastery Level of the Instructional Design Course Knowledge Competencies
To test the impact of the instructional design teaching strategy on Arabian Gulf University Diploma and Master Students' mastery level of the instructional design course knowledge competencies, a closed book final exam was administered at the end of the course (last day of week 4). The final test was answered by 21 out of 22 students. Descriptive statistics results revealed that the sample' lowest score was 28 out of 40 and the highest score was 38 (i.e., with a range equal to 10). The sample mean of the final exam was equal to 35.5238 with standard deviation equal to 2.452. One sample t-test was used to compare the sample mean to a mean of 34 out 40 repressing the proposed level of mastery (85%).  Table 4 shows the sample grades' distribution as well as their histogram graph.
The results above showed that the final exam grades were negatively-skewed where the distribution has a greater number of higher values of students' grades with the tail heading off to the left. Such increase could be attributed to the adoption of the mastery learning strategy used for teaching the course.

Results Related to the Impact of the Course on Developing Arabian Gulf University Diploma and Master Students Instructional Design Competencies
To test the impact of the instructional design module on developing students' instructional design general competencies, the instructional design general competencies scale was administrated twice: at the beginning of the course (first day of week 1) and at the end of the course (last day of the 4 th week); after the students successfully completed the course requirements. The general instructional design competencies scale was pre and post answered by 19 out of 22 students enrolled in the course. Table 5 Table 5 provides the means, standard deviations of students' own ratings for each instructional design competency. The table also displays the ranking of the competencies based on the degree of judgment. From Table 5  for the statement # 6 (I can write statements of performance objectives).
2) As indicated in Table 5  3) The overall mean score of the students' instructional design general competencies is above average (m = 3.4112, Sd. = 0.83679) which is judged as possessing an average level of ID competencies.
In the post ID general competencies survey a noticeable development in students' competencies was revealed. From the post ID general competencies survey results (Table 6)   To test the impact of the proposed instructional design course on developing students` ID general competencies, a paired samples t-test was conducted to evaluate whether students' ID general competencies were affected by the course or not as presented in Table 7.
Notes: IDGCpr refers to Instructional Design General Competencies pre assessment, and IDGCpo refers to Instructional Design General Competencies post assessment.
From Table 7

Results Related to Satisfaction with the Instructional Design Course
Seventeen (17) out of the 22 students who participated in the study answered the satisfaction with the learning scale. Data analysis showed a very high degree of satisfaction with the course (i.e. overall satisfaction is 92%) as revealed in Figure   3.
With regards to the open-ended questions of the scale of satisfaction with learning, Table 8 summarizes comments made by the students.    -Is considered to be the core of the field of distance teaching and training.
-Though the course content is huge, the content is distributed in a useful and easy manner that helps students' learning.
-Topics and information are scheduled in a sequence that contributes to clarity of the learning materials.
-The course instructor is a unique, very nice and keen, a type of a professor who likes his students and considers their needs.

Discussion of Results and Conclusion
One of the reasons that the quality of much instructional material is poor is because it is not carefully planned and well designed. This fact led the Arabian Gulf University distance teaching and training program curriculum developers to consider a course in instructional design to help the diploma and master degree graduates to deal with the essential principles for designing instructional materials and developing essential competencies of instructional design.
The present study aimed to investigate the impact of an instructional design graduate course (module) on the Arabian Gulf University graduate students' learning outcomes represented in the instructional design general competencies generated by the IBSTPI 1986, and to explore satisfaction of the students with the learning experience.
The researchers developed a course in instructional design as a core program requirement, taught the course using a mastery learning strategy developed for enhancing the participants' mastery of the course competencies and then as- Data analysis revealed that the proposed mastery learning strategy used for teaching the course content proved to help the participants reach the estimated level of mastery (85% out of 40 = 34). Thus, the mean difference between the mean of the sample (35.5238) is statistically significantly different from the estimated mean of 34 (proposed level of mastery 85).
Data analysis revealed statistically significant differences (P ≤ 0.05) in the students' instructional design basic competencies between means of pre and post-survey. Students' ratings of their instructional design competencies were found to be higher in the post survey and ranged from high to very high (i.e. all statements' means scored more than 4.00 out of 5.00 in the post assessment). This significant development in students' instructional competencies is due to studding the course and the learning material, the strategy used for developing the course as a kind of deep design approach. This result asserted thoughts proposed by [3] which ensured that "deep design" was a term used by purposefully to distinguish student-and learning-centred lesson planning from the classroom-centred activity-oriented planning that is common among beginning teachers. These results were also supported by [8] who found that factors of competency were: 1) the persistence in learning management and work practice, 2) competency in planning, goal setting, learning management and work practice, 3) competency in ICT use in learning management and work practice, 4) creativity of learning management, 5) competency in following up and evaluation of knowledge management and work practice, and 6) competency in improving and developing the learning management and work practice.
The study also used a satisfaction instrument developed by the AGU quality and excellence center administrated at the end of the course. Analysis of data revealed a high level of satisfaction with the course material i.e. (around 92% overall satisfaction level). The students reported that the course content was realistic, clear and useful and links the theory of instructional design with its real applications, contributes to developing useful knowledge, skills and competencies for graduate students and helps them to deal with how to carry out learners' needs, learning environments and learning context. Moreover, the course helped them to understand how systematic design of instruction was guided by ADDIE, or Dick and Carey instructional design model works which in turn led to students' realization that "instructional design" be the core of the field of distance teaching and training.
Besides these advantages of the course students highlighted some difficulties To overcome these difficulties students suggested activation of the broken links or replacing them with active ones, planning for a field visit to do real analysis of learners and environments and contexts, taking into account all students' learning styles when developing the course learning materials and reducing the load of the course to help students learn the material better.
The development of the course and the assessment of its impact on AGU distance learning graduate students revealed that competencies were critical for instructional designers working in the field of distance learning and teacher education. Developing such competencies can be a research topic for future studies in training and certification of instructional designers and distance learning professionals to determine which competencies have the greatest impact on the success of an instructional design course and efforts. Based on these findings, the researchers suggested further training in the field of instructional design and further studies with a large group of students to assess the impact of graduate courses in developing educational competencies in graduate and postgraduate programs. This study used the competencies statements without the performance statements. Therefore, further studies can use the full statements with their performance indicators for postgraduate students to undertake a self-assessment to determine areas in which they may need to focus on for future professional development.