Identification of Key Performance Indicators through E-Learning Critical Success Factors for an Online Course of English for Football ()
1. Introduction
The usability as well as applicability of distance learning through e-learning or online classes are becoming a widely accepted and popular practice in education. With advancements in IT and communication technologies with high popularity among people of every age everywhere, as well as the pandemic COVID-19, e-learning is becoming so pervasive and popularized and globally accepted as an educational approach with its different tools, teaching methods, tasks, and learning style from traditional physical classes. According to Dust (2007), the authors approve that e-learning is good when it serves the right learners with the right skills at a reasonable expense in a timely manner. As there are some interchanging terms, Urdan and Weggen (2000) saw e-learning as a subset of distance learning, online learning as a subset of e-learning, and computer-based learning as a subset of online learning. Hall (1997) defined web-based training as an instruction that is delivered over the Internet or a company’s intranet. Urdan and Weggen (2000) defined e-learning as computer-based learning, web-based learning, virtual classrooms, and digital collaborations. They concluded that e-learning is inclusive and synonymous to all computer-related applications, tools and media channels that strategically enhance learning and teaching.
An issue in e-learning is the comparability of its pros and cons with the traditional physical classes to ensure the success of e-learning and online classes. E-learning can secure sustainability if using computers, applications, and online platforms, generally, information and communication technologies can enhance teaching and learning that are able to do so through traditional physical classes. One of the major concerns in e-learning, online classes, or distance learning is its effectiveness and efficiency in comparison with traditional physical classes that need to be identified, secured, and followed to be continued until reaching success. One solution is to benefit from critical success factors (CSFs) and key performance indicators (KPIs). Based on Leidecker and Brunto (1984) CSFs are “characteristics, conditions, or variables that, when properly sustained, maintained, or managed, can have a significant impact on the success of a firm competing in a particular industry”.
Although there are many studies in the literature done for the identification and proposition of the e-learning CSFs in education, e.g. Selim (2007), Cheawjindakarn et al. (2013), Prougestaporn et al. (2015), Bhuasiri et al. (2012), Soong et al. (2001), Govindasamy (2001), etc., few studies have used these e-learning CSFs existed in the literature to arrive at KPIs for an educational program such as an ESP course. The review of literature shows a gap for the use of the e-learning CSFs classified by different scholars to be prioritized for an online course of study and processed to yield KPIs.
The purpose of this research was to prioritize the e-learning CSFs in the literature specifically for the online course of English for Football through Analytical Hierarchy Process (AHP) from the participants’ perspective and then to set down the KPIs. This research was an attempt to identify KPIs for the e-learning course of English for Football with SMART features proposed by Badawy et al. (2016) based on the e-learning CSFs identified and proposed by Alqahtani and Rajkhan (2020).
2. Literature Review
2.1. Critical Success Factor
The CSFs defined by Freud (1988) as cited in Caione et al. (2017) are those things that must be done if a company is to be successful. Rockart (1979) defines them as those few crucial areas where the company has to perfectly work to succeed in business. According to Selim (2007), the CSFs are grouped into four categories of instructor, student, university support, and IT that need to be rich, reliable, and capable of providing the course with necessary tools comprised of network bandwidth, network security, and accessibility, internet availability, audio, and video plug-ins, courseware applications, instructional multimedia services, videoconferencing, courseware management systems, and user interface. Based on Prougestaporn et al. (2015), there are four key factors that determine the success of an e-learning model: human deliberation, instructional design, development of technology, and social delivery. Papp (2000), proposed a set of CSFs including intellectual property, the suitability of the course for the e-learning environment, the building of the e-learning project, and e-learning project content. Dillon and Guawardena (1995) and Leidner and Jarvenpaa (1993) as cited in Caione et al. (2016) defined three variables that were effective in the efficiency of the environment of e-learning as follows: technology, instructor’s characteristics and student’s characteristics. Another proposed CSF classifications as cited in Caione et al. (2016) was by Volery and Lord (2000) comprised of “technology (ease of access, interface design and level of interaction); instructor (attitudes towards students, instructor technical competence and classroom interaction), and previous use of technology from the student’s perspective”. The CSFs classification of Soong et al. (2001) was also cited in Caione et al. (2016) including human factors, technical competency of both instructor and student, level of collaboration, and perceived information technology infrastructure. Based on Benigno and Trentin (2002), learning materials, learning environment, characteristics of students, student to student interaction, effective support and IT were identified as CSFs.
As cited in Caione et al. (2017), e-learning CSFs include 1) Information Technology that has a significant role to deliver the course online. IT facilities, availabilities, and competencies are of crucial importance to the success of the course and students. IT tools are comprised of the bandwidth of the network, the security of the network, accessibility of the network, web 2.0 software, plug-ins of audio and video, videoconferencing, the interface of user and course management systems. 2) Human Factor based on Soong et al. (2001) includes technical competency, e-learning mindset and collaboration level between students and instructors. Moreover, the students and instructor’s computer skills are critical and affect the success of e-learning. Referring to the suggestion from Volery and Lord (2000), instructors should take on an interactive teaching style and excite student-student interaction. Also, the characteristics of the learners such as their preparation for e-learning, perception of the system and contents, their collaboration, interaction and motivation are important. 3) Instructional Design that Prougestaporn et al. (2015) describes it to maximize the appeal, effects, efficiency of teaching and learning. Cheawjindakarn et al. (2013) names objectives, contents, learning psychology and strategies included in instructional design. For a meaningful learning and implementation of e-learning, the materials, well-designed course contents and teaching and learning materials are very essential. 4) Cost-Effectiveness should be to the benefits of the students to save money and also generate benefits to the business, said Prougestaporn et al. (2015). Due to the problems regarding the costs such as the budget for investment in the course and considering sustainability in the long run, expenses shall be reduced that IT advancements and facilities make it possible. 5) Course Evaluation to make sure that the system of e-learning is successful in the achievement of course objectives.
With reference to Papp (2000), critical success factors involved in e-learning include intellectual property, the suitability of the course for e-learning environment, e-learning course structure, e-learning course content, e-learning course maintenance, e-learning platform and the measuring success of e-learning. Graf and Caines (2001) offered six criteria for robust contents to weigh the successfulness of e-learning involving the online availability of contents, course structure, images and graphics, the extent of interaction among students, the degree of interaction between students and teacher, and type and quality of evaluation and assessment.
Bhuasiri et al. (2012) surveyed, concluded and prioritized six dimensions of CSFs for e-learning from eighty-two e-learning experts in 25 developing countries including learners’ characteristics, instructors’ characteristics, institution & service quality, infrastructure and system quality, course and information quality, and extrinsic motivation.
In a study done by Alqahtani and Rajkhan (2020), ten CSFs were identified for e-learning during COVID-19 pandemic through AHP and TOPSIS techniques by participation of 69 e-learning managers in educational institutions. The identified CSFs include student characteristics, instructor characteristics, learning environment, instructional design, support, IT, technology knowledge, course, level of collaboration and knowledge management. They also found that among five learning systems of blended learning, flipped classroom, ICT supported face- to-face learning, synchronous learning and asynchronous learning, the blended learning was the most suitable learning system to practice.
2.2. Key Performance Indicator
According to Parmenter (2015), there are two groups of performance measures including Results Indicators and Performance Indicators that their measures should not be taken wrongly. As Results Indicators are reported too late to make changes to the direction and they do not specify what are needed to be done to improve the results, so they are of little use to management. But Key Performance Indicators tell how the management or organization is doing within its critical success factors to make changes if necessary to be aligned with CSFs. Results Indicators replace outcome measures that they usually observe the activities in the past but KPIs include the present and future measures.
With reference to Parmenter (2015), “KPI represent a set of measures focusing on those aspects of organizational performance that are the most critical for the current and future success of the organization” (p. 4). KPIs should be tracked down regularly and if they are not meeting the target, then the system or process needs to be modified as stated by Arif and Smiley (2004). According to Lyddon and McComb (2008), as cited in Ballard (2013), KPI measures have several components: the actual results of the indicator; the target for which the indicator is striving; and the difference between actual results and target results; and signal values or benchmark.
KPIs are to conduct measurement of performance in an organization. KPIs determine what is important and what needs to be done for them. According to Hilgarth (2011), 9 KPIs were identified for e-learning as follows: 1) effectiveness: the contribution of the program to the extent of reaching goals; 2) costs; 3) satisfaction; 4) effects of business processes; 5) cost-benefits ratio; 6) efficiency; 7) materials; 8) project progress; and 9) learning outcome.
Based on Parmenter (2015), KPIs are nonfinancial measures, measure frequently, reported to senior management by description, simple to be understood, simple for identification of corrective actions, andconcentrate on a specific activity.
The KPIs as mentioned by Badawy et al. (2016) need to be SMART. These criteria were originally proposed by Doran (1981) to write management goals and objectives that includes: specific, measurable, attainable, real and time-bound.
Types of Performance Indicators
According to Chalmers (2008) citing from Borden, & Bottrill, (1994); Carter et al, (1992); Cave et al, (1997); Richardson, (1994), there exist a general census on four types of performance indicators including Input, Process, Output, and Outcome, which Input and Output are quantitative and Process and Outcome are qualitative indicators.
As defined by Chalmers (2008), “Input indicators reflect the human, financial and physical resources involved in supporting institutional programs, activities, and services”. According to Burke (1998), “Output indicators reflect the quantity of outcomes produced, including immediate measurable results and direct consequences of activities implemented to produce such results”. So as Burke et al. (2002) says, these two indicators that are quantitative do not demonstrate the quality of education but rather its outcomes’ quantities.
Based on the definitions by Burke (1998) and Warglien & Savoia (2006) cited in Chalmers (2008), the measure of outcome concentrate on the quality of the program, service, and activity. This qualitative indicator does not generate numerical data as do indicators of output performance. With reference to the definitions in the literature such as Romainville (1999) and Bruwer (1998), the qualitative indicators as outcome are more insightful, accurate, and meaningful in measuring the teaching and learning quality and satisfaction. Based on Burke (1998), those means that are used to deliver the educational program, activities, or services are Process Indicators. These indicators look at the operation of the system in its specific context. They collect information regarding the policies and practices concerning teaching and learning, management of performance, staff development, quality of curriculum, and assessment of learners, quality of facilities, and services and technology. Process indicators provide a comprehensive perspective for the identification of the strengths and drawbacks or weaknesses for making improvements in further initiatives.
Research Framework
The top-down CSF/KPI framework proposed by Jahangirian et al. (2017) was used as the road map in this study (Figure 1).
The purpose of this framework is going from uncertain goals and objectives toward CSFs and finally arriving at concrete and measurable indicators, KPIs. As stated by Jahangirian et al. (2017), “CSFs represent strategic focus areas and KPIs represent operational performances”. The researchers intended to set objectives for an online course of English for Football, then find the best e-learning critical success factors in the literature to establish SMART key performance indicators for the course to be measurable by defined and determined performance metrics.
3. Methodology
This research was a survey study that was carried out through administration of an open-ended questionnaire and a ratio scale. According to Griffee (2012, p. 52),
![]()
Figure 1. Top-down CSF/KPI hierarchical framework.
“A survey design uses various data collection procedures to enable the researcher to investigate a construct by asking questions of either fact (descriptive) or opinion (explanatory) from a sample of a population for the purpose of generalizing to the population”.
In order to determine the KPIs, which play a key role in performance and implementation of the e-learning course of English for Football as an ESP course, the below research procedures were employed as detailed in the following sections.
3.1. Participants
The participants were selected through a non-probability purposeful sampling method because the researchers sought licensed football coaching instructors who have content knowledge and teaching experience in the field to share their emic perspectives. A total of 104 football coaching instructors who were registered with Iran Football Federation were invited to participate in the survey because their responses and perspective carried content knowledge, teaching experience and expertise in football. The list of the participants was requested from the education department of Iran Football Federation. The participants were from all over Iran and all possessed coaching certificates awarded by the Asian Football Confederation and qualified to teach football coaching courses under the auspices of Iran Football Federation. All the participants were over 30 years old including males and females. Since the participants’ opinion, their emic perspective and teaching experience in the context of football were important to be drawn on the survey of e-learning critical success factors (CSF), their English language proficiency was not under the study to be assessed or tested. The survey was opinion-based and their football coaching instruction experience in context of football was important as insiders to share their opinions through writing statements of success for each CSF and ranking them in priorities based on their views, experience and expectations.
3.2. Instruments
With reference to Parmenter (2010) mentioned in Jahangirian et al. (2017), the KPIs should be aligned with the objectives, and a secure approach to reach this alignment is through the CSFs.
The CSF dimensions proposed by Alqahtani & Rajkhan (2020) were adopted in this study to make the CSF questionnaire on Google Form as an online survey to collect the participants’ Statements of Success for each CSF dimension in the context of online course of English for Football. The CSFs proposed by Alqahtani and Rajkhan (2020) were identified and drawn from previous studies in the literature that were found more comprehensive for this study and inclusive of similar CSFs in comparison with other identified e-learning CSFs and classifications in the literature such as Bhuasiri et al. (2012), Selim (2007), Frimpon (2012), Sun et al. (2008), Malik (2010), Prougestaporn et al. (2015) etc.
Out of the multi-criteria decision-making (MCDM) methods summarized in Chaibate et al. (2021), the AHP ratio scale of Saaty (1980) was used because of its easiness, flexibility, and ability to simplify multi-criteria and multi-alternative problem into a hierarchical structure. This ratio scale is from 1 to 9 with 2, 4, 6 and 8 values in between and includes Extremely Preferred (9), Very Strongly Preferred (7), Strongly Preferred (5), Moderately Preferred (3) and Equally Preferred (1). Based on the AHP ratio scale, an AHP questionnaire was designed in Microsoft Excel with the CSFs. Appendix 1 is the AHP questionnaire of this study. The questionnaire compares the importance of the elements in relation to the objective to demonstrate which element of each pair is more significant, A or B, and how much more on a scale of 1 to 9 (Appendix 1).
3.3. Procedure
All the participants were personally contacted online through WhatsApp and invited to receive the link of the CSF questionnaire prepared on Google Form. The first and second authors, who are employed in Iran Football Federation, prepared a text as an invitation and sent to the participants via WhatsApp and informed them of the purpose of the questionnaire for the research and organizing an online course of English for Football.
The CSF questionnaire consisted of 10 essay-type questions, CSFs, translated into the participants’ L1 with a short explanation for clarification of the meaning and purpose of each question (CSF). In the rubrics of the form, the respondents were requested to write one or more recommendations or their expectations for each question (CSF) that could lead to the success of the e-learning course of English for Football. The answers were the Statements of Success.
After almost a week waiting for the responses, the received responses, which were in participants’ L1 on the Google Form were exported and translated into English and then reviewed by the researchers. During codifying the responses, some answers were found irrelative because of the respondent’s misunderstanding of the question and some others were rejected by the respondent to be answered because of lack of idea. As a result, those responses, which were unrelated or not meaningful in response to each question (CSF) were removed. Then, among the responses for each CSF, some words were found repeated by a number of respondents. They were recognized as Common Features to be reworded and encapsulated in one Statement of Success since they had the same meaning. Finally, the most frequent encapsulated responses in each CSF were picked as the selected Statements of Success for each CSF.
In the second round of the survey, in order to avoid any misunderstanding of the AHP ratio scale to answer, 38 participants who had submitted their responses for the CSF questionnaire were personally contacted on the phone by the researchers to rate the 10 CSFs on the AHP ratio scale from 1 to 9 based on their importance and priority in their view. The researchers explained the AHP scale and the pairwise comparison to the participants and guided the respondents to fully understand the questions and answer properly with full comprehension on the scale.
To answer the AHP questionnaire, the participants had to choose one option in a pairwise comparison of two CSFs. Choosing the more important option (CSF) in comparison to the other CSF, then rating the significance and priority of the chosen option (CSF) over the other on a scale from 1 to 9.
After collecting the Statements of Success, grouping and encapsulating them based on their common features and selecting based on their frequency among the responses, and ranking and prioritizing the CSFs through the AHP, the researchers wrote the key performance indicators for each critical success factor.
4. Data Analysis
Having launched the CSF questionnaire, 38 responses, Statements of Success, were received and codified by the researchers. After codifying the responses and encapsulating in each CSF based on Common Features and removal of unacceptable responses, 24 Statements of Success were selected and paraphrased for a better wording to be placed in each CSF. Table 1 shows the Statements of Success, which were selected for the e-learning course of English for football.
![]()
Table 1. Statement of success of the CSFs.
Table 2 shows the AHP ratio scale of the AHP questionnaire, designed by Saaty (1980), given to the respondents to rate the CSFs based on their importance and priority in their views. The AHP priority calculator demonstrated through pairwise comparison that which criterion is more important and how much more on a scale from 1 to 9.
Pairwise Comparison Values
In this study, totally, 45 comparisons were done on 10 critical success factors and the consistency ratio (CR) of 7.7% was reached that is acceptable. The results showed 92.9% consensus. Table 3 and Table 4 show the resulting weights and pairwise comparison matrix of the CSFs. The priority vector is obtained from normalized Eigen vector of the matrix (Figure 2).
![]()
Table 2. Ratio scale of analytical hierarchy process.
Note: 2, 4, 6, 8 can be used to express intermediate values.
![]()
Table 4. Pairwise comparison matrix.
![]()
Figure 2. Axis major gridlines of the AHP results.
5. Results and Discussion
The data could be collected from 38 participants who were certified football coaching instructors. The response rate for the questionnaire given to 104 invited participants was 36.53% and it was 100% for the AHP questionnaire given to 38 respondents of the first survey. Table 3 shows that Course with 30.9% and Instructor’s Characteristics with 21.0% and Learner’s Characteristics with 14.2% were rated the top 3 important critical success factors (CSFs) for the e-learning course of English for Football. Respectively, Learning Environment, Instructional Design, Support, Level of Collaboration, IT, Knowledge Management, and Technology Knowledge were chosen as the next critical success factors in the expert panelists’ view and opinion.
After collecting the Statements of Success, grouping and encapsulating them based on their common features and selecting based on their frequency among the responses, and ranking and prioritizing the CSFs through the AHP, the researchers who have higher education in English language teaching wrote the key performance indicators for each critical success factor with consideration of SMART characteristics. In the end, the KPIs and their metrics were proposed for the e-learning course of English for Football as shown in Table 5.
![]()
![]()
Table 5. Identified KPIs for an e-learning course of english for football.
The prioritization of the CSFs in the context of this study was different from the results of Alqahtani & Rajkhan (2020) because the responses in that study were from a managerial perspective, but in this study they were from the participants’ perspective. The results of the study by Alqahtani & Rajkhan (2020) demonstrated knowledge management and support as the most influential factors, while the same factors were ranked the ninth and sixth critical success factors in this study. While Technology Knowledge was ranked the last and IT the eighth in this study, the results of Bhuasiri et al. (2012) and Siritongthaworn et al. (2006) revealed that computer skills, technical background, training programs and computer literacy were the key factors to implement e-learning in developing countries. It can be drawn that since English for Football is an ESP course, the contents and teacher’s content knowledge were recognized and ranked as more significant factors than the other CSFs.
6. Conclusion
Identifying and developing a set of standard KPIs to guide and measure the performance of an educational program or a class can lead to a higher quality, more fruitful and efficient class with successful learners. To do so, we prioritized and ranked the e-learning CSFs in the literature from an emic perspective, and then identified KPIs and set metrics for them. This research proved that each online course can have its own specific priority of e-learning critical success factors and accordingly, exclusive key performance indicators.
The e-learning CSFs proposed in the literature have many names and groupings but almost all of them have many similarities. The results of the present study indicate that the hierarchy of the e-learning CSFs is dependent on the program and the respondents’ role in the program. Interestingly, based on the results of this study, IT or technology knowledge even in a course to be delivered online can be prioritized lower than the factors including the course, instructor’s characteristics and learners’ characteristics. Since the course of English for Football is an ESP, regardless of the mode of the class, the factors associated with e-learning such as IT or technology knowledge were not recognized as more important than contents and materials of the course, contents knowledge and experience of the teacher and learners’ attitude and motivation. This study tried to apply CSF and KPI concepts into organizing a course of study in order to demonstrate that CSFs and KPIs can be significant factors to be taken into account in educational programs and online courses in specific contexts.
Appendix 1
AHP Questionnaire