Towards Automated Assessment of Learning Management Systems in Higher Education Institutions in Zambia ()
1. Introduction
1.1. Background of the Study
According to the Higher Education Authority (HEA), Zambian higher education institutions were unprepared for the changes that occurred during the pandemic [1] , and most learning institutions were unable to conduct teaching and learning on learning platforms. HEA in a report attributed the severe impact of Covid on teaching and learning to inadequate technological competencies and technological tools by HEIs. Additionally, the abrupt onset of the lockdown did not accord HEIs any chance to prepare systems for teaching and learning. HEIs had to fall back to their existing systems to salvage the loss of learning hours. HEIs who were already offering distance learning at the very least managed to conduct lectures on Learning Management Systems [2] . According to a study conducted by the Zambia Information and Communications Technology Authority (ZICTA), mobile services were the most prominent activities of households during the pandemic. While this was true for sector-specific services such as financial services accessed via mobile services, the same could not be said about online learning services [3] . In this report, the authority revealed that, despite the world coming to a stop in movements, sectors such as finance were going about their business with online services. The same could not be said for the education sector as learning systems were not widely used across the country. Some HEIs were able to pre-record content and share on their student record management systems. However, students had challenges accessing this content due to a lack of e-learning knowledge. Some HEIs could not conduct end-semester examinations due to lack of assessment platforms; as a result, their academic calendars were distorted. This is just a snippet of various struggles HEIs faced. There were many challenges faced by Zambian HEIs implementing and administering learning management systems.
Platforms and technologies such as student management systems, learning management systems and finance systems bridge the gap between the growing demand for higher education and the limited number of resources available, such as lecturers and classroom space [4] . Although educational technology is critical for streamlining educational delivery in higher learning institutions, many studies have demonstrated that there are numerous obstacles to its deployment in underdeveloped countries, including Tanzania and Zambia. Some of the challenges mentioned include a lack of ICT infrastructure, a lack of technical and managerial support and lack of education technology knowledge among facilitators [5] .
1.2. Assessment of Educational Technology in Zambian HEIs
The HEA in Zambia, before the Covid pandemic, had developed audit tools for assessing the capacity of HEIs to deliver quality-learning services to the public. These audit tools included the general infrastructure of a HEI, financial structure, program content with prescribed course materials and human capital required to run a successful HEI. However, when it came down to educational technology assessment, HEA focuses on the hardware infrastructure and the availability of a student record management system. HEIs and regulatory authorities in Zambia do not have a model for assessing LMSs away from other integrated systems. This has resulted in HEIs taking LMSs as an option, developing them to their own standards and at their own priority. Consequently, quality of teaching and learning on LMSs is compromised as evidenced by challenges HEIs encountered while HEIs tried to conduct learning activities on LMSs during the Covid 19 pandemic. The introduction of a model to achieve automated assessment of learning management systems will be a step towards improving education technology in HEIs in Zambia. When HEIs adopt a model for assessing LMSs, their systems will be designed based on set standards that consider the quality of teaching and learning, thereby improving learners learning outcomes in Zambian HEIs.
2. Literature Review
This study reviewed various literature related to this work, one of the key literature reviewed was the technology acceptance model which describe factors affecting the acceptance of technology among users. Despite this topic being exhausted in other parts of the world, it is still a relatively new concept when applied to educational technology in the Zambian setup.
2.1. Educational Technology
Education Technology refers to the application of scientific advancement in the field of science and technology to the process of teaching and learning [6] . The evolution of education post-21st century is inseparable from the advancement in modern technology. The application of technological advancements is meant to revolutionize education systems. For instance, a model by [7] proves educational technology can go as far as assisting in the award of marks in classroom assessments. Educational technology is a concept high learning institutions must learn to familiarize with in order to develop. As the world shapes itself into a global village, education without borders will keep skyrocketing. This calls for a deeper dive into educational technology to understand how its different aspects can help build learning environments for learners and instructors.
There are numerous processes in education where technology can be applied. Figure 1 above shows a few. This study concentrated on the application of technology to the process of teaching and learning in systems called; learning management systems (LMS). Nishtar defined an LMS as a “software application that
automates the administration, tracking, and reporting of classroom and online training events, enabling detailed analysis of the effectiveness of your training investment” online [8] . Higher learning institutions must embrace LMSs to realize their investment in online learning. As [9] projected, the process of teaching and learning when subjected to appropriate technologies will record increased effectiveness of the whole education system. With its focus on learning processes, this study adopted some concepts of the Unified Theory of Acceptance and Use of Technology (UTAUT) model proposed by Venkatesh. This theory stipulates that technology acceptance is affected by perceived usefulness, ease of use and some external factors. To tailor our theory to education, we also included some concepts from the community of inquiry model to incorporate the creation of a conducive environment for learners to thrive in. By integrating the concepts of technological features and environmental factors, we can enhance our understanding of their collective impact on teaching and learning.
2.2. E-Learning in Higher Education Institutions in the Region
A study was done by [10] with the aim of describing the levels at which e-learning platforms are utilized by the tertiary learning institutions in Zambia. From this study, it was discovered that the assimilation of learning management systems differ from one institution to another. Furthermore, assimilation of LMSs in Zambian HEIs is hindered by a lot factors among which are lack of hardware equipment to run LMS on as well as the functionalities of the LMS [11] . From the universities sampled, utilization of LMS varied from one institution to another because HEIs management regarded the LMS with different priority levels. Another study was done to assess the factors affecting the adoption of e-learning platforms in higher learning institutions in Tanzania. Adam [12] discovered that the implementation of e-learning methods and technology enhances access to education; reaches the majority of learners and expand learners’ capacity to access education from a variety of locations rather than being bound to a physical classroom [13] . Semlambo et al. also attest to the very same factors to have affected the implementation and adoption of LMS in Tanzania. However, Semlambo goes further to suggest that the best way HEIs can implement LMSs is by using the multi-factored adoption model, a concept this study adopted to develop a model for assessing LMSs in HEIs.
2.3. Unified Theory of Acceptance and Use of Technology
Researchers have proposed and tested several competing models such as the technology acceptance model (TAM) to explain and predict user acceptance and use of Information Technology (IT). Venkatesh, Morris, Davis, and Davis [14] synthesized various models into the unified theory of acceptance and use of technology (UTAUT). UTAUT identifies four key constructs; the performance expectancy, effort expectancy, social influence and facilitating conditions and four moderators that is, age, gender, experience and voluntariness are related to predicting behavioral intention to use a technology and actual technology used primarily in organizational contexts [15] . In UTAUT, performance expectancy, effort expectancy, and social influence were theorized and found to influence behavioral intention to use a technology, while behavioral intention and facilitating conditions determine technology use [16] . UTAUT’s five constructs also embraces other models constructs such as perceived usefulness, extrinsic motivation, job-fit, relative advantage and outcome expectations form the performance expectancy in the UTAUT model while effort expectancy captures the notions of perceived ease of use and complexity [17] .
2.4. Community of Inquiry Model
Another educational technology model that this research builds on is the Community of Inquiry (CoI) framework. CoI is a theoretical model that provides a framework for understanding and assessing the quality of online learning environments [18] . It was originally developed by Garrison, Anderson and Archer in 2000. It has since been widely used in the field of online education and distance learning. According to [19] the CoI framework is based on the premise that successful online learning involves three interdependent elements or “presences” that create a meaningful learning experience. Cognitive presence refers to the extent to which learners are able to construct and confirm meaning through sustained reflection and discourse. Social presence refers to the degree of perceived connectedness and interaction among participants in an online learning environment. Teaching presence refers to the role of the instructor or facilitator in designing, guiding, and facilitating the online learning experience. This study adopts concepts from UTAUT and CoI to create as yardsticks for understanding and developing educational technology; learning management systems.
3. Methodology
A mixed methods research design was adopted for this study. Probability sampling method was used and the target population was users of LMSs in Zambian HEIs. The sample size involved students, instructors and administrators in five (5) selected HEIs in Lusaka. Random Sampling was used to select students and instructors, purposive Sampling used to select officers. Instructors were selected based on their experience in designing and delivering learning programs online. ICT and Quality Assurance Officers were selected based on their expertise in implementing LMSs. The sample size was arrived at using the statistical formula by Cochran; with 5% error and 95% confidence level and population proportion of 0.5.
A total of 366 questionnaires were administered with a response rate of 87%. 64% of responses were from Students, 24% from Instructors, 7% from Quality Officers and 5% from IT Officers. The resultant data obtained was analyzed using Statistical Package for Social Science (SPSS) and Excel. Using the primary data obtained from surveys, there was overwhelming evidence of factors affecting the development and assessment of learning management systems in Zambia’s higher education institutions.
Data Collection Methods
Primary data collection involved both surveys and interviews to gather quantitative and qualitative data on the challenges faced by HEIs when implementing LMSs. Data on models used to implement LMSs was collected. Respondents were also asked whether these LMSs are assessed for quality of service, compliance with standards and usability. The survey included questions on system design and features, usability and user support, security and privacy, effectiveness and impact, and cost-effectiveness and sustainability. The interviews were conducted with a subset of the sample to gather more in-depth insights into their experiences and perspectives on the learning management system.
Surveys were distributed to students, instructors, and administrators to gather quantitative data on system usage, satisfaction, and perceived effectiveness. Interviews were conducted with key stakeholders, such as IT staff, instructional designers, and system administrators, to gather qualitative data on system design, implementation, and impact. The questionnaires were designed in four parts based on the respondents targeted. There were questionnaires focused on systems features and user perception administered to students, another based on course design and administration was administered to instructors, another on the technical areas of implementation administered to IT officers and last one to do overall performance review of LMSs administered to quality Assurance officers. Quantitative data was analyzed using descriptive statistics to summarize the responses to the survey questions. The qualitative data from the interviews was analyzed using thematic analysis to identify patterns and themes in the responses. The data from the surveys and interviews was triangulated to provide a more meaningful insight into the construct. Parameters for assessment were derived from respondents and a model was developed from the parameters.
4. Findings and Discussions
With the key objectives of identifying factors affecting implementation and assessment of learning management systems, designing an assessment model and a tool to test this model, this article discusses key areas that are critical to achieving these objectives in this chapter.
4.1. Factors Affecting the Implementation of Learning Management Systems
We discovered that technology acceptance among students accounts for 35.6% of challenges institutions face while implementing LMSs, followed by instructor technology acceptance which accounts for 24.1% while the cost of implementing 20.2%, regulatory requirements with 15.1%, and lastly expertise needed for implementing LMSs 5%.
Technology acceptance among students and instructors remains one determinant factor to the success of an LMS as discovered in this research. We investigated technology acceptance to understand major factors that affect technology acceptance among the stakeholders of LMSs. Table 1 shows factors affecting technology acceptance among HEI system users
This study discovered that the pedagogical capability of the system accounts for a significant 40.1% of factors affecting technology acceptance. Learners will only utilize the LMS if it meets their learning needs. Usefulness of the system accounts for 31.6% this entails that the users only utilize the system if they are satisfied with the features and the quality of learning otherwise it is not useful to them. User-friendliness accounts for 13.3%, followed by creative design with 10% and communication with 5%. Usefulness and pedagogical capability remain the most prevalent factors that affect technology acceptance which is a major factor that affects implementation of LMSs in HEIs. The study further discovered that 45% of respondents were dissatisfied with the services of the learning management system and 40% were somewhat dissatisfied. Only 5% indicated they were very satisfied with their system and 10% were somewhat satisfied. This means users’ adoption of LMSs is likely to be low. If learners are dissatisfied with their systems, there is little chance they would want to use it unless when circumstances force them like what happened during the pandemic.
When asked how user-friendly LMSs are, 36% did not find LMSs user-friendly, 19% found LMS somewhat not user-friendly, 21% found LMSs user-friendly and 24% found LMSs very user-friendly. This entails that 55% of respondents did not find their LMSs user-friendly.
According to a focused group discussion with Quality Assurance Officers in these institutions (as shown in Table 2), 40% revealed their LMS was able to provide both summative and formative assessment tools. 15% agreed the LMS has prescribed content while 10% revealed the LMS class schedules meet the mandatory prescribed contact hours. Surprisingly, 30% of the QA officers revealed there are no monitoring tools on LMSs for assessing teaching and learning. This is a source of concern for HEIs as the quality of the LMS can only be assessed if the systems come with monitoring tools
![]()
Table 1. Factors affecting technology acceptance.
![]()
Table 2. Compliance with regulatory standards.
The study also considered the pedagogical capability of the LMSs to gage their ability to transmit intended course content to the learners. The study also considered availability of relevant facilities for learners to access course content as well as discuss content with instructors and fellow learners and the ability for instructors to deliver relevant content as prescribed by the regulatory bodies. Table 3 shows responses from various respondents across the participating universities. About 28% of respondents revealed that LMSs had the capacity to deliver content, 31.9% felt the capacity was there but could be improved, 28.1% were neutral and only 11.1% believed their LMSs were not fit to be used to deliver content to learners. Based on this evidence, this study inferred that HEIs have developed systems that are capable of disseminating content to learners.
4.2. Discussions
This study investigated factors influencing the implementation and assessment of Learning Management Systems (LMSs) in Higher Education Institutions (HEIs) in Zambia’s Lusaka district. It identifies major factors affecting LMS implementation, including User Technology Acceptance, Cost of Implementation, Regulatory Requirements, and Expertise needed. Technology acceptance issues constitute a significant portion (35.6%) of these factors, indicating challenges in user adoption. When primary users such as students and staff find LMSs not useful, adoption becomes difficult, affecting the ability of HEIs in Lusaka to administer online lessons during the Covid pandemic.
Another crucial factor identified is the user-friendliness of learning management systems (LMSs). A significant 55% of respondents found their LMSs not user-friendly, indicating difficulties in navigating system functions. This lack of user-friendliness contributed to the challenges faced by HEIs in administering online lessons during the pandemic. When students struggle to engage with pedagogical activities on an unfamiliar system, effective teaching and learning are hindered. Additionally, 40.1% of instructors reported being unable to conduct lessons online, highlighting issues with LMS implementation. Given that learners and instructors are primary users of LMSs, it is essential to ensure that system functionalities are implemented in a manner acceptable and usable to them. Therefore, this study suggests the development of a model to validate system features for compliance and technology acceptance before deployment to users.
4.3. Categorization of Assessment Areas of LMSs
The assessment areas of LMSs are divided into four categories of assessment. The rationale behind the categorization of the key assessment areas was based on various impacts of the LMSs on the actual users. The responses to the survey questions by our respondents provided an insight into what areas needs to be focused on to effectively assess an LMS for its performance, which has direct impact on the user. For instance, students’ were interested in the features of the systems that could allow them to learn and download content uploaded by their instructors. Instructors were interested in the features that could allow them to disseminate content for their students to access, features that could enable them to teach and interact with their students. The platform offers various features, including integrated audio and video conferencing tools, discussion group forums, and assessment portals. IT Officers were interested in the technology acceptance among system users. They revealed that their biggest challenge while implementing LMSs is the user technology acceptance. This gave an insight into the area of assessment focus. Quality assurance officers gave an insight towards the capability of the systems to help them comply with regulatory standards from regulatory authorities as well as the quality of service the system presents such as its availability to users, responsiveness to user needs and its reliability.
Table 4 below highlights some relevant regulatory standards across different regulatory authorities.
Table 5 shows some of the important regulatory requirements in assessment of HEIs. There are assessment parameters such as summative and formative assessment tools which were incorporate into the developed model. However, these standards had little done with the assessment of LMSs. Despite most of them pointing to technology, none of the above standards is focused on assessing LMSs. Hence, there is the need for this study to propose the model in Figure 2 below.
Firstly, the functionalities of the learning management systems are evaluated to ensure the assessment includes how the systems are designed and administered. Secondly, the Quality-of-Service category assesses the system for availability, reliability, responsiveness, security, interoperability, and susceptibility to technical challenges. Thirdly, compulsory standard features required for regulatory compliance are considered, including formative and summative assessment tools, student attendance monitoring, course scheduling, quality assurance tools,
![]()
Table 4. Regulatory standards relevant to e-learning.
![]()
Table 5. Assessment parameters for LMSs.
and content dissemination capabilities. Lastly, parameters for fostering technology acceptance of the learning management systems are included, focusing on user-friendliness and user satisfaction features. Table 5 provides a summary of assessment parameters feeding into the proposed assessment model in Figure 2, incorporating major factors affecting LMS Implementation and assessment.
4.4. Proposed Model
Figure 2 below illustrates the logical view of the proposed model, depicting how these parameters are integrated. The assessment model examines whether the learning management system meets required features, achieves quality of service, complies with regulatory standards, and is designed for technology acceptance
among users. We have systematically synthesized a model from challenges we discovered to be affecting implementation of LMSs in this study, in order for us to operationalize this model, there is need for a tool to test this model. To achieve this, the parameters discussed in Table 5 were used to identify evaluation criteria.
4.5. Evaluation Criteria for Assessment Parameters
In Figure 2 above, we have questions along each category of parameters. The design of the model allows the evaluator to interrogate each parameter in each category using relevant questions whose answer determines the strength or weakness in that area of assessment. For example, when an evaluator is given expanded audit tools for system features, they check the system features for the availability of monitoring tools, pedagogical tools, communication tools and mobile learning tools on that particular LMS by answering specific questions tied to each parameter. Each question in each category is assigned a mark. Depending on the answer selected in the evaluation question, the mark is awarded or deducted. A mark is awarded when the answer is positive and deducted when the answer is negative. At the end of the evaluation, a total mark is calculated based on the answers in each category of parameters. The institution can set a threshold of pass mark decide if an LMS is good enough to be adopted by users. If the total marks obtained are less than the set threshold, the key areas can be rechecked for weak links, changes can be made to improve the sections of the system until an improved outcome that is acceptable and above the set threshold is met.
Tables 6-9 show the assessment criteria for each category of parameter and for each parameter.
4.6. Automated Assessment Tool
To automate the assessment process in each category, a system was developed. Each category of parameters had a list of questions entered into a database. A value was assigned to each answer. Based on the answered selected by the system assessor, the system recorded the value and incremented by the next value based on the answers selected. A threshold was set such that when assessor submits the assessment, the system is able to calculate the total score and gives feedback to the assessor based on the score obtained. Figure 3 shows a sample step into assessing compliance while Figure 4 shows the system feedback given to an assessor after they have completed assessing all key areas.
![]()
Table 6. Assessing parameters for system features.
![]()
Table 7. Assessing parameters for quality of service.
![]()
Table 8. Assessing parameters for compliance with regulatory standards.
![]()
Table 9. Assessing parameters for technology acceptance.
![]()
Figure 3. Sample assessment on automated tool.
![]()
Figure 4. System response to an assessment.
5. Conclusions and Recommendations
The main purpose of the study was to identify factors affecting the implementation of LMSs in Zambia’s HEIs and propose a model for assessing LMSs in Zambian HEIs. The study discovered overwhelming factors. These factors include: 1) User Technology Acceptance, 2) Cost of Implementation, 3) Unclear Regulatory Requirements and 4) Expertise needed to develop and administer LMSs. This discovery cements the reports from HEA and ZICTA that HEIs in Zambia are struggling to implement LMSs. We discovered that these challenges are arising because HEIs do not have a model for assessing these LMSs before they are deployed to users. HEIs need a proper model for assessing LMSs in order for online learning systems to be up to a set standard.
As the world becomes a global village united by the digitization of processes including learning, Zambian HEIs have the responsibility of ensuring they keep the negative impact of these factors on teaching and learning to the minimum. The onus is on Zambian HEIs and regulatory bodies to make an informed decision to address these challenges. When technology acceptance issues are addressed and learners are receptive to LMSs, HEIs will see an increase in the use of LMSs. This alone will safeguard HEIs from the impact of unforeseen circumstances that may lead them into digital modes of service delivery. If these challenges continue to be left unchecked, HEIs (higher education institutions) risk encountering similar disruptions as those faced during the Covid lockdown if such circumstances recur.
We recommend that Zambian regulatory bodies and HEIs should adopt this model focused on assessing learning management systems as a standalone system as opposed to absorbing LMS’s assessment in the general IT infrastructure assessment, a process that is currently being used by Zambia Higher Education Authority to assess compliance. Further, regulatory authorities should develop prescribed guidelines for system features for LMSs, predefined guidelines for assessing quality of service of LMSs and predefined guidelines for LMS compliance to standards. HEA should incorporate the assessment parameters of this model in their audit tools while HEIs invest in customizing the developed assessment tool for assessing LMSs.
There is need for an independent study into technology acceptance among users of LMSs in Zambian HEIs. This factor should be further explored to understand the correlation (if any) between technology acceptance and overall implementation of LMSs in HEIs. Future research could explore additions to this model, including the integration of learning management systems for general education, to provide a sustainable alternative to traditional classroom instruction.
Acknowledgements
We would like to acknowledge Dr. Jackson Phiri and all academic members of staff in the University of Zambian, Department of Computer Science for providing guidance during this study. We also acknowledge staff at HEA who took their time to assist with this research as well as University leadership who opened doors for us to conduct this study. Special acknowledgement to Chreso University where the system assessment was conducted to test our model.