Effects of Health Record Computerization in the Context of the COVID-19 Pandemic in Burundi’ Hospitals: Propensity Score Matching ()
1. Introduction
The new technology of information has a progressively significant and prominent role in the healthcare sector. A great number of software are currently used to digitalize hospital information systems in order to optimize the quality of care and services offered to patients [1]. In this context, Information and Communication Technologies seem to be a vector favoring the coordination of health professionals, the quality of services offered by a good organization of care processes and close cooperation to allow better management of patients [2].
The research has proved that the use of Electronic Health Records (EHR) improves the quality of care and services, as the electronic management and storage of the patient’s data facilitates continuity of care, correct diagnoses and good decisions regarding prescriptions guided by the decision support tools (International Classification of Diseases 10/11, Systematized Nomenclature of Medicine Clinical Terms) [3] [4]. In addition to clinical effectiveness, the hospital must respond to the objectives of efficiency, equity, safety, patient satisfaction. The studies proved a positive correlation between quality of care and hospital performance indicators [5].
With the emergence of New Information and Communication Technology, most sub-Saharan African countries have implemented EHR. However, the expected effects are often not observed due to multiple factors, the main ones being related to the availability of crucial resources (infrastructure, equipment, human resources); the lack of strategic planning before the project’s implementation; the absence of a needs assessment before the beginning of the project [6] [7] [8]. In addition, the occurrence of potential shocks such as epidemics negatively influences the expected effects of implementing health interventions. Many studies have proved that the COVID-19 pandemic strongly affected the health care system. On one hand, containment and the risk of exposure to the coronavirus caused patients to modify their use of care. On other hand, the incidence of COVID-19 disease among providers has also affected the supply of care and services in hospitals [9] [10] [11] [12] [13].
In Burundi, in order to strengthen the Hospital Information System (HIS) by using New Information and Communication Technology, many activities have been carried out, including the implementation of District Health Information Software version 2 (DHIS 2), which is a routine health data management tool. This software was only limited to the management of aggregated data, should be completed by other software that allows to manage the individual patient data and networking within a hospital. That is the reason why the health record digitalization project using Open Clinic GA software was initiated in Burundi hospitals. The period between 2015 and 2017, eleven hospitals including five district hospitals were supported by the Belgian Cooperation Agency Project [14] [15] [16]. All countries in the world as well as Burundi, have been affected by the COVID-19 pandemic. The results of national studies on the impact of the COVID-19 pandemic and the services continuity in health facilities proved the decreased of the service utilization indicators [9] [17].
In light of this, the aim of this study was to evaluate the effects of health record computerization in the context of the COVID-19 pandemic in Burundi’ hospitals. The findings of the study add to the current body of knowledge and enhance the understanding of 1) the EHR effects to increase the use of district hospital services, 2) the EHR effects to increase the overall quality of district hospital services, 3) the EHR effects to increase the amounts earned by the hospital under the performance-based financing strategy and 4) the COVID-19 pandemic impact on the effects of health records digitalization project.
2. Materials and Methods
2.1. Setting
The study was conducted in ten hospitals in Burundi. Five hospitals had adopted the EHR system based on the OpenClinic GA software, and five others hospital that had not.
2.2. Design
We conducted a quasi-experimental study based on difference in difference method to assess the EHR effects on district hospital indicators performances.
2.3. Sample
2.3.1. Choice of Experimental Group Hospitals
Five hospitals were selected as the experimental group. These were all district hospitals whose health records were digitized with OpenClinic GA software between 2015 and 2017.
2.3.2. Choice of Group Control Hospitals
The district hospitals in the control group were selected by the propensity score-based matching method. The propensity score refers to the probability of each hospital in the control group having the similar observable characteristics at least one hospital in the experimental group. The five district hospitals control group were selected among the twenty-six that were not digitized until 2020.
2.3.3. Calculation of the Propensity Score
The propensity score was calculated by logistic regression using the observable characteristics [18]. Variables that relate to the use of outpatient consultation and hospitalization services, surgery and maternal health were identified. Five variables were selected: number of new outpatient consultation, number of dystocic deliveries, number of caesarean sections, number of surgeries performed and number of hospitalizations. The two variables that were excluded to obtain equilibrium in the final model were hospital status and eutocic deliveries. The data used to calculate the propensity score were extracted from the National Health Information System. The data quality was verified under the Performance-Based Financing (PBF) approach from 2014 corresponding to the period before the EHR implementation in all hospitals studied.
2.3.4. Matching
Two matching methods were used: nearest neighbor matching and caliper matching. For the first “nearest neighbor” matching method, each district hospital with the HER adopted was matched to a district hospital with the paper health record whose propensity score value was more or less close. The matching without replacement and with replacement were used to select district hospitals from the total of twenty-six that were not digitized until 2020. For the second matching method “caliper”, the group with no HER adopted was selected based on the closest propensity score, subject to a certain maximum distance (0.25). The matching was done with replacement, one hospital with EHR adopted was matched to the two control hospitals [19].
2.3.5. Quality of Matching and Bias Reduction Test
Means comparison tests were performed between the experimental and control district hospital groups to check the matching quality. The verification of bias percent was obtained by calculating the difference between the calculated bias before and after matching, the data used were from 2014. By comparing percent bias before and after matching, the first method of nearest neighbor matching without replacement had less bias (9.6%), whereas nearest neighbor matching with replacement had a bias of 19.0% and the caliper matching method had a bias of 18.6% bias (Table 1).
![]()
Table 1. Hospital characteristics before and after matching (2014).
Source: Open RBF-2014.
2.4. Intervention
The OpenClinic GA software chosen to be used in this project is Open Source. It was developed by a Belgium company MXS SA since 2006. The software code source was introduced in the public domain in December 2008. The main modules of the OpenClinic GA software concern the patient’s administrative file, the patient’s financial file. It’s also used for the patient’s medical file, the health insurance, the funds, the pharmacy (including the store), the laboratory, the X-ray, the health statistics in the form of periodic reports, the human resources, the clinical thesaurus with coding assistance validated for ICD-10/11, SNOMED CT and the multimedia supports (images, video, audio). Currently, the number of components developed differs from hospital to hospital. From 2015 to 2017, five district hospitals received support from the Institutional Support Program to the Public Health Sector in this electronic health computerization process [14] [15] [16].
2.5. Outcome (Hospital Performance Indicators)
Several strategies were adopted to improve the health system performance. One of them is the Performance-Based Financing (PBF) strategy. It was scaled up nationwide in 2010 and contributed to increase the use and quality of health services as well as the governance and management of health care facilities [20] [21] [22].
In this study, the effect of the health records digitalization project was evaluated using the hospital performance indicators collected and validated through the PBF strategy. Three aspects were taken into account: 1) the indicators of services utilization; 2) the quality of services represented by the overall quality score of services and 3) incomes from PBF [20].
The indicators of service utilization concerned outpatient consultations, surgery, deliveries and hospitalizations. At this level we used the indicators associated with the aggregate data collected in the Routine Health Information System (RHIS) whose quality is verified in the context of PBF.
The overall quality score is obtained from two scores: 1) the technical quality assessed in the context of the PBF strategy is derived from a peer review that is conducted quarterly for each hospital. The technical quality evaluation concerns the hospital management, the administrative and logistics department, the reception, emergency and ambulatory care department and the hospitalization department. It represents 70% of the overall quality score; 2) the perceived quality score is obtained from the results of the community surveys conducted every six months. The evaluation of perceived quality involves to verify the existence of registered users in the hospitals, the health care service provided and the satisfaction of hospitals’ clients. It represents 30% of the overall quality score.
The incomes from the counted and approved benefits corresponding to the care services are recorded on the bills monthly. In this study, it was also tested whether the EHR adopted had an effect on the income allocated to quantity verification from PBF.
2.6. Data Analysis
The “double difference” method was used to calculate the effects induced by the use of HER. For each group of hospitals, the averages of the above indicators were calculated for the year 2014 (before the intervention) and for the year 2019 and 2020 (after the intervention). The difference between the two periods was calculated for each indicator. Then, we proceeded to calculate the differences (gaps) between the value of the gap found for the group of experimental hospitals and that of the group of control hospitals. Significance of the effect was analyzed with the Wilcoxon-Mann-Whitney test. The analysis of the evolution of the overall service quality score and the amount received by the hospitals was also done from 2014 to 2020.
2.7. Ethical Considerations
The research protocol has received approval from the National Ethics Committee for the Protection of Human Subjects of Biomedical and Behavioral Research. Authorization to use the validated SNIS data in the framework of Performance-Based Financing (PBF) was given by the Minister of Public Health and Fight against AIDS. Administrative approval was obtained from the head of the PBF Technical Management Unit.
3. Results
This section presents the effects of EHR on service utilization indicators, overall quality scores of service’ hospitals, and income received in the context of the PBF, calculated as a double difference.
3.1. Hospital Service Utilization Indicators
Considering the pre- and post-intervention period corresponding to 2014 and 2019, for the five district hospitals with HER adopted, all service utilization indicators evaluated in this study showed an increase. The use of delivery services increased by more than 100% (332%) while major surgery increased by 43%. Hospitals with EHR adopted in 2017 had two indicators that experienced the decrease (outpatient consultation decreased by 43% and cesarean section decreased by 43%) while those with EHR adopted in 2015 had one indicator that experienced the decrease for major surgery (Table 2).
![]()
Table 2. Effect of EHR use on service utilization indicators in hospitals (period before 2014 and period after 2019).
For the pre- and post-intervention period corresponding to the years 2014 and 2020, outpatient consultation and cesarean sections decreased by 3% and 5%, respectively. Hospitals with HER adopted in 2015 had a decrease of one indicator compared to those with HER adopted in 2017 (Table 3).
The Figures 1-4 shows that the two groups of hospitals had broadly the same trend from 2014 through 2020. The correlation coefficients calculated between hospitals with EHR and those without EHR are higher and represent 0.91 for outpatient consultation; 0.75 for major surgeries; 0.74 for deliveries; and 0.88 for cesarean sections (Figures 1-4).
3.2. The Overall Quality Score of Services
For the pre- and post-intervention period corresponding to the years 2014 and 2019, the average of the overall quality score of services increased by 37%. Considering the beginning of EHR use, the hospitals with EHR adopted in 2015 proved a strong increase of more than 100% of the overall quality score of services while those with EHR adopted in 2017 showed a decrease of 13% (Table 4).
The same trend of increasing in the overall quality score was observed when considering the pre-intervention period from 2014 to 2020 after the introduction of EHR. The five district hospitals experienced a 25% increase in the average overall quality score. Hospitals with EHR adopted in 2015 experienced an increase of 155% while those with EHR adopted in 2017 experienced a decrease of 19% (Table 5).
![]()
Table 3. Effect of EHR use on service utilization indicators in hospitals (period before 2014 and period after 2020).
![]()
Table 4. Effect of EHR use on overall quality scores in hospitals (period before 2014 and period after 2019).
*DID: Difference in difference.
![]()
Table 5. Effect of HER use on overall quality score in hospitals (period before 2014 and period after 2020).
*DID: Difference in difference.
![]()
Figure 1. Comparison of outpatient consultation average number between hospitals with EHR and their control hospitals.
![]()
Figure 2. Comparison of major surgeries average number between hospitals with EHR and their control hospitals.
![]()
Figure 3. Comparison of deliveries average number between hospitals with EHR and their control hospitals.
![]()
Figure 4. Comparison of average number of cesarean sections between hospitals with EHR and their control hospitals.
The analysis of the evolution for the two groups of hospitals shows that all the district hospitals (5 hospitals with EHR and 5 hospitals without EHR) included in the study increased in the average overall quality score from 2015. However, the hospitals without EHR decreased in 2017 while the hospitals with EHR increased gradual from 2014 to 2017 and decreased from 2018 to 2020. The calculated correlation coefficient between the average overall service quality scores of two groups of hospitals was higher and represents 0.85 (Figure 5).
3.3. Income from the Performance-Based Financing Context
Considering the pre- and post-intervention period corresponding to 2014 and 2019 respectively, the average income from the PBF increased by 31% for five district hospitals with EHR adopted. The two hospitals with EHR adopted in 2015 recorded an increase of 106% while three hospitals with EHR adopted in 2017 recorded an increase of 38% (Table 6).
Considering the pre- and post-intervention period corresponding to 2014 and 2020 respectively, the incomes from the PBF evaluation decreased with 15% for the five district hospitals with EHR adopted. The two hospitals with EHR adopted in 2015 recorded an increase of 59% while three hospitals with EHR adopted in 2017 recorded a decrease of 20% (Table 7).
Trend analysis shows that the income average received by the five hospitals without EHR was higher than the five hospitals with EHR from 2014 to 2018. However, the five hospitals with EHR received the higher income average than the five hospitals without EHR in 2019. In 2020, there was a decrease compared to 2019 for all hospitals with EHR adopted and regardless of the period they started using EHR. The calculated correlation coefficient between the average amounts received in the PBF context between the two hospital groups is very high at 0.97 (Figure 6).
4. Discussion
The evaluation of the EHR effects was carried out with the propensity score matching method. The propensity score was calculated by logistic regression using observable characteristics of the study hospitals. Among the twenty-six
![]()
Table 6. Effect of EHR use on the income from FBP (period before 2014 and period after 2019).
*DID: Difference in difference.
![]()
Table 7. Effect of EHR use on the income from FBP (period before 2014 and period after 2020).
*DID: Difference in difference.
![]()
Figure 5. Comparison of average overall quality score between hospitals with EHR and their control hospitals.
![]()
Figure 6. Comparison of the average amount obtained between hospitals with EHR and their control hospitals in the PBF context.
district hospitals not digitized before the year 2020, five hospitals were identified to be matched with the five district hospitals with HER adopted in 2015 and 2017. Two matching methods were used: nearest neighbor matching and caliper (based on the nearest propensity score approach, subject to a maximum distance of 0.25). Means comparison tests were performed between the two groups of hospital to check the matching quality. Assessment of bias reduction was obtained by calculating the difference between the calculated bias before and after matching. The first method of nearest neighbor matching without replacement was chosen to be use by the fact that it gave the match with less than 10% bias at the time the nearest neighbor matching with replacement had 19.0% bias and the caliper matching method had shown 18.6% bias (Table 1). Thus, the propensity score-based matching method and the choice of nearest neighbor without replacement minimized selection bias in choosing the control hospitals [18] [23]. The pre-intervention period used to calculate the effect was 2014 and the post-intervention period was 2019 and 2020. The year 2020 was taken into account by the fact that it was found that the performance indicators of health care structures were affected by the COVID-9 pandemic [9] [10] [11] [12] [13].
The effects of EHR use on service utilization indicators, overall quality scores and on income obtained in the context of the PBF were calculated in double difference. The significance of the effect was tested using the Wilcoxon-Mann-Whitney test adapted for small samples. The main limitation of the double-difference method is that it is based on the assumption that the relevant indicators follow the same trajectory in the experimental and control groups. Because of this, we compared trends in performance indicators for hospitals with EHR and hospitals without EHR and found that both groups of hospitals had performance indicators following the same trajectory from 2014 to 2020 (see Figures 1-6). In addition, the correlation coefficients between the two groups of hospitals for the mean values of all indicators were high (p > 0.70).
4.1. Effect of EHR in Improving Quality of Health Care Services and Indicators of Services Utilization
By taking a broader approach than other studies that focused on specific services or particular pathologies, our study allows the effect of the adoption of EHR to be seen from a pluridisciplinary. The comparative approach in hospitals with EHR and those without EHR was used and focused on indicators of service utilization, the overall quality score, and the income from performance-based financing (PBF) strategy. Under performance-based financing, the overall quality score is obtained using the results of the hospital’s technical quality assessment of services and the results of the community satisfaction assessment. If the quality of services and care provided by hospitals increases, the overall quality score should also increase. In this study, the evaluation of EHR effects on the overall quality score of health services showed that for five hospitals with EHR adopted, an increase of 37% was recorded. Taking into account the period of EHR beginning, the hospitals with EHR adopted in 2015 recorded a strong increase of more than 100% while those with EHR adopted in 2017 recorded a decrease of 13%. Thus, the improvement in overall service quality score changes with the number of years of electronic patient record use.
Many authors have evaluated the effect of EHR use on the quality of health services by using different approaches [24] - [29]. This is because EHR help to improve patient care and the work environment, which leads to improved quality of health care services. The EHR use improved continuity of care through accessibility to patient data, which guided providers toward correct and complete care. A study evaluated the effects of EHR on the quality of health care dimensions (tangibility, reliability, responsiveness, assurance and empathy) found that the EHR explained 29.5% of the variance in the quality of health care services [26]. A cross-sectional study evaluating the impact of EHR on ambulatory quality in a community setting found that the use of EHR was associated with higher quality of care (0.4, p = 0.008) [30]. Another study showed that medical doctors’ use of EHR had significantly higher ratings of care quality by increasing the level of attention and care given to patients, reducing wasted time and increasing time spent on patient care, reducing variation and errors in the work process, and improving the level of trust between staff and patients through structured guidelines as well as the use of up-to-date equipment (Beta = 4.83, p < 0.01) [31]. A study found a relationship between use of EHR and a decreasing number of errors (p < 0.001) and a higher level of adherence to guidelines (p = 0.049) [32].
As the quality of the supply of services and care increases in health facilities, the use of these services also increases. In our study, the evaluation of the EHR effects on the four indicators (Outpatient consultation, Major surgery, Delivery, Cesarean section) showed an increase for the period before and after the intervention corresponding to the years 2014 and 2019. It was found that hospitals whose EHR were adopted in 2015 had experienced less decrease than those whose HER were adopted in 2017. As a result, it is noted that the improvement in service utilization indicators changes with the number of years the Electronic Health Record has been in use.
4.2. Effect of EHR in Increasing the Amounts Received in the Context of BPF
The five hospitals experienced a 31% increase in the amount of money earned in the PBF assessment. Considering the year of EHR use start, hospitals with an EHR in 2015 had a 106% increase, while those with an EHR in 2017 had a 38% increase. Other studies have also shown that the use of EHR increases revenue. A study in a hospital proved that the electronic health record-based system interoperable with a smart pump in the infusion influenced the increase in revenue in outpatient consultation departments [33] [34].
4.3. Effect of EHR on Performance Indicators in the Context of COVID-19
The indicators that experienced a decrease decline in effect in 2020 were outpatient consultations (3%), cesarean sections (5%) and amounts received (15%). The results of this study show that the COVID-19 context had a negative impact on the interventions carried out in the health facilities and consequently on the performance indicators (Figure 7).
Many studies have proved that the COVID-19 epidemic strongly affected the health care system. Containment and the risk of exposure to the coronavirus prompted patients to change their use of care. A study carried out in the Provence-Alpes-Cote d’Azur region showed that consultations with general practitioners decreased by 23%, whereas they decreased by 46% among specialists [9] [10]. Another study conducted in Niamey health centers showed that there was a 34% reduction (95% IC: −47% to −21%) in outpatient consultation care and also showed that the COVID-19 pandemic negatively impacted the provision of services to the most at-risk groups of the population, namely women and children [11]. Another study in Mozambique also compared quantitative indicators of access to maternal health care in 2019 with those in 2020 and found that cesarean sections dropped by 28%; first prenatal visit and hospital deliveries decreased by 4% [12].
4.4. Limitation of the Study
The evaluation was performed on small sample of hospitals (five hospitals with EHR and five hospitals without EHR). Using the nonparametric Wilcoxon-Mann-Whitney test adapted to compare the distribution of small sample, no indicator had a significant effect. Although the significance of the effect was not found, it can be concluded that there is an association between the health record digitalization and the improvement of the performance indicators of the study hospitals considering the variations observed from 2014 to 2020 and the positive effects found in favor of the hospitals with the EHR. Furthermore, it would be relevant to conduct in-depth research on a sufficient number of hospitals to extrapolate the results and confirm or refute the hypothesis of an association between the health record computerization and the improvement of hospital performance indicators in general.
![]()
Figure 7. Comparison of the effect between the years 2019 and 2020.
5. Conclusion
The evaluation of the effects of the use of the ECD proved that digitalization of the health record improved indicators of service use in hospitals, service quality, and revenue. However, it was noted that the number of years of EHR use correlates with the effect induced. In addition, health emergencies such as the COVID-19 pandemic negatively impacted the effects induced by the computerization of the patient record. Strengthening resilience strategies in the face of potential shocks that may affect the interventions will make it possible to consolidate the achievements and guarantee the success and sustainability of the health record computerization project in Burundian hospitals.