Effectiveness of Remote Monitoring and Evaluation by Development Agencies during COVID Pandemic: A Case of IFAD Supervision/Implementation Missions in Uganda, Rwanda and Kenya ()
1. Introduction
The coronavirus pandemic started at the end of 2019, and spread like wild fire across the globe. By end of May 2020, a total of nearly six million people were infected and a third of a million people died (Paul et al., 2020). COVID 19 was pronounced as a health crisis affecting people’s lives and health, threatening public service especially where the affected numbers surpassed the capacity of hospitals, threatening economies given that a number of economic activities such as transport, industries, and workplaces came to stand still.
Literature (Paul et al., 2020; Chowdhury & Jomo, 2020; Martínez-Córdoba et al., 2021) on governance of the pandemic show that each country developed and implemented country specific strategies with varying success levels. People were advised to adhere to public health guidelines (wash hands, test, vaccinate and or get treated) while the entire society was advised to limit social interaction. Lockdown was the common tool used to control the spread of COVID 19, however, it is limited interaction and implementation of socio-economic activities (Paul et al., 2020).
Kauzya (2020) maintains that the pandemic challenged the values and principles enshrined in the 2030 SDG agenda, which include equity, inclusion, effectiveness, accountability, resilience innovation, collaboration and leaving no one behind. There is no doubt that development interventions were disrupted in many ways, implementation of some development programmes was halted due to scarcity of workers and funding and other development programmes operated minimally because the world was focused on handling the pandemic and its aftermath.
Project supervision, monitoring and evaluation processes were also disrupted and Dube et al. (2021) call it the changing face of monitoring and evaluation. Given the challenges of travelling across the globe, remote monitoring and evaluation models and practices were adopted by many development agencies to ensure continuity in providing technical support to programme implementers, validate performance of projects, develop lessons and make informed decisions. Monitoring and evaluation shifted from conventional monitoring and evaluation to COVID 19 tailored approaches which minimised face to face data collection, increased remote data collection through use of computer apps, online methods like telephones, maximised used of secondary data, simplified data collection methods as primary data collection focused on essential and critical areas and adopted simple sampling designs aimed at promoting inclusion (Ibid). According to Vindrola-Padros (2021) in the health sector, the use of apps for monitoring and evaluation allowed the follow-up of a higher number of patients compared to paper-based models and the use of telephone calls was more inclusive (i.e. including patients without internet access or technological literacy).
Remote monitoring and evaluation are not a new subject, it has always been used in health projects and its processes and success well documented. The Covid 19 opened remote monitoring and evaluation more to sectors that were traditionally not practicing it. Literature on remote monitoring and evaluation during COVID 19 focus on explaining the changing approaches from the conventional to those tailored to the COVID 19 context (Dube et al., 2021), the processes and stages of undertaking remote monitoring and evaluation (Vindrola-Padros, 2021), the technological advancements and tools (Ji et al., 2023) and their limitations in maintaining data quality and project oversight (Dube et al., 2021). The systematic review undertaken by Vindrola-Padros (2021) maintains that it was not possible to assess the costs, outcomes and impacts of remote monitoring and evaluation because the studies utilised in the analysis did not capture such information.
Effective monitoring and evaluation identify project strength and weaknesses as well as potential project risks and if well-handled improves project quality and accountability. There is a paucity of published literature on the effectiveness of the remote monitoring and evaluation carried out by development agencies during COVID 19 in terms of enhancing accountability and learning, stakeholder participation. The aim of this paper is to analyse the effectiveness of remote supervision, monitoring and evaluation missions undertaken by the International Fund for Agricultural Development (IFAD) in Rwanda, Uganda and Kenya and draw implications for the evaluation theory and practice. More specifically the study sought to answer the following questions:
Did the remote M&E enhance accountability and learning?
Did remote M&E enhance stakeholder participation?
Did the remote M&E help programmes to run efficiently?
2. Literature on Remote Monitoring and Evaluation
Remote Monitoring and Evaluation is a Remote Management (RM) approach that became apparent as organizations aimed to ensure the effective execution of projects irrespective of geographical constraints. According to Donini & Maxwell (2013) and IOM (2020), remote management is frequently employed by humanitarian organizations. This strategy allows aid agencies to maintain their operations in situations where having international staff on-site is no longer feasible or allowed. Numerous studies (including Stoddard, Harmer, & Haver, 2006; Donini & Maxwell, 2013; Brahmbhatt & Cowie, 2019; Armstrong, Dubben, & Karas, 2022; IOM, 2020), have identified security concerns and pandemics like COVID 19 as crucial factors driving the shift towards remote management.
According to Jan, Harmer & Stoddard (2011), remote monitoring as “an adaptation to insecurity, the practice of withdrawing international (or other at-risk staff) while transferring increased programming responsibilities to local staff or local partner organizations”. The authors added that utilizing remote management has proved to be a reliable and practical strategy for humanitarian organizations in situations where human lives are at risk. As a result, humanitarian organizations such as the International Organization for Migration (IOM), UNICEF, The War Child Canada, the International Committee of the Red Cross (ICRC) and the Women Refugee Commission recommended a number of changes to the monitoring and evaluation systems of their partners in which Remote Monitoring and Evaluation was deemed most appropriate during pandemic and insecurity or political barriers.
Remote Monitoring and Evaluation refers to using technological methods to review project progress data from locations separate from project sites (Jan et al., 2011). For example, according to Vegesna et al. (2017), Remote M&E technologies such as mobile phones, satellite imagery, and mapping have significantly improved clinicians’ ability to monitor and manage patients in nontraditional healthcare settings.
Remote M&E has proved to be particularly useful in sectors where projects are spread out, challenging to access or require real-time data insights. Examples of such sectors include international development, healthcare, and disaster response. Remote M&E enables organizations to efficiently monitor projects in diverse locations (Stoddard, Harmer, & Haver, 2006; Brahmbhatt & Cowie, 2019; Vegesna, Tran, Angelaccio, & Arcona, 2017; Dube et al., 2021; Armstrong et al., 2022). IOM (2020) also pointed out that RM&E has important implications for the organizational setup, accountability, monitoring, and assurance of intervention quality.
Various studies (Jan et al., 2011; Brahmbhatt & Cowie, 2019; Dube et al., 2021; OECD, 2021; Vegesna et al., 2017; Armstrong et al., 2022; IOM, 2020) have highlighted the benefits of Remote Monitoring and Evaluation (RM&E), including real-time feedback, improved efficiency, cost reduction, timely emergency medical care, enhanced safety of staff and communities, positive relations with grassroots organizations. However, RM&E can face obstacles such as competing priorities, budgetary constraints, limited staff capacity, infrequent monitoring visits, un standardized procedures and monitoring tools, challenges of data integrity and confidentiality, and low-quality data collection and lack of information triangulation mainly due to restrictions in social interaction.
IOM (2020) echoed that Remote M&E can be temporary or permanent depending on the degree of remoteness, response to security, and logistical challenges; hence, different strategies may need to be applied. Consequently, it is important for the office overseeing the intervention to establish a set of standard parameters and thresholds that determine when direct implementation is no longer feasible.
The above the literature review on Remote M&E lays a solid foundation but leaves critical gaps in understanding how RM&E specifically addresses accountability and learning, fosters stakeholder participation, and contributes to programme efficiency.
Despite the numerous capacity building efforts on the African continent, most evaluations are majorly conducted by International consultants, because they are believed to be objective and independent (Tarsilla, 2014).
Remote Supervison Missions in Ifad
During the COVID 19 pandemic, IFAD was running a number of projects within in East Africa. For Instance, in Uganda some projects included; the National Oil Palm Project (NOPP), Vegetable Oil Development Project (VODP) phase 2, and the Project for Restoration of Livelihoods in the Northern Region (PRELNOR); in Kenya IFAD was also running a number of projects including: the Kenya Livestock Commercialization Project (KeLCoP), Aquaculture Business Development programme (ABDP), Kenya Cereals Enhancement Programme-Climate Resilient Agricultural Livelihoods Window (KCEP-CRAL), Smallholder Dairy Commercialization Project (SDCP) and the Upper Tana Natural Resource Management Project (UTaNRMP); while in Rwanda IFAD was implementing project like: the Climate Resilient Post-Harvest and Agribusiness Support Project (PASP), Partnership for Resilient and Inclusive Small Livestock Markets (PRISM) and the Kayonza Irrigation and Integrated Watershed Management Project phase 1.
Three weeks into the COVID 19 pandemic and with the introduction of social distancing and lockdowns, IFAD recognised that COVID 19 had drastically altered the daily lives of IFAD staff in Rome, those stationed around the world and those of the projects they supported. IFAD recognised that there was a slowdown of world economies with a possibility of accelerating poverty and hunger necessitating greater commitment to IFAD’s mission. IFAD introduced regular virtual meeting aimed at enabling staff to “catch up on the work that is being carried on and to support each other”.
“Only by working closely together can we find a way to serve the rural poor in such an unprecedented context... We are working tirelessly to mobilize our resources in a quicker and more efficient way, in order to support governments and rural people in their struggle against the effects of the outbreak… We are battling against time to find ways to provide adequate responses to governments’ requests and to continue sustaining livelihoods in rural areas. It is obvious that business as usual is not an option: We need to be innovative, agile and flexible… Despite all the obstacles the COVID 19 outbreak is putting in our path, IFAD will continue to work relentlessly, recasting our business model to better cope with the new challenges… We are doing our best to avoid business disruption by engaging in ministerial-level dialogues and continued follow-ups with project management units, including remote supervision of projects in areas where local conditions allow field work to be carried out by local consultants. So far, we have confirmed several remote missions that will be virtually fielded. Not being able to travel has also given us the opportunity to focus on non-lending activities and to develop remote capacity-building for our project staff”.
Guidance notes for remote supervision (Extract from IFAD’s Note, 2020)
Traditionally, IFAD has had three different supervision modalities including; 1) full supervision missions (looking at all performance indicators of the project), 2) partial supervision missions (assessing fewer performance indicators of the project), and 3) implementation support missions. Often these missions involve field work, face to face data collection and engagement with stakeholders. In the context of COVID 19, two new supervision modalities were introduced and include; remote (full) supervision, and remote partial supervision. IFAD defined remote supervision to mean the supervision which does not include fieldwork or meetings in country, and which leverages digital tools and communication channels to allow teams to review project activities, outputs and outcomes at distance. According to IFAD, remote supervision presents an option for monitoring project implementation when crises such as COVID 19 make direct supervision problematic.
IFAD developed a guidance note on how remote supervision should be conducted. IFAD maintained that the choice between full or partial remote supervision will depend on the availability of data that can facilitate each supervision modality. The Country Directors were given responsibility to decide to conduct remote supervision missions or not and to also follow-up on the implementation of supervision recommendations. According to the guidance note for remote supervision;
“in case it is not possible to conduct any supervision, because all country operations have stopped, or because PMUs are not able to perform their work and no local consultants or local service providers can be hired to support remote supervision….” For countries with partial stop and marginal disruption, remote supervision mission modalities may apply.
IFAD also provided for reclassification of the mode of supervision by the mission depending on the context in the country. For instance, a full remote supervision can be re-classified to a partial supervision mission, or a partial remote supervision can be re-classified as implementation support. Refer to Table 1 below for provisions and limitations for different categories of remote supervision.
According to IFAD the remote supervision mission has the same planning process as a normal supervision mission where the Country Director undertakes the following activities;
Table 1. Summary of supervision modalities by IFAD.
Summary table of supervision modalities |
Remote Full
supervision mission |
Limitations for in-country direct supervision mission for IFAD staff and int. consultant, however the PDT can assess all performance indicators remotely, with the help of PMU, local consultants and third parties. Same reporting requirements apply as for normal full supervision mission (including all Performance Score and Key SIS ratings). |
Remote Partial
supervision missions |
Limitations for in-country direct supervision mission for IFAD staff and int. consultant, however the PDT can assess only a single or a few performance indicators remotely, with the help of PMU, local consultants and third parties. As an example, a partial remote supervision mission is when there is only a remote procurement supervision review conducted without in-country mission, or only a FM remote supervision review, or an. effectiveness and progress supervision review conducted. Also, a combination of supervision review areas might be conducted. Same reporting requirements apply as for normal partial supervision mission. |
Implementation support mission |
Performance indicators are not assessed or updated in ORMS/FMDB. No specific reporting mission requirements mandated, although tailored reports or findings/observations can be provided according to the individual mission TORs. Can be organized with international travel and in-country missions and also virtually and remotely. |
Develop TORs and mission schedule and get it approved.
Communicate the timing and TORs to mission team members.
Fix appointment for virtual meetings with project implementation team, government and implementing agency officials, local consultants and/or local service providers, with some beneficiaries.
Inform the implementing agency to prepare updated physical and financial progress reports, including the implementation status of the previous mission’s agreed actions.
Request the implementing agency to prepare a more comprehensive progress report than usual, including visuals from implementation sites and targeted communities, and statements by beneficiaries. Alternatively, more in-depth details on the progress and field evidence can be also arranged by local consultants or contracted third parties.
Use savings from travel budget to 1) finance equipment/internet data costs for local level virtual meetings, 2) recruit local consultants/NGOs for data collection and interviews, etc.
If possible, remote wrap-up meetings should be organized to agree on the supervision outcomes, recommended actions and the Aide-Memoire. If not possible, also phone calls and/or exchange via e-mail is possible to agree on the final version of an Aide-Memoire, including actions to improve different aspects of project performance.
After having conducted a remote supervision mission a management letter is issued by IFAD and sent to the government.
If possible, remote implementation support can be organized and conducted virtually, with e-learning, or follow-up calls to help establish operational and fiduciary systems, recruit technical assistance and build the capacity of key project implementers.
It is evident that IFAD has had elaborate guidelines on remote monitoring and evaluation processes for its programmes. This study sought to analyse whether the remote monitoring and evaluation missions conducted by IFAD enabled organisations to achieve the normative agenda of M&E including enhancing accountability/learning, stakeholder participation, and efficiency of the projects?
3. Methodology
The study adopted an exploratory study design aimed at investigating whether the remote monitoring and evaluation missions enabled organisations to achieve the normative agenda of M&E including enhancing accountability/learning, stakeholder participation, and efficiency of the projects. The study draws the implication of remote monitoring and evaluation to the evaluation theory and practice.
The study utilised experiences and Insights from IFAD consultants and project implementers. “The International Fund for Agricultural Development (IFAD) is an international financial institution and a specialized agency of the United Nations dedicated to eradicating poverty and hunger in rural areas of developing countries” (https://www.ifad.org/en/). IFAD supports Agricultural and poverty eradication projects in several regions including the Eastern and Southern African region. This study was conducted in the Eastern Africa region i.e. Rwanda, Uganda, and Kenya because the researcher has worked in these countries for over 20 years. Rwanda is a small landlocked country with limited natural resources and Agriculture is a major economic sector, employing about seven in ten people. Three quarters of Rwanda’s agricultural production comes from smallholder farmers. Kenya became a low-middle-income country in 2014. However, poverty and income inequality remain persistent and about 3.5 million Kenyans need food assistance. “IFAD extends its support beyond productive rural areas in Kenya to the country’s arid and semi-arid lands”. On the other hand, Uganda has experienced high economic growth and a significant decline in poverty. Agriculture accounts for 24 per cent of GDP and employs seven in ten of the working population. Agriculture remains an important source of income and employment for the rural population. For each of these countries, IFAD supports an average of three projects at a time for an average of seven years per project.
The researcher an IFAD consultant in the East and Southern African regions, she has participated in remote monitoring and evaluation missions of IFAD and has established working relationships with many other IFAD consultants and project implementation units in Rwanda, Kenya and Uganda.
The study was conducted using an online self-administered questionnaire among evaluation managers, mission teams, and evaluators who participated in IFAD supervision/implementation support monitoring and evaluation missions in Uganda, Rwanda and Kenya during COVID 19 in Rwanda, Uganda. The 12 participants who participated in the study were purposively selected because they participated in IFAD remote missions during 2020-2021.
Data drawn from semi structured self-administered questionnaires was analysed using thematic analysis by identifying patterns and similar themes.
4. Findings
Did the remote M&E enhance accountability and learning?
Data gathered from study respondents show mixed finding on whether remote monitoring and evaluation enhances accountability and learning. Importantly there was a general agreement that remote monitoring and evaluation enhanced accountability in 3 ways. It enabled stakeholders to receive updates/reports on the performance of the projects. As noted by the respondent;
Remote monitoring and evaluation provided an opportunity for the Development Agencies to receive updates on the progress and guide on how to improve performance, to track and report on specific projects even when physical follow up was difficult due to the outbreak of COVID 19 (Respondent 3).
Remote M&E aided continuous reporting on project performance despite the lockdowns and limited interaction with project sites.
Remote monitoring and evaluation also kept the funders and the project implementers focused towards the anticipated results of the project and maintain contact with project beneficiaries. As stated by one of the respondents;
Some stakeholders were affected by COVID 19 yes, but overall the systems within IFAD and those of implementing agencies demanded results. Even if a staff member was sick, another could stand in and continue to implement. We endeavoured very much to stay in touch with beneficiaries because we knew reports would be required (Respondent 6).
Accountability was enhanced by deciding on new ways to improve the implementation of the project as being informed by supervision reports and aide memoirs. As noted by one of the project implementers;
For every supervision mission we often have a table of agreed upon actions with related timelines. Yes, during COVID 19 lock downs timelines were relaxed and we took on few activities but the support given by the missions helped us to continue implementing few activities especially those which were in easy control of the beneficiary. The online wrap up meetings helped to build consensus on what needed to be adjusted (Respondent 1).
Remote M&E facilitated continuous discussions and decision making on areas of the projects that needed improvement.
However, remote monitoring and evaluation was said not to have enhanced accountability because of inherent challenges of lack of physical contact. Respondents maintained that it was difficult to validate results reported by the project implementers since there were no opportunities to triangulate the reports. As noted by a respondent;
It is difficult to make a definitive judgment, but I don’t believe that remote M&E has had a positive impact on enhancing accountability. The primary purpose of most in-country missions is to validate the progress reported in project reports; yet if remote M&E becomes more prevalent, there may be limitations in cross-validating the figures presented in those reports through alternative means (Respondent 2).
This may mean that online tools used to undertake monitoring and evaluation to certain extent would not allow triangulation with sources of information. Were the means of verification adjusted during COVID 19? Possibly there was no opportunity to do this.
Accountability to the beneficiaries was difficult since some of them could not be accessed online. As noted by a respondent,
The wrap up meetings often include a wide category of stakeholders including farmer’s representatives. But during COVID 19 lockdown some farmers could not be reached through data collection and also giving them feedback. Beneficiaries were tasked to monitor their own progress. However, this works best for stakeholders who are self-motivated and have a drive to perform. Those with no drive may not even report on their progress unless we visit them through fieldwork (Respondent 12).
Remote monitoring limited effective communication with categories of beneficiaries who would not access the relevant technologies. Even with the introduction of teleconferences, it worked well where internet coverage was good.
In terms of learning remote M&E enabled project implementer to invest in high technologies, and software to aid communication. Project implementers had to rapidly learn how to use these technologies and sustain conversations with funders and support supervision missions. Project beneficiaries were mainly engaged through telephone conversations with the support of project staff based at project sites.
Did remote M&E enhance stakeholder participation?
On whether remote monitoring and evaluation enhanced stakeholder participation, finding indicate that Remote M&E enhanced participation of the remote supervision teams and project implementation teams, to a smaller extent there was seeming participation of farmers through the use of teleconference and telephones but largely remote supervision fell short on enhancing meaningful participation of stakeholders. As stated by a mission member;
When we employed teleconference in the field, it was fascinating for the farmers. They were able to talk to mission members who were based in Rome and those in Rome were also able to see the demo gardens and ask some questions. Of course, the conversation was limited and took a short time but in places where internet services are available and the environment is conducive, teleconferencing can create some virtual interaction (Respondent 1).
There was direct communication and engagement of project beneficiaries and other stakeholders through teleconferencing and telephone calls. This enabled access to real time information and possibly verify some information however application of teleconferencing and telephones have limitation in terms of scope. This necessitates use of various other methods for the support supervision missions to draw conclusions.
Another respondent who is the project implementer echoed;
For us we used mobile phones to call farmers and engage them on some areas. We engaged them on aspects such as; dairy extension issues, even agronomy issues and access to extension services. We also engaged them on marketing issues of produce through a phone conversation. We were able to identify some of their challenges during the remote mission. We asked them questions related to implementation, on results and progress of the projects. We documented this information and used it to develop project reports. Most of these reports were used by support supervision missions (Respondent 5).
The implementers participated in implementation as well as monitoring of project activities and development of project reports which were relied on by project supervision, monitoring and evaluation missions. There is potential for internal project staff to generate useful information from the internal monitoring and evaluation exercises.
Respondents maintained that remote supervision, monitoring and evaluation has a potential to enhance participation if it is able to enhance accessibility and inclusion of different categories of stakeholders however, quality of participation may not be guaranteed. As noted by one of the respondents;
Remote M&E may have the potential to enhance stakeholder participation by increasing accessibility and inclusivity. Since the remote M&E is done mostly through virtual meetings and online platforms, this have made it easier a broader spectrum of stakeholders from various institutions to participate in the discussions. This does not necessarily mean the quality of engagement is equally improved; however, it primarily serves to lower the barriers for those who are geographically or logistically constrained to engage in the M&E activities (Respondent 2).
Remote supervision, monitoring and evaluation missions enhanced engagement between the mission and the implementing units compared to other stakeholders. This is because the project implementation units were better equipped with internet and communication facilities compared to farmers. Even then the farmers may not be conversant with the indicators to track and document. As one of the project implementers noted;
Remote M&E was mainly facilitating discussions between the Development Agency Supervision teams and the Project Implementation Unit but it was difficult to engage other stakeholders. In the cases where efforts were done for online engagements, these were not effective due to connectivity problems but also controls around meetings arranged in such a format and stakeholders were not very informed about the key indicators of the project, they could not make meaningful contributions (Respondent 6).
Did the remote M&E help programmes to run efficiently?
Generally remote supervision, monitoring and evaluation missions were perceived to be relevant and to have enhanced efficiency projects in terms of project time management and saving resources from travel. Exemplified by response from one of the mission members;
I would say yes, particularly in terms of time management. The remote M&E became a new normal during the pandemic, save project from travel time, and time to mobilise communities to meet with supervision team members. Remote monitoring and evaluation allowed projects to maintain its implementation momentum without significant disruptions. Particularly, the flexibility of having the remote M&E enabled the project teams to efficiently use their time and resources (Respondent 9).
Physical supervision missions require more time to plan, prepare, and mobilise finance and human resources to participate on the missions. Often physical missions occupy project staff during and after the missions. On the other hand, remote missions require time and internet resources from the project but because it does not involve movement to the field, time could be saved.
Another respondent emphasised that projects saved time and money and they were able to continue to run and improve. As noted by one of the respondents;
I cannot mention in terms of money what was saved due to limited travel to the field and also limited international travels. As you know a big percentage of expenses in projects relate to fieldwork but, this time there was no fieldwork and projects continued to survive. Again, you know, many mission members always are international consultant’s travelling from different countries, but this time round they did not travel (Respondent 4).
Another respondent echoed;
When local travel limitations were eased in this country, we often used local consultants/evaluators to go to the field. We selected fewer field sites to visit and reduced the number of days we would spend in the field. At the same time, we limited numbers of beneficiaries to engage and practiced social distancing. Use of local evaluators saved on travel costs.
There is no doubt that there were savings on travel costs i.e. inland and international travels and possibly on repair and maintenance of project vehicles and machinery since supervision was remotely done. However, this could not be quantified across the countries and my need a different analysis.
There was a perception that remote monitoring and evaluation improved the operations of the project through implementation of the recommendations made by the remote teams. As noted by a respondent;
Again, remote M&E provided a platform to discuss good performance and areas which needed improvement. Just like during the physical missions, the supervision mission teams were able to provide guidance on implementation challenges which helped to improve operations of the project including remote interventions (Respondent 3).
5. Discussion and Implications to Evaluation Theory and Practice
Remote Monitoring and Evaluation is an old tradition which has been deployed by organisations working in emergency situations, insecure situation and geographically remote located areas. In these situations, Donini & Maxwell (2013) and IOM (2020) maintain that remote M&E has always been useful in providing information to make decisions necessary to maintain operations of the project and inform stakeholders on the performance of the project thus enhancing continuity and accountability. The users of such evaluation processes were mainly project funders and project implementation teams. From the findings of this study, remote monitoring and evaluation became the norm. Evidently during the COVID 19 pandemic this tradition of remote M&E was scaled up and applied by other organisations which were originally not using it. While it used to be applied by humanitarian organisations working on emergency issues and those dealing with health projects, it was scaled up to apply to agriculture and developmental projects within the context of the pandemic. It was mainly applicable to routine supervision and monitoring compared to evaluations. This is in line with the Utilisation Focused Evaluation by Michael Quinn Patton who emphases the importance of making evaluations useful and relevant to stakeholders.
Remote M&E practices in Kenya, Rwanda and Uganda enhanced efficiency in terms of time and cost saving. Just as Vegesna et al. (2017); Armstrong et al. (2022); IOM (2020) noted, remote monitoring and evaluation had several benefits including real-time feedback, improved efficiency, and cost reduction. However, these benefits could not be quantified.
With remote monitoring and evaluation, some stakeholders like project field staff members, and IFAD country-based officers participated more than the others especially farmers, international consultants and other IFAD international and regional officers because of the limitations associated with technology and low-quality data collection as well as lack of information triangulation and validation due to limited social interaction. Due to limitations of not being able to visit the field in person, there are constraints in collecting and analysing all the information through the figures in reports or interviews with the stakeholders. Sustaining interviews online was difficult due to connectivity and time difference issues. It thus requires evaluators to exercise greater caution on evaluation conclusions.
There are two continuous debates in the field of monitoring and evaluation. First, is whether projects should rely more on internal evaluations compared to external evaluations. The findings in this report show that internal/implementers successfully undertook monitoring and evaluation activities, used information to support operations of the projects and used this information to compile monitoring and evaluation reports which were relied on by the IFAD supervision missions. Although according to Jan, Harmer & Stoddard (2011), this is an added responsibility, it was useful for the implementation teams to wholesomely and objectively take control of project processes and learnings. This means that internal monitoring and evaluation can comparatively be useful and can provide unbiased information to guide the operations of the projects.
The second debate has been on the use local evaluators compared to international evaluators. Despite the numerous capacity building efforts on the African continent, most evaluations are majorly conducted by International consultants, because they are believed to be objective and independent (Tarsilla, 2014). The African Continent has been depending on international or external consultants to undertake supervision, monitoring and evaluation. The COVID 19 gave more importance and prominence to locally available evaluators compared to international evaluators. The power dynamics between the local and the international consultants changed swiftly to give more importance and recognition to local consultants. Local consultants were hired independently to support specific IFAD activities. This practice worked perfectly well in IFAD projects, it saved costs of international travels and gave a chance to locally available evaluators to participate independently in IFAD projects. There is no doubt that use of local evaluators is advantageous compared with international evaluators. For instance, local evaluators are often more familiar with cultural nuances, languages, and traditions, reducing the risk of cultural misunderstandings especially in the COVID 19 context which was very uncertain. It was more likely for the local evaluators to appreciate and understand the local political landscape and community dynamics to aid interpretation of the data collected especially during the pandemic restrictions compared to International Consultants. More so, local evaluators are part of the community with a likelihood of fostering trust and encouraging honest responses during interviews. Therefore, engaging local evaluators needs to continuously be encouraged because it contributes to building long-term evaluation capacity within the country or region.
In addition to technology innovations, remote M&E introduced other innovations in the process of execution. For instance, during some IFAD missions, the supervision teams used the concept and practice of delegation in some cases. External evaluators unable to travel developed a set of questions which were administered by a local team member and thereafter the external evaluators depended on lenses and interpretation of a local team member or sometimes IFAD locally available staff to appreciate the progress of the project. This was an innovation to provide effective remote evaluation support and still provide an unbiased assessment on the performance of projects and programmes. The success of delegation in the field of monitoring and evaluation may need to be further studied. Perhaps the existing theories and/or practices might need to be adjusted to the different context and methodologies of remote M&E.
According to IOM (2020) remote M&E can be temporary or permanent depending on the degree of remoteness, response to security, and logistical challenges. IMO maintained that offices overseeing the intervention to establish a set of standard parameters and thresholds that determine when direct implementation is no longer feasible. Indeed, IFAD followed suit and provided guidelines of implementing remote M&E however, the guidelines seem not to anticipate re-occurrence of emergencies, meaning that remote M&E practiced by IFAD was envisaged to be temporary. The question is, even after the world has slowly recovered from COVID 19, to what extent can these tools continue to be relevant to the discipline to further increase efficiency of evaluation and practice? There might be need to expand on the tools and skill set for evaluation to ensure that evaluation as a practice is better resilient to shocks such as the COVID 19.
Drawing lessons from remote M&E practices, especially how development organisations adapt and learn from challenges and success of its implementation would be useful to continue to shape the field of monitoring and evaluation. Systematic analysis and accumulation of knowledge on various practices of remote M&E might be interesting to guide the future practices. Therefore, development organisation ought to deploy organizational Learning Theories which emphasizes the importance of learning from experience for organizational success.
6. Conclusion
This study maintains that remote M&E largely achieved the basic objectives of monitoring and evaluation including enhancing performance of projects, use of M&E results in decision making, accountability and learning at the same time adoption of modified evaluation methodologies and practices. However, it did not provide space for quality participation of project beneficiaries due to technological gaps. The implementers also became evaluators as the secondary information accessed by the evaluation teams could only be provided and verified by the project implementers. The concept of delegation was introduced into the field monitoring and evaluation as locally available mission teams who could possibly access the project sites additionally wore lenses team members unable to access project sites. COVID 19 gave more importance and prominence to locally available evaluators compared to international evaluators. The power dynamics between the local and the international consultants changed swiftly to give more importance and recognition to local consultants.
7. Areas for Further Research
What have we learnt from remote M&E practices? What have organisations learnt from remote monitoring and evaluation during the COVID 19 pandemic? What were the good practices?
To what extent can the tools used in remote M&E continue to be relevant to the discipline to further increase efficiency of evaluation and practice? Can remote M&E be sustainable? What are the implications?