Reducing impact evaluation capacity disparities through holistic engagement

1 March, 2022
Evaluation Matters Magazine:
The Future of Impact Evaluations in Africa

Bio Mama, 3ie
Elsadig Idris Abdelsalam Mohammed Zin, 3ie
Paul Thissen, 3ie
Rosaine N Yegbemey, 3ie
Sara Ansari , 3ie

Abstract:

Vast disparities in impact evaluation capacity exist between regions of Africa, and closing this gap requires more than just one-off academic-style training in evaluation methodologies. The experiences of the International Initiative for Impact Evaluation’s (3ie) West Africa Capacity-building and Impact Evaluation (WACIE) program and our participation in the PeaceFIELD initiative in Sudan have provided some useful lessons on the best ways of building evaluation capacity in contexts where it is lacking. We have found that a holistic approach that fits impact evaluation training into people's existing work is most effective, especially when combined with active follow-up.

Key Messages:

  • Efforts to build evaluation capacity are most effective when integrated into real-world work streams.
  • Understanding the landscape of evaluation and the policy actors involved is essential in designing applicable training materials.
  • Ongoing in-country engagement is key to understanding stakeholder interests and coordinating capacity-building work.
  • Impact evaluation methods training should be complemented with broader discussions regarding their place among other evaluation tools and their role in the policy cycle.

Impact evaluation capacity in Africa varies widely across the continent. Research work at the Africa Centre for Evidence highlights the disparities in terms of geography and sector (Erasmus, Jordaan and Stewart, 2020). Indeed, most impact evaluators come from English-speaking countries in East and Southern Africa. In terms of substance, most impact evaluations by African evaluators are focused on the health and nutrition sectors, with few intervening in social sciences.

Our scoping work for the West Africa Capacity-building and Impact Evaluation (WACIE) program revealed a similar pattern: limited existing impact evaluation capacity in many parts of Francophone West Africa, combined with a strong desire by stakeholders in those countries to improve that status quo (Amouzou et al., 2020).

Closing these gaps requires more than just training sessions in impact evaluation methods. Our recent experiences with the WACIE program in West Africa and with partners of the UN Peacebuilding Fund in Sudan show that building capacity is most effective when integrated into real-world work that policymakers, project managers, and evaluation professionals are already engaged in. When we show people impact evaluation tools, methods, and findings relevant to their work, the lessons seem more likely to stick.

We have found that offering holistic support works best. Rather than “one-and-done” training sessions, we work to help our colleagues build relationships with other evaluation professionals, researchers, and the training team, which guides the trainee on to the next steps. In some cases, those next steps involve finding funding opportunities, helping with the often Byzantine process of submitting proposals, and offering technical assistance through the research lifecycle. Through these steps, helping to foster linkages creates a community of practice and support.

WACIE's approach in West Africa

To meet the need for capacity-building and to fill the gap in the practice of impact evaluation in Francophone West Africa, the WACIE program set out to develop a strategy for the various actors in the national evaluation systems of the countries where it works in the West African Economic and Monetary Union (WAEMU), namely Benin, Burkina Faso, Côte d’Ivoire, Guinea-Bissau, Mali, Niger, Senegal, and Togo. We targeted those who commission impact evaluations, those who conduct them, and those who use the results.

The first step was to map the actors and understand their needs. The demand side includes the actors who commission the evaluations and use the results, including decision-makers, technical staff in government agencies, and donors. The supply side consists of the actors who produce the evaluations, including universities, consulting firms, and research centers and institutes. We then tailored our capacity-building efforts to the needs of each stakeholder group and established partnerships to bolster our training work.

Training for demand-side actors covers the steps in conducting an impact evaluation, how to think through a high-quality theory of change, what types of evaluation questions can be answered, and how impact evaluation evidence can be used to develop better policies and programs. Training for supply-side actors addresses the more technical aspects of impact evaluation methodologies, emphasizing quantitative techniques and hands-on work with econometric software.

Several types of training were organized for high-level officials in countries including Benin, Côte d'Ivoire, and Senegal, and at the request of certain organizations such as eBASE in Cameroon. In total, more than 500 actors in the evaluation ecosystem have benefited from capacity-building in impact evaluation through the WACIE program. In addition to high-level officials, the participants include students, lecturers, researchers, consultants, and evaluators. The capacity-building events are conducted in several formats, such as in-person or online meetings, and sometimes following a semi-virtual model. One of the latest events implemented in Benin in Laboratoire d'Analyse et de Recherche sur les Dynamiques Économiques et Sociales (LARDES) at the University of Parakou engaged with participants in two phases. The first was a one-week virtual phase, and the second was a one-week in-person phase. As a best practice, the WACIE program collects participant assessments of the training and follows up with them whenever possible.

holistic engagement

WACIE and CLEAR for Francophone Africa's partnership to build capacity

Cooperating with other organizations in Francophone Africa is another aspect of WACIE's strategy. A partnership with the CLEAR Center for Francophone Africa is a particularly successful example, which enabled the implementation of the Regional Programme for Training and Support to Impact Assessment (Programme Régional de Formation et d’Appui à l’Évaluation d’Impact, PREFA-EI). This initiative supported 20 members of public administrations of countries with impact evaluation projects in the agriculture and rural development sector, who were coached in the development of impact evaluation protocols in four projects. The teams then submitted their proposals for funding to the Fund for Innovation in Development (FID), which is still reviewing the proposals. This approach ensured that we could avoid the situation in which impact evaluation projects conducted by researchers do not correspond to the priority needs of decision-makers. The PREFA-EI initiative allowed us to start from the needs of decision-makers in specific sectors to promote the usefulness of the data that impact evaluations will produce.

As one illustrative example of how this process looked, one of the projects submitted for FID financing aims to evaluate the Market Gardening Development Support Project in Benin. This project was initiated and implemented by the Ministry of Agriculture, Livestock and Fisheries of Benin in partnership with the International Fund for Agricultural Development (IFAD) to contribute to the sustainable improvement of food and nutritional security and poverty reduction in Benin. The evaluation seeks to understand and estimate how the project has impacted the yield of market garden produce and the annual income of market gardeners in the target area. PREFA-EI helped develop the mixed-method research design that would respond to these specific policymaker needs. The quantitative analysis will use the matching method to construct the comparison group and estimate the effects. The qualitative analysis will focus on analyzing the project's theory of change using focus groups with project stakeholders.

The organization of a regional conference on impact evaluation in Francophone Africa from October 25 to 27, 2021, with the additional support of the Independent Development Evaluation (IDEV) of the African Development Bank (AfDB) and FID, was an opportunity to bring together practitioners and policymakers from various backgrounds to discuss the role and contributions of impact evaluations in the implementation of public policies. It succeeded in raising awareness in some countries and organizations on the use of impact evaluations to produce more rigorous evidence to inform decision-making. For example, after participating in the conference, the Republic of Gabon requested support and technical assistance from the WACIE program to develop the use of evaluative evidence in the country.

WACIE Helpdesk: Linking decision-makers to impact evaluation evidence

Beyond these efforts to promote the capacity to commission and conduct impact evaluations, our work has also aimed to build decision-makers' capacity to use evidence in their policy choices. While significant evidence exists, it is often scattered in various databases, divided into technical reports, and difficult to find or understand. A World Bank report on the use of their reports, for example, found that one-third of the reports had never been downloaded (Doemeland and Trevino 2014).

The WACIE program's response to this problem has been to develop an evidence use assistance service called the WACIE Helpdesk. On the premise that decision-makers need quick answers and a way to transform technical and sometimes contradictory data into practical and actionable information, the helpdesk service delivers evidence in the form of Rapid Response Briefs drawn from existing literature.

Several Rapid Response Briefs have already been developed and disseminated through capacity-building activities. For example, one recent brief addresses a question from stakeholders in the Government of Benin about that country's technical and vocational education strategy and has led to follow-up discussions on improving the program. The success of this initiative is a testament to the growing interest in evidence among decision-makers in Francophone Africa.

Working with UN partners in Sudan

Although the stakeholders and circumstances are quite different, our impact evaluation capacity-building efforts with UN staff and partners as part of the PeaceFIELD initiative in Sudan have highlighted similar lessons. With our work starting in early 2020, the fast-changing political and security situation has consistently kept all evaluation and project staff busy. Arranging to fit impact evaluation training into people's packed schedules required making it relevant to the work they were engaged in. Integrating impact evaluation capacity-building, impact evaluation design efforts, and work on ongoing M&E has yielded benefits for all three workstreams.

The PeaceFIELD initiative is a collaboration between the UN Peacebuilding Fund (PBF), the International Security and Development Center (ISDC), and 3ie, with support from the German Federal Foreign Office. The initiative aims to conduct three impact evaluations of pro-peace interventions supported by the PBF, increase engagement with evidence in the peacebuilding policy community, and build impact evaluation capacity among PBF partners. With Sudan as one of our identified country cases, we are working closely with in-country UN colleagues to design and implement an impact evaluation of PBF-supported projects in Darfur. Six other UN organizations implemented these projects, namely, the Food and Agriculture Organization (FAO), the International Organization for Migration (IOM), the United Nations Development Programme (UNDP), the United Nations Human Settlements Programme (UN-Habitat), the United Nations High Commissioner for Refugees (UNHCR), and the United Nations International Children’s Emergency Fund (UNICEF). In parallel, we are working to build capacity within these UN organizations and their local partners.

Coordinating efforts with our in-country colleagues required recognizing the day-to-day challenges they were facing. There was ample demand for training in evaluation methods and practices, but the logistics were challenging. With a process involving staff at multiple UN agencies, delays were inevitable due to conflicting schedules. We had multiple consultations and bilateral meetings with in-country partners to identify the right audience and ensure maximum attendance. The target audience also started with varying capacities and professional backgrounds. For example, we had technical M&E staff, as well as implementation staff, as participants. Therefore, we needed to calibrate our efforts to help more advanced colleagues learn new skills while bringing everyone up to the same level of understanding. Recognizing the infrastructure limitations faced by colleagues in Darfur, recordings of each session were available for those who could not attend.

Despite the virtual nature of the training, it was important to have in-country engagement because we wanted to build lasting professional relationships with the participants. Having an on-the-ground presence for key meetings was essential. This rapport has helped us maintain contact since the training and has opened new information channels on PBF project implementation. We can now better identify the interests of our in-field stakeholders, gaps in organizational capacities, and the best ways to build them up moving forward.

The result of these coordinated planning efforts was our PeaceFIELD Sudan virtual training, which ran for five days with one two-hour session each day. The training syllabus was structured to start with the theoretical aspects of impact evaluation tools as an overview and then go into how these tools applied to the Darfur evaluation and the ongoing M&E activities staff are engaged in. The latter two sections were especially interactive with applied case studies and exercises that linked theory to Darfur evaluation work's specific case. As a result, we saw increased engagement from participants as we moved to applying impact evaluation tools to the specific case in Sudan. In particular, the interactive exercises based on the team's monitoring indicators were essential. Participants were very active in these sessions because the discussion directly related to their work, how the tools can be applied to their M&E plans, the challenges of application in the Darfur context, and how the tools could be adapted to overcome these challenges.

This training simultaneously helped us move forward with the impact evaluation, helped UN partners improve their M&E practices, and built capacity for subsequent evaluation work. Following up the training, we have contacted key personnel to discuss the political, social, and ethical challenges of data collection in Darfur and how to overcome them. As a result of this training, UN partner M&E staff updated their existing monitoring tools to provide the more detailed data necessary for impact evaluation design. Furthermore, the training informed our ongoing collaboration with IOM on designing the endline survey for the impact evaluation.

We hope to engage policymakers and practitioners in discussions on evaluating peacebuilding efforts, challenges of evaluation methods, and what approaches can be advantageous in such contexts. By discussing the issues that matter to busy colleagues now, we are also aiming to build evaluation capacity for the future.

Ideas for the future

Meeting the Sustainable Development Goals across Africa requires the capacity to implement—and evaluate—development policies across the whole continent, not just in a handful of high-capacity countries. This process requires qualified human resources trained in the design, implementation, and use of the results of rigorous impact evaluations. Despite this real need, the concepts of evaluation in general, and impact evaluation, are rarely or not at all integrated into academic training. It is important to set up a training series system. One-off training sessions are generally suitable for providing an overview but are limited to in-depth learning. In addition, potential actions to improve capacity in impact evaluation include a scholarship system to support training efforts and advocating for the introduction of evaluation concepts into university training programs. Establishing an expert pool or mentoring system would allow more advanced local researchers to create a center of excellence in evaluation. This center could host student interns and even conduct capacity-building activities. Another area to explore would be the placement of research assistants in impact evaluation studies. This process would allow for induction of promising young students.

These proposals must cater to the needs of the actors in evaluation systems in regions where capacity is currently limited. In many cases, actors in evaluation are stretched to find time for on-site training in impact evaluation approaches unless it is directly applicable to their work. Both the WACIE program and the PeaceFIELD initiative show that recognizing the constraints of low-capacity environments can lead to win-win efforts that solve immediate problems and build skills to reduce capacity disparities in the long term.

Author(s)


Bio

Bio MAMA is a Research Associate at 3ie and is involved with the implementation of the West Africa Capacity-building and Impact Evaluation (WACIE) program, in addition to supporting 3ie’s London office in conducting systematic review and Evidence and Gap Map research. He has eight years of experience in designing, planning, implementing and evaluating public policies. Prior to joining WACIE, Bio worked as research officer at the Benin Prime Minister’s Office, contributing to the institutionalization of evaluation in Benin and ensuring that policymakers got access to relevant evidence for decision-making. Bio holds a master’s degree in Public Economy and Applied Statistics from the African School of Economics (ASE) and a master's degree in Economics from the University of Abomey Calavi. He has an interest in research on poverty dynamics, inequalities and inclusive growth.

 

Elsadig

Elsadig Idris Abdelsalam Mohammed Zin is a consultant for the 3ie PeaceFIELD initiative. Elsadig has an extensive background working on peacebuilding, humanitarian, and development programming in Sudan, including UN programming and programming in Darfur. He has over 19 years of experience in areas of coordination, democracy, civic education, and WASH. Prior to joining the PeaceFIELD initiative, he has worked in different capacities with the Danish Refugee Council, IOM, UNHCR, and Mercy Corps. He also has experience advising and working with the governor of South Kordofan State, Sudan. Elsadig is currently pursuing a Master’s in International Cooperation and Humanitarian Aid with Kalu Institute, Italy. He holds a MSc and BSc in chemical engineering from the University of Gezira.

 

Paul

Paul Thissen is an Evaluation and Communication Specialist at 3ie where he provides technical and management leadership of and support for impact evaluations, synthesis, and other evidence programs. He is a social scientist with extensive experience as a multi-methods researcher. His research has focused on the Sahel region, where he has extensive experience in Cameroon and Chad. Before joining 3ie, he served as team leader for an evaluation of USAID/OTI peacebuilding and stabilization programming in Northern Cameroon. His research has also been supported by the National Science Foundation and the US Institute of Peace. Paul holds a PhD and MA in Political Science from the University of California, Berkeley, and a BS in Economics and Journalism from Northwestern University.

 

Rosaine

Rosaine N Yegbemey is a 3ie Research Fellow and an Assistant Professor at the Department of Rural Economy and Sociology of the Faculty of Agronomy, University of Parakou, Benin. His research and teaching interests include impact evaluation, monitoring and evaluation systems, mixed-methods research and climate change adaptation. His regional focus is on West Africa. Rosaine has been a consultant and advisor for the National Agricultural Research System of Benin, the Africa Rice Center, the World Bank’s Gender Innovation Lab, Care Canada, TechnoServe Benin, the GIZ, the United Nations Development Programme in Benin, 3ie and the Center for Evaluation and Development (C4ED), among others. He has published in leading journals, including Land Use Policy, Agronomy for Sustainable Development, Climate Risk and Management, Climate and Development, Renewable & Sustainable Energy Reviews, and Agricultural Systems. He holds a PhD in Agricultural Economics and Rural Development from the University of Giessen, Germany.

 

Sara

Sara Ansari is a Research Assistant at 3ie where she provides research assistance support for projects affiliated with the UN Secretary-General’s Peacebuilding Fund (PBF). She also supports capacity building and creation of engagement materials. She has prior experience working on monitoring and evaluation for the education, labor, and health sectors in India, Myanmar, and Papua New Guinea. She has led various capacity development exercises with NGOs and research organizations in Myanmar. She has also supported mixed-method research projects for ActionAid, Save the Children, World Vision, and others. Sara holds an MSc in Development Economics from the School of Oriental and African Studies (SOAS), University of London, and a BA in Economics from St. Xavier’s College, Mumbai. Her education has focused on development policy in South Asia and sub-Saharan Africa.