Insights from the World of Impact Assessment

aa-taj1

A former Bocconi student, Paola Mallia, 25 years old, now works as a Research Analyst at the International Fund for Agricultural Development (IFAD). I’ve known Paola since our first days of the bachelor’s degree in Economics and Social Sciences and when I found out that she had jumped from classes in Policy Evaluation and Development Economics to a position at IFAD, I thought that she could share some insights on her hands-on experience in Development.

Paola, first of all, can you provide us with a bit of background information on IFAD?

The internal structure of IFAD can be broadly divided into two main parts: one which is more operational and another one that is more research oriented. The operational departments are responsible for the design and implementation of the projects, even though the implementation is not entirely managed by the Fund. When IFAD decides to finance a project, the funds are allocated to the local government, which has the task of setting up an implementation unit, called Project Management Unit (PMU). IFAD will then continue to monitor the project and to disburse funds in several tranches. So, IFAD plays an essential role in the initial stage, when, after being approached by a member state seeking for a solution to a development problem, it helps the local government to design the project. Basically, the country brings the development problem, while IFAD offers its expertise. So, the design of a project always moves from a specific request and tries to follow a theory of change that, starting from the development problem,  provides inputs and activities to be implemented, which lead to the outputs, then the outcomes, and eventually (and hopefully) the impacts. The intended impacts, such as poverty reduction and the achievement of food security, are strictly linked to the Strategic Objectives of IFAD: increasing rural people’s productive capacities, increasing benefits from market participation, and strengthening the environmental sustainability and climate resilience of the economic activities. Although nutrition improvement is not a direct objective, it is gaining increasingly more attention since it is clear that it’s closely related with income and poverty.

I’ll give you an example of this approach. I’m currently working on a project in the poorest region of Tajikistan. The country was one of the poorest members of the former URSS and after independence, in 1991, the poverty trends increased sharply due to the abrupt termination of economic support from the Soviet Union and to an extended civil war, which compromised economic development.  What makes the situation even worse is the geography of the country: more than 90% of the surface of Tajikistan is covered by mountains, so the key coping strategy for the smallholder farmers is livestock ownership. After the fall of the Soviet Union, there has been a transformation of the livestock husbandry system from one based on intensive livestock farming to an extensive one, which has worsened an already compromised situation of limited availability of feed, low animal yields and degraded pasture land. So, with the help of the government, IFAD has set up several projects, which have been implemented over the years. First of all, they reformed the land titling system in order to end conflicts over land between villages. Then, they focused on the rehabilitation of degraded land by setting up a system of pasture rotation. They have also provided farmers with improved livestock breeds and offered training on fodder production. Moreover, IFAD always tries to add a women empowerment component to the project. In this context, since in the majority of the households men migrate to work in Russia, women were offered training programs in order to learn additional activities, such as wool processing and production of dairy products.

Can you tell us a bit more about your current job?

I am currently working for the Department of Research and Impact Assessment. The impact assessment part of our work is strictly related with the projects implemented by the Fund. We cannot carry out an evaluation for every single project, but IFAD tries to set up a portfolio of impact assessments that covers the heterogeneity of interventions and places. For the selected projects, the impact assessment is aimed at understanding whether they really triggered an effect on the target population, coherently with the original design, and whether they met the strategic objectives of the Fund.

In practice, we step in at the beginning or at the end of the project depending on whether we are carrying out an ex-ante or an ex-post evaluation. We often try to combine the two things: for instance, when there is a project which is about to end and at the same time a similar one is about to start in the same region, the latter can provide a good control group for the former. When IFAD decides to conduct the impact assessment for a project, we first get in contact with the Country Project Managers (CPMs), who are the direct link between the PMU and IFAD and have to give their consent to it. The biggest challenge is precisely to convince them about the utility of an impact assessment and to convey that it’s not an inspection, but rather something they can gain from. We are only now starting to see a growing willingness to learn from what was done and to understand what could be changed and improved for future projects, but of course not everyone wants to do it. Once the assessment has been agreed upon, the next step is to go to the field and talk with the PMU to investigate whether the implementation actually reflected what was reported in the design document.

The first thing we do is explain to the PMU how the evaluation will be carried out, how important it is to have a good control group, and how to select it. We also start conducting interviews and focus groups with the beneficiaries of the treatment, which are aimed at gaining further insights on the actual implementation of the project, for instance by asking them how they found out about the intervention or why they took part in it. At this stage, it is essential to gather as many different perspectives as possible. The people in the PMU usually know very little about impact assessment and do not fully understand its importance, for example the benefits of randomization or of selecting an adequate control group. So, during the first meetings they often try to be as accommodating as possible, for instance trying to convey that they adopted a scientific approach. It’s really hard to understand to what extent they are reporting things truthfully. This is why it is so important to maintain a well-rounded perspective and to talk with many different actors. Later stages concern the design of all the different aspects of the impact assessment plan, from the sample and survey design, to the outline of a timeline and budget. We then go back to the field to conduct a pilot and focus groups to test the survey questions and then train the enumerators responsible for the data collection.

What is the most challenging aspect of your job?

What I’ve found most difficult isn’t concerned with the tasks I’ve been asked to perform, but rather with the communication part of my job. Having to interact with and explain my work to people who don’t have a background in economics has turned up being especially challenging, as well as something university hadn’t really prepared me to. While in academia – say when you are presenting a paper – you tend to talk with people who share your background and who know what you are talking about if you mention propensity score matching. Instead, when I talk with people who are in charge of more operational aspects, they may know little about impact assessment. Evaluation is often understood as simply collecting data at the baseline and in a subsequent follow up only for the treatment group. You then have to explain that a control group is needed and make them understand its importance for the validity of the impact assessment. Communication is definitely a huge part of our job. To give you an example, when you present the results of your impact assessment, you must be able to communicate them effectively, so that your audience is able to understand them and in particular to understand what are the limits of the evaluation and to what extent the findings are applicable to other contexts. And it’s not so easy to make your work understandable in non-technical terms. Finally, even within your own team, you are surrounded by people with diverse educational and cultural backgrounds and this makes you appreciate the importance of those soft skills that were mentioned in management classes: being able to bond with your team, and to show leadership skills while at the same time positively relating with others and respecting their different views. In a nutshell, being able to work in team and having great interpersonal and communication skills proves to be essential.

What has been the most rewarding aspect of this experience so far?

As I mentioned, I am part of the Research and Impact Assessment Department. I’ve told you about the Impact assessment part, which is more related to the projects carried out by IFAD, but then there is the research part of our job. Going back to the project I was telling you about, it may happen that we go to a country and carry out an intervention with the specific purpose of increasing livestock productivity, but later find out that children’s education also improved significantly and male migration to Russia declined thanks to increased employment opportunities in their own village. So, we end up using the same data collected for the impact assessment to answer other research questions, which are also relevant from a development point of view, even if they do not represent the expected impact of the project. They are more like unintended outcomes, which are important because they allow us to pursue independent research projects. So, one of the most rewarding aspects is given by the fact that my job is not limited to the application of impact assessment methodologies, but it also offers me exciting opportunities to investigate other research questions.

Do you have any piece of advice for our aspiring development practitioners to conclude this interview?

I would recommend them to always be proactive and perseverant, to keep asking themselves questions and try not to discard ideas that seem too challenging or too trivial to pursue, but rather to discuss them with professors or other practitioners. Moreover, to cultivate relationships and contacts, since opportunities to work on interesting projects may arise in the most unconventional ways.

Michelle Acampora

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s