Insights from the World of Impact Assessment

aa-taj1

A former Bocconi student, Paola Mallia, 25 years old, now works as a Research Analyst at the International Fund for Agricultural Development (IFAD). I’ve known Paola since our first days of the bachelor’s degree in Economics and Social Sciences and when I found out that she had jumped from classes in Policy Evaluation and Development Economics to a position at IFAD, I thought that she could share some insights on her hands-on experience in Development.

Paola, first of all, can you provide us with a bit of background information on IFAD?

The internal structure of IFAD can be broadly divided into two main parts: one which is more operational and another one that is more research oriented. The operational departments are responsible for the design and implementation of the projects, even though the implementation is not entirely managed by the Fund. When IFAD decides to finance a project, the funds are allocated to the local government, which has the task of setting up an implementation unit, called Project Management Unit (PMU). IFAD will then continue to monitor the project and to disburse funds in several tranches. So, IFAD plays an essential role in the initial stage, when, after being approached by a member state seeking for a solution to a development problem, it helps the local government to design the project. Basically, the country brings the development problem, while IFAD offers its expertise. So, the design of a project always moves from a specific request and tries to follow a theory of change that, starting from the development problem,  provides inputs and activities to be implemented, which lead to the outputs, then the outcomes, and eventually (and hopefully) the impacts. The intended impacts, such as poverty reduction and the achievement of food security, are strictly linked to the Strategic Objectives of IFAD: increasing rural people’s productive capacities, increasing benefits from market participation, and strengthening the environmental sustainability and climate resilience of the economic activities. Although nutrition improvement is not a direct objective, it is gaining increasingly more attention since it is clear that it’s closely related with income and poverty.

I’ll give you an example of this approach. I’m currently working on a project in the poorest region of Tajikistan. The country was one of the poorest members of the former URSS and after independence, in 1991, the poverty trends increased sharply due to the abrupt termination of economic support from the Soviet Union and to an extended civil war, which compromised economic development.  What makes the situation even worse is the geography of the country: more than 90% of the surface of Tajikistan is covered by mountains, so the key coping strategy for the smallholder farmers is livestock ownership. After the fall of the Soviet Union, there has been a transformation of the livestock husbandry system from one based on intensive livestock farming to an extensive one, which has worsened an already compromised situation of limited availability of feed, low animal yields and degraded pasture land. So, with the help of the government, IFAD has set up several projects, which have been implemented over the years. First of all, they reformed the land titling system in order to end conflicts over land between villages. Then, they focused on the rehabilitation of degraded land by setting up a system of pasture rotation. They have also provided farmers with improved livestock breeds and offered training on fodder production. Moreover, IFAD always tries to add a women empowerment component to the project. In this context, since in the majority of the households men migrate to work in Russia, women were offered training programs in order to learn additional activities, such as wool processing and production of dairy products.

Can you tell us a bit more about your current job?

I am currently working for the Department of Research and Impact Assessment. The impact assessment part of our work is strictly related with the projects implemented by the Fund. We cannot carry out an evaluation for every single project, but IFAD tries to set up a portfolio of impact assessments that covers the heterogeneity of interventions and places. For the selected projects, the impact assessment is aimed at understanding whether they really triggered an effect on the target population, coherently with the original design, and whether they met the strategic objectives of the Fund.

In practice, we step in at the beginning or at the end of the project depending on whether we are carrying out an ex-ante or an ex-post evaluation. We often try to combine the two things: for instance, when there is a project which is about to end and at the same time a similar one is about to start in the same region, the latter can provide a good control group for the former. When IFAD decides to conduct the impact assessment for a project, we first get in contact with the Country Project Managers (CPMs), who are the direct link between the PMU and IFAD and have to give their consent to it. The biggest challenge is precisely to convince them about the utility of an impact assessment and to convey that it’s not an inspection, but rather something they can gain from. We are only now starting to see a growing willingness to learn from what was done and to understand what could be changed and improved for future projects, but of course not everyone wants to do it. Once the assessment has been agreed upon, the next step is to go to the field and talk with the PMU to investigate whether the implementation actually reflected what was reported in the design document.

The first thing we do is explain to the PMU how the evaluation will be carried out, how important it is to have a good control group, and how to select it. We also start conducting interviews and focus groups with the beneficiaries of the treatment, which are aimed at gaining further insights on the actual implementation of the project, for instance by asking them how they found out about the intervention or why they took part in it. At this stage, it is essential to gather as many different perspectives as possible. The people in the PMU usually know very little about impact assessment and do not fully understand its importance, for example the benefits of randomization or of selecting an adequate control group. So, during the first meetings they often try to be as accommodating as possible, for instance trying to convey that they adopted a scientific approach. It’s really hard to understand to what extent they are reporting things truthfully. This is why it is so important to maintain a well-rounded perspective and to talk with many different actors. Later stages concern the design of all the different aspects of the impact assessment plan, from the sample and survey design, to the outline of a timeline and budget. We then go back to the field to conduct a pilot and focus groups to test the survey questions and then train the enumerators responsible for the data collection.

What is the most challenging aspect of your job?

What I’ve found most difficult isn’t concerned with the tasks I’ve been asked to perform, but rather with the communication part of my job. Having to interact with and explain my work to people who don’t have a background in economics has turned up being especially challenging, as well as something university hadn’t really prepared me to. While in academia – say when you are presenting a paper – you tend to talk with people who share your background and who know what you are talking about if you mention propensity score matching. Instead, when I talk with people who are in charge of more operational aspects, they may know little about impact assessment. Evaluation is often understood as simply collecting data at the baseline and in a subsequent follow up only for the treatment group. You then have to explain that a control group is needed and make them understand its importance for the validity of the impact assessment. Communication is definitely a huge part of our job. To give you an example, when you present the results of your impact assessment, you must be able to communicate them effectively, so that your audience is able to understand them and in particular to understand what are the limits of the evaluation and to what extent the findings are applicable to other contexts. And it’s not so easy to make your work understandable in non-technical terms. Finally, even within your own team, you are surrounded by people with diverse educational and cultural backgrounds and this makes you appreciate the importance of those soft skills that were mentioned in management classes: being able to bond with your team, and to show leadership skills while at the same time positively relating with others and respecting their different views. In a nutshell, being able to work in team and having great interpersonal and communication skills proves to be essential.

What has been the most rewarding aspect of this experience so far?

As I mentioned, I am part of the Research and Impact Assessment Department. I’ve told you about the Impact assessment part, which is more related to the projects carried out by IFAD, but then there is the research part of our job. Going back to the project I was telling you about, it may happen that we go to a country and carry out an intervention with the specific purpose of increasing livestock productivity, but later find out that children’s education also improved significantly and male migration to Russia declined thanks to increased employment opportunities in their own village. So, we end up using the same data collected for the impact assessment to answer other research questions, which are also relevant from a development point of view, even if they do not represent the expected impact of the project. They are more like unintended outcomes, which are important because they allow us to pursue independent research projects. So, one of the most rewarding aspects is given by the fact that my job is not limited to the application of impact assessment methodologies, but it also offers me exciting opportunities to investigate other research questions.

Do you have any piece of advice for our aspiring development practitioners to conclude this interview?

I would recommend them to always be proactive and perseverant, to keep asking themselves questions and try not to discard ideas that seem too challenging or too trivial to pursue, but rather to discuss them with professors or other practitioners. Moreover, to cultivate relationships and contacts, since opportunities to work on interesting projects may arise in the most unconventional ways.

Michelle Acampora

Satellite Data – A New Star in Development Economics

image1

“Without data you’re just another person with an opinion” – A friend of mine eagerly quoted Deming while I was complaining how data scarcity had been such a pain in development economics. The quote became my favorite for a few days.

The lack of data is a nightmare, and this nightmare is particularly scary for researchers in development economics. A lot of time developing countries lack resources or incentives to construct and maintain reliable databases – For example, data are often not available at a subnational level; data are not collected during wartime or recent post-conflict periods; or for the worst part data have never existed (and probably will never do…). Moreover, at times development topics themselves can be too sensitive or too “irrational”, which makes finding a good data source a tricky task. How could you conduct a survey with the local Indonesian government on their corruption through illegal logging?

Luckily many economists have broken the spell of data scarcity: some have creatively designed and collected field data, while others have found ingenious statistical methods to trace the forgotten trove of data. Among these techniques, satellite data is a new star. Coupled with the advancement of computer science, the technological evolution in satellite and remote sensing instruments has created a bust of novel spatial datasets, which cover diverse topics that are beyond the scope of traditional empirical data – data on Uppsala armed conflicts locations, historical Chinese boundaries and tribal areas, South African drainage network, to name a few interesting spatial datasets. Moreover, since 2014, for the first time American companies are allowed to sell images with resolution below 0.5 meters to non-governmental agencies. This has allowed researchers to gain access to a higher degree of variations in a wider range of topics.

However, satellite data has its own pitfalls. Like any other types of data, satellite data can be subject to measurement errors, and privacy concerns. Further, spatial dependence can plague the whole empirical analysis if it is not well treated. And lastly, from my experience, extracting and managing spatial data are unfortunately not a piece of cake.

Given its pros and cons, satellite data has become an increasingly common tool to answer unlikely questions in development economics. To name a few, Burgess et al (2012) use satellite data of deforestation to prove that an increase in political jurisdictions increases deforestation in Indonesia. Harari and La Ferrara (2015) exploit 1-degree gridded cells of weather data to pin down the spatially contagious effects of droughts on armed conflicts in Africa. Marx, Stoker and Suri (2015) use satellite data of sunlight reflection on the roofs to show that ethnicity matters for the housing market in Kenyan slums – co-ethnicity between residents and the tribal chief reduces the price and increases services of the slums. And lastly, by combining weather data with georeferenced data from Demographic and Health Surveys, Kudamatsu, Persson, and Stromberg (2016) estimate the effects of weather variations on infant mortality.

If satellite data sparks your interests, Donaldson and Storeygard (2016) provide a great systematic literature review on the recent applications of satellite data in economics. For now, read on to find out some hands-on experience with satellite data in development economics projects.

Long Hong – Master graduate, Bocconi:

  1. In which development economics project have you used spatial (satellite) data?

I have used spatial data for one chapter of my thesis. The research question is: at a disaggregated level, how ethnic segregation and fractionalization affect local conflict. In particular, by using “Geo-referencing ethnic groups” (GREG) and high-resolution population data from Gridded Population of the World v.4, I have calculated the ethnic segregation and fractionalization at 1 x 1 degree cell level. Also, the conflict data from the Armed Conflict Location and Event dataset (ACLED) contains detailed geographic information, which allows me to locate each conflict event.

  1. Why did you decide to use spatial data instead of “traditional” economic datasets?

For my project, using spatial data allows me to conduct my analysis at a subnational level instead of the traditional country level for ethnicity and conflict. And in general, there are no sub-national data for many variables such as population and GDP.

  1. Some advantages and disadvantages in using spatial data for your project?

Advantage: It helps me understand how the spatial distribution of ethnic groups, which is measured as ethnic segregation, affects conflict. Also, it gives a very nice visual representation of the data and helps me gain a better sense of the data simply by looking at the maps.

Disadvantage: First, the software, ArcGis, is not very user-friendly, although it is not hard to pick up. Second, since the spatial data is usually very large, sometimes it takes time to process.

  1. Any favourite Stata commands that help specifically dealing with spatial data?

You can use -shp2dta- and two other packages to import georeferenced data to Stata. For more information, please read http://www.stata.com/support/faqs/graphics/spmap-and-maps

  1. Do you have any recommendations for spatial data sources?

Yes, there are many sources: DIVA-GIS, UCDP-GED, NASA website, GRID-PRIO v2, Gridded Population of the World, etc.

Lara Engelfriet – Consultant, OECD:

  1. In which development economics project have you used spatial data?

For my thesis, using crowd-sourced datasets I analyzed the effect of urban form (city size, urban density, land-use mix, polycentricity and spatial clustering) on the cost of commuting expressed in distance and time in large Chinese cities. Studies on European and the U.S. cities have demonstrated that travel behavior is influenced by urban form. Based on these findings, policies steering the shape of cities have been proposed to reduce urban transport emissions and limit congestion. Such policies can also be relevant for the rapidly growing and motorizing Chinese cities. Yet, empirical evidence on the relationships between urban form and car usage is scarce for the specific Chinese context.

  1. Advantages and disadvantages of using spatial data for your project?

The advantages were that we could include more indicators for urban form than just aggregate measures such as population density or city size. Those indicators are essential to define the form of cities, because the internal resources distribution in a city can be an important determinant, say for commuting and accessibility. To name an example: the cities of Los Angeles and New York have the same average population density. However, those cities are very different in terms of the internal distribution of resources: New York has much higher density clusters of people/jobs/businesses than L.A. This, in turn, determines for a large part the length of commuting trips and CO2 emissions.

Disadvantages were that the availability of spatial data is scarce for Chinese cities. Therefore, we had to make use of crowd sourced datasets, which may be subject to certain biases.

  1. Do you have any recommendations for spatial data sources?

For studies on urbanization, built-up area and population data can be very useful, which are currently publicly available for a worldwide coverage and with high resolution (as detailed as cells of 12 meters). By combining these two types of data, it is possible to infer for each cell whether it is urban built-up area or not and how many people live there.

Furthermore, crowd-sourced datasets can be helpful in regions where data sources are scarce (in my research on Chinese cities I used crowd sourced data from the Beijing City Lab).

  1. Any additional suggestions or comments?

Use software that is specially developed to analyze spatial data! ArcGIS is a very good and easy to use program. It makes very nice maps as well! However, licenses are very expensive. So QGIS, the open source version, is a very good alternative with almost the same functionalities, however, a bit more difficult to use. Both QGIS and ArcGIS have built-in Python consoles, which can be very useful if you want to automate your tasks.

Robert Grundke – Economist, OECD:

  1. For which project that have you used spatial data?

The title of the paper is “Coerced Labor in a Global Supply Chain: How Higher Commodity Prices (Don’t) Transmit to the Poor”. In this project, I investigate whether the land privatization reform has affected the pass-through of cotton world market prices to rural labor markets. 

  1. Any specific benefits from satellite data for your project?

A more robust identification. The empirical strategy is to identify which municipalities that can and cannot grow cotton. In the household survey, there is information pre-shock on whether municipalities grow cotton or not. But this status might be endogenous to other municipality-level variables like labor supply and wages, which we use as dependent variables. So using georeferenced data on the suitability of land for cotton production from the FAO Global Agricultural Ecological Zones (GAEZ) database was an important step to exogenously identify cotton and non-cotton municipalities (treatment and control group).

  1. Any shortcomings in using spatial data for your project?

A potential disadvantage is that matching geographical data to communities in the household survey might be time-consuming (e.g. wrong coordinates for communities in the survey, imprecise information in the GEO data).

  1. Any additional comments?

Always check the coordinates for communities and the preciseness of the GIS data! In my case, the database was highly erroneous with respect to coordinates for communities. More than 50% of the communities had wrong coordinates and I had to retrieve the correct coordinates from old Russian maps and other Tajik website information.

To conclude, satellite data can mitigate some parts of data limitation in development, but it is not a cure-all solution to approach development economics. The other side of the coin is that without an opinion, you’re just another person with data. A great interest in development is key after all.

Jacqueline Nguyen

How Economics saved me from a cultural shock

india-i-mese-119

On the 28th of February 2014, I opened my eyes in the pale light of a 7 a.m. morning as a lullaby came from the street. It was the song of a junk dealer that every day woke up the sleeping streets of Kolkata, and now it had become my personal alarm clock. Every day I lingered for a while under my bed net, trying to recall to my mind where, why and how I got there. The former question was simple to answer, I was in India. But why was I there? I didn’t know it yet. If someone had asked me the same questions a month before, when I was still in Italy in my comfort zone, no doubt I would have spoken as fast as I could in order to motivate my imminent exotic trip. At that period, I was going to complete in few months my B.A. in Economics and I was feeling a bit frustrated about all the stuff that I had learnt in classrooms. Ok, let’s tell the truth. I was not slightly frustrated… I was truly disappointed by Economics, and I was wondering why I did not choose Philosophy after high school. Sound familiar, doesn’t it? To make a long story short, I was in my just-a-bit-in-delay-rebel-period and society was the first enemy, as always. By chance, I attended a course in which the professor spoke about Grameen Bank and microcredit and… that’s it; no course of development, no academic framework provided. I decided that this was the way to go and that I needed to leave. I organized everything in a rush: the VISA, the internship, the vaccinations and I contacted a professor to do my thesis on India and microcredit. Yes, I was going to save the world, to find illumination, to become wise as a monk, and I was going to write a thesis with my own data in a developing country.
Oh, Sweetheart.

Two things became clear in my mind as the end of the first month spent in Kolkata was approaching. First, I was having a great time; each day was completely different from the previous one at IIMC-Indian Institute for Mother and Child, the local NGO settled in the rural areas around the city of Kolkata where I was doing my internship. I was mainly interested in the microcredit activities, collecting data from the microcredit branches of the Mahila Udyog Unit, their Grameen-inspired microcredit bank. Several times I followed the COs in their daily trips to the villages and I had the chance to see microcredit coming to life from books to reality. It was every time a charming vision seated on the floor of a bamboo house, a pale face in a storm of colorful saris. I will always remember the concentration stamped in the eyes of the women while handling their money, the feeling of understanding among them and that electric shiver of strong resolution that was in the air. Aside from microfinance, other countless daily experiences were shaking me; from taking a cow away from the traffic jam, to listening to the stories of the women I had the pleasure to speak with, children running free in the paddy fields; all these stimula were teasing me to arrive at a conclusion I was not able to grasp.

And here we are to the second thing I realized at the end of February: I was not any closer to figuring out it all! Despite the intensity of the past days, I felt that I was lacking a personal interpretation of my being there. I felt like a silent bystander who was watching an interesting movie, unable to gain his own place in the plot. I know now that this feeling is far from being unusual among people in the field. It comes from what my Indian friends in Kolkata used to call “a cultural shock”, a kind of difficulty to deeply adapt and make sense of a different world. The cultural shock is a very subtle “disease” as the sick person is apparently plain sailing, enjoying his exotic permanence without there-is-a-mouse-in-my-bed-stuff hysterics. However, he is just not really into it. He can make fun of odd events and actions because he cannot understand them, and he cannot interact with people on a peer-to-peer basis as he always has to position himself. Well, I was sick. And last but not least, I had wasted a full month torturing the microcredit responsible to obtain data and my basket was sorely empty. I did tell you I was sick, didn’t I?

I don’t remember the exact day I started to “heal”. But I remember the person who unintentionally inspired me the interpretation key of that experience. He was the Branch Manager of Hogolkuria’s bank and his name was Sabir, a Muslim with primary education. I had already met him several times without giving him much credit but this time was different, as I had come to know that he had the data I desperately needed. I rushed cockily into the dusty room of the bank and I blurred out something like “Listen Sir, Doctor Sujit (the President of IIMC) gave me the authorization so you have to provide me these data!”. Sabir, an amused light in his eyes, calmly replied: “Listen Madame, I don’t know if I can help you. It would take time and I can’t ask my COs to work more for something they don’t understand at their wages. I suggest you to come back here, many times, so that they can become familiar to you. And then we’ll see”. I tried everything, from international repercussions to praying Ramakrishna, but he was firm with his decision. That night, my fury against all Indians was keeping me awake. However, two words he had said somehow kept coming back to my mind: time and wage. Indeed, I was quite familiar with them from my Microeconomics I course. Turning in my bed, I remembered that the professor told us that time is a commodity with its own value, and that people can decide to give up part of their time for wage; in this exchange, time is the opportunity cost of wage. But what if you want people to give you extra time and the wage is set? I was starting to understand the point. Maybe motivation could be the way, if we think of motivation as good. “Something they don’t understand” Sabir said, motivation was out of discussion. So, how could I push these people to work more? The following weeks, I went every day to Hogolkuria. I spent every morning there, speaking to people, getting to know them and their lives. We chatted a lot about everything, we sang, ate Nutella, laughed together and I worked with and for them. Lastly, we had become friends, people who care and trust each other. And my last week, data literally flowed to me, hundreds of observations collected in a week of hard work, cooperation and fun. Sabir was right from the very beginning and Economics had been my unexpected translator. If you cannot offer neither money nor professional motivations in exchange for people’s time, you have to share something that will ensure them their time is not going to be lost. Something very fashionable in academia. Can you guess?
Trust.

Economics saved me many other times during that beautiful month of growth. From the crossing of the streets to workplace, I was now able to see how trust and human bond were of paramount importance in those contexts of limited resources. Sabir and I had a lot of fun using the Kanheman and Tversky theory to design the data collection sheets for the COs, mixing nudges and framing effects to improve their work. And it incredibly worked one more time. One day, I was helping in the hospital and I came after an 18 years old girl who was in danger of life. The mother wanted me to stay there with her, and while she was sleeping I had a lot of time to touch by hand the tragic reveal of what economists call choices under uncertainty. The girl was born with a congenital disease and the mother told me that her and the father were to retire the girl from school to concentrate more on the education of the other children. This conversation drove me to visit the schools of IIMC, where I had the strongest and most inspiring confirmation of how Economics hit the bull’s eye in saying that education is the way.  A lot of lively girls told me how their older sisters were married at their age while they were planning to go to college. Many children were able to speak in English better than their parents did. Some children wanted to know if I was African, which made me laugh and ask why. They explained to me that their geography teacher taught them that in Africa there was a desert called Sahara, a desert of golden, fluffy sand; they were completely blown away and wanted me to give more information. I took my mobile, connected to the internet and googled Sahara Desert. And there I saw the light brought by discovery sparkling in their curious eyes, while more and more children were coming around us to see the hills of the Sahara.

When I landed in Milan, I was coming back. I left to escape from a disappointing university choice, but India taught me Economics all over again. In particular, I discovered the inner beauty of our dismal science and its ability to provide an all-around framework to interpret life.
Ok, let’s be honest again. When I landed, I was not so philosophical and deep. But I remembered the first thing Doctor Sujit told us when we arrived at Sonarpur IIMC headquarters: “Please guys, do not have a cultural shock!” and how I skeptically laughed at him.
Oh, Sweetheart.

Giulia Buccione