Why don’t we know what we know?
This week I have been at the Evidence to Action (E2A) conference in Accra organised by the International Centre for Evaluation for Development (ICED). On the last day, Dean Karlan argued we should not do studies which address questions we already know the answers to – such as do cash transfers reduce poverty – but rather seek to answer more nuanced questions such as the timing, size and delivery mechanisms for cash transfers. I agree. I have long argued that impact evaluations should address what I call (and I do think this is my terminology) second generation questions of how to design interventions rather than the first generation question of whether an intervention ‘works’ or not.
But in an earlier presentation by Mercy Kamau of Tagemeo I made just one note: why don’t we know what we know? Mercy presented findings which illustrate findings I call mid-level theory principles in my presentations of mid-level theory (see for example, here). One is, don’t tell people to do things they are already doing. You can’t bring about behaviour change in such cases. And, second, don’t tell people to do things they can’t do. My favourite example of that is an MCC impact evaluation of the impact of farmer education for irrigated agriculture when the irrigation infrastructure was not yet built. But the more usual cause is the need for complementary inputs which are not available.
A presentation after Dean’s talk illustrated another favourite weak link in causal chains – not everyone is an A student. The common assumption in nearly every intervention that training ‘works’ is simply wrong, More and repeated training is needed. As the communication adage goes: you need ro tell someone something seven time in seven different ways before they get it.
The same study illustrated another more difficult problem. The intervention was a fish pond project and the fish died. This is the issue of top down externally imposed solutions being inappropriate. There is a long history of development failures such as the Tanganyika Groundnut Scheme. This problem beset colonial settlers importing cows from Surrey to Kenya, and has continued in both grand and small-scale schemes across the world. But it’s not always or necessarily so. The answer is formative evaluation rather than jumping into the a full scale RCT.
To return to the main point of this blog – why don’t we know what we know? Because the incentives for social science researchers are not aligned with accumulating bodies of evidence. As an economist, when I first started working with public health researchers I was surprised at the sort of things they would publish – descriptions of studies, prevalence data from baseline surveys and so on. What is original about that?, I asked myself. I then realised that the criterion for publication in natural and health sciences is different from that in social sciences. In health papers are published because they are useful. They don’t have to be clever in terms of contributing new ideas or methods. But in social science papers have to be clever; usefulness be damned.
The motivation for publication is in fact antithetical to accumulating bodies of evidence. A typical social science paper will start by saying there is paper A doing this and paper B which has the opposite finding. I am going to do C which is different, In health researchers would come together to say, how can we explain the difference between A and B, so we understand what the evidence actually says
This is why systematic reviews have made more headway in health, and now in the natural sciences than they have in the social sciences, particularly economics, The discipline is set up for controversy and debate. Authors have to show their contribution by being different – clever – not by building the body of evidence. The incentive system is not aligned to advancing policy relevant knowledge. This is also why null findings are not published, But a null finding is a finding -that’s a subject for another blog.
This blog is brought to you by the Research and Evaluation Centre (REC), As the saying goes: please like, share and follow, Our mission is to promote the use of evidence, and to be inclusive in our approach. If you want your voice to be heard there is space for you in our blog.
And watch our for E2A next year, It is the place to be seen. REC will be there and we hope to see you.
