I went to an aid conference last year where the speakers took turns ridiculing each other's work. The speakers in question were all among the top in their field, and all had tenure, so to my mind they had little reason to be insecure. And many of them are even personal friends! The atmosphere seemed almost childish - though most everyone there was over forty.
I asked one of them afterwards why they all seemed intent on attacking and triumphantly exposing a shortcoming or flaw in the others' presentations, with the implication being the whole approach was baloney. Why not build on what was valid in others' presentations and then refine one's own view, so as to more quickly advance their understanding of the world?
"That's not the way it works," my friend told me. "You advance knowledge by mud-slinging, not by listening. That's the way it has always been and always will be."
I just can't accept this. People need to get past their own egos and actually talk to - and learn from - each other, especially when the stakes are high. In that context, I was really happy to have a good conversation about RCTs (Randomized Controlled Trials) with some friends and highly qualified colleagues on Facebook. I learned a lot from it, and it may even generate a paper. I am reproducing my original post, along with their comments (with their permission) below.
That is Alanna Shaihk guest blogging at AidWatch. She describes two limitations to evaluation discussed by Steve Lawry of the Hauser Center at Harvard. Excessive reliance on evaluation, Lawry says, stifles innovation and artificially constrains aid agencies to initiatives that can be easily measured with data.
It’s a standard trope of this blog to point out that there’s no panacea in global development. That’s true of impact evaluation, too. It’s a tool for identifying worthwhile development efforts, but it is not the only tool. We can’t go back to assuming that good intentions lead to good results, but there must be room for judgment and experience in with the quantifiable data.
I would add a third limitation. Formal evaluations, including the gold standard of randomized controlled trials, are not scalable. We simply do not have the time and resources to do centralized, in-depth evaluations of everything. The only way forward is to establish a decentralized, implicit form of evaluation in which beneficiaries and other stakeholders can provide feedback about quality and relevance of aid projects.
This is how markets work. The magazine Consumer Reports does a great job of evaluating products. But it evaluates a miniscule proportion of all the products produced each year in the economy. So who evaluates the other 99.9999% of products? The consumer. If the consumers buy a product, it keeps getting produced. If not, it doesn't. Does this system work perfectly? Of course not. Does it work better than any alternative we have found? By far.