Wednesday, February 25, 2009

Bingeing at the Information Buffet

My friend Jeff Koeze (who also makes the world's best peanut butter) sent me a link to a talk by Barry Schwartz at the TED conference. In his talk , Barry notes the exceptional abundance of intelligence and knowledge among those present at the conference. But he argues that good judgment, not intelligence, is the binding constraint these days.
I agree with Barry that raw intelligence and wisdom are not necessarily correlated. In fact, my own experience is that the variance in wisdom around the mean grows as raw IQ rises. Really smart people can have insights that can be breathtakingly innovative - or completely disastrous.
So I wonder about the effect of the huge increase in knowledge coming from a particular dimension of the web. I am not talking about Facebook. I am talking about sites such as TED Talks, which makes available hundreds of riveting talks by leading (and often photogenic) scientists and intellectuals. Or Edge (my favorite) , a website that publishes short papers, interviews, and debates with some of the smartest (but often less camera-ready) people in the world. Or Marginal Revolution, a blog written mainly by the super-human Tyler Cowen, who each day spews forth an alarming volume of links and his own strong opinions on everything from the economic crisis to Mexican painting to the best cheap restaurants in Northern Virginia.
I partake of these sites like I used to partake of all-you-can-eat buffets when I was in college: I cram in a huge array of different appetizers, meat, fish, and pasta courses, followed by a brownie, a piece of apple pie, and a blackberry cobbler to finish it all off. Then I stagger home, completely bloated and feeling queasy, swearing off the buffet forever. But the next week I am back, bingeing again.
We have made big strides in computing capacity - both hardware and software. Gmail went down the other day for a few hours, and it made news, because these days it is rare that even large, highly networked websites crash. Such websites can process information that is growing at an exponential rate, with nary a hiccup.
But have we made similar strides in either the hardware or software of the human brain? Ray Kurzweil argues we will augment human cognition with machine intelligence at an accelerating pace. But few would argue that the human brain's capacity is growing as fast as in the increase in information available to it.
So the question is this: how will we process all this knowledge that is available to us? The good news is that much innovation comes from the recombination of existing technologies and approaches, so the increase in knowledge about these can in principle lead to accelerating innovation. And heaven knows we are in dire need of to address the economic crisis and challenges such as climate change. I also believe in the power of the Wisdom of Crowds in many contexts; that is one of the principles on which GlobalGiving is founded.

But what can we do to both increase the mean and reduce the variance of the wisdom emanating from all this knowledge? Sometimes I wonder whether the sheer quantity of new information we now "consume" will increase the variance in our judgement and wisdom, leading to unforeseen consequences.