Earlier this week Tim Harford, also known as the Undercover Economist, gave a fantastic talk at ODI on the topic of ‘Development as Trial and Error’. Drawing on his latest book, Adapt: Why Success Always Starts with Failure, Tim provided the audience with a compelling account of the need for a different way of thinking about and navigating complex problems. (click here and scroll down for videos of the talk and
One of his starting points was to highlight the considerable complexity of the economy, drawing on the work of Eric Beinhocker. This compared the ‘product space’ in different societies – the number of distinct products that are on offer. In hunter gatherer societies, it amounts to some 300 products. In the average Wal-Mart, it is 100,000. In a city like New York or London, it is 10 billion. (you can see Eric’s explanation of this in his presentation at a UKCDS workshop in May)
Tim used this concept, and a story about an unusual project to build a toaster from scratch, to argue that this kind of complexity can most effectively be navigated through evolutionary principles of ‘variation, selection and amplification’ that enable lots of small ‘trial and error’ experiments to be aggregated as effectively as possible.
The market was given as one example of such an aggregator, but there are obvious shortcomings. It has worked for some kinds of problems – Western affluence, most notably. But there are other problems where the market has failed, or where its influence has been far from desirable. Other examples include the scientific enterprise, consisting of diverse means of submitting and peer review of new ideas and experiments. But of course the scientific establishment is also prone to conservatism and inhibiting innovation.
So what can we say about situations where the market is not working or cannot work, where peer review mechanisms are not in place, and where tolerance of failure is impossibly high? The public sector was the example that everyone kept coming back to. How do you get the process of trial and error working in such settings? There are numerous ways Tim touched upon, including innovation prizes, creating artificial marketplaces, finding ways of supporting entrepreneurs, and so on. This is a great article he wrote for Slate, drawing from the book, on exactly this topic. As he puts it there:
Here’s the thing about failure in innovation: It’s a price worth paying. We don’t expect every lottery ticket to pay a prize, but if we want any chance of winning that prize, then we buy a ticket. In the statistical jargon, the pattern of innovative returns is heavily skewed to the upside; that means a lot of small failures and a few gigantic successes.
The paradox at the heart of the need to be more systematic (i.e. controlled,
managerial) about innovation (i.e. disruptive, entrepreneurial) is one that is
not easily overcome. Tim’s remarkable achievement in Adapt is in his clear and
eloquent articulation of this challenge, along some excellent accounts of how it
has been addressed. Let’s hope we in development and humanitarian aid are able to take his ideas and lessons on board.