I read Jim Manzi’s Uncontrolled over the weekend. Manzi is the co-founder of Applied Predictive Technologies, a software company that empirically tests new promotions and products for consumer-focused businesses. Two insights from Uncontrolled stand out.
***
1) Imagine you’re a senior executive of a company that operates 10,000 convenience stores, 8,000 QwikMarts and 2,000 FastMarts. On average, QwikMart earns $1.1 million in annual revenue while FastMart earns $1 million. The difference might seem small, but it’s statistically significant enough to investigate. You’re tasked with recommending if the company should rebrand FastMart stores to QwikMart. Can you attribute the extra $100,000 in revenue to the QwikMart brand? Or are other variables at work?
It’s a simple question with a complicated answer. There are hundreds of variables to consider, including “physical size of the store, relative quality of merchandise at each competitor store, number of parking places, traffic count on the road in front of the store, ease of access from the road, distance to nearest highway, visibility of store and signage, … and so on, in practical terms, ad infinitum.”
If you controlled for every variable, Manzi writes, a model that predicts consumer behavior based on historical data would still run into three problems: omitted variable bias, prevalent high-order interaction effects, and variable intercorrelation. In other words, the convenience stores exist in “an environment of high causal density.” No matter how diligent you are, you’ll never know if your analysis is correct.
Full article -250words.com – more below
Uncontrolled – Description
Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society by Jim Manzi
How do we know which social and economic policies work, which should be continued, and which should be changed? Jim Manzi argues that throughout history, various methods have been attempted—except for controlled experimentation. Experiments provide the feedback loop that allows us, in certain limited ways, to identify error in our beliefs as a first step to correcting them. Over the course of the first half of the twentieth century, scientists invented a methodology for executing controlled experiments to evaluate certain kinds of proposed social interventions. This technique goes by many names in different contexts (randomized control trials, randomized field experiments, clinical trials, etc.). Over the past ten to twenty years this has been increasingly deployed in a wide variety of contexts, but it remains the red-haired step child of modern social science. This is starting to change, and this change should be encouraged and accelerated, even though the staggering complexity of human society creates severe limits to what social science could be realistically expected to achieve. Randomized trials have shown, for example, that work requirements for welfare recipients have succeeded like nothing else in encouraging employment, that charter school vouchers have been successful in increasing educational attainment for underprivileged children, and that community policing has worked to reduce crime, but also that programs like Head Start and Job Corps, which might be politically attractive, fail to attain their intended objectives. Business leaders can also use experiments to test decisions in a controlled, low-risk environment before investing precious resources in large-scale changes – the philosophy behind Manzi’s own successful software company.
In a powerful and masterfully-argued book, Manzi shows us how the methods of science can be applied to social and economic policy in order to ensure progress and prosperity.
Uncontrolled – Review
David Brooks, New York Times
“[Manzi’s] tour through the history of government learning is sobering, suggesting there may be a growing policy gap. The world is changing fast, producing enormous benefits and problems. Our ability to understand these problems is slow. Social policies designed to address them usually fail and almost always produce limited results. Most problems have too many interlocking causes to be explicable through modeling. Still, things don’t have to be this bad. The first step to wisdom is admitting how little we know and constructing a trial-and-error process on the basis of our own ignorance. Inject controlled experiments throughout government. Feel your way forward. Fail less badly every day.”
Wall Street Journal
“[O]ffers much to digest….
Uncontrolled is at its most provocative…when Mr. Manzi considers the largely unmet potential of controlled experimentation to improve outcomes in social science and government policy…. A vigorous book, pulsing with ideas.”
Arnold Kling, National Review
“The ideas in this book are important…. This is a provocative book for people who are interested in how social science relates to public policy.”
Forbes
“One of Hayek’s “old truths” is that individual freedom is an indispensible means to both human flourishing and material progress and that it is threatened by misguided government bureaucracy. We are fortunate to have it restated extraordinarily well in today’s language in…Jim Manzi’s
Uncontrolled…His observations offer genuinely original insights into longstanding political and social problems.”
Tyler Cowen, Marginal Revolution
“This is a truly stimulating book, about how methods of controlled experimentation will bring a new wave of business and social innovation.”
The American
“This book is one of the most powerful challenges to progressive political impulses I’ve read in a while.”
The New Republic
“In the first two thirds of his book, Manzi describes the historical development of the RFT [randomized field trial] and its philosophical basis, and includes a digression on the philosophy of science. The argument will be familiar to empiricists and philosophers, but it may interest a popular audience, and is well done and readable…. A more ambitious argument emerges in the last part of the book. Manzi argues that the RFT – or more precisely, the overall approach to empirical investigation that the RFT exemplifies – provides a way of thinking about public policy. This is the most imaginative and interesting part of Manzi’s book.”
Andrew Sullivan, The Daily Beast / The Dish
“It’s a fresh, dense and fascinating exploration of what the policy implications of a true ‘conservatism of doubt’ would mean. I hope it can jumpstart a conservative intellectual renaissance.”
Kirkus Reviews
“A thoroughly argued, powerful study based on principles independent of the author’s own conservative-libertarian views.”
Library Journal
“If social scientists entrusted with informing policymakers utilize more experiments, Manzi argues, the policies they create will be more effective over the long term. Simply put, adopting a trial-and-error methodology can help businesses, policymakers, and society as a whole. Backed by numerous pertinent examples, Manzi’s arguments are convincing. Recommended for anyone interested in policymaking or in how businesses make decisions.”
Booklist
“This challenging book highlights the astounding advances in science and technology that have started to be used in social-program evaluations.”
Conor Friedersdorf, The Atlantic
“If
Uncontrolled were merely a restatement of the need for epistemic humility among wonks and legislators, interest in it might be confined to the right. The book is of broader interest, and may turn out to be important, because its author makes a compelling argument for an ideologically neutral method for improving policy, one that left and right might both plausibly embrace, even as it challenges both sides to rethink some of their reflexes…. [
Uncontrolled is] the rare political book that goes out of its way to raise the most powerful objections to its arguments and to point out the limits of the reform program that it recommends.”
Kenneth Silber, The Daily Beast
“Jim Manzi’s
Uncontrolled is an intriguing investigation of the power, limits, and varieties of empirical knowledge…. [A] substantial part of
Uncontrolled’s value is in its sharp thinking about how various disciplines seek reliable knowledge….
Uncontrolled offers useful advice for navigating a hard-to-know world.”
Arnold Kling, National Review
“The ideas in this book are important…. This is a provocative book for people who are interested in how social science relates to public policy.”
The American Conservative
“[A]s Jim Manzi persuasively argues in his insightful and well-written new book,
Uncontrolled, humanity is terrible at foresight, and trial-and-error is the chief way humans develop reliable knowledge…. In
Uncontrolled, Manzi provides an incisive and highly readable account of how trial-and-error experimentation in science and free markets lessens human ignorance, uproots bias, and produces progress.”
Steve Sailer, Taki’s Magazine
“In his impressive first book,
Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society, entrepreneur/intellectual Jim Manzi has the makings of an airport best seller in the genre of Steven Levitt’s
Freakonomics and Malcolm Gladwell’s
Blink. Indeed,
Uncontrolled is far more reliable than those two sometimes-dubious tomes….
Uncontrolled offers one of the most lucid and sensible historical overviews of the philosophy of science I’ve ever read.”
Reihan Salam, National Review Online
“[T]he most important book of 2012 (read it now so you can be ahead of the curve).”
David McKenzie, World Bank, Development Impact blog
“[I]nteresting reading…. The lesson of trial-and-error with thousands of relatively low-cost experiments designed to make marginal improvements is one that could be useful in many government bureaucracies (and indeed in our own bureaucracy).”
Gary Gutting, The New York Times, Opinionator, The Stone
“[Jim Manzi] offers a careful and informed survey of the problems of research in the social sciences.”
About the Author
Jim Manzi is the founder and chairman of Applied Predictive Technologies (APT), an applied artificial intelligence software company. Prior to that he was a vice-president at Mercer Management Consulting. He is currently a contributing editor at National Review, where he writes about science, technology, business and economics, a senior fellow at the Manhattan Institute, and serves on a number of other corporate and non-profit boards. He has also written articles for a variety of political publications including the New York Post, the Weekly Standard, the Atlantic, and Slate. His work is regularly covered widely in the blogosphere, and his articles on why Republicans should acknowledge global warming and “Keeping America’s Edge” have become much-debated “must reads.” He lives in Paris.
