"All models are false, . . .

. . . but some are useful.” The famous quote from George Box, a pioneer in quality control and time series analysis, perfectly sums the risks associated with advanced analytics and decision modeling/automation. The sub-prime mess is an example of the misadventures in modeling. Where financial models were seen, not as tools, but as answers. Where individual mortgages were like grains of sand – and studying the individual grains under the microscope didn’t give a clue as to what was going on in the whole sand pile. Where small changes in individual grains of sand can trigger huge and dynamic changes in the overall pile. Where complex systems tend to become more complex as time goes on – the systems never get simpler.

Modeling and decision science is a delicate balance of the core subject matter and material – combined and blended with deduction, insight, and inference. It is having the ability to recognize the difference between too simple and simply wrong. In the November 2009 issue of Harvard Business Review, Thomas Davenport outlines his thought on balancing decision tools with human intuition in the article entitled, “Make Better Decisions”:

Warn managers not to build into their business analytical models they don’t understand. This means, of course, that to be effective, managers must increasingly be numerate with analytics. At as the Yale economist Robert Shiller told the McKinsey Quarterly in April 2009, “You have to be a quantitative person if you’re managing a company. The quantitative details really matter.” Make assumptions clear. Every model has assumptions behind it, such as “Housing prices will continue to rise for the foreseeable future” or “Loan charge-off levels will remain similar to those of the past so years.” (Both these assumptions, of course, have recently been discredited.) Knowing what the assumptions are makes it possible to anticipate when models are no longer a guide to effective decisions.

Practice “model management,” which keeps track of the models being used within an organization and monitors how well they are working to analyze and predict selected variables. Capital One, an early adopter, has many analytical models in place to support marketing and operations. Finally, cultivate human backups. Automated decision systems are often used to replace human decision makers – but you lose those people at your peril. It takes an expert human being to revise decision criteria over time or know when an automated algorithm no longer works.