We’ve moved to http://www.haleyAI.com

May 8, 2008

Probabilities are Better than Scores

Strategic Analytics slide from Fair Isaac Interact on 2007 mortgage meltdownDuring a panel at Fair Isaac’s Interact conference last week, a banker from Abbey National in the UK suggested that part of the credit crunch was due to the use of the FICO score.  Unlike other panelists, who were former Fair Isaac employees, this gentleman was formerly of Experian!  So there was perhaps some friendly rivalry, but his point was a good one.  He cited an earlier presentation by the founder of Strategic Analytics that touched on the divergence between FICO scores and the probability of default.  The panelist’s key point was that some part of the mortgage crisis could be blamed on credit scores, a point that was first raised in the media last fall.

The FICO score is not a probability. 

Fair Isaac people describe the FICO score as a ranking of creditworthiness.  And banks rely on the FICO score for pricing and qualification for mortgages.  The ratio of the loan to value is also critical, but for any two applicants seeking a loan with the same LTV, the one with the better FICO score is more likely to qualify and receive the better price.

Ideally, a bank’s pricing and qualification criteria would accurately reflect the likelihood of default.  The mortgage crisis demonstrates that their assessment, expressed with the FICO score, was wrong.  Their probabilities were off.More...

  • Was the FICO score a useful metric of creditworthiness before the crisis but not during?
  • Is the FICO score a reliable metric going forward?

In these mid-crunch days, Fair Isaac is reminding its customers that the FICO score is a ranking not a probability.  The underlying point they seek to make is that the relationship between the FICO score and the probability of default is more complex and dynamic than their banking customers understood last year.  (Another post on predictive analytics also discussed stationarity.)

It’s the probability that matters, not the score!

In his keynote, Ian Ayres also focused on the inadequacy of scores.

He was explicit that bankers need the probability of default and, further, that they need to know how reliable such probabilities are.  As an example, he cited polls where one candidate is leading by 6 points within a margin of error of 3 points as almost meaningless.  More meaningful would be the probability that the leading candidate will win.  Even better would be an estimate of the probability of default along with an assessment of the reliability or accuracy of that probability.

Bankers increasingly understand that the FICO score is not the probability of default that they need when originating and underwriting credit.  As a result, bankers increasingly understand that there is no adequate external source of the probabilities they need in order to optimize their portfolio performance. 

This realization has several ramifications:

  • A market opportunity for predictive analytics in credit has opened on Fair Isaac’s turf.
  • Scorecards have lost much of their “solutions” luster, becoming just another technique.

But several things also became clear as I talked with numerous practitioners last week.  Fair Isaac doesn’t have much competition.  In fact, it is shocking how little competition they have in such a large and lucrative market. 

What decisioning market?

Although there is a market opportunity for more rigorous decisioning solutions, there is no significant challenger to Fair Isaac.  I expected to hear more about the Experian Group, but the only direct competitor identified by more than one person was Austin Logistics.  Several people indicated that they were using statistical tools directly, especially SAS, and Fair Isaac itself is placing a great deal of emphasis on its own predictive analytic tools, especially Model Builder.

Note that Vantage Score is really having an impact on Fair Isaac scoring revenues, as reflected in their most recent earnings call transcript.  So there is more competition than may seem apparent to the audience that attends Interact.

Another chasm to cross

Generally speaking, this market needs the benefits of broader machine learning techniques, such as statistics, and a more rigorous understanding and emphasis on probabilities.  The audience, however, is not technically sophisticated enough to become aggressive adopters, despite recent harsh lessons.  In the same panel, every banker in turn solicited risk analysts to join their organizations, headquartered in Asia-Pacific, London, Canada, and on the west coast.  They also agreed that it is easier to learn finance than analytics.

The market for analytics is crowded with sophisticated tools and intellectually demanding techniques that are simply too hard for most people to understand and use, let alone to use effectively and reliably.  This is precisely the circumstances that decision management was in during the late nineties when business rules technology started going mainstream.  In 2000, we crossed that chasm by introducing natural language business rules (see Haley’s Authority).  At the same time, Blaze Advisor, now owned by Fair Isaac, was crossing that chasm using a form based approach called “Innovator”.

Similar advances will be forthcoming in analytics.  As with business rules, this will not eliminate the need for highly skilled consultants, but their criticality and marginal value will diminish as analytics becomes more effective in the hands of non-experts (and as better solutions develop in key markets, such as in credit, risk, fraud and other criminality or terrorism).

Until it’s easy, use expertise

If you are in this market and could use some help with modeling, analytics, or adaptive decision management, feel free to get in touch.  We have some excellent capabilities and partners in these areas.  We are happy to help recommend approaches or products, or simply to make referrals.  Of course, there are also highly specialized consultancies, such as Strategy Analytics that can give excellent implementation-agnostic advice. 

One thing worth noting, but only in passing for now, is Fair Isaac’s acquisition of Dash Optimization.  This  reflects the increasing trend towards broader and deeper application of technology within credit decisioning.  it is also a response to the decline of scoring and the increasing need for decision optimization, which is a broader subject than decision management, with or without predictive analytics and adaptation. 

Nonetheless, optimizing portfolios will not optimize profits if the scores used are not reliably correlated with probabilities.

It is also interesting how Ilog and Fair Isaac continue to converge from a technological perspective. 


Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: