Analysis

Charles, from the excellent and recommended Wallpapering Fog blog, had an excellent post on the strengths of analysis.

I realise that I should seek JG's approval before linking to a post as in his, and apparently ONLY his, opinion linking to a post without his prior approval is a criminal matter, not just a civil one.

However, despite his self-reportedly having had two interviews with the police on the matter of my linking to a Hejik post, I am yet to receive an invitation from the Met to present my side of the story, so I shall risk doubling my sentence and fine by linking to yet another unapproved post. What a rebel.

Crime: Linking to a post

If this blog goes quiet for a few months, please send comments to me c/o Her Majesty.

Of course, it is always possible that Jonny has an over-active imagination and is as clueless about the law as he is about betting "a predictive model cannot react to events such as a red card" (see below Jonny), but Mrs. Cassini has the kettle on ready for tea should the police turn up to arrest me.

Anyway, here is the post on What Analysis Is Good At - although I would have preferred that the subject did not end in a preposition - call me old-fashioned:

It should be an easy question to answer. It should be, but it's not.

What is statistical analysis consistently good at?

I'm talking here about it's real use to the Managing Director of a company, or to the Chairman of a professional sports team, or to a politician. To somebody who has choices to make and is looking for help to make the best choice that they can.

A sceptic can easily reel off a list of things that your analysis can't do. Your analysis probably can't account for human frailty, or random chance, or a whole host of things that it was never designed to measure in the first place.

Your analysis can't forecast the effect of something that's never been tried before.

Your analysis says 'trust my numbers', but offers no guarantees of success.

And your numbers can't spontaneously volunteer new ideas; only tune up the effectiveness of old ones.

When you come right down to it, complex statistical analysis is a waste of time and effort, right?

As an analyst, I hear some of these arguments a lot. It's true that statistical analysis can't come up with the perfect strategy on its own, but it's still a hugely important tool. Here's what I think statistical analysis is really good at.

Analytics will conclusively reject a multitude of bad strategies that you might otherwise employ.

Analytics stops you making avoidable bad decisions.

Does that sound a overly negative? It doesn't have to be.

This is the scientific method applied to business and its tremendously powerful. Scientists know that you can't ultimately prove the truth of anything; that there's always the possibility that you're wrong. What you can do is falsify what definitely isn't true. All of our scientific knowledge about the world is based on theories that we're only working with for now, until we prove that they're wrong.

All of it. But just look at the progress we've made by rejecting ideas that don't work...

It's this scientific method that means we've found ways to cure many diseases, which were previously terminal. And it's the rejection of this evidence-based method that can kill people who believe strongly in homeopathy.

Do you reject analytics because the answers are obvious and it will just tell you what you already know? You're a corporate homeopath.

Rejecting ideas that don't work is real progress and a truly valuable exercise. It's how we learn; we try something, we reject it, we have a think and then we try something else until we find a method that works.

You can often spot a good analyst by the way that they approach problem solving. If you ask a good analyst why sales are declining, they'll come up with a whole host of different possibilities and then work with data to disprove them - one at a time - until they're left with the most plausible explanation. It's a process and it's the true value of analysis. It stops us from accepting hypotheses that aren't true; from blaming bad weather, or bad luck for under-performance, when really our business has systematic problems.

Sam Allardyce (the West Ham manager) talked this week about using statistical analysis in football and it's fantastic to see this type of discussion starting to gain real traction. Something that he said struck me as slightly jarring though.

"You can take out of it what you want. You can find your best performance in each area. You can find your best performance on fitness level, you can find your best performance in possession…"

It might just be throwaway phrasing from the interview, but that could also be heading firmly in the direction of confirmation bias. If you analyse your best performances, you'll find the occasions when what you tried appeared to work. Your worst performances are often a lot more valuable, because you're forced down a route of working out why they were bad and then coming up with ideas to fix them.

Very often, I find that analytics sceptics are those who are looking to confirm the effectiveness of the strategy that they're already employing. It's self-fulfilling then, that your analysis won't be able to teach you anything new. At best, analysis like this is an internal marketing tool; a way to 'prove' you're right and end any debate about other options and in the short term - until everybody works out that's what you're doing - it might be somewhat effective at that job. EMI was determinedly doing using analytics like that for the short time I was employed there. Before reality struck and it was broken up and sold.

Good analytics...

Proves conclusively that bad ideas aren't working

And so forces you to think up new ideas

Which you can then analyse to see if they're an improvement

Good analytics...

Gets you there faster. Of course you'll work out eventually that a bad idea isn't working, but wouldn't you rather know now, before it's too late?

And finally, good analytics will prompt new ideas, by giving you details about what went wrong with the old ones.

There are so many other benefits of taking an analytical approach to a problem, but this is the big one. This is what statistical analysis is really good at and this is my answer when faced with scepticism. Of course analytics can't solve every problem, but used correctly, it can solve a very, very big one.