Firstly, apologies for the title of this blog but I’ve had some internal feedback that my usual titles have been somewhat deficient in the key words that might attract the sort of pensions oriented Googlers and Tweeters we’re hoping for to our website, so I’ve decided to go for it this time.
Anyway, we’ve blogged before about the abuse of statistics that seems commonplace in the press, including the specialist pensions press.
But I couldn’t resist the headline describing some recent research by Friends Provident which boldly declaimed that “27 million Brits confess they’re clueless about private pensions”.
Confession is by its nature a very personal act. The only reasonable interpretation of the headline is that 27 million “Brits” (which I assume is lazy shorthand for people resident in the United Kingdom, regardless of how they define themselves) have personally declared to Friends Provident that they are clueless about private pensions.
The latest population statistics for the UK suggest that there are approximately 61.4 million “Brits”, of which roughly 20% are under 16. So that’s roughly 49 million adults in the UK.
Friends Provident tell us that the 27 million “Brits” in the headline are the equivalent of 58% of those surveyed. Assuming Friends Provident ignored the under 16s, and I think that is something that is to be generally encouraged, I worked out that the survey population is about 46.6 million. This is quite a relief as I didn’t recall being interviewed by Friends Provident, and was bit worried that I was either forgetful or unpopular. Clearly, I’m one of the 2.4 million “Brits” that Friends Provident didn’t manage to speak to. Did they manage to speak to you? No? What a coincidence!
The truth behind the headline is that Friends Provident, or more accurately, a company acting on their behalf, surveyed 2,462 people. So a more accurate headline is “1,428 people who may or may not define themselves as British said in a survey that they were clueless about private pensions”. Not quite as dramatic.
And we don’t even know who these people are or how the sample was selected. I remember carrying out similar surveys whilst at university. They highlighted a few flaws in the whole attitude survey process. At least 25% of any survey was generally completed by the students themselves as a result of failing to get their quota of real respondents. Another 50% consisted of people that the students had met in the pub while having lunch. Any remaining balance was made up of people who didn’t have anything better to do in their lives other than to agree to answer inane surveys from spotty students, about topics which they had never really considered, but were still prepared to express a, usually incoherent, opinion on.
There is also the Bradley effect to take account of, which basically states that people lie in opinion polls and surveys all the time.
I’ve read all the counter arguments from pollsters and statisticians about random sampling and techniques to overcome all the flaws inherent in surveys of this kind and, whilst I’m not suggesting that the Friends Provident survey was carried out in the manner of some of my university experiences, I remain unconvinced about the value of simply extrapolating results from a small sample to the UK population as a whole, on any subject.
It’s easy to overlook that there is a real issue behind the headline which needs to be addressed. We are a financially illiterate nation and as the world increases in financial complexity then we need to be ensuring that people are provided with sufficient education and information to make informed decisions about financial transactions. This is a challenge for all of us working in the financial services sector and the wider world of education.
But keeping matters in a lighter vein on this occasion, the Friends Provident headline represents a pretty spectacular leap from the data. I’d be delighted to hear of your favourite example of the appalling misuse of statistics.
The best example (and I will be the sole unchallengeable judge in this matter) posted on our website or via our Twitter account, by 31 October 2009 will receive a cheap bottle of bubbly courtesy of Spence & Partners.
To paraphrase Sergeant Stan Jablonski, I’d encourage you to do it to them before they do it to you!