I started life as an accountant, and luckily, recognised before anyone else that I would be the world’s worst. However, from my trials, I do have respect for numbers, proof, real outcomes tested and validated by the scientific method.
As a marketer, I have always tried to find the quantitative base of the stuff I was doing, rather than being seduced by the hyperbole, supposition, and self-interested ‘data’ presented by someone with an interest in the outcome.
Usually that someone has had a pecuniary interest in the decision. They are selling something, from a piece of machinery to an advertising campaign, to a bottle of shampoo sitting on a supermarket shelf.
Science starts with a hypothesis that you set out to prove by trying to disproving it. Having failed to disprove it, the result must be the truth. As Arthur Conan Doyle via Sherlock Holmes said ‘When you have eliminated the impossible, whatever remains, however improbable, must be the truth.‘ A sample of 12 carefully selected personnel from your ad agency does not constitute proof that your made up new natural sounding ‘extract’ will make your hair shine.
‘Data’ as used by whole ranges of marketers, advertisers, and perhaps worst of all, politicians, is often nothing like a reliable representation of the truth, it is just the opposite. It is the selective use of bits of pieces of information, (real or imagined) and contextual engineering that suits the pre-ordained conclusion that is presented. The opposite to the scientific method, in that the result is determined, then data is constructed that does as good a job as possible to ‘prove’ the outcome.
In academic and scientific circles, this is a heinous crime that will end your career.
From time to time I have not been popular as a result of asking what to me seems to be reasonable questions of those presenting ‘data’ in an effort to sell something.
What is the size and structure of the sample used?
How does the methodology replicate actual behaviour?
What controls were used to manage the data?
Where did the outliers come from, and where are they in the stats?
Have the results been substantiated by independent repeat studies?
There are a few more, but usually I only get one or two out before I am dismissed as some sort of data cretin who does not understand these things.
It is amazing to me how often I see major decisions taken on the basis of flawed, incomplete and inconsistent ‘data’ where the vested interest is clear to all who choose to look closely.
Hi Allen, I can’t tell you how much of this is true in the Australian real estate business in which I work. Most of the property focused media use Median Price data to make generalizations about how much a property has gone up in an area. Too often, a median has changed due to a big change in mix of properties sold – think of a new housing estate just startup with an an older area or a bunch of new units coming to market. A median price for units typically includes studio, 1bd, 2bd, 3bd etc… So in some cases, you may be comparing a 1 bd sale 1 year to a 2bd the next.
So what is the solution? Only use valuations done by licensed valuers vs real estate agent? Even that subjects the result to the choice of comparable properties used in the valuation.
In the US, they use an index called Case Shiller which only compares sales of the same property over time. It’s the closest I’ve seen to an index that might provide a buyer with useful data for marking their decisions.
Scott,
There is I understand legislation that makes pumping prices illegal.
Not sure how it works in the advertising end, but face to face, there are large penalties, applied to the franchise owner to close the ‘not me’ loophole.