Data mining as it is evolving in retail is a fascinating exercise in identifying behavior characteristics that apply to very small percentages of the shopper population, and doing something with them. Progressively retailers are getting better at leveraging the data, and as the penetration of cards increases past a critical mass, so will the effectiveness of the marketing and promotional programs. Of course, consumers are well aware of this, and have well developed “relevance meters” built in.

Consider the category management of potatoes. Pretty dull stuff? no, fascinating stuff.  I am making these numbers up to illustrate the point, but consider, of 100 customers using their cards at the checkout,  perhaps 10% have potatoes in their trolleys, and 10% of that 10% have a particular variety, and of that 10% (now down to 0.1%), they also have sour cream and chives in their trolley.  Pretty reasonable guess that the potatoes will be cooked in their jackets, with sour cream and chives garnish, particularly if the shopper is single, no kids, and also buys steak.  An opportunity to offer the consumer a deal on a bottle of red wine on her way out of the shop, or in the associated retailer across the way? Multiply that by 5 or 6 million cards, and you have a pile of data to mine.

The gold standard of retailer card data mining is Dunhumby, now owned by UK retailer Tesco. They did such a great job in the development stages of the Tesco loyalty card, that the retailer bought them to keep their competitors away from them. In a move that recognises the future, Dunhumby is now crowdsourcing ideas via Kaggle, a fascinating startup that turns data mining into a competition for data nerds.

This is Category Management on steroids, and represents a monumental change in the skills needed by FMCG suppliers deal with dominant retailers. In the Australian context, very few FMCG suppliers have any idea of the power of the data tsunami coming at them, and how this will impact on their brand marketing strategies. It is also the realisation of the vision of category management the few of us who were playing with this stuff  30 years ago had when the data was warehouse withdrawals, we had a bit of U&A consumer research, and managed it all with calculators.