How to use big data to make smart decisions

Big data is fast becoming the board room's hottest topic of conversation – everybody talks about it and everybody wants it.

Capturing and making sense of big data is a challenge, but when done right it can give your company an upper hand against your competition. There is so much unused information that can help you understand your customers and improve your proposition – the big question is how do you get it?

In this post I will explain what big data is, tell you how you should use it and what you should use it for with examples.

What is big data anyway?

Let's first tackle the question that baffles CEOs and divides opinion in the analyst community.

What is big data?

Look it up in Wikipedia and they will tell you that big data is a reference to data sets that are too large to be captured, stored, processed or analysed using traditional data processing software.

What is traditional data processing software anyway?

MySQL was only released in 1995 and most banks still use a mix of Excel and SAS to go about their daily jobs.

The two largest data platforms available to retail banks – First Data and TSYS provide access to data through green screens that look like they were last updated in 1982 (in fact they were – it says so when you log in).

When talking to data science gurus, the definitions of big data generally narrow into 2 camps:

#1 – Data that takes more time or resources to be captured, stored, processed and used than the frequency with which the data is available.

Let's say you can capture a piece of data that changes once a second. If you take 2 seconds to save that data, you then have 2 more bits of data to process once you're finished.

Once you have processed these 2, there are 4 more bits of data – see what's happening? You've got yourself into a data storage pyramid scheme.

Big data can prove to be so big that conventional ways of storing it in databases can take too long compared to how quickly new data turns up. And no amount of money can get you processors that will fix this.

#2 – Data that is not generally considered available but could be with the right setup and integration

This is the more common way for people to explain big data. For a retail business, examples would be social media data about your customers, sophisticated on-screen behaviour data or cross-functional data about a customer's interaction with thousands of other websites and businesses

Data that has never before been available to an industry can be the difference between the companies that succeed and fail. If you know that much more about your customers then surely you should be able to have a competitive advantage in sourcing your customers, targeting your marketing and offering services that better meet your customers' needs.

What is the value of big data?

Everybody is telling you that you really have to use big data, but few people say exactly what you should do with it or why it is that you need to use it.

What is the actual value of using big data?

To answer this question, let's cover what you might want to use big data for.

In most cases, using big data will not help you design an amazing product. It is unlikely to help you build an efficient operational base or streamline financial planning processes. In fact, big data is unlikely to significantly affect almost any part of your business.

Except the bits that directly interact with your customer.

Big data is all about your customers – all that information and data describes and tells you that little bit more about where you customers are, what it is that they look like and how they may interact with your or other products.

The value of introducing big data will affect your customer journey and all the key touch points where your business and your customer directly interact.

How can big data affect customer journeys?

The primary points at which your business will interact with the customer are:

· Initial marketing

· Acquisition funnel

· Product selection

· Direct product use

· Customer retention

We don't have enough time to dive too deep into each of these steps, but how exactly can you use big data to improve these interaction points?

For initial marketing and acquisition funnels, the more you know about your customers, the more you can zone in on each individual customer's path and tailor it to suit their needs and wants.

Marketing 20 years ago was a one size all approach – film one advert for your product, stick it everywhere you can on TV and billboards and wait for the right customers to see it at the right time and come buy your product.

Today, the advertising market is changing fast. Suddenly the majority of online advertising is tailored to your profile and advertisers use hundreds of variables about you to sell you products.

Do you like things to do with your favourite music band on Facebook? Prepare to see concert ads for similar bands playing in your local town.

Have a beard? Prepare for photo parsing AI software to advertise beard trimmers to you on every site you visit. This crazy science fiction is happening right now and fast expanding to TV and display advertising.

How can you improve acquisition funnels with big data?

There are two key differences between the marketing strategies of the past and new approaches using big data.

Firstly, you are able to source a large part of your new customers from channels where big data can be directly used in targeting.

Facebook pixels, Google tags and other tools are becoming virtually ubiquitous on big websites. This means that before you even click through to visit a website through their ad, the company already knows far more about your profile than you might think.

They know your social circle, what websites you visit, what kind of things you like and even your political affiliation.

That means that you can segment your inbound customer journeys into a large number of targeted pathways.

Imagine you're a retail bank selling the full suite of typical banking products – current accounts, loans, credit cards, mortgages, insurance and the like.

When your super-targeted customers are landing from these ads that already segment them based on big data, you will know whether the prospect is a young student with a part-time job or couple in their 50s.

Do you need to show everybody the same landing pages where the majority of products have no relevance?

If you already know the customer is employed, is there any point in asking the question in your application form? Are they more likely to give the more accurate answer on LinkedIn or your loan application form?

So how DO you capture and make use of big data?

Big data is a vast pool of information that is too large to capture, too large to process and certainly too large to keep in a traditional database. So how do you go about storing and using it?

The answer is through smart service and system integration.

In the past, every bit of information organisations stored would live in a standardised relational database. Every customer would have their line in the relevant table and all of their data would be stored for every day, transaction or customer-level entry.

But what happens if big data relates to large customer groups or doesn't lend itself to being stored in tables?

Let's say you want to capture what the customer's twitter activity looks like to understand what they are interested in. Are you going to store 29.5k tweets with all of their photos and metadata.

Are you going to store the links from those tweets? What about the people who liked and shared those tweets? Do you want to store their profile data and all of their tweets too?

This is where suddenly the volume and usability of data becomes a major problem. Even if you did manage to store all of this data in a SQL database. What exactly are you going to do with it? Ever tried running queries parsing long text strings on a large table? No processing power will help you here.

The answer is both fairly straight-forward and complex at the same time.

Big data can be stored and processed simultaneously by picking out key information from the data at the point of extraction and it can also be stored differently.

Today's data storage and manipulation techniques have evolved exactly for this reason. If you heard about things like Hadoop and Spark, these are the tools that people use to manage information from big data. A lot of the way these systems work is still relational, but the data is captured and processed in customised clusters and does not have to be structured.

I regularly help big companies make effective use of these systems in their journey to making big data work for them. I think there's a whole separate post on getting this set up that I will write later on.

What other sources of big data are there?

There are a lot more sources of big data than it might seem on the surface. The two most commonly referred to sources are social media and the broader public web.

These have heaps of information that are hard to process – the perfect example of big data sources.

But there are a lot more places where you can look for big data.

Document archives for your organisation, other publicly accessible organisations or government bodies are a big data source that is sparingly used. Technology limitations are probably the main reason why, but nonetheless, there is a lot of information there.

Photos are a huge source of big data in the future. A picture tells a thousand words is certainly something that applies to making use of data in photos. Device used to capture the image, key details about what is in the photo, the way the photo is taken and what is going on in the background can provide a huge amount of information.

On top of these, there is log data from various machines, servers and systems; all manner of sensor data and general public media that is accessible.

For a while yet, there is far more information out there than is being processed or used. The more of this information your organisation learns to use, the more competitive advantage you will get.

How can a big organisation integrate big data?

So you're ready to make use of big data – you want to reap the benefits of building smarter products and being better at marketing. But how do you actually go ahead and integrate it?

A lot of organisations run on technology stacks that are a couple of decades old if not even older. Not only is the server setup unlikely to be friendly to huge swathes of data passing through them, the likelihood is that that data warehouse solutions are not suitable either.

The majority of large organisations still run on MSSQL or similar structures sitting on a physical server somewhere within their premises.

The answer is that this is the time to make the leap and upgrade your platform.

You don't have to move all of your data in one go. When working this type of project with our clients, most often the best and fastest way to get going is to build a brand new data environment in the Cloud of your choice.

Then build a live link and either replicate existing data as necessary or build a robust API to call for the data that you need. Now go ahead and build new environments for your big data capture and processing solutions in the same cloud structure.

This approach means you should not face any disruption to your BAU business and the timescales for implementing big data are not dependent on your entire business having to rebuild and migrate platforms.

Even better, in the process of building this new stack, you have already done a lot of the work required to make the necessary upgrades so why not make improving data platforms your next big target?

Do you actually use the data you already have?

Here is an amazing fact: your organisation already has a tonne of big data available to you. And you most probably make no use of it whatsoever.

It's already sitting there, somewhere in the depths of your data structures covered by a thick layer of dust and cobwebs.

It's easy to chase the popular sources of big data such as social media networks and browsing behaviour, but dig into your business and you can uncover some real gems that can tell you a lot of information that you didn't previously know.

Here's a few examples.

Let's say that you are a big online retailer. You have a customer who accesses the service after 6pm or at weekends only. They also order items to be delivered to the address of their credit card at weekends and a different address in the centre of their local city for weekday deliveries. They frequently log onto the site between 8.00am and 8.30am in the morning from their phone from different IPs but the same device cookie.

Does your organisation know that this is a commuter who uses some form of public transport and works 5 days a week in the city? Do you target products based on this information? Do you offer the customer incentives based on when they are going to log in next? Do you tailor their mobile experience in the morning based on their desktop activity the night before?

Can you dig further? Sure. HTML headers and app data will tell you exactly what phone and software they use to access your store.

You already know that this customer is in the market for a new phone right now. Their handset is just over two years old has been fully paid up and their battery is declining fast.

You know all of this and the customer hasn't even logged in yet today.

Think about your business. How much data do you already capture that you just never properly look at? Do you analyse transaction timestamps? Do you actually use information about the customer's browsing activity on your site?

If you don't already, take a look and see what you can find. It is amazing how quickly you can improve conversion rates and increase ROI by making use of information you already have.

How to make smart decisions at the right time

The process of making the right decisions is a whole topic in itself, but let's briefly cover what you can do on the back of knowledge that you're about to obtain from using big data.

The main difference to your organisation can be the method in which certain customer decisions are made.

Today, most customer management strategies or changes in marketing strategy are taken on an ad-hoc basis.

This means that managers make changes or run programmes when it suits them, NOT when it suits their customers.

You know when you get one of those emails offering you a discount on new suits. It will come at midnight on Saturday at a point in time when you have no need in a suit whatsoever.

The company probably approved and created this email the week before and some random software sent it out when the clock ticked over.

You go on to ignore this email. Worse still, you might well unsubscribe from the mailing list and you might even feel more negative about the store brand that sent you this email.

But what if that same email hit you at a time that was right for you? What if the store used a mix of big data and found out you just got promoted at work through updated statuses and social media content? Might you want to make the most of your new role and dress the part?

Or perhaps things went the other way and you just lost your job/are scheduling interviews. Can a great suit at a decent price make all the difference to your confidence?

Suddenly the people you are microtargeting feel that the product is super relevant, and your response is likely to be very different.

You can also be much smarter with your marketing budgets and target broader groups before your micro targeting kicks in while returning much greater ROIs.

Conclusion

Big data is here and you can start your journey in using it today. Market leaders are dipping their toes in already and the availability of cloud technology means your business can take full advantage fast.

From marketing strategies or optimising the value of your existing customers, companies that take big data onboard and make the most out of it will come out on top.

Those who have more information and use that information better always win.

So why not make sure you are that winner?

Sure, it can feel daunting working with new concepts and information, tools and systems that may be hard to understand. But that is exactly what separates the innovators from the dinosaurs. Big data is that differentiator so don't get left behind.

How are you going to use big data in your organisation?