In my previous article on the topic of Big Data I explained how we need Smart Questions, Smart Interfaces and Smart Analytics to make Big Data worth our while. The response was overwhelmingly positive and there was excellent feedback, some of which I am including in this article.
Having large amounts of data is one thing, but if not implemented properly within a Big Data framework it comes with some really big risks. The picture above is a poster that explains in broad terms the Smart Facets of Big Data (you can download the A1 version for free – link below).
Each facet requires its own technologies and experts to function effectively. The facets consist of Data Collection, Integration, Analytics (Question Set and Answer Set) as well as Explication (which is really just another word for detailed explanation). When these facets are not properly developed and managed they lead to potentially disastrous risks. Here are some of the biggest Big Data risks:
The Data Collection Facet
Heather Jack commented on the previous article that we need Smart Governance – that is to say, we need to ensure that the data that goes into the Big Data system can be trusted
- Your Big Data lacks integrity – Data is generated by people and people make mistakes, which is why things like typing errors and empty fields exist. These small errors create data integrity risks – I always remind my clients that the difference between 10 000 and 100 000 is a single zero.
- Your Big Data lacks metadata – This is an issue that is becoming increasingly critical in Big Data applications. Developers sometimes use shorthand to describe the fields in their databases, thinking that it is only for their reference, but more and more Big Data applications will require metadata to make sense of what is contained in each field. Without metadata your analysts may be using incorrect fields to build queries!
The Integration Facet
- The back-end is cheap – I am not saying that you should buy the most expensive back-end tech available, but this is one area where size definitely matters, and it is a matter of getting the right size. Crunching the numbers requires good hardware and tested software that is optimised for Big Data applications. The risk here is twofold – you may spend too much money on buying systems you may not need, or your systems cannot handle the load and crash at critical junctures.
- The front-end is confusing – Analysts are generally very smart people and they find their way around some of the most complicated software on the market. The risk here is threefold – firstly complicated systems that require many steps will inevitably lead to missteps, secondly they slow the decision making process down, and thirdly they create “knowledge kingdoms” run by analysts that may leave your employ and take their knowledge with them.
The Analytics Facet
- Your analysts don’t understand your question – This is more of a soft issue. Analysts don’t always have the skill to translate “normal language” into “query language” – they are often technically adept, but they may lack insight into practical business.
- The analysis is incomplete (and you don’t know it) – Analysts may feel pressure to provide information that fulfils an expectation. Selective reporting is possibly one of the most critical issues executives face today – rather than risk their careers, analysts may obscure or leave out critical elements in analyses and deliver positive results. This will always lead to decisions that perpetuate problems and possibly lead to your company’s demise.
The Explication Facet
Arden Manning highlighted this point – Big Data requires clear interpretation. Without it you are bound to make ineffective or even negative decisions. Sufyan Barghouti also indicated that Big Data makes sense if it is properly contextualised.
- You lack a means to interpret the analyses – The analysts may produce wonderful reports that are disconnected from reality. I have known analysts that simply shrug and say “That is what the data says” and leave it at that. If you don’t understand the outputs of the analysis then you may risk reading things into the results that may not be there.
- You don’t act on the analyses – Mark Doutre responded to the previous article by pointing out that every analysis must lead to a clear a decisive action. If you are simply gathering reports on your big data and throwing them in the bottom drawer, you might as well be throwing cash in the shredder.