There is certainly a large amount of noise at the moment regarding big data, especially around what it can do, its challenges and how it could change the world for the better.
However, like most new concepts and ideas, one has to maintain a certain amount of suspicion around any new technology idea. This is because a) new ideas often have a large amount of hype and therefore under-deliver; b) people cannot see anything wrong with new idea and tend to overlook its shortfalls and c) people often jump on the bang wagon and ‘re-badge’ other ideas as the one, typically for commercial reasons.
This article investigates what big data is, what it can be used for and the challenges with its implementation.
What is big data?
If one were to search the internet, you would likely find hundreds, if not thousands, of different definitions of big data. However, the following three trends seem to underpin most definitions:
- There is a massive volume of data. While size and volume are often relative to circumstances, we are talking in the range of millions of data items, often with hundreds of data variables within each data item.
- The data is constantly changing; often at a rapid pace. New items are being added, updated and removed quickly.
- Finally, the data is stored in a variety of different formats. This will cover the more ‘traditional’ pre-defined structured database formats but also a wide range of unstructured formats, such as videos, audio recordings, free format text, images, social media comments, etc.
Once this data is collected, then it is possible to undertake various forms of analysis. This analysis will find patterns, trends, themes and correlation between variables. This analysis can then be used to explain historical behaviours as well as to predict and shape future behaviours. A few simple examples are listed below is illustrate this point:
- Governments obtain insights to help them with healthcare analysis.
- Meteorologists can use big data to predict and understand weather conditions.
- Political parties can utilise big data to understand voting intentions.
- Medics can try to understand the cause and spread of diseases. This will allow preventative measures to be implemented. (Very topical at the time of writing in regard to the COVID-19 pandemic.)
- Finally, big data can help with the ‘normal’ functions of a business. For example, cost/profit management, marketing / product management, improving the clients’ experience and internal process efficiencies.
In fact, big data can be used to efficiently monitor, analyse and predict trends in most areas of life.
