What Does Big Data Mean?

Big Data is the term used for huge quantities of collected data and the methods used to analyse and make sense of it all.  The problem really arose when the Sloane Digital Sky Survey began its redshift study in 2001 and started to collect data from the whole sphere of space surrounding us.  Within a few weeks, it had acquired more information on the stars it observed than had ever been collected in the history of astronomy and the team running it were at a loss as to how to deal with the information. The Large Hadron Collider at CERN in Switzerland has 150 million sensors that collect data 40 million times every second in its attempts to find new and exotic sub-atomic particles and understand how matter is made up.  Both of these examples illustrate how the world of data collection and analysis has to adapt to accommodate increasingly large amounts of information, and the challenges that it represents.

With sizable chunks of information now becoming commonplace in science, medicine, Defence, and manufacturing, industries have to change how they collect and analyse data that they are collecting. This is a factor that is receiving increasing attention from the Oil & Gas industries where exploration either on land or undersea, may require the collection and sifting of huge amounts of data in order to pinpoint potential new resources.  Big Data can be characterised by a number of specific features;

Volume – Relating to the quantity of data produced, the volume is the size variable that relates to the overall size of the collected information.  It is estimated that US consumer giant Walmart deals with over a million customer transactions every hour and runs stock and consumer information databases that are in the order of two and a half petabytes, or 2560 Terabytes, of information in size, which is a pretty big volume to keep on top of.

Variety – Variety is used to catergorise the received information and may relate to database entries, chemical data, statistical information or physical aspects such as size, position, movement etc. The greater the variety, the more analysis that needs to be carried out to reconcile the information and understand it.

Velocity – This parameter dictates the speed with which the data is amassed.  While some data sets may be large they may have been gathered over many years, others may be obtained as a huge mass of data that then requires analysis.  It is expected that the Large Synoptic Survey Telescope will amass 140 terabytes of information every five days, all of which will need to be sifted and catergorised.

Veracity – This refers to the quality of the obtained information.  A geological survey may capture many terabytes of information regarding the make-up of rock formations and gas analysis data, but if it is of insufficiently quality then there is little point in analysing it. Data needs to be of the best quality, which unfortunately tends to increase its size.

Added to these factors is another called “Complexity” which dictates how data management and analysis can become a very intensive process, especially when large volumes of data come from multiple sources and at high velocity. Big Data analysis is now being utilised in an increasing number of fields and experts try to understand the data coming in from many different sources.  Knowing something is one thing but actually understanding what the data is telling you is key to getting the most from it, and helping us advance in many areas.

Written exclusively for Enigma Consulting Group

Check out more of what we can do for you at www.enigma-cg.com

Read More

It's All About The Big Data & Analytics!

EXPLAINING WHAT BIG DATA IS ALL ABOUT

The buzz at the moment is all about Big Data. But how many of us really know what it is? In layman's terms big data is a set of valuable unstructured data collected from various sources such as surveys, research, feedback and internal sources. Most data collected are from social networks, online website analytics, and customer tracking system s to name a few The big data is later processed to reveal recognizable patterns and useful information for the benefits of businesses which is also known as customer insight. The data produced is pivotal in determining the businesses’ future decisions namely on what customer to target, how and what the eventual ROI would be.

Data scientist are usually the one that will possess this knowledge after analyzing the big data using statistical software such as SAS. SPSS or KXEN. ‎

Companies are increasingly looking for useful insights from the big data they collect to further assist them in directing the path of their businesses to grow portfolios whether it be in size, revenue which will ultimately lead to increased profits via a loyalty. Big data exists from the needs of answering the important questions related to the business needs. With the right manipulation of the insights, companies might be able to boost their sales, keeping up with the competitive landscape and derive new business strategy to meet with their business goals. This will later help in improving the operation, customer service whist reducing the risks of the businesses later. By applying big data analytics, businesses can effectively increase customer retention and add up to the new customer base after they successfully tap into their new niche market.

Normally big data is essential in improving performance of the business while accelerating processes in it for more efficiency. The MD of EnigmaCG Abeed Rhemtulla highlights that this is the whole point of big data. It plays a functional role in optimizing the business performance to the optimum point as stated.

THE CHALLENGES OF BIG DATA

Despite the widely known benefits of the big data to business, according to Enigma CG many marketers and business owners still skeptical with the increasing popularity of big data. These data to them impose challenges that they have to deal in a daily basis which require them to tackle in a timely manner to avoid bigger troubles to arise later. Considering the sheer volume of data will contain almost every different shapes and form either unstructured or structured form. These data is collected across various departments and need variety of processes for the different types of data to be combined later to analyze and find patterns that leads later to useful information. This massive volume of data is proven difficult to process with the traditional databases and typical software.

These types of data required a high performance analysis from specific analytics tool that can help the mass volume of data to be processed in detail accurately from one vend to another. This process will need to go through various analytical process before a quality information and pattern able to be detected from the big data processed. Tools and processes such as predictive analysis, prescriptive analysis, forecasting and descriptive analytics will help to determine the use of the data later as it will release a recognizable pattern and insights that is vital to answer the business needs. These processes will help to separate which data is relevant and which is not to devise a better corporate decision further.

WHAT ARE THE VOLUMES OF BIG DATA WE ARE SEEING TODAY

Based on the current reports gathered by Enigma CG  that we are getting these days, it is estimated that Facebook itself approximately released over 30 billion pieces of contents generated from this past few months from over 600 million plus users. And Zynga processed over 1 petabytes of content for players every day. And just from yesterday alone it is counted over 2 billion videos have been watched. An average teenager is sending approximately more than 4.7K text messages from his phone per month. And finally over 32 billion searches are performed this month just on Twitter alone.

THE BIG DATA BUZZ IS GETTING LOUDER

As time goes by, the big data is creating quite a loud buzz everywhere and from the online searches itself you can see how many people has start talking and discussing about the new phenomenon of big data. If you observe carefully you can notice that over 112 million blog posts are written specifically to discuss about this big data phenomenon. And if you go to Google and type “What Is the Big Data” on the search term and hit enter, you will notice that there are over 1.35 billion search results that is specifically discussing things related to the big data. Over the Twitter itself if you see, there are over 120 accounts dedicated to the big data which formally discussed and share things about big data.

You can also notice that by this year, there are already more than 2 million pieces of PDF released all over the web that is thoroughly discussing about big data and these are gathered from the search results when you search for the terms “big data”. There are over 50 infographics illustrated the process and discussing about the whole thing of what is big data all about throughout the internet. That doesn’t count the increasing hits on the Wikipedia itself in the topics that just discussing about “big data” alone which currently reaching 70K at the moment. Starting from year 2012 and beyond over 9000 job searches on data scientists are performed on the internet showing the increasing awareness on the benefits of the big data, as from the recent research done by Enigma CG.

PREDICTIVE ANALYTICS

Predictive analytics is one of the branches from the advance analytical procedures that will be used in making future forecast of unknown circumstances. This analytical method derived from the multiple techniques that also includes data mining, statistics, modeling, machine learning and artificial intelligence in order to analyze current unstructured data and process them to enable data scientist to create forecast on the future. This method bring together multiple analytical techniques to bring together the management, information technology and modeling processes to enable them to make predictions of the future. Patterns identified will be later used to predict the historical transactional data which will be later used as a platform to predict the future risk and outcome. By successfully interpreting big data from the predictive analysis will help the company to better manage the future of their business in their benefits.

DESCRIPTIVE ANALYTICS

Descriptive analytics will tell us what actually the data is. This process will summarize what happen and giving you an actual insights of the data. This process will later translate into trending past and future events which will be used to devise a plan on what should they do later. Descriptive analytics will drill down the data to learn the frequency of the event occurrence, the cost of the events and the root cause of the failures which will be beneficial to get a workaround plan later on to overcome the previous failure in which the strategy often can be seen in sales division especially the one that is specifically customer-oriented. Descriptive analytics often triggered alerts and indicators that will help the team to measure up the process. This type of analytics is what usually being used in order for the company to get the bigger picture of the occurrence for their future actions.

Read More