The Dangers of #BigData are Human and so is The Solution
By it’s very nature Big Data is huge volumes of information in varying formats.
With the help of technology we have transitioned from data being scarce to a state of superabundance .
When you consider the speed at which people are now creating data.
From the 175+ million tweets that are created every day to the 48 hours of new video uploaded to YouTube every minute… and that’s only social media!
More than 215 million high-resolution MRI scan are created a day, while Whole Slide Imaging scanners used in pathology laboratories create terabytes of images per day.
Data is truly superabundant!
The challenge with this unimaginably vast amount of digital information being captured, processed and shared by sensors, computers, mobile phones and IoT devices is that –
- it’s beginning to exceed reasonable storage allocations
- it’s hard to protect and keep secure
- it’s shared ever more widely through more systems globally
- it’s easier to lose the Single Source of Truth (SSOT)
At a business and decision-making level the abundance of data is actually creating practical issues.
Problems with Big Data
Business Intelligence projects are regularly faced with the challenges of bringing together incompatible big data from siloed systems and uncovering problems with incomplete data.
Workarounds create suspect and largely untrustworthy information; and how many C-level Executives do you know would willingly put their reputation on the line by making a decision with this sort of information?
Then there’s the other problem of analyzing data to produce data findings that aren’t actionable or the discovery of an incidental finding (finding additional data that you weren’t actually looking for).
In business, big data incidental findings happen a lot – the data is collected for a particular purpose and during analysis a significant pattern or trend is detected.
The serendipity of incidental findings is magical.
A business can’t survive and thrive on opportunistically finding a treasure now and again.
It needs real information that you’re intentionally pursuing, have KPIs for and is supposed to be driving your decision-making.
Information from Big Data is Rarely a Fact
They say “a little knowledge is a dangerous thing”.
In reality, “treating information from your big data as Fact is a dangerous thing”.
It’s not that your data is lying, it’s that you need to have the full context for the data in the data set.
In everyday language, that means ensuring everyone’s understanding of what needs to be entered in a particular field is the same; that everyone is calculating a specific metric in the same way.
Even with careful analysis, the risks are high that your information from big data is wrong.
From medical research we’ve discovered that if you have enough data to examine that eventually you’ll find a statistically significant relationship where no such relationship actually exists .
As Greg Layok from West Monroe Partners once said in an interview on the topic – “Data did not lead us down the wrong path when most models predicted Hillary Clinton to win the (US) presidency in 2016: The underlying assumptions, were incomplete, and there were significant data collection issues, leaving predictions open to wide margins of error.
It was a case of bad data and unchecked models.”
Overcoming the Human Problem in Big Data
You may have begun to realize that quite a lot of the problems with big data actually stem from the people within the process.
From laziness to prejudice, human behavior impacts on the Veracity of your Big Data.
Moving forward with Productivity Improvements, Sales Growth and other Business Initiatives using automation and artificial intelligence is dangerous if you don’t overcome the big data human problems first.
- Getting the data entry right from the start,
- Making sure the context is appropriate for the data set used,
- Undertaking analysis within intended purpose,
- Using inferred and derived conclusions appropriately (as a test hypothesis for example),
- Letting go of obsessions about answers only being in the data.
Adding Human Insights to Give Big Data Value
Your people know a lot about their jobs, your company, the industry you’re in, your customers and so much more.
This arms them with useful insights to test, verify and validate against your big data findings.
The serendipity of an informed human insight can change everything and, just like an incidental finding, bring you a revolutionary scenario out of which you must make a decision.
Because the truest value of big data is its ability to reveal information upon which you must make a decision.
If your big data isn’t helping you find out what’s going to happen next so that you can make the right decisions, arrange an obligation free 90-minute confidential consultation with Semantia.
Call 1300 766 328.