We are drowning in data, and with the rise of the machines — sensors/Internet of Things, not Skynet — we will no doubt look back upon merely 100% data-volume growth-per-year with nostalgia. However, in addition to Big Data’s Vs — volume, velocity, variety, veracity and value [also, the more Vs, the varier: verbality, verbosity, versatility, viscosity and visibility] — we have a relatively new Big Data phenomenon called perishable data, information that can substantially decrease in value over a period of time.
A decade ago, high-value data was put into data warehouses, said Mike Flannagan, VP/GM, Data & Analytics, Cisco. Over the last 5 years, that data has been dumped into Hadoop, but today, “data is not going to be stored in Hadoop or data warehouses at all”, he said. “We will have data in three locations… data warehouse… Hadoop… and real-time data that will likely have to be processed and stored very near to the location.”
Industries like oil and gas that collect a lot of data locally, at wells, can use that data to extend the life of these assets. Perishable data is not for everyone, but Flannagan said a recent Cisco survey found that 37% of customers said three years from now most data (generated by IoT) will processed at the edge. Just the IoT is expected to be worth $19 trillion business, with analytics a key component of that ($7.3 trillion is tied to data, analytics and data in motion). Currently, according to IDC, less than 1% of data is analyzed.
The volume of data is one challenge, he said, i.e. sensors in oil & gas wells generate 1-10Tb a day. Another challenge “is the timeframe I need to analyze and get value from that data.” For many applications, days, weeks, months, or a quarter can be sufficient, but for others, you need to know right away.
“If you have very low latency for getting insight from your data, and very high volumes… moving that data back to the datacenter is unfeasible. It just becomes impossible.”
Edge analytics is very focused, very low latency or real time-sensitive, said Flannagan. He believes this will also create a big access problem, with storing data at the edge, and in the datacenter, with the best result generated by combining that data. “When it comes to edge analytics… no solution will operate entirely at the edge”.
More data, whether processed locally, at the datacenter, or some combination is a huge opportunity for Cisco, for its networking, datacenter and relatively new analytics businesses. Addressing the perishable data issue means the company can solve customer problems that are really high-value, said Flannagan. It also gives them ability to monetize not just infrastructure, but the opportunity to move up the stack. “However none of those matter if you’re not solving problems that customers care about.”
Clearly, data-based decision making is a huge challenge. According to IBM:
-1 in 3 business people are making decisions based on data they don’t trust;
-1 in 2 people lack the information they need to support their decisions; and,
-60% say they have too much data.
As companies start to digitize, there’s a pretty significant disruption that’s happening, agreed Flannagan. However, while a lot of challenges are being created… “the promises of benefits are huge.”