Tooling Up For Big Data, Cloud Storage With Erasure Coding

The mushrooming data explosion and the accelerating adoption of software-everything are combining to help drive rapid growth in the file and object-based storage market, according to new research from International Data Corporation. However FOBS needs a helping hand to address cloud storage and Big Data, said Paul Carpentier the CTO and Co-Founder at Caringo, a developer of storage software that combines ease of management, intelligent automation and elastic data protection.

Carpentier agreed that the object-based storage market has reached an inflection point. In the last quarter of 2012 and company sold three times what it had sold since starting , and that growth is continuing. “Finally, everyone has seen the light,” he said, and “erasure code is to object storage as to what RAID is to file storage.”

Traditional file systems simply cannot provide the ease of management and accessibility required for cloud storage, nor will they provide the massive scalability and footprint efficiency required for Big Data repositories, he said. The future of both cloud storage and Big Data remain firmly entrenched in an object storage solution that incorporates both replication and erasure coding into its architecture to overcome the limitations of either one technology.

Erasure coding, which started life in RAID 6, provides redundancy by breaking objects up into smaller fragments and storing the fragments in different places, enabling data to be recovered from any combination of a smaller number of those fragments. However there are no free lunches, said Carpentier. The underlying objective is clear: provide protection against failure at lower footprint cost, but there are trade-offs to be considered when compared to replication.

IDC just reported that FOBS revenues are expected to exceed $23 billion this year, and reach $38 billion in 2017. “The future of storage is software based,” said Ashish Nadkarni, Research Director, Storage Systems. “FOBS solutions are much more versatile and will quickly outpace more rigid, hardware-based options. Scale-up solutions, including unitary file servers and scale-up appliances and gateways, will fall on hard times throughout the forecast period, experiencing sluggish growth through 2016 before beginning to decline in 2017. On the other hand, scale-out file- and object-based solutions – delivered either as software, virtual storage appliances, hardware appliances, or self-built for delivering cloud-based offerings – are forecast to show robust growth with a compound annual growth rate (CAGR) of 24.5% from 2012 to 2017.”

Object-based systems can be made more scalable, more efficient and more economical than traditional scale-up or even scale-out storage architectures, and using erasure coding and data dispersion can also greatly increase the reliability of data while improving storage efficiency and reducing cost, wrote Eric Slack, Senior Analyst, Storage Switzerland. ‘But this combination of storage technologies can provide another significant advantage as well, one that’s especially important to cloud providers – data security.’

Last month the OpenStack development community decided to introduce erasure codes to its open source Swift object storage. The default setting of Swift software calls for three replicas of every object to be stored in separate locations, so erasure coding would offer customers the chance to reduce the number of servers and drives they need to buy.

However, this requires a tradeoff, compute versus storage. “Do you want to use processing power as a cheaper way of providing protection than extra disks? That’s really the tradeoff,” said David Floyer, CTO at Wikibon, a community-focused research and analyst firm, noting that Moore’s Law will continue to reduce the cost of processing power. While the introduction of erasure coding represents a significant enhancement for OpenStack Swift object storage, early adoption is not going to be fantastic, he predicted.

IDC’s Nadkarni said OpenStack Swift poses a threat to object storage vendors such as Amplidata, Cleversafe, EMC and Scality. Although erasure coding represents good news for proponents of Swift object storage, potential users will need to wait for the actual implementation to come through before they get too excited because “there are different ways to do erasure codes, and ultimately the devil is in the details,” he said.

This will take time to settle out, but Carpentier said object-based storage and erasure coding are critical to the future of cloud storage. “New challenges can’t be met with old technologies. We’re not talking 10% changes, we’re talking 10X.”


Author: Steve Wexler

Share This Post On

Leave a Reply