Small Business Technology News
Big Data Still Is Not Fast Enough
Written by Tim Morral
Published: 8/29/2016
Big Data needs to be faster. Fortunately, new hardware and a new memory-centric distributed storage system may kick things up a notch.
Big Data isn't big news anymore.
It's now standard for smart companies to use Big Data to mine data and develop actionable business insights.
The problem is that Big Data is slower than it needs to be. If a Big Data query takes fifteen minutes, or even hours, to complete, that slow throughput can really limit a company's ability to extract meaningful information from massive datasets.
Better Hardware Isn't Enough
I've written previously about Fast Data and how 3D Xpoint technology is poised to make Big Data analytics much faster.
By unifying storage and memory, 3D Xpoint eliminates data access bottlenecks that have slowed Big Data down to date. The technology was jointly developed by Intel and Micron and is just now getting to market.
But while faster hardware can certainly improve things, the real trick is figuring out how to keep the right data in memory and to get it into memory quickly. This is especially challenging when data is distributed across multiple data centers in different geographies.
Even with better hardware like 3D Xpoint, the ultimate solution to speed up Big Data must include some really great software.
Enter Alluxio
Alluxio (formerly name Tachyon) is an open source project that provides the needed interface between between Big Data applications and the storage systems.
It's being used by big companies like Baidu, Barclays, Alibaba and Rackspace with promising results.
Indeed, Senior Architect Shaoshan Liu of Baidu recently ran some Alluxio tests and told ReadWrite that the "performance was amazing. With Spark SQL alone, it took 100-150 seconds to finish a query; using Alluxio, where data may hit local or remote Alluxio nodes, it took 10-15 seconds."
What It Means for All of Us
Big Data can solve myriad business and societal challenges.
There are thousands of examples of Big Data in action.
One that I just learned about this week is technology that is combing all the data on the Dark Web to find zero day exploits that cybercriminals are using to hack people and organizations.
Another is the use of Big Data to identify possible Zika outbreaks.
Some even think that Big Data could cure cancer.
While some suspect that Big Data will be used for nefarious purposes (of course it will), that's true of any technology. For the purposes of this discussion, let's assume Big Data is a good thing.
If it's a good thing and it's helping people, it goes without saying that speeding it up is also a good thing.
Even those of us who are not Big Data scientists (I'm certainly not one) will likely benefit from the outputs of Big Data. We already are.
More importantly, Big Data continues to get easier to use, which means that even we mere mortals may be able to do some interesting things with it soon.
How soon?
Not fast enough.
But, like Big Data memory management, everything about Big Data is getting faster with each passing day.
Share this article
About Our Small Business Magazine
Our reporters cover all aspects of entrepreneurship. Our coverage is dedicated to helping you to start and grow your business.
Additional Resources for Entrepreneurs