Global Sources
EE Times-India
Stay in touch with EE Times India
EE Times-India > Memory/Storage

How to handle Big Data

Posted: 29 Jan 2015     Print Version  Bookmark and Share

Keywords:Big Data  analytics  visualisation  entropy 

Big data is a hot topic. Our daily interactions with the Internet—through laptops, smart phones, tablets and IoT devices—generate vast amounts of data that are thought to encode our buying preferences, our political and economic moods and more. This attracts entrepreneurs who seek new ways to make money with new methods to manage and analyse the sheer volume and distributed nature of the data. Along the way, discussion around the potential benefits of big data analysis seems to have morphed from enthusiasm to almost religious fervour, asserting that that big data analytics will become the dominant method of analysis for almost everything. The apex (or more exactly the nadir) of this adulation may be an article in Wired magazine—from the editor-in-chief, no less—titled The end of theory: the data deluge makes the scientific method obsolete. Let that one sink in—we didn't need Galileo or Newton or Einstein, we just needed big data analytics.

Big Data

In what follows, it may seem that I'm just another engineer who doesn't get it, but I'm actually a fan. However, I'm wary of claims that seem too over the top, especially when they start leaking into popular media. I'm also concerned that the expectations being heaped on a rather slender framework of substance will inevitably lead to collapse and disappointment, which would be a shame because it will overshadow the real value that might come from this direction. In the spirit of encouraging healthy debate, here are a few pins in the hyperbole balloon. Several of these ideas are from, or inspired by, Nate Silver's The Signal and the Noise.

Large amounts of data do not guarantee large amounts of accessible information

If you have any physics background, you know that information is related to entropy, not the size of the dataset. As entropy goes up, information goes down. At one extreme, a volume of gas in equilibrium contains, in principle, a huge amount of information—the positions and momenta of each molecule in the gas. This number easily dwarfs any measure of information on the Internet. And yet we can only extract two numbers from all that data—the pressure and temperature of the gas. Entropy continues to limit available information as you move away from this extreme, because it is difficult to extract a signal from noisy data. Differential Power Analysis (DPA) is an example where an encryption key can be extracted from power data though clever statistical sampling over many large datasets, but it is not a technique for the faint of heart. DPA requires strong mathematical underpinnings, and from all that analysis and math you are only able to extract one 256bit encryption key (valuable to be sure, but not "a lot" of information).

1 • 2 • 3 • 4 Next Page Last Page

Comment on "How to handle Big Data"
*  You can enter [0] more charecters.
*Verify code:


Visit Asia Webinars to learn about the latest in technology and get practical design tips.


Go to top             Connect on Facebook      Follow us on Twitter      Follow us on Orkut

Back to Top