Data sources - Hadoop

We live in the data age.It’s not easy to measure the total volume of data stored electronically,but an IDC estimate put the size of the “digital universe” at 0.18 zettabytes in 2006, and is forecasting a tenfold growth by 2011 to 1.8 zettabytes.* A zettabyte is 1021 bytes, or equivalently one thousand exabytes, one million petabytes, or one billion terabytes. That’s roughly the same order of magnitude as one disk drive for every personin the world.

This flood of data is coming from many sources. Consider the following:

  • The New York Stock Exchange generates about one terabyte of new trade data perday.
  • Facebook hosts approximately 10 billion photos, taking up one petabyte of storage.
  •, the genealogy site, stores around 2.5 petabytes of data.
  • The Internet Archive stores around 2 petabytes of data, and is growing at a rate of 20 terabytes per month.
  • The Large Hadron Collider near Geneva, Switzerland, will produce about 15 petabytes of data per year.

→ From Gantz et al., “The Diverse and Exploding Digital Universe,” March 2008.

So there’s a lot of data out there. But you are probably wondering how it affects you. Most of the data is locked up in the largest web properties (like search engines), orscientific or financial institutions, isn’t it? Does the advent of “Big Data,” as it is being called, affect smaller organizations or individuals

I argue that it does. Take photos, for example. My wife’s grandfather was an avid photographer, and took photographs throughout his adult life. His entire corpus of medium format, slide, and 35mm film, when scanned in at high-resolution, occupies around 10 gigabytes. Compare this to the digital photos that my family took in 2008,which take up about 5 gigabytes of space. My family is producing photographic data at 35 times the rate my wife’s grandfather’s did, and the rate is increasing every year as it becomes easier to take more and more photos.

More generally, the digital streams that individuals are producing are growing apace. Microsoft Research’s MyLifeBits project gives a glimpse of archiving of personal information that may become common place in the near future. MyLifeBits was an experiment where an individual’s interactions phone calls, emails, documents were captured electronically and stored for later access. The data gathered included a photo taken every minute, which resulted in an overall data volume of one gigabyte a month. When storage costs come down enough to make it feasible to store continuous audio and video, the data volume for a future My LifeBits service will be many times that.

The trend is for every individual’s data footprint to grow, but perhaps more important,the amount of data generated by machines will be even greater than that generated by people. Machine logs, RFID readers, sensor networks, vehicle GPS traces, retail transactions all of these contribute to the growing mountain of data.

The volume of data being made publicly available increases every year, too. Organizations no longer have to merely manage their own data: success in the future will bedictated to a large extent by their ability to extract value from other organizations’ data.

Initiatives such as Public Data Sets on Amazon Web Services,, and the exist to foster the “information commons,” where data can be freely (or inthe case of AWS, for a modest price) shared for anyone to download and analyze. Mashups between different information sources make for unexpected and hitherto unimaginable applications.

Take, for example, the project, which watches the Astrometry groupon Flickr for new photos of the night sky. It analyzes each image and identifies which part of the sky it is from, as well as any interesting celestial bodies, such as stars orgalaxies. This project shows the kind of things that are possible when data (in this case,tagged photographic images) is made available and used for something (image analysis)that was not anticipated by the creator.

It has been said that “More data usually beats better algorithms,” which is to say that for some problems (such as recommending movies or music based on past preferences),however fiendish your algorithms are, they can often be beaten simply by having more data (and a less sophisticated algorithm).

The good news is that Big Data is here. The bad news is that we are struggling to store and analyze it.

All rights reserved © 2018 Wisdom IT Services India Pvt. Ltd Protection Status

Hadoop Topics