Home Home Home

Big Data Becoming Bigger Data— Fast

Date posted: November 19, 2015

A recent article on tdwi.org suggests that “big” may be too small a word to describe what is happening with data, as a trio of factors are combining to accelerate the exponential growth of the volume of data that enterprises must contend with: the Internet of Things (IoT); social media; and improvements in database management technology.

  • The Internet of Things
    Sensors, meters, biochips, transponders, controllers, appliances, wearables, etc., will constitute 50 billion devices by 2020 that will generate 50 to 100 trillion objects of data. Just a decade ago a terabyte was considered a lot of data, but now we routinely talk about analyzing petabytes of data, which is 10 to the 15th power or 1000 terabytes. Tomorrow we will be talking about exabytes of data, which is 10 to the 18th power or one million terabytes.
  • Social media
    Interactive marketing spend is growing at 17 percent per year and social media spend is growing at 34 percent. Last year, companies spent $55 billion on social media ads… 6.4 quintillion bytes of Internet data were created in 2014 alone; 90 percent of all Internet data was generated in the last two years, and 43 percent of data created on personal social media accounts is gathered and analyzed. In general, structured and unstructured data is growing 60 percent annually, and as third-world countries become more connected, don’t expect that to slow down any time soon.
  • Improvements in Database Management Technology
    The relational database hasn’t changed much since it was invented in 1970. Most investment and innovation by vendors was been incremental, and the brand new models never lived up to the success of the relational model. Then document-based databases showed up about eight years ago and standards such as Javascript Object Notation (JSON) became accepted standards. Suddenly companies such as MongoDB and CouchDB were off to the races, adopted by independent software companies and large companies with Java development staffs. They introduced horizontal scaling and large file storage and processing. These products did not rely on SQL, but rather let the application itself define the queries. NoSQL is not where all the new innovation is— SQL database companies such as IBM, Microsoft, and Oracle picked up the pace to accommodate large transactional database tables and newcomers such as NuoDB invented scale-out distributed SQL databases for cloud deployment. This start-up claims it can do 1,000,000 new order transactions per minute.

Individually, any one of these factors would accelerate the pace of data growth, but collectively they are driving its explosion. As Eric Schmidt of Google reminded us at the decade’s outset, “There were 5 exabytes of information created between the dawn of civilization through 2003; now that much information is created every two days.”