In my June 2013 post "Byte Off More Than You Can Chew", we looked at the inconceivable quantity of data collected by the NSA. Maybe this collection is onlymetadata, but recent revelations, depending on who you believe, may indicate otherwise. This time we wrangle with Big Data.

"Big data refers to our burgeoning ability to crunch vast collections of information, analyze it instantly, and draw sometimes profoundly surprising conclusions from it. This emerging science can translate myriad phenomena—from the price of airline tickets to the text of millions of books—into searchable form, and uses our increasing computing power to unearth epiphanies that we never could have seen before." (1) We can now draw logical conclusions regarding relationships that heretofore we would have never considered. These capabilities are not free. They cost us more than money. Given enough relevant data, predictive models are quite accurate. We can discern the probabilities of events yet to unfold, and we can now do so with frightening accuracy.  "It also poses fresh threats, from the inevitable end of privacy as we know it to the prospect of being penalized for things we haven’t even done yet, based on big data’s ability to predict our future behavior." (1) Queue "Minority Report" montage.

"Leaders in every sector will have to grapple with the implications of big data, not just a few data-oriented managers. The increasing volume and detail of information captured by enterprises, the rise of multimedia, social media, and the Internet of Things will fuel exponential growth in data for the foreseeable future." (3)

A yardstick for the popularity of a given topic is the "...For Dummies" book. "Big Data for Dummies" has got it covered. "Big data management is one of the major challenges facing business, industry, and not-for-profit organizations. Data sets such as customer transactions for a mega-retailer, weather patterns monitored by meteorologists, or social network activity can quickly outpace the capacity of traditional data management tools." (2)

Then there is the privacy concern. Who has access to, who collects, who analyzes big data. These questions, again depending on who you believe, have yet to be answered with veracity. One can assume beneficial goals as easily as nefarious pursuits. "Much of what constitutes Big Data is information about us. Through our online activities, we leave an easy-to-follow trail of digital footprints that reveal who we are, what we buy, where we go, and much more." (1)

Theresa Payton, former White House CIO, offers some food-for-thought on the issue of privacy: "Digital devices have made our busy lives a little easier...we get just-in-time coupons, directions, and connection with loved ones.... Yet, these devices...send and collect data about us whenever we use them, but that data is not always safeguarded the way we assume it should be to protect our privacy. Privacy is complex and personal. Many of us do not know the full extent to which data is collected, stored, aggregated, and used. As recent revelations indicate, we are subject to a level of data collection and surveillance never before imaginable. While some of these methods may, in fact, protect us and provide us with information and services we deem to be helpful and desired, others can turn out to be insidious and over-arching." (4)


(1) "Big Data: A Revolution That Will Transform How We Live, Work, and Think" by Viktor Mayer-Schönberger and Kenneth Cukier (Mar 5, 2013)

(2) "Big Data For Dummies" Paperback by Hurwitz, Alan Nugent, Fern Halper, Marcia Kaufman

(3) "Big data: The next frontier for innovation, competition, and productivity " by James Manyika, Michael Chui, Brad Brown, Jacques Bughin, Richard Dobbs, Charles Roxburgh, Angela Hung Byers - McKinsey Global Institute

(4) "Privacy in the Age of Big Data: Recognizing Threats, Defending Your Rights, and Protecting Your Family" by Theresa M. Payton and Ted Claypoole