How Big Data Became So Big

theodp writes "The NYT's Steve Lohr reports that his has been the crossover year for Big Data — as a concept, term and marketing tool. Big Data has sprung from the confines of technology circles into the mainstream, even becoming grist for Dilbert satire ('Big Data lives in The Cloud. It knows what we do.'). At first, Jim Davis, CMO at analytics software vendor SAS, viewed Big Data as part of another cycle of industry phrasemaking. 'I scoffed at it initially,' Davis recalls, noting that SAS's big corporate customers had been mining huge amounts of data for decades. But as the vague-but-catchy term for applying tools to vast troves of data beyond that captured in standard databases gained world-wide buzz and competitors like IBM pitched solutions for Taming The Big Data Tidal Wave, 'we had to hop on the bandwagon,' Davis said (SAS now has a VP of Big Data). Hey, never underestimate the power of a meme!"

Big Data is the new place where magic happens

[sco08y]

The NYT's Steve Lohr reports that his has been the crossover year for Big Data — as a concept, term and marketing tool.

"Big Data" is another way to put data into a cylinder or a fluffy cloud and avoid the messy task of actually thinking about it.

We don't need structure, we don't need logic, we'll just throw a metric crap-ton of data at it and hope something works!

Perspective please

[Anonymous Coward]

Recently I was at a University in town here talking to one of the PhD students. He showed me a server where they store several dozens of TB of data that come from one of the space telescopes. He said that the data they had on-site was just a small fraction of the overall amount of data that gets collected each week, for which they write algorithms to analyze.

To me, that put into perspective what Big Data really means. I think for the most part, most people in tech. today still use it as a buzz-word without a real concept or understanding of what it means.

How big is 'big data'?

[shic]

And how are we measuring the size? What sizes are measured for typical 'big data'?

Are we talking about detailed information, or inefficient data formats?
Are we talking about high-resolution long-term time series, or are we talking about data that is big because it has a complex structure?

Is the data big because it has been engineered so, or is it begging for a more refined system to simplify?

BigData != "standard databases"

[THE_WELL_HUNG_OYSTER]

... had been mining huge amounts of data for decades. But as the vague-but-catchy term for applying tools to vast troves of data beyond that captured in standard databases

Big Data has nothing to do with standard databases and "mining of huge data" for decades. Data is modeled fundamentally differently than in relational systems. Indeed, that is why one invariably doesn't use SQL with the likes of Hadoop and Cloudera. Think of them more like distributed hash tables [wikipedia.org] and you'll be closer to the mark.

How 'bout Big Salespeople

[EmperorOfCanada]

Have you ever met one of the sales people from these companies? They are really really good. They take closing a sale to a whole new level. These salespeople don't walk in off the street and say, "Hey would you guys like a 50 million dollar data analysis package?" In governments they work at the highest levels. Then the directive to put out a tender that only fits one company suddenly comes out of nowhere and poof a mega project takes off. With companies they work at the board of directors level. So again suddenly a team of "consultants" shows up and determines what is needed is a multi million dollar data analysis system. Other approaches is that they buy out a consulting company that is already entrenched with a government or large corporation. If you fight the system their "consultants" will discover that you are a useless tool and recommend your replacement. If you are reluctant then they offer you a crazy training package and that you should come to their booth at some in a trade show in an exotic local.

If all that doesn't work then they always just have the buy out. That is where they find a decision maker they can't take out but they offer her a juicy job that she will take shortly after the contract is signed: http://en.wikipedia.org/wiki/Darleen_Druyun [wikipedia.org]

So big data may or may not be a complete fad but it is another way for sales people to fool upper management into buying a zillion dollar system instead of running a few well crafted python scripts on a dedicated machine and feeding them into an open source graphing solution such as Graphite.

Re:Big Data

[Nerdfest]

(Jim Davis, the name of the CEO of SAS, has the same name as the guy who did the 'Garfield' comics, although is not the same person. Off topic? Perhaps. Funny? On a Sunday evening ... I thought so. Modded over-rated as am initial mod? Not so much).

I need to get a life.