The Rise of Big Data

By: Brad Buchanan | bbuchanan@newzgroup.com

Big data has become ubiquitous. Amazon uses it to tell us about other products we might like, iTunes points us toward other music we might enjoy, and Google indicates other searches that might be useful. Retailers use big data to predict future buying behaviors. They know that if you buy products X, Y and Z together, that there is a high probability you are pregnant, and begin showering you with coupons and promotions for diapers, cribs, strollers and other baby products. Trends, and predictions based on those trends, are becoming more accurate using big data.

The eminent paleontologist Stephen Jay Gould once said that humans are “story-telling and pattern-seeking creatures.” News organizations offer good examples of both. Reporters are trained to seek out patterns. A series of fires may indicate arson. Similar murders may link back to a serial killer. Gifted journalists then bundle the quantity of facts into a compelling story, connecting people to current events.

On another level, big data can be used to analyze the news from diverse outlets across wide geographic areas. Analytics can tease out patterns that are not readily discernable. Using rigorous statistical techniques on huge data sets made available through community news outlets, regional trends can be identified and acted on. Using community news as source material, aggregated information can offer unique insights on a spectrum of topics, such as political sentiment, construction booms and busts, the spread of agricultural disease, and municipality undertakings. The wealth of information contained in local news provides fertile ground for statistical analysis on a myriad of subjects.

One of the primary tools for discovering trends from diverse sources is the emerging field of data science. Analyzing the rich information available today creates new insights. The new emerging trend emphasizes the need for statistical results to be relevant and actionable in nature.

Sports have changed dramatically over the last few decades as a result of data science. Olympians are better informed on what preparation methods deliver results, football analytics helps teams at the annual combine in determining the success of future player, and baseball has become immersed in sabermetrics and the predictive qualities it produces.

In the old days, baseball managers ran their teams through intuition and guess-work. They “felt” like a pitcher was getting tired, or that a batter was not seeing the ball well. With the advent of big data statistics in baseball, managers no longer have to run their teams by the “seat of their pants.” Instead, they can use statistical analysis as a basis for moment-by-moment decision making. The application of higher mathematics to baseball data has lead to small market teams competing with the perennials, such as the Yankees. The use of analytics in baseball inspired the book and film, Moneyball.

Through information aggregation and the application of data science, other industries are leveraging the power of analytics. Politicians, educators, architects, city-managers, law enforcement, and virtually anyone who needs to base their decisions on quantifiable insights rather than intuition, predispositions, and feelings.

When it comes to making a decision, the feeling of intuition is palatable and undeniably influential. But as the saying goes, knowledge is power. And data science is the new global authority on emerging knowledge.