Why you shouldn't fall in love with data

Story highlights Kakaes: Many people and institutions are intoxicated by the potential of big data He warns data can mislead us and not every valid judgment can be summed up in a number

"Big data" and "evidence-based policy" are the dominant ideas of our moment. A May 2014 White House report put it this way: "Big data will become an historic driver of progress, helping our nation perpetuate the civic and economic dynamism that has long been its hallmark."

The White House report presents big data as an analytically powerful set of techniques. It says the social and economic value created by big data should be balanced against "privacy and other core values of fairness, equity and autonomy."

Konstantin Kakaes

But the White House effort to balance the costs and benefits of big data misses the bigger picture. There are limits to the analytic power of big data and quantification that circumscribe big data's capacity to drive progress.

Data-driven techniques are only one part of how government, industry and civil society should make important decisions. Bad use of data can be worse than no data at all. As a December 2014 New York Times Magazine story about Marissa Mayer, Yahoo's chief executive, pointed out:

"Mayer also favored a system of quarterly performance reviews, or Q.P.R.s, that required every Yahoo employee, on every team, be ranked from 1 to 5. The system was meant to encourage hard work and weed out underperformers, but it soon produced the exact opposite. Because only so many 4s and 5s could be allotted, talented people no longer wanted to work together; strategic goals were sacrificed, as employees did not want to change projects and leave themselves open to a lower score."

As the Yahoo example shows, the presumption that quantitative techniques objectively assess "what works" is deeply flawed. Many attempts to collect and interpret data not only miss key factors, but transform for the worse the systems they claim only to be measuring.

Sheri Lederman, a fourth grade teacher on Long Island, sued the New York State Education Department in October 2014 in what is perhaps the clearest legal test case of the dangers of big data. Lederman is highly regarded by her peers and superiors, an "exceptional educator" in the words of her school district's superintendent.

Yet a statistical technique called "value-added modeling" that purports to evaluate teachers based on students' standardized test scores said Lederman was ineffective. The American Statistical Association has criticized value-added modeling as an ineffective measure. "Ranking teachers by their VAM scores can have unintended consequences that reduce quality," the statisticians said.

Read the original:
Why you shouldn't fall in love with data