“This transformation presents an entirely new menace: penalties based on propensities,” he writes in his book Big Data: A Revolution That Will Transform How We Live, Work and Think, which is co-authored by Kenneth Cukier, The Economist data editor.
“That is the possibility of using big-data predictions about people to judge and punish them even before they’ve acted. Doing this negates ideas of fairness, justice and free will. In addition to privacy and propensity, there is a third danger. We risk falling victim to a dictatorship of data, whereby we fetishize the information, the output of our analyzes and end up misusing it. Handled responsibly, big data is a useful tool of rational decisionmaking. Wielded unwisely, it can become an instrument of the powerful, who may turn it into a source of repression, either by simply frustrating customers and employees or, worse, by harming citizens,” Mayer-Schonberger writes.
Mayer-Schonberger presents two very different real-life scenarios to illustrate how algorithms are being used. First, he explains how the analytics team working for US retailer Target can now calculate whether a woman is pregnant and, if so, when she is due to give birth.
“They noticed that these women bought lots of unscented lotion at around the third month of pregnancy and that a few weeks later they tended to purchase supplements, such as magnesium, calcium and zinc. The team ultimately uncovered around two dozen products that, used as proxies, enabled the company to calculate a ‘pregnancy prediction’ score for every customer who paid with a credit card or used a loyalty card or mailed coupons. The correlations even let the retailer estimate the due date within a narrow range, so it could send relevant coupons for each stage of the pregnancy,” he writes.
Harmless targeting, some might say, but what happens, as has already reportedly occurred, when a father is mistakenly sent diaper discount vouchers instead of his teenage daughter, who a retailer has identified is pregnant before her own father knows?
Mayer-Schonberger’s second example throws up even more potential dilemmas and pitfalls.
“Parole boards in more than half of all US states use predictions founded on data analysis as a factor in deciding whether to release somebody from prison or to keep him incarcerated,” he writes.
Christopher Steiner, author of Automate This: How Algorithms Came to Rule Our World, has identified a wide range of instances where algorithms are being used to provide predictive insights — often within the creative industries.
In his book, he tells the story of a Web site developer called Mike McCready who has developed an algorithm to analyze and rate hit records. Using a technique called advanced spectral deconvolution, the algorithm breaks up each hit song into its component parts — melody, tempo, chord progression and so on — and then uses that to determine common characteristics across a range of No. 1 records.
McCready’s algorithm correctly predicted — before they were even released — that the debut albums by both Norah Jones and Maroon 5 contained a disproportionately high number of hit records.
The next logical step — for profit-seeking record companies, perhaps — is to use algorithms to replace the human songwriter, but is that really an attractive proposition?