Cathy O’Neil’s Weapons of Math Destruction, when it was first published in 2016, sounded an early alarm about the big data algorithms and their potential for social evil. The cover is adorned with a robotic death’s head and the subtitle reads “How Big Data Increases Inequality and Threatens Democracy.”
O’Neil’s book begins with stories that are about data, but don’t really tap into the dangers posed by “big data algorithms.” She relates the origins of baseball analytics, describing the now-common infield shift to the right side to cope with the left-handed hitting Ted Williams in 1946. The modern incarnation of baseball analytics is really all about descriptive statistics and the equivalent of pivot tables – for example, what types and locations of pitches Bryce Harper is good at hitting, and where he hits the ball. A detailed understanding of the data, certainly, but not really a “dangerous” big data algorithm. The same is true of a 1997 legal case she recounts, in which a self-styled expert testified in the sentencing phase of a trial in Texas that blacks were more likely to commit crimes by virtue of their race. A simplistic interpretation of data, and unchanged for a century. Another chapter describes how universities gamed the U.S. News and World Report rankings.
The chapter “Propaganda Machine” comes closer to the author’s premise, describing large scale predatory online advertising by for-profit colleges, targeting people who are emotionally vulnerable. The plan to target the emotionally-fragile came not from an algorithm, but rather from a college training manual, which led to mass marketing. By 2016 the direction of digital advertising was already clear – algorithmically driven microtargeting. Constantly morphing experiments generate individual-level data about what messaging motivates you as a person, and what doesn’t. It won’t be the same as what motivates your wife or your brother, so you’ll see different ads. The era of mass marketing is being supplanted by the era of individual targeting at scale. Brad Parscale, who handled digital ads for President Trump’s 2016 campaign, described on 60 Minutes the system he set up to make and test ads automatically:
Average day: 50-60 thousands ads. Programmatically. In one day. In one day.
What it is is: what can make people react? What catches their attention? Remember, there’s so much noise on your phone. You know, or on your desktop. What is it that makes it go: Poof! “I’m gonna stop and look.”
Mass media advertising as it evolved in the 1950’s and 1960’s sparked concern that Madison Avenue was manipulating the behavior of consumers. And yet that era pales in comparison with the scope of action for today’s digital advertisers. O’Neil pictures you as a lab animal subjected to a constant flow of subtle yet unremitting manipulative signals and experiments that are informed by your constantly-evolving digital footprint, and by how you reacted to the prior experiment.
Weapons of Math Destruction spends most of its time discussing the bad things people do with data, without benefit of algorithms. The really scary weapons, though, lie at the intersection of this unholy trinity:
-
CEO’s setting direction for their programmers
-
Huge amounts of personal data
-
Powerful algorithms