Weapons of Math Destruction

5 years after my first read, I am discovering new aspects of Cathy O’Neil’s Weapons of Math Destruction (2016). Some of her examples have lost their urgency – by now, most people understand that they are the ‘product’ rather than the consumer, and in Europe, the GDPR has addressed some of the excesses of automated decision-making and profiling – but O’Neil’s widely cited work remains highly relevant. I read it consecutively with Meredith Broussard’s Artificial Unintelligence; the books complement each other perfectly.

Toxic proxies Although O’Neil doesn’t explicitly define it, a ‘weapon of math destruction’ is an algorithm that is opaque (‘black box’), damaging (harmful to individuals or society), and scalable (applied broadly). Closely associated are automated decision-making, predictive modelling, profiling, and the use of proxies. Especially the last of these struck me this time. Since the truth is often too difficult to quantify, models are rely on measurable aspects instead. This makes outcomes unreliable, susceptible to manipulation, and often toxic. University rankings are a good example. Because ‘educational excellence’ is too complex, rankers use whatever data approximates it. As a result, universities focus on improving those metrics rather than the quality of education itself.

We are ranked, categorized, and scored in hundreds of models, on the basis of our revealed preferences and patterns. This establishes a powerful basis for legitimate ad campaigns, but it also fuels their predatory cousins: ads that pinpoint people in great need and sell them false or overpriced promises. They find inequality and feast on it. The result is that they perpetuate our existing social stratification, with all of its injustices. The greatest divide is between the winners in our system, like our venture capitalist, and the people his models prey upon.

Predictive modelling As a contrast, O’Neil mentions Michael Lewis’ Moneyball as an example of safe predictive modelling: the algorithms are transparent, regularly updated, and based on actual game performance rather than proxies. Stakeholders understand the process and share the objective (winning the league). WMD’s, however, work differently. Fairness and public values are not part of the equation. In practice, they harm people with disadvantaged backgrounds, notably poor people. In segregated countries, geography serves as a ‘highly effective proxy’ for race. Imagine the police prioritising surveillance in a wealthy area, O’Neil suggests. This would inevitably lead to an increase in recorded crime, as affluent individuals currently evade scrutiny. New data would then reinforce the perception of the area as a high-risk zone, justifying even more surveillance. Another example of a WMD is predictive marketing – such as marginal universities targeting people with a lower income, persuading them to pay fees for worthless certificates.

Automated vs. human decision-making Towards the end, the book becomes somewhat sluggish, with less engaging examples. Compared to my first reading, it lost a ★. Overall, I’m glad to have revisited it. O’Neil’s conclusion that WMD disproportionately punish the poor, who are often processed by machines, while the rich can rely on human decision-making, remains all too relevant, as does her remark that proxies should be limited to positive feedback loops. I will keep those in mind as I continue with Madhumita Murgia’s Code Dependent.

Phrenology was a model that relied on pseudoscientific nonsense to make authoritative pronouncements, and for decades it went untested. Big Data can fall into the same trap.

This book takes you on a journey through all areas of life and shows how Big Data systems cause harm in all of them. Through the examination of these case studies, it also gets to the fundamental issues with Big Data and proposes ways to change our perspectives on it.

This book is really good. It is clear, understandable for a layperson and very well-rounded. I would give it a 5/5 if there weren't these two points:

• it is completely US-centric. The case studies are all domestic. This weakens its explaining power for the rest of the world, imo. (this isn't to say that it doesn't make sense or that it's wrong for a US citizen to only write about the US)
• it's 8 years old now, and while it's analyses are not at all outdated, the world of Big Data has evolved since 2016. I often wondered what ended up happening to some of the real-world cases discussed in the book. I would be super curious to read a "follow up", maybe a "10 years later" kind of addendum or revision.

If you wish to understand better what is at stakes with the algorithms and the global digital maze we seem to be prisoner of, then, this is a must read. Cathy O'Neil very clearly unravel the intricate technical, philosophical, political and socioeconomic logic at play. She shows how they work together to force an even more violent neoliberalism in every area of our daily lives, whether social, cultural, professional, judiciary, educational or political and economic. It is both quite scary and at the same time empowering, as it gives us the basics to look for collective ways to progressively get out of the maze we've lost ourselves in. As the say goes, knowledge is power, and I believe it has never been more true than today.

I would recommend to read it along Shoshana Zuboff's (bookwyrm.social/author/39159/s/shoshana-zuboff) "The Age of Surveillance Capitalism" (bookwyrm.social/book/260163/s/the-age-of-surveillance-capitalism). They nicely complement each others.

Math! And social justice! Two of my favorite things! What's not to like?

Unfortunately, kind of a lot. Look: people who read math books for fun are math nerds. Dumbing down math concepts with cutesy terms is not needed. It will not make people who would not otherwise read math for fun read your book and it will piss off the rest of us. Also, it's lazy. And it's bad math -- O'Neil uses the term "weapon of math destruction" (over and over) very vaguely, so that she doesn't have to define exactly what she's talking about. Oh, she claims that she has a clear definition, but then she calls things like Racial Profiling a WMD (cringe). Racial Profiling isn't an algorithm; it's a cognitive heuristic and it doesn't relay on Big Data.

More problematically, I think she uses this term to obscure that a lot of her points are actually about cognitive biases, racial inequality and socioeconomic inequality, rather than the data science used to enforce these. She herself acknowledges that some things (like, e.g. racial profiling) have happened to exactly the current degree long before data science was available.

Overall, I found her approach really shallow. She's a former tenured ivy league math professor! I wanted her to write a book that only she could write -- full of nuance and equations I needed a scratchpad to struggle through.

Nonetheless, I think some of her points were good: that machine-learning algorithms are dense and require supervision and critical thinking as to their results rather than blind trust. It's an important book for the math-phobic.

A WMD, Weapon of Math Destruction, is an algorithm that is a block box (opaque), used at scale, and damages the lives of people, generally poor minorities. Cathy O'Neil goes through a lot of detail describing several of these WMDs and how they are ruining people's lives. Hate Clopening? (working at Closing and then Opening up the next morning). It's likely an algorithm created that schedule. Hate the fact that employers now use opaque personality tests to look for mental illness while you're applying for a job? Another WMD.

This book is important, and I think it should be read by anyone concerned about how Big Data can be used to harm us all. As someone whose future career depends upon algorithmic learning, statistics, and mathematics, I can say this book was eye opening. I'm used to hearing about the power of algorithms and modeling, but really, a model is not the thing that it models (as every mathematician knows).

This book is a lot more accessible than Derman's Models.Behaving.Badly, even if it is in the same vein. It has a much clearer focus, and it very clearly explains the traps mathematical modeling has created. I highly recommend this book to everyone. It doesn't require an understanding of math (there are no models or equations in this book). Just an understanding of how algorithms can contain bias through the use of proxies. Read it and share it.

So often when someone starts a Twitter message with the label “Must read” I get defensive. You’re not my teacher. I’m a grown up. I get to decide what I’m going to read, thank you very much. But I’m really tempted to start this post with “Must read” because Cathy O’Neil’s book, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy is important and covers issues everyone should care about. Bonus points: it’s accessible, compelling, and – something I wasn’t expecting – really fun to read.returnreturnO’Neil is a data scientist who taught at Barnard before being seduced by the excitement of applying mathematics to finance, working for a Wall Street hedge fund before the crash of 2008. One of the things she quickly learned was different from academic mathematics was that employees were treated like members of an Al Qaida cell: the amount of information they could share was strictly limited so that if anyone was captured by a competing firm, they couldn’t reveal too much. Also, the scale of their collective if obscure work was ginormous. Subprime mortgages were a three trillion dollar market, but the markets created around them through credit default swaps, synthetic CDOs, and other weird financial inventions based on math and baloney was twenty times that size. As it all began to collapse, the damage cascaded, and people, lots of people, got hurt.returnreturnThese risky financial instruments, like many other proprietary big data projects – what O’Neil calls “weapons of math destruction” - have features in common. They are opaque (few people could understand them even if they weren’t trade secrets that cannot be examined by those who are subject to the decisions they make); they work at large scale, and because they are sealed systems, they can’t learn from their mistakes. They can do a lot a damage and are bizarrely unaccountable for it, often claiming greater objectivity than the fallible humans who encode them. Her experience in high finance is a cautionary tale because the features that crashed the world economy are present in big data systems that affect our lives in myriad ways, from education to jobs to the criminal justice system to how we are persuaded to vote.returnreturnTwo of her early chapters deal with the effect of these big data systems on higher education. On the one hand, they feed the hysterical and costly race among elite schools to game U.S. News’s ranking system, making colleges “manage their student populations almost like an investment portfolio” and enhance their status, often at the expense of actual students. She also examines how for-profit schools manipulated personal data to target the vulnerable.returnreturn"The marketing of these universities is a far cry from the early promise of the Internet as a great equalizing and democratizing force. They find inequality and feast on it. If it was true during the early dot-com days that “nobody knows you’re a dog,” it’s the opposite today. We are ranked, categorized, and scored in hundreds of models, on the basis of our revealed preferences and patterns. This establishes a powerful basis for legitimate ad campaigns, but it also fuels their predatory cousins: ads that pinpoint people in great need and sell them false or overpriced promises. The result is that they perpetuate our existing social stratification, with all of its injustices."returnreturnThat injustice is scaled up in programs like predictive policing, which magnifies human prejudices, and algorithm-based sentencing: “we criminalize poverty, believing all the while that our tools are not only scientific but fair.” Companies that have turned to Big Data to manage the tiresome business of hiring low-wage workers use personality tests to “exclude as many people as possible as cheaply as possible” – and they don’t correct for their mistakes because they never check to see if that excluded employee actually turns out to be very good at something. Of course, these tests are used more on the poor than on the wealthy, who would never put up with it. Big data also is used to schedule workers to maximize efficiency, giving them a few days’ notice, making it impossible to schedule daycare or take courses and get ahead in life. Systems that rate teachers are based on nonsense, and secret and flawed formulas are used to determine whether we are credit-worthy.returnreturnHer penultimate chapter looks at ways these WMDs undermine democracy, and it's particularly timely. By analyzing personal data and targeting us with highly personalized political messages, we never know what promises a candidate is making to our neighbors or see the same news they read that persuades them of something that seems to us nonsensical. We are increasingly living in different worlds, sculpted to fit different world views, micro-targeted to the point "e pluribus unum" no longer pertains. Rather, divide, divide, divide and conquer.returnreturnO’Neil doesn’t hate data. She loves mathematics, but hates the ways its being deployed on us. In her conclusion she suggests the ways we could use data to make lives better, if we didn't use money as a proxy for goodness. With transparency, open audits, and a willingness to take up the problems these systems expose, we could do great things for the public good.returnreturnDare I say it? I will. For anyone who cares about information systems and how society works these days, it's a must read.returnreturn(reposted from Inside Higher Ed)