Weapons of math destruction

5 years after my first read, I am discovering new aspects of Cathy O’Neil’s Weapons of Math Destruction (2016). Some of her examples have lost their urgency – by now, most people understand that they are the ‘product’ rather than the consumer, and in Europe, the GDPR has addressed some of the excesses of automated decision-making and profiling – but O’Neil’s widely cited work remains highly relevant. I read it consecutively with Meredith Broussard’s Artificial Unintelligence; the books complement each other perfectly.

Toxic proxies Although O’Neil doesn’t explicitly define it, a ‘weapon of math destruction’ is an algorithm that is opaque (‘black box’), damaging (harmful to individuals or society), and scalable (applied broadly). Closely associated are automated decision-making, predictive modelling, profiling, and the use of proxies. Especially the last of these struck me this time. Since the truth is often too difficult to quantify, models are rely on measurable aspects instead. This makes outcomes unreliable, susceptible to manipulation, and often toxic. University rankings are a good example. Because ‘educational excellence’ is too complex, rankers use whatever data approximates it. As a result, universities focus on improving those metrics rather than the quality of education itself.

We are ranked, categorized, and scored in hundreds of models, on the basis of our revealed preferences and patterns. This establishes a powerful basis for legitimate ad campaigns, but it also fuels their predatory cousins: ads that pinpoint people in great need and sell them false or overpriced promises. They find inequality and feast on it. The result is that they perpetuate our existing social stratification, with all of its injustices. The greatest divide is between the winners in our system, like our venture capitalist, and the people his models prey upon.

Predictive modelling As a contrast, O’Neil mentions Michael Lewis’ Moneyball as an example of safe predictive modelling: the algorithms are transparent, regularly updated, and based on actual game performance rather than proxies. Stakeholders understand the process and share the objective (winning the league). WMD’s, however, work differently. Fairness and public values are not part of the equation. In practice, they harm people with disadvantaged backgrounds, notably poor people. In segregated countries, geography serves as a ‘highly effective proxy’ for race. Imagine the police prioritising surveillance in a wealthy area, O’Neil suggests. This would inevitably lead to an increase in recorded crime, as affluent individuals currently evade scrutiny. New data would then reinforce the perception of the area as a high-risk zone, justifying even more surveillance. Another example of a WMD is predictive marketing – such as marginal universities targeting people with a lower income, persuading them to pay fees for worthless certificates.

Automated vs. human decision-making Towards the end, the book becomes somewhat sluggish, with less engaging examples. Compared to my first reading, it lost a ★. Overall, I’m glad to have revisited it. O’Neil’s conclusion that WMD disproportionately punish the poor, who are often processed by machines, while the rich can rely on human decision-making, remains all too relevant, as does her remark that proxies should be limited to positive feedback loops. I will keep those in mind as I continue with Madhumita Murgia’s Code Dependent.

Phrenology was a model that relied on pseudoscientific nonsense to make authoritative pronouncements, and for decades it went untested. Big Data can fall into the same trap.

This book takes you on a journey through all areas of life and shows how Big Data systems cause harm in all of them. Through the examination of these case studies, it also gets to the fundamental issues with Big Data and proposes ways to change our perspectives on it.

This book is really good. It is clear, understandable for a layperson and very well-rounded. I would give it a 5/5 if there weren't these two points:

• it is completely US-centric. The case studies are all domestic. This weakens its explaining power for the rest of the world, imo. (this isn't to say that it doesn't make sense or that it's wrong for a US citizen to only write about the US)
• it's 8 years old now, and while it's analyses are not at all outdated, the world of Big Data has evolved since 2016. I often wondered what ended up happening to some of the real-world cases discussed in the book. I would be super curious to read a "follow up", maybe a "10 years later" kind of addendum or revision.

If you wish to understand better what is at stakes with the algorithms and the global digital maze we seem to be prisoner of, then, this is a must read. Cathy O'Neil very clearly unravel the intricate technical, philosophical, political and socioeconomic logic at play. She shows how they work together to force an even more violent neoliberalism in every area of our daily lives, whether social, cultural, professional, judiciary, educational or political and economic. It is both quite scary and at the same time empowering, as it gives us the basics to look for collective ways to progressively get out of the maze we've lost ourselves in. As the say goes, knowledge is power, and I believe it has never been more true than today.

I would recommend to read it along Shoshana Zuboff's (bookwyrm.social/author/39159/s/shoshana-zuboff) "The Age of Surveillance Capitalism" (bookwyrm.social/book/260163/s/the-age-of-surveillance-capitalism). They nicely complement each others.

Math! And social justice! Two of my favorite things! What's not to like?

Unfortunately, kind of a lot. Look: people who read math books for fun are math nerds. Dumbing down math concepts with cutesy terms is not needed. It will not make people who would not otherwise read math for fun read your book and it will piss off the rest of us. Also, it's lazy. And it's bad math -- O'Neil uses the term "weapon of math destruction" (over and over) very vaguely, so that she doesn't have to define exactly what she's talking about. Oh, she claims that she has a clear definition, but then she calls things like Racial Profiling a WMD (cringe). Racial Profiling isn't an algorithm; it's a cognitive heuristic and it doesn't relay on Big Data.

More problematically, I think she uses this term to obscure that a lot of her points are actually about cognitive biases, racial inequality and socioeconomic inequality, rather than the data science used to enforce these. She herself acknowledges that some things (like, e.g. racial profiling) have happened to exactly the current degree long before data science was available.

Overall, I found her approach really shallow. She's a former tenured ivy league math professor! I wanted her to write a book that only she could write -- full of nuance and equations I needed a scratchpad to struggle through.

Nonetheless, I think some of her points were good: that machine-learning algorithms are dense and require supervision and critical thinking as to their results rather than blind trust. It's an important book for the math-phobic.

A WMD, Weapon of Math Destruction, is an algorithm that is a block box (opaque), used at scale, and damages the lives of people, generally poor minorities. Cathy O'Neil goes through a lot of detail describing several of these WMDs and how they are ruining people's lives. Hate Clopening? (working at Closing and then Opening up the next morning). It's likely an algorithm created that schedule. Hate the fact that employers now use opaque personality tests to look for mental illness while you're applying for a job? Another WMD.

This book is important, and I think it should be read by anyone concerned about how Big Data can be used to harm us all. As someone whose future career depends upon algorithmic learning, statistics, and mathematics, I can say this book was eye opening. I'm used to hearing about the power of algorithms and modeling, but really, a model is not the thing that it models (as every mathematician knows).

This book is a lot more accessible than Derman's Models.Behaving.Badly, even if it is in the same vein. It has a much clearer focus, and it very clearly explains the traps mathematical modeling has created. I highly recommend this book to everyone. It doesn't require an understanding of math (there are no models or equations in this book). Just an understanding of how algorithms can contain bias through the use of proxies. Read it and share it.