Ben Waber reviewed Noise: A Flaw in Human Judgment by Daniel Kahneman
An Ironically Flawed Book
3 stars
This book presents the best and tendencies of behavioral economics. At its best, it exposes the flaws in human decision-making and reasoning and the implications for those flaws across a wide variety of areas. At its worst, it falls into the trap of failing to interrogate the nature of accuracy, data labels, and systems that cause outcomes. This emerges from the shockingly uninformed usage of many thoroughly debunked metrics as supposedly “unbiased,” “objective” performance metrics - intelligence tests and bail data particularly stand out.
The time spent trying to unconvincingly distinguish what they call “noise” from bias could have been devoted to addressing ethical issues that come with scaling up a single metric of performance, issues of distribution shift, etc. From the extremely limited coverage of the actual issues in AI systems, however, I'd be unsurprised if they were mostly unaware of these issues. If you’re already familiar with these …
This book presents the best and tendencies of behavioral economics. At its best, it exposes the flaws in human decision-making and reasoning and the implications for those flaws across a wide variety of areas. At its worst, it falls into the trap of failing to interrogate the nature of accuracy, data labels, and systems that cause outcomes. This emerges from the shockingly uninformed usage of many thoroughly debunked metrics as supposedly “unbiased,” “objective” performance metrics - intelligence tests and bail data particularly stand out.
The time spent trying to unconvincingly distinguish what they call “noise” from bias could have been devoted to addressing ethical issues that come with scaling up a single metric of performance, issues of distribution shift, etc. From the extremely limited coverage of the actual issues in AI systems, however, I'd be unsurprised if they were mostly unaware of these issues. If you’re already familiar with these problems this book can provide some helpful cases and framings, otherwise reading this book will probably paradoxically increase your misunderstanding of these fundamental issues.
