The minimum description length principle
by Peter D. Grünwald · first published 2007
6mentions
2recommended by
6HN commenters
2008–2026first – last mentioned
33% recommend33% neutral33% critical
Affiliate links. As an Amazon Associate I earn from qualifying purchases.
Jaynes, Probability Theory: The Logic of Science (https://books.google.com/books/about/Probability_Theory.html...) Grunwald, The Minimum Description Length Principle.
on “Ask HN: What is your favorite book for learning statistics?” ↗…exasperation about the Darwinist analogy, and for my claims about the flabbiness of 10-year-old climate predictions, which also happens to be interesting reading for hackers interested in machine learning: _The Minimum Description Length Principle_. Lots of math to help you think precisely about questions like "by how many bits did this hypothesis reduce the surprisingness of the world compared to alternative hypotheses?" For sources more specific…
on “Why Politicized Science is Dangerous” ↗…0% probability of being "by chance". Then, you say "that's a complicated rule: no one can memorize the list", and that's fine. But finding the simplest rule that has the best fit can be shown, via the Minimum Description Length principle, to be equivalent to models with deep connections to statistical learning. The gist of the matter is: even if you have a single dataset, statistical considerations are very important to make sure that…
on “Here's Waldo” ↗Some ideas like the Minimum Description Length principle derived from information theory turn out to be just rephrasings of already-known statistical ideas. This can occasionally provide more insight, but also can lead to ridiculously irrelevant things like…
on “Statistical Computing for Scientists and Engineers” ↗…so-called "universal" coding. It's ok* to depart from that starting point in creating subtheories but if you don't start there you'll end up with garbage like the last 50 years of confusion over what "The Minimum Description Length Principle" really means. *It is, however, _not_ "ok" if what you are trying to do is come up with causal models. You can't get away from Turing complete codes if you're trying to model dynamical systems even…
on “Learning Theory from First Principles [pdf]” ↗…https://arxiv.org/abs/1811.06965 Keeping Neural Networks Simple by Minimizing the Description Length of the Weights - https://www.cs.toronto.edu/~hinton/absps/colt93.pdf A Tutorial Introduction to the Minimum Description Length Principle - https://arxiv.org/abs/math/0406077 The First Law of Complexodynamics - https://scottaaronson.blog/?p=762 Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton - https://arxiv.org/abs/1405.6903…
on “30papers.com – Ilya's 30 essential ML papers, in a beginner friendly format” ↗