Information Theory, Inference & Learning Algorithms
by David J.C. MacKay · first published 2003
83% recommend17% neutral
Why HN recommends it
Well-paced, comprehensive and wittily written text tying information theory to Bayesian inference and machine learning, and free online. Suits readers with solid math who want a rigorous, physics-flavored foundation.
Summarised by Claude from recommending comments
Book Jacket: This textbook introduces theory in tandem with applications. Information theory is taught alongside practical communication systems, such as arithmetic coding for data compression and sparse-graph codes for error-correction. A toolbox of inference techniques, including message-passing algorithms, Monte Carlo methods, and variational approximations, are developed alongside applications of these tools to clustering, convolutional codes, independent component analysis, and neural networks. Publisher Description: This textbook offers comprehensive coverage of Shannon's th
Affiliate links. As an Amazon Associate I earn from qualifying purchases.
Mentions per year
What Hacker News says
Related books
2007–2026Pattern Recognition and Machine Learning
Christopher M. Bishop
Recommended by 80 · mentioned by 10480 recommend · 104 mention80 recommend · 10480 recommend
2011–2023Bayesian reasoning and machine learning
David Barber
Recommended by 15 · mentioned by 2015 recommend · 20 mention15 recommend · 2015 recommend
2019–2022Introduction to Topological Manifolds
John M. Lee
Recommended by 2 · mentioned by 22 recommend · 2 mention2 recommend · 22 recommend
2014–2016Adventures in stochastic processes
Sidney I. Resnick
Recommended by 1 · mentioned by 21 recommend · 2 mention1 recommend · 21 recommend
2007–2026The Elements of Statistical Learning
Trevor Hastie, Robert Tibshirani, Jerome Friedman
Recommended by 144 · mentioned by 227144 recommend · 227 mention144 recommend · 227144 recommend
2009–2023A logical approach to discrete math
Gries, David.
Recommended by 6 · mentioned by 66 recommend · 6 mention6 recommend · 66 recommend
2008–2026All of Statistics
Larry Wasserman
Recommended by 40 · mentioned by 6040 recommend · 60 mention40 recommend · 6040 recommend
…Lifschitz's series - Vladimir Arnold's books on ODEs and Classical Mechanics - Winning Ways for your Mathematical Plays (by Berlekamp et al.) - Elements of Information Theory (by Cover & Thomas) - Information Theory, Inference, and Learning Algorithms (by MacKay) - Network Optimization (by Bertsekas) - Convex Optimization (by Boyd & Vandenberghe) - Nonlinear Control Systems (by Isidori) - Visual Complex Analysis (by Needham) - Lasers (by Siegman)…
on “Reading list for those who love to learn” ↗…grounding in multi-variable calculus, linear algebra, probability theory and information theory. It will also be helpful to be well versed in graph theory. In my opinion one of the best starting points is "Information Theory, Inference and Learning Algorithms" by David MacKaye. It's a bit long in the tooth now, but it is still one of the most approachable and well written books in the field. Another old book that stands up very well is "Probability Theory:…
on “Ask HN: What maths are critical to pursuing ML/AI?” ↗…Practical Common LISP - Seibel Programming Perl - Wall, Christiansen and Orwant ANSI Common Lisp - Graham Refactoring - Fowler UML Distilled - Fowler Introduction to the Theory of Computation - Sisper Information Theory, Inference, and Learning Algorithms - MacKay Modern Operating Systems - Tanenbaum On Lisp - Graham Hackers & Painters - Graham Paradigms of Artificial Intelligence - Norvig Pattern Recognition and Machine Learning - Bishop The Java…
on “ASK HN: Hacker Bibles” ↗David Mackay covers MCMC in his exhaustive book titled "Information Theory, Inference, and Learning Algorithms" available for free here: http://www.inference.phy.cam.ac.uk/itprnn/book.html They are covered near the end of the book. It should be enough to familarize yourself with and understand the basic concepts…
on “Free Python ebook: Bayesian Methods for Hackers” ↗i second this. this is probably my favorite text book behind Mackay's "Information Theory, Inference, and Learning Algorithms". if you take the time to honestly follow the text you come away with a great understanding of convex optimization... both the theory and the methods.
on “Nice book on convex optimization techniques” ↗…with Caltech video lectures: http://work.caltech.edu/telecourse.html Pattern Recognition and Machine Learning, by Bischop http://research.microsoft.com/en-us/um/people/cmbishop/prml/ Also noteworthy Information Theory, Inference, and Learning Algorithms, by Mackay, free. http://www.inference.phy.cam.ac.uk/itprnn/book.html Classification, Parameter Estimation and State Estimation, by van der Heijden. http://prtools.org Computer Vision: Models, Learning,…
on “A Tour of Machine Learning Algorithms” ↗…misleading; the book seems to be mostly about "information" in the colloquial sense, and only peripherally about information theory. For anyone seriously interested in the latter, David Mackay's book Information Theory, Inference, and Learning Algorithms is available online. It's great.
…the kinds of idea statisticians bring to the party (significance, parameter intervals, chained analysis like ANOVA). Hope that helps :-) Edit* spelling errors PS: Almost forgot MacKay's excellent "Information Theory, Inference, and Learning Algorithms" (free as pdf). It does a great job linking coding, ML, inference, and bayesian probability together. Read at least as far as the section linking K-means clustering to Expectation-Maximization over a…
on “Ask HN: Machine Learning? Where to start?” ↗…exercise/fun project for the author, I think this is ok. But for any serious study, I fail to see what this offers over the completely free, very readable, more carefully written, and more thorough "Information Theory, Inference, and Learning Algorithms" by David MacKay: http://www.inference.phy.cam.ac.uk/itila/book.html. As an example of the problems with the pdf in its current stage, Theorem 2 (asymptotic source coding) includes the term "negligible…
on “Information Theory for Machine Learning [pdf]” ↗David MacKay: Information Theory, Inference, and Learning Algorithms Kevin Murphy: Machine Learning: a probabilistic perspective. Christopher Bishop: Pattern Recognition and Macine Learning
on “Computational and Inferential Thinking – The Foundations of Data Science” ↗Lindley’s Understanding Uncertainty Jaynes’s Probability Theory: The Logic of Science MacKay’s Information Theory, Inference, and Learning Algorithms (http://www.inference.org.uk/itila/book.html)
on “Ask HN: What are your favorite statistics and probability textbooks?” ↗…unconventional entry point, I recommend the seminal text 'Elements of Information Theory' by T. Cover (skipping chapters like Network Information/Gaussian channel should be fine), paired with David MacKay's 'Information Theory, Inference and Learning Algorithms'. Both seem available online: http://www.cs-114.org/wp-content/uploads/2015/01/Elements_of... http://www.inference.org.uk/itprnn/book.pdf They cover some fundamentals of what optimal inference looks…
on “Model-Based Machine Learning Book” ↗I'm not sure whether David Mackay's Information Theory, Inference, and Learning Algorithms (freely available at http://www.inference.org.uk/mackay/itila/book.html) counts as a textbook, but it's one of my favorite books, on any topic, in any genre.
on “Ask HN: What, in your opinion, are the greatest and most useful textbooks?” ↗…breathtakingly beautiful. * The shape of space by Weeks * Solid Shape by Koenderink * Analyse fonctionnelle by Brézis * Tristan Needhams "visual" books about complex analysis and differential forms * Information theory, inference, and learning algorithms by MacKay (great book about probability, plus you can download the .tex source and read the funny comments of the author) And finally, a very old website which is full of mathematical jewels with an…
on “Ask HN: Math books that made you significantly better at math?” ↗if you want to read a good mathematical exposition, check out chapter 6 in Mackay's (the guy giving the presentation) book "Information Theory, Inference, and Learning Algorithms". It covers stream codes and, specifically, arithmetic coding. I've recommended this book in other threads... its an amazing read.
on “Single finger text input without a keyboard (using language models for efficiency)” ↗