Elements of Information Theory
by T. M. Cover, Joy A. Thomas · first published 2006
71% recommend29% neutral
Why HN recommends it
Clearly written information theory textbook with elementary starts, direct examples and readable notation. Covers compression, Kolmogorov complexity and gambling/Kelly. Suits math-inclined readers wanting fundamentals.
Summarised by Claude from recommending comments
Affiliate links. As an Amazon Associate I earn from qualifying purchases.
Mentions per year
What Hacker News says
Related books
2009–2024An introduction to Kolmogorov complexity and its applications
Ming Li, Paul Vitanyi
Recommended by 5 · mentioned by 105 recommend · 10 mention5 recommend · 105 recommend
2010–2020Neuro-dynamic programming
Dimitri P. Bertsekas
Recommended by 3 · mentioned by 43 recommend · 4 mention3 recommend · 43 recommend
2009–2023A logical approach to discrete math
Gries, David.
Recommended by 6 · mentioned by 66 recommend · 6 mention6 recommend · 66 recommend
2007–2026Information Theory, Inference & Learning Algorithms
David J.C. MacKay
Recommended by 77 · mentioned by 9377 recommend · 93 mention77 recommend · 9377 recommend
2010–2026Reinforcement Learning
Richard S. Sutton
Recommended by 18 · mentioned by 3018 recommend · 30 mention18 recommend · 3018 recommend
2009–2026Options, futures, & other derivatives
John C. Hull
Recommended by 25 · mentioned by 3525 recommend · 35 mention25 recommend · 3525 recommend
Winning Ways for Your Mathematical Plays
Elwyn R. Berlekamp, John Horton Conway, Richard K. Guy
Recommended by 16 · mentioned by 1916 recommend · 19 mention16 recommend · 1916 recommend
2009–2026Convex optimization
Stephen P. Boyd
Recommended by 40 · mentioned by 5740 recommend · 57 mention40 recommend · 5740 recommend
…and Tardos) - Feynman Lectures on Physics - Landau & Lifschitz's series - Vladimir Arnold's books on ODEs and Classical Mechanics - Winning Ways for your Mathematical Plays (by Berlekamp et al.) - Elements of Information Theory (by Cover & Thomas) - Information Theory, Inference, and Learning Algorithms (by MacKay) - Network Optimization (by Bertsekas) - Convex Optimization (by Boyd & Vandenberghe) - Nonlinear Control Systems…
on “Reading list for those who love to learn” ↗…of the material in 10.) Golub, G., and Van Loan, C. (1996). "Matrix Computations" Johns Hopkins. It's good to know some information theory. The classic is: 11.) Cover, T. and Thomas, J. "Elements of Information Theory" Wiley. Finally, if you want to start to learn some more abstract math, you might want to start to learn some functional analysis (if you haven't already). Functional analysis is essentially linear…
on “Learning About Statistical Learning” ↗…1) The Information by Gleick is a book covering everything from biographies of the key historical figures and contributors to contemporary applications of information theory in quantum mechanics. 2) Elements of Information Theory by Joy Thomas and Thomas Cover is a thorough and engaging textbook. 3) Quantum information and quantum computation by Nielsen and Chuang explaining both classical and quantum information with some practical…
on “Information Theory: A Tutorial Introduction” ↗…For algorithms I don't have a preference. There is Dasgupta-Papadimitriou-Vazirani; Cormen-Leisersen-Rivest-Stein; Kleinberg-Tardos; and more. A bit farther from core CS, I also like: - Cover, Thomas. Elements of Information Theory. - Li, Vitanyi. An Introduction to Kolmogorov Complexity and Its Applications
on “Ask HN: What are the most fundamental books on computer science?” ↗Elements of Information Theory, by Thomas Cover is also a very good read for fundamentals.
on “Journey into Information Theory” ↗If you are math inclined, Cover & Thomas’ “elements of information theory” is a very clear and readable introduction (more for coding, less for channels, but great overall). If you consider giving it another chance.
on “What is the Kullback-Leibler divergence?” ↗If you're looking for a bit of an unconventional entry point, I recommend the seminal text 'Elements of Information Theory' by T. Cover (skipping chapters like Network Information/Gaussian channel should be fine), paired with David MacKay's 'Information Theory, Inference and Learning Algorithms'. Both seem available online:…
on “Model-Based Machine Learning Book” ↗…surprising, to me at least, that the length of a thing is related to its probability. From that relationship come a number of fascinating connections to many other fields. Check out Cover and Thomas' "Elements of Information Theory" for a very approachable introduction to all these connections.
on “Information Theory with TensorFlow” ↗…Commutative Algebra. * Bourbaki (in French). * Gasper, Rahman - Basic Hypergeometric Functions. * Hasti et al - Elements Of Statistical Learning. * Rudin - Real & Complex Analysis. * Thomas, Thomas - Elements of Information Theory.
on “Ask HN: Which mathematics books taught you the most in your career?” ↗For information theory, a really recommend: Thomas M. Cover, Joy A. Thomas "Elements of Information Theory" - it starts elementarily, has direct examples, straightforward definitions and readable notation. (And first recommendation by Cosma Shalizi: http://vserver1.cscs.lsa.umich.edu/~crshalizi/notabene/infor...)
on “A Software Engineer’s Adventures in Learning Mathematics” ↗Elements of Information Theory - Cover, Thomas Network Information Theory - El Gamal, Kim Adaptive Wireless Communications - Bliss, Govindasamy Wireless Communications - Goldsmith
on “Ask HN: What are the best textbooks in your field of expertise?” ↗Elements of Information Theory by Cover and Thomas. Very accessible overview of the field.
on “Ask HN: Books you read in 2018?” ↗Chapter 6 of "Elements of Information Theory" by Cover and Thomas is "Gambling and data compression". It explains that maximizing expected utility almost always leads to ruin. I didn't believe it when I first read it. Of course it is exactly…
on “The Ergodicity Problem in Economics” ↗> 2) Elements of Information Theory by Joy Thomas and Thomas Cover is a thorough and engaging textbook. Thomas & Cover is THE textbook on information theory -- no doubt -- but you and I have very different definitions of engaging. For…
on “Information Theory: A Tutorial Introduction” ↗Try the textbook Elements of Information Theory by Cover and Thomas (2006)
on “Six (and a half) intuitions for KL divergence” ↗