Managing gigabytes
by Ian H. Witten, Alistair Moffat, Timothy C. Bell · first published 1999
74% recommend23% neutral3% critical
Why HN recommends it
Reference on compressing and indexing large text collections: inverted indexes, entropy coders, and ranking, with an engineering slant toward implementation. Suits newcomers to search, IR, and compression.
Summarised by Claude from recommending comments
"In this fully updated second edition of the highly acclaimed Managing Gigabytes, authors Witten, Moffat, and Bell continue to provide unparalleled coverage of state-of-the-art techniques for compressing and indexing data. Whatever your field, if you work with large quantities of information, this book is essential reading - an authoritative theoretical resource and a practical guide to meeting the toughest storage and access challenges. It covers the latest developments in compression and indexing and their application on the Web and in digital libraries. It also details dozens of powerful te
Affiliate links. As an Amazon Associate I earn from qualifying purchases.
Mentions per year
What Hacker News says
Related books
2009–2022Modern information retrieval
R. Baeza-Yates
Recommended by 4 · mentioned by 84 recommend · 8 mention4 recommend · 84 recommend
2015–2022Lucene in action
Michael McCandless
Recommended by 5 · mentioned by 65 recommend · 6 mention5 recommend · 65 recommend
2017–2023Relevant Search
Doug Turnbull, John Berryman
Recommended by 2 · mentioned by 52 recommend · 5 mention2 recommend · 52 recommend
2009–2024Introduction to information retrieval
Christopher D. Manning, Prabhakar Raghavan, Hinrich Schütze
Recommended by 41 · mentioned by 4441 recommend · 44 mention41 recommend · 4441 recommend
2009–2015Data-intensive text processing with MapReduce
Jimmy Lin
Recommended by 5 · mentioned by 65 recommend · 6 mention5 recommend · 65 recommend
2007–2021Data mining
I. H. Witten, Ian H. Witten, Eibe Frank
Recommended by 9 · mentioned by 149 recommend · 14 mention9 recommend · 149 recommend
2008–2026Speech and language processing
Dan Jurafsky, Daniel Jurafsky, James H. Martin
Recommended by 41 · mentioned by 4541 recommend · 45 mention41 recommend · 4541 recommend
2008–2025More programming pearls
Jon Louis Bentley
Recommended by 12 · mentioned by 1512 recommend · 15 mention12 recommend · 1512 recommend
This book has a nice treatment of that kind of compression: https://www.amazon.com/Managing-Gigabytes-Compressing-Multim... For instance you might be keep track of facts like the word "the" is contained in document 1 the word "john" is contained in document 1 the word "the" is contained in…
on “Sorted Integer Compression” ↗This site rocks. Some pirated books well worth reading (buy the book if you can afford it): Witten, Moffat, Bell. Managing gigabytes: compressing and indexing documents and images (How to build a search engine) Aho A.V., Lam M.S., Sethi R., Ullman J.D. Compilers: Principles, techniques, and tools (Dragon book) Anyone else have any…
This is a great intro to full-text search. These techniques were the basis of some of the early web search engines (pre Google). More of this stuff is explored in Witten Moffat and Bell's book "Managing Gigabytes" which dates from the era of those early search engines and when "Gigabytes" was something unimaginably large that certainly wouldn't fit in RAM and perhaps not even on a single disk.
on “Let's build a full-text search engine” ↗Managing Gigabytes (Witten) Information Retrieval (Manning) Text Compression (Bell) Natural Language Processing (Manning) Natural Language Understanding (Allen) Speech and Language Processing (Jurafsky) The Text Mining…
on “Ask HN: Best Text Mining Resources” ↗…Segaran, Programming Collective Intelligence Toffoli and Margolus, Cellular Automata Machines Niklaus Wirth, Project Oberon: The Design of an Operating System and Compiler Witten, Moffat, & Bell, Managing Gigabytes (mostly from my old list at http://wry.me/~darius/personal/books.html)
on “Which programming book you just couldn’t stop reading from start to end?” ↗If you want to know some of the techniques, read "Managing Gigabytes: Compressing and Indexing Documents and Images" by Ian H. Witten, Alistair Moffat, and Timothy C. Bell. If you want to actually implement search for your users, use solr. And use Tika for extracting…
on “Full Text Search Algorithms” ↗…"ranking functions" (how queries are matched to documents). The former is basically a solved problem. Lucene/ElasticSearch and Google are using basically the same techniques, and you can read about them in Managing Gigabytes [1], which was first published over 2 decades ago. Google may be a generation or so ahead - they were working on a new system to take full advantage of SSDs (which turn out to be very good for search,…
on “What every software engineer should know about search” ↗I always have to plug Managing Gigabytes whenever a discussion of computer books comes up. Great reference for anyone dealing with searching or compressing large amounts of information: http://www.amazon.com/Managing-Gigabytes-Compressing-Multime...
on “Five Memorable Books About Programming” ↗- Modern Information Retrieval by Baeza Yates - Data Mining Book by Jiawei Han et al - Managing Gigabytes by Witten et al - Hypertext Mining book by Chakrabarti
on “Ask HN: Best Text Mining Resources” ↗I recommend reading 'Managing Gigabytes' by Witten, Moffat and Bell: http://www.amazon.com/Managing-Gigabytes-Compressing-Multime...
on “Build your own web search engine” ↗…enjoy reading articles about the rediscovery of indexing large amounts of read-only data, I'd highly recommend reading this book which is a treasure trove about this kind of work: http://www.amazon.com/Managing-Gigabytes-Compressing-Multime...
on “Impossible Engineering Problems Often Aren't” ↗"Managing Gigabytes" by Witten et al is _the_ book that got me into the field of compression back in the day. Granted, it’s a bit dated now as there’s been a lot of progress in the field, but I’d still heartily recommend…
on “Modern LZ Compression (2019)” ↗…documents (essentially what the terms mean in the traditional Information Retrieval, e.g Lucene, SOLR, Elastic) then this is a pretty good on the subject that taught me a lot: https://www.amazon.com/Managing-Gigabytes-Compressing-Multim...
on “Ask HN: I want to dive into how to make search engines” ↗Thanks for the link. I was motivated to write the post after reading Moffat’s book ‘Managing Gigabytes’. A pearl from the 90’s. The authors mention this technique in the second edition.
on “Building a data compression utility in Haskell using Huffman codes” ↗…a textbook on information retrieval co-authored by the head of Yahoo Research and the authors of 'Foundations of Statistical NLP' (another great textbook). Kind of like a more up to data edition of 'Managing Gigabytes', and it's just as good if not better.
on “How to Build a Search Engine” ↗