Build a Large Language ModelSebastian Raschka
Build a Large Language Model
by Sebastian Raschka · first published 2024
26mentions
12recommended by
26HN commenters
2024–2026first – last mentioned
46% recommend54% neutral
Why HN recommends it
A hands-on, step-by-step build of a GPT-style model that explains each of the transformer's building blocks clearly. Suits anyone serious about understanding how LLMs work inside, from the ground up.
Summarised by Claude from recommending comments
Affiliate links. As an Amazon Associate I earn from qualifying purchases.
…requires two or three book chapters at least if you really want to start grokking it! For anyone serious about coming to grips with this stuff, I would strongly recommend Sebastian Raschka's excellent book Build a Large Language Model (From Scratch), which I just finished reading. It's approachable and also detailed. As an aside, does anyone else find the whole "database lookup" motivation for QKV kind of confusing? (in the article, "Query (Q):…
on “The Q, K, V Matrices” ↗I highly recommend Build a large language model from scratch [1] by Sebastian Raschka. It provides a clear explanation of the building blocks used in the first versions of ChatGPT (GPT 2 if I recall correctly). The output of the model is a huge vector of n elements,…
on “I don't know how you get here from "predict the next word."” ↗I stumbled across https://www.manning.com/books/build-a-large-language-model-f... today - it's not complete yet, but sounds like exactly what you're after.
on “Ask HN: Is it feasible to train my own LLM?” ↗…working on publishing a X From Scratch series, which currently includes: * Container Orchestrator: https://www.manning.com/books/build-an-orchestrator-in-go-fr... * LLM : https://www.manning.com/books/build-a-large-language-model-f... * Frontend Framework: https://www.manning.com/books/build-a-frontend-web-framework... The first book might be most interesting, given your particular subject interest.
on “Ask HN: What are some books/resources where we can learn by building” ↗…https://www.oreilly.com/library/view/designing-data-intensiv... - https://www.manning.com/books/100-go-mistakes-and-how-to-avo... - https://www.amazon.com/Systems-Performance-Brendan-Gregg/dp/... And more recently: - https://www.manning.com/books/build-a-large-language-model-f... - https://www.manning.com/books/the-creative-programmer - https://www.manning.com/books/the-programmers-brain - https://www.amazon.com/Understanding-Software-Addison-Wesley... I also find books…
on “Programmers Don't Read Books – But You Should (2008)” ↗…playlist: https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThs... He is building new learning materials under his new company "Eureka Labs": https://eurekalabs.ai Sebastian Raschka's book "Build a Large Language Model (From Scratch) just released: https://www.manning.com/books/build-a-large-language-model-f... All of these resources are excellent.
on “Tutorial on diffusion models for imaging and vision” ↗This is a great book (parts of it are available as blog posts from the author if you want to get a taste of it): https://www.manning.com/books/build-a-large-language-model-f...
on “Open models by OpenAI” ↗…enough to write implementations. I really miss the feeling that I could develop at least simpler LLMs on my own. I am slowly working through Sebastian Raschk's excellent book https://www.manning.com/books/build-a-large-language-model-f... but I will probably never finish it (to be honest).
on “GPT-OSS vs. Qwen3 and a detailed look how things evolved since GPT-2” ↗Not OP but I worked through Sebastian Raschka's "Build a Large Language Model (From Scratch)" [0] and Raj Abhijit Dandekar's "Build a DeepSeek Model (From Scratch)" [1] books. I don't think there is anything in a transformer I couldn't explain in the smallest detail now. [0]: https://www.amazon.com/Build-Large-Language-Model-Scratch/dp...…
on “How LLMs Work” ↗Similarly, I’ve been working through Raschka’s Build a Large Language Model (From Scratch) https://sebastianraschka.com/llms-from-scratch/ and it’s been well worth it (particularly paired with his explainer videos)
on “AI by Hand” ↗…the web that can help with this. One good one is Andrej Karpathy's "Let's build GPT: from scratch, in code, spelled out.": https://www.youtube.com/watch?v=kCc8FmEb1nY A somewhat more intensive one is "Build a Large Language Model (From Scratch)" by Sebastian Raschka: https://www.sebastianraschka.com/llms-from-scratch/ And if you really want to go crazy, Stanford's "CS336: Language Modeling from Scratch" is online: https://cs336.stanford.edu/…
on “Show HN: A working Nintendo in ~1GB of Qwen weights” ↗Sebastian is the author of two excellent books related to LLMs Build a Large Language Model (From Scratch): https://sebastianraschka.com/llms-from-scratch/ Build a Reasoning Model (From Scratch): https://sebastianraschka.com/reasoning-from-scratch/
on “Language models for text classification: From bag-of-words to Jev” ↗I have pre-ordered this book: Build a Large Language Model (From Scratch) https://www.manning.com/books/build-a-large-language-model-f...
on “Ask HN: Textbook Regarding LLMs” ↗Most likely this one. https://www.manning.com/books/build-a-large-language-model-f... (I've taken it from the footnotes on the article)
on “Building LLMs from the Ground Up: A 3-Hour Coding Workshop” ↗Dear all, Recently I purchased "[Build a Large Language Model (From Scratch)](https://www.manning.com/books/build-a-large-language-model-from-scratch)" by Sebastian Raschka, so that I could learn more about how to build and/or fine-tune a LLM, and even developing some applications…
on “Ask HN: How can I experiment with LLMs with a old machine?” ↗