Programming Massively Parallel Processors
by David Kirk, Wen-Mei W. Hwu · first published 2010
15mentions
8recommended by
15HN commenters
2011–2026first – last mentioned
53% recommend47% neutral
Why HN recommends it
Hands-on introduction to GPU programming with CUDA. It explains how GPUs work and focuses on performance, and it is approachable for engineers moving from CPU programming to GPU architecture.
Summarised by Claude from recommending comments
Affiliate links. As an Amazon Associate I earn from qualifying purchases.
If you like textbooks, I would recommend "Programming Massively Parallel Processors" by Hwu, Kirk, and Hajj. [1] The most recent edition was published in 2022. If you like lecture videos, I would recommend Hajj's YouTube playlist of 2021 lectures [2]. He works through a subset of the…
on “Ask HN: Best way to learn GPU programming?” ↗this book: Programming Massively Parallel Processors by Wen-mei W. Hwu , David B. Kirk , Izzat El Hajj seems to be tailor mode for folks transitioning from cpu -> gpu arch.
on “Introduction to CUDA programming for Python developers” ↗…Programming for Video Games (not CUDA specific, but covered): http://users.ece.gatech.edu/lanterma/mpg/ UIUC ECE498AL - Applied Parallel Programming: http://courses.engr.illinois.edu/ece498/al/ Books: Programming Massively Parallel Processors: http://www.amazon.com/Programming-Massively-Parallel-Process... CUDA By Example: http://www.amazon.com/CUDA-Example-Introduction-General-Purp... If you decide to go the CUDA route, as opposed to OpenCL,…
on “Ask HN: Best GPU (scientific) programming tutorials?” ↗To ones who are interested: "Programming Massively Parallel Processors: A Hands-on Approach" is a great book to learn CUDA programming, and it talks mostly about performance because, after all, GPU is about speed. Unlike normal programming books, it talks a lot about how…
on “Optimization Techniques for GPU Programming [pdf]” ↗…direction too. GPU race is getting really hot and there is a lot of work being done to squeeze every ounce of performance especially for LLM training and inference. One resource I would recommend is “Programming massively parallel processors” [1] I am also learning it as my hobby project and uploading my notes here [2] [1] https://shop.elsevier.com/books/programming-massively-parall... [2] https://github.com/mandliya/PMPP_notes
on “Ask HN: Going low-level – what to learn next?” ↗My go to is Programming Massively Parallel Processors by Wen-Mei Hwu, excellent really approachable introduction. [0] [0] https://a.co/d/9fmbZqg
on “DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling” ↗These should keep you busy for months: https://www.gpumode.com/ resources and discord community Book: Programming massively parallel processors nvidia cuda docs are very comprehensive too https://github.com/srush/GPU-Puzzles
on “Ask HN: How to learn CUDA to professional level” ↗Thanks! I highly recommend the book Programming Massively Parallel Processors
on “Micrograd-CUDA: adapting Karpathy's tiny autodiff engine for GPU acceleration” ↗Programming Massively Parallel Processors: A Hands-on Approach by David Kirk and Wen-mei Hwu.
on “Initial CUDA Performance Lessons” ↗…shouldn't hit any constraints. Also i recommend "Cuda for Engineers" by Storti /Yurtoglu as a good first cuda book, and the Wrox Pro C Cuda programming as 2nd book. There's another that just came out, Programming Massively Parallel Processors, Third Edition by Kirk /Hwu, that looks good but i haven't read it
on “An Even Easier Introduction to CUDA” ↗There are also video lectures which are almost 1-1 mapping of the book Programming Massively Parallel Processors: https://www.youtube.com/watch?v=4pkbXmE4POc&list=PLRRuQYjFhp...
on “Optimization Techniques for GPU Programming [pdf]” ↗Currently "Programming Massively Parallel Processors" (Hwu, Kirk, El Hajj). It is full of typos, but once I look over that, it has some quite educational techniques. The next one waiting on the book shelf is "Compilers" (Lam, Ullman). Why did I buy them?…
on “Ask HN: What technical book did you buy and read?” ↗The Programming Massively Parallel Processors textbook has a chapter on histograms. You have basically mentioned the main trick though (privatization). For this problem I think coursening would also help (so everything isn’t queued up on the copy…
on “The One Billion Row Challenge in CUDA” ↗The book - PMPP - Programming Massively Parallel Processors The YouTube Channel - CUDA_MODE - it is based on PMPP I could not find the channel, but here is the playlist https://www.youtube.com/watch?v=LuhJEEJQgUM&list=PLVEjdmwEDk... Once done, you would be on…
on “Ask HN: How to learn CUDA to professional level” ↗First one I clicked on is 404: Programming Massively Parallel Processors: A Hands-on Approach (3rd Edition) https://www.cambridge.org/core/books/programming-in-parallel...
on “CUDA Books” ↗