Numerical Optimization
by Jorge Nocedal, Stephen Wright · first published 2006
67% recommend22% neutral11% critical
Why HN recommends it
Covers nonlinear optimization, including line search, BFGS, and Newton-type methods, with implementation detail. Suited to engineers and machine learning practitioners who need the algorithms in depth.
Summarised by Claude from recommending comments
Affiliate links. As an Amazon Associate I earn from qualifying purchases.
Mentions per year
What Hacker News says
Related books
2013–2024Operations Research
Wayne L. Winston
Recommended by 3 · mentioned by 43 recommend · 4 mention3 recommend · 43 recommend
2009–2025Matrix computations
Gene H. Golub
Recommended by 19 · mentioned by 2419 recommend · 24 mention19 recommend · 2419 recommend
2009–2026Convex optimization
Stephen P. Boyd
Recommended by 40 · mentioned by 5740 recommend · 57 mention40 recommend · 5740 recommend
2010–2026Numerical linear algebra
Lloyd N. Trefethen
Recommended by 26 · mentioned by 2926 recommend · 29 mention26 recommend · 2926 recommend
2007–2026The art of computer programming
Donald Knuth
Recommended by 336 · mentioned by 1,125336 recommend · 1,125 mention336 recommend · 1,125336 recommend
He said "your #1", making reference to the first book in OPs list: "Numerical Optimization" by Jorge Nocedal and Stephen J. Wright. It's pretty fun to read.
on “Ask HN: Literature for mathematical optimization?” ↗…Paul https://www.springer.com/us/book/9780387922799 For modeling tricks for convex programming, Convex Optimization by Stephen Boyd and Lieven Vandenberghe https://stanford.edu/~boyd/cvxbook/ For basic nonlinear optimization algorithms, Numerical Optimization by Nocedal, Jorge, Wright, S. https://www.springer.com/us/book/9780387303031 I've never really found the ultimate optimization book, but the references above tend to have the right modeling techniques,…
on “Applied Mathematical Programming (1977)” ↗Nocedal's text, 'Numerical Optimization' is the standard for that field. As he notes, I've always been surprised that more techniques in ML do not leverage the Hessian to get quadratic convergence rates. Nevertheless, the most interesting…
on “Optimization Methods for Large-Scale Machine Learning” ↗…infimum. The background necessary to understand this book is real analysis and as a quick aside I think Rudin's Principles of Mathematical Analysis is the best for this. Second, is Nocedal and Wright's Numerical Optimization book. It provides a good overview of the powerful algorithms in optimization that we should be using. Now, it's weakness is that it often cheats and uses a stronger constraint qualification than we're…
on “Confession as an AI researcher; seeking advice” ↗…implementation details of Linnainmaa's BP, but there are all sorts of improvements to plain reverse mode auto-grad through stuff like Jacobian-Vector products that save an enormous amount of memory. Nocedal's "Numerical Optimization" covers a great deal on how to implement autograd, and I can easily recommend it to anyone interested. [1] https://www.reddit.com/r/MachineLearning/comments/e5vzun/d_j...
on “LeCun's 2022 paper on autonomous machine intelligence does not cite prior work” ↗…Recipes offers very quickly. There are some good books in the list you linked, but that list is also very outdated. Notably absent: any of Trefethen's books (ATAP in particular), Nocedal and Wright's "Numerical Optimization", Stoer & Bulirsch, anything about convex optimization, anything about the finite element method, anything about geometry or CAD, anything about Monte Carlo, anything about special functions, and on and…
on “Matters Computational (2010) [pdf]” ↗…Scipy's default line search method -- used for cg and bfgs -- is a python port [2] of the dcsrch routine from MINPACK2. A good reference covering line search methods & BFGS is Nocedal & Wright's 2006 book Numerical Optimization. [1] https://en.wikipedia.org/wiki/Wolfe_conditions [2] https://github.com/scipy/scipy/blob/main/scipy/optimize/_dcs...
on “Why Momentum Works (2017)” ↗…you know any books on math optimization that are essential for anybody getting into this field? Is there any classical literature for optimization related to ML? Thanks! My current list includes: 1. Numerical Optimization by Jorge Nocedal Stephen J. Wright 2. Algorithms for Optimization - introduction to optimization with a focus on practical algorithms 3. Algorithms for Decision Making - a broad introduction…
on “Ask HN: Literature for mathematical optimization?” ↗…hand. Even if only a single Krylov iteration is done, there are many benefits to using this methodology to help handle the scaling of the problem. These algorithms are described fully in a book such as Numerical Optimization by Nocedal and Wright. The current paper does not directly affect most ML algorithms because it speaks directly to convex optimization. Most ML models are highly nonlinear, so globalization of the optimization…
on “Regularized Newton Method with Global $O(1/k^2)$ Convergence” ↗Though I have complaints with it, Numerical Optimization by Nocedal and Wright is probably the best reference for modern optimization techniques. My complaint with it is that they also present many historical techniques that I would argue should not be used…
on “Why “Gradient” Descent?” ↗Next time mention the name of the book first. Numerical Optimization (Jorge Nocedal, Stephen J. Wright).
on “Algorithms for Optimization [pdf]” ↗…Python ecosystem, BFGS is used as the default nonlinear optimisation method by scipy.optimize.minimize [2] if your problem doesn't have any constraints or bounds. Scipy optimize cites Nocedal & Wright - Numerical Optimization (2nd ed) 2006 as a reference. BFGS is Algorithm 6.1 on page 140. Nocedal & Wright say "the most popular quasi-Newton algorithm is the BFGS method, named for its discoverers Broyden, Fletcher, Goldfarb,…
on “Ink and Switch Constraint System (2023)” ↗