Newton Method

You've seen gradient descent: a method that inches toward the best answer by taking small steps downhill. The Newton Method is the express version. It uses extra information — the shape of the curve, not just its slope — to jump almost directly to the optimum, especially when you're already close. This is why it matters: it's the reason some optimization problems are solved in seconds instead of hours. It's the foundation for understanding advanced optimizers like Adam, which power deep learning. It also prepares you for the next big ideas: line search, conjugate gradient, and how to handle massive parameter spaces where computing the full shape is impossible. If you've ever wanted to know why some algorithms 'just know' where to go, this is it. Master this and you'll understand the difference between crawling and jumping in optimization — a leap that separates beginners from practitioners.

Imagine you are trying to find the lowest point in a valley. Gradient descent is like walking downhill, one careful step at a time, always checking the slope under your feet. It works, but it's slow — especially when you're close to the bottom, where the slope gets shallow and you must take tiny steps to avoid overshooting. The Newton Method is different. Instead of just feeling the slope, it also senses the curvature — how quickly the slope is changing. With that extra information, it can estimate where the bottom of the valley is and jump almost directly to it. If the valley is shaped like a perfect bowl, the Newton Method lands on the bottom in one step. If the valley is irregular, it takes a few jumps, each one closer, but still vastly faster than gradient descent. That's the core: gradient descent walks; Newton Method leaps.

Think of searching for the bottom of a waterfall basin in a dark cave. You can only feel the ground with your feet. Gradient descent: you feel the slope underfoot and take a step downhill. If the slope is steep, you take a big step; if it's shallow, a small step. Each step gets you closer, but when you're near the bottom, the slope becomes almost flat, so you're forced to take very small steps to avoid overshooting, and progress crawls. Newton Method is like having a rough map of the entire basin's shape — not just the slope underfoot, but the curvature all around you. You can estimate where the bottom must be, and you jump there. If the basin is perfectly smooth, you land exactly. If it's bumpy, you might need a few jumps, but each jump is far more efficient than a hundred small steps. The map isn't perfect, though. If you've never been there, your map might be wrong — you could jump to a ledge or a small dent that looks like the bottom but isn't the deepest point. That's the risk: the Newton Method is powerful but can be fooled by irregular terrain. When you're close to the true bottom, it's unbeatable; when you're far away, it's a gamble. Gradient descent is safe; Newton is quick. That's why modern optimizers mix both — they start with safe steps, then switch to fast jumps when they're near the solution.

The Newton Method is an optimization algorithm that uses both the slope (first derivative) and the curvature (second derivative) of a function to jump directly toward a minimum or maximum, converging quadratically — meaning the number of correct digits roughly doubles with each step — when starting close to the optimum.

You already have notes on Gradient Descent, which is a first-order method — it uses only the slope. The Newton Method is a second-order method — it also uses the curvature. It's like having both speed and a map. In the learner's library, this connects to Convex Optimization, because on convex problems (where the function is shaped like a single bowl), Newton's Method shines: the 'map' is always accurate, and it converges extremely fast. In nonconvex problems (like deep learning), the map can be misleading, so Newton Method is rarely used directly — instead, you see quasi-Newton methods (like Adam) that approximate the Hessian. This is also related to Line Search, which decides how big a step to take along a chosen direction — Newton's Method actually determines both direction and step size in one shot, based on curvature. Understanding this one concept will prepare you for advanced optimizers you haven't yet studied, and for the broader topic of Optimization.