67

I understand what Gradient Descent does. Basically it tries to move towards the local optimal solution by slowly moving down the curve. I am trying to understand what is the actual difference between the plan gradient descent and the newton's method?

From Wikipedia, I read this short line "Newton's method uses curvature information to take a more direct route." What does this intuitively mean?

rehctawrats
  • 201
  • 2
  • 15
London guy
  • 24,942
  • 40
  • 110
  • 169
  • 3
    curvature relates to how Newton's method uses the fuction's second order derivative. Gradient descent is typically first order. – akk Nov 30 '12 at 13:01
  • 2
    Watch this lecture from start to finish: https://www.youtube.com/watch?v=sTCtkkqrY8A&index=15&list=PL3940DD956CDF0622 – ahwillia Jun 04 '15 at 03:51
  • Very similar, also with a good answer: https://math.stackexchange.com/q/1085436/407385 – rehctawrats Dec 20 '18 at 08:21

4 Answers4

70

At a local minimum (or maximum) x, the derivative of the target function f vanishes: f'(x) = 0 (assuming sufficient smoothness of f).

Gradient descent tries to find such a minimum x by using information from the first derivative of f: It simply follows the steepest descent from the current point. This is like rolling a ball down the graph of f until it comes to rest (while neglecting inertia).

Newton's method tries to find a point x satisfying f'(x) = 0 by approximating f' with a linear function g and then solving for the root of that function explicitely (this is called Newton's root-finding method). The root of g is not necessarily the root of f', but it is under many circumstances a good guess (the Wikipedia article on Newton's method for root finding has more information on convergence criteria). While approximating f', Newton's method makes use of f'' (the curvature of f). This means it has higher requirements on the smoothness of f, but it also means that (by using more information) it often converges faster.

Florian Brucker
  • 7,641
  • 3
  • 37
  • 62
  • 1
    I always see mentions of choosing the 'steepest descent'. What does that mean? Is that the most negative number of `f'(x)`? – Terence Chow Feb 14 '14 at 02:33
  • 3
    @Chowza: If your domain is multi-dimensional, e.g. if `f` maps 2D points to real numbers, then the gradient of `f` at any point is not a scalar number but a vector. The reason is that the "steepness" of `f` at that point depends on the direction that you're looking in. It's like standing on a mountain top: If you look north the mountain may drop off very sharply, but to the other sides it may be less steep. Choosing the steepest descent hence means choosing that direction which causes the greatest change in your target function. – Florian Brucker Feb 14 '14 at 08:33
13

Put simply, gradient descent you just take a small step towards where you think the zero is and then recalculate; Newton's method, you go all the way there.

dashnick
  • 1,772
  • 16
  • 30
  • 1
    Is "all the way" true for a non-quadratic function? – bers Oct 17 '16 at 16:19
  • 3
    Yes, for non quadratic functions you are just approximating the first derivative with a line. This is a bit hand wavey but I think it's fine for intuition. – dashnick Nov 07 '16 at 22:23
  • Ok, I agree. All the way to "where *you think* the zero is" is undoubtedly correct. – bers Nov 07 '16 at 22:28
  • If the main difference as you say is "small steps" vs "all the way", could you elaborate on how the size of the "small step" is determined? – Mr Purple Mar 19 '19 at 20:57
  • @MrPurple it's not very well defined, small enough that the gradient doesn't change too much (so you don't keep zigzagging) but large enough that you make progress. A lot of research is around how to optimize this adaptively. For intuition, think like on the order of .1% of the x value. – dashnick Mar 20 '19 at 01:50
4

Edit 2017: The original link is dead - but the way back machine still got it :) https://web.archive.org/web/20151122203025/http://www.cs.colostate.edu/~anderson/cs545/Lectures/week6day2/week6day2.pdf

this power point the main ideas are explained simply http://www.cs.colostate.edu/~anderson/cs545/Lectures/week6day2/week6day2.pdf

I hope this help :)

MimiEAM
  • 2,170
  • 1
  • 23
  • 27
1

If you simply compare Gradient Descent and Newton's method, the purpose of the two methods are different.

Gradient Descent is used to find(approximate) local maxima or minima (x to make min f(x) or max f(x)). While Newton's method is to find(approximate) the root of a function, i.e. x to make f(x) = 0

In this sense, they are used to solve different problems. However, Newton's method can also be used in the context of optimization (the realm that GD is solving). Because finding maxima or minima can be approached by finding f'(x) = 0 which is exactly Newton's method is used for.

In conclusion, two methods can be used in optimization: 1)GD and 2)find x so f'(x)=0 and Newton's method is just a way to solve that second problem.

Cheng
  • 61
  • 7