Yihui Xie

grad.desc()

Demonstration of the Gradient Descent Algorithm

Yihui Xie & Lijia Yu / 2017-04-04


This function provids a visual illustration for the process of minimizing a real-valued function through Gradient Descent Algorithm.

Gradient descent is an optimization algorithm. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or the approximate gradient) of the function at the current point. If instead one takes steps proportional to the gradient, one approaches a local maximum of that function; the procedure is then known as gradient ascent.

The arrows are indicating the result of iterations and the process of minimization; they will go to a local minimum in the end if the maximum number of iterations ani.options('nmax') has not been reached.

library(animation)
## default example
ani.options(interval = 0.3, nmax = 50)
xx = grad.desc()
xx$par  # solution
##          x          y 
## -0.0675852  0.0009736
xx$persp(col = "lightblue", phi = 30)  # perspective plot
## define more complex functions; a little time-consuming
f1 = function(x, y) x^2 + 3 * sin(y)
xx = grad.desc(f1, pi * c(-2, -2, 2, 2), c(-2 * pi, 2))
xx$persp(col = "lightblue", theta = 30, phi = 30)
## need to provide the gradient when deriv() cannot handle the
## function
grad.desc(FUN = function(x1, x2) {
  x0 = cos(x2)
  x1^2 + x0
}, gr = function(x1, x2) {
  c(2 * x1, -sin(x2))
}, rg = c(-3, -1, 3, 5), init = c(-3, 0.5), main = expression(x[1]^2 + 
  cos(x[2])))
## Warning in grad.desc(FUN = function(x1, x2) {: Maximum
## number of iterations reached!
## or a even more complicated function
ani.options(interval = 0, nmax = 200)
f2 = function(x, y) sin(1/2 * x^2 - 1/4 * y^2 + 3) * cos(2 * 
  x + 1 - exp(y))
xx = grad.desc(f2, c(-2, -2, 2, 2), c(-1, 0.5), gamma = 0.1, 
  tol = 1e-04)
## click your mouse to select a start point
if (interactive()) {
  xx = grad.desc(f2, c(-2, -2, 2, 2), interact = TRUE, tol = 1e-04)
  xx$persp(col = "lightblue", theta = 30, phi = 30)
}