This practical work aims to provide the necessary elements to understand the implementations of gradient descents.
Here is an overview of the points covered :
-
Defining a set of test functions
-
Calculation of the gradient of a function in an approximate way (to cover cases where the explicit calculation of the gradient is impossible or cumbersome).
-
The descent in the direction of the Gradient at constant pitch and the Pitch optimization by Backtracking
-
Choosing a direction other than the gradient
- Steepest slope as standard
- Conjugate gradient
-
Acceleration:
Momentum,Nesterov,Adam. -
Newton's Method and Quasi-Newton's Method
Throughout the course of the practical work, to goal was to be very careful to validate the validity of the written programs by means of a set of tests. We had to look at the influence of the parameters on the convergence and we had to compare the advantages of the diferent methods against each other.