Backtracking line search

In (unconstrained) mathematical optimization, a backtracking line search is a line search method to determine the amount to move along a given search direction. Its use requires that the objective function is differentiable and that its gradient is known. The method involves starting with a relatively large estimate of the step size for movement along the line search direction, and iteratively shrinking the step size (i.e., "backtracking") until a decrease of the objective function is observed that adequately corresponds to the amount of decrease that is expected, based on the step size and the local gradient of the objective function. A common stopping criterion is the Armijo–Goldstein condition. Backtracking line search is typically used for gradient descent (GD), but it can also be used in other contexts. For example, it can be used with Newton's method if the Hessian matrix is positive definite. Given a starting position x {\displaystyle \mathbf {x} } and a search direction p {\displaystyle \mathbf {p} } , the task of a line search is to determine a step size α > 0 {\displaystyle \alpha >0} that adequately reduces the objective function f : R n → R {\displaystyle f:\mathbb {R} ^{n}\to \mathbb {R} } (assumed C 1 {\displaystyle C^{1}} i.e. continuously differentiable), i.e., to find a value of α {\displaystyle \alpha } that reduces f ( x + α p ) {\displaystyle f(\mathbf {x} +\alpha \,\mathbf {p} )} relative to f ( x ) {\displaystyle f(\mathbf {x} )} . However, it is usually undesirable to devote substantial resources to finding a value of α {\displaystyle \alpha } to precisely minimize f {\displaystyle f} . This is because the computing resources needed to find a more precise minimum along one particular direction could instead be employed to identify a better search direction. Once an improved starting point has been identified by the line search, another subsequent line search will ordinarily be performed in a new direction.