This section provides a theoretical framework for framing most optimization problems in machine learning, as well as intuition/derivations of some commonly used algorithms
Problem Statement
We wish to find the minima of $f$ given a vector of parameters $\bm \theta$. Here, we only consider the 1D case.
Suppose we have a function $f(\theta) : \mathbb R \rightarrow \mathbb R$
Taylor Series Expansion
<aside> 📖 The Taylor series expansion of a function $f$ centered around a point $\theta_0$ is:
$$ f(\theta) \approx f(\theta_0) + f'(\theta_0)(\theta - \theta_0) + \frac{1}{2}f''(\theta_0)(\theta - \theta_0)^2 + \dots $$
More generally to the $n$-th order:
$$ f(\theta) \approx \sum_{i=0}^n \frac{f^{i}(\theta_0)}{i!}(\theta - \theta_0)^i $$
</aside>
Main takeaway: We can approximate $f$ near $\theta_0$
Critical Points
<aside> 📖 A critical point $\theta^* \in \mathbb R$ is a point where:
$$ f'(\theta^*) = 0 $$
</aside>
Critical points can correspond to minima, maxima or saddle points
We can use the second derivative to test whether a point is a maximum or minimum: