• Table of Contents

For all later sections assume we have:

• $\Omega \subset \mathbb R^n$ - The domain of the function we wish to optimize, i.e. our feasible region/solution space
• $\mathbf f : \Omega \rightarrow \mathbb R$ - The function we wish to optimize (notice 1d real)

$$ \text{minimize }\mathbf f(\mathbf x) \\ \text{subject to } \mathbf x \in \Omega $$

• Notation

Setup

In this section, we set up the basic mathematical framework we are working with, specifically:

1. Things that we consider ‘solutions’
2. Directions we can ‘move’ in
3. Exploring a class of functions which will come in handy later

Solution Points

Feasible Directions

Convex & Concave functions

Optimizing Differentiable Functions

We now derive conditions that are satisfied by a relative minimum point $\mathbf x^*$.

• We’ll consider movement away from the point along some direction.

Next, we develop a theory towards characterizing global minima, rather than local minima

• We do this by analyzing this through the lens of convexity we covered earlier

TLDR:

• All extrema must satisfy necessary conditions
• Points which satisfy sufficient conditions guarantee optimality

First-order necessary conditions