• Table of Contents

(Very) Simplified Overview

This page goes over matrix differentiation concepts often used in ML. Below are some possible functions which are covered by the table:

• $\mathbf f : \mathbb R \rightarrow \mathbb R^n$, e.g vector valued functions
• $f : \mathbb R^n \rightarrow \mathbb R$, e.g vector norms
• $f : \mathbb R^{n \times n} \rightarrow \mathbb R$, e.g determinant
• $\mathbf f: \mathbb R ^n \rightarrow \mathbb R ^n$, e.g linear transformations

Vector Differentiation (Review)

Differentiation as a Linear Operator

For simplicity, assume:

• Our domain is an open set

Warning, this part is a bit hand-wavey

<aside> 📖 Main idea:

$$ \begin{align*} d \mathbf f &= \mathbf f(\mathbf x + d \mathbf x) -f(\mathbf x) \\ &= \mathbf f'(\mathbf x)d\mathbf x \end{align*} $$

Where:

• $d \mathbf f$ is $m \times 1$
• $\mathbf f’(\mathbf x)$ is the $m \times n$ Jacobian
• $d \mathbf x$ is $n \times 1$ </aside>

Linearization

Recall the linearization of a single variable function, which allow us to approximate values near a point $x$:

$$ f(x + \delta x) = f(x) + f'(x)\delta x + \dots $$