• Table of Contents

Intro

Supervised learning can be usually seen as picking one function (model) $f$, from a set of possible functions $F$ - umass sml2011

Loss Function and Risk

Loss function

<aside> 📖 Given some particular pair of inputs, $\mathbf x$ and outputs $\mathbf y$, the loss is defined as follows:

$$ L(f(\mathbf x), \mathbf y) $$

Where:

• $f$, our ‘model’ is a function is chosen from a hypothesis space $F$ </aside>

This tells us how much the prediction, $f(\mathbf x)$ “hurts”, given that the true output is $\mathbf y$

Risk

<aside> 📖 Assuming we have some ‘true’ distribution over inputs $\mathbf x$ and $\mathbf y$, the risk is defined as:

$$ \begin{align*} R_{true} &= \mathbb{E}[L(f(\mathbf x), \mathbf y)]\\ &= \int \int p(\mathbf x, \mathbf y)L(f(\mathbf x), \mathbf y)d\mathbf xd\mathbf y \end{align*} $$

i.e. the expected loss, true loss

</aside>

Risk measures how much on average, it ‘hurts’ to use $f$ as for our predictions.

Choosing a loss function

“What loss function is appropriate?”

• This question cannot be purely answered by math with the given information

The loss function depends on the priorities of the user

• e.g. user decides how much certain outcomes ‘hurt’

Empirical Risk

Assume we have:

• $D = \{(\mathbf x_1, \mathbf y_1), (\mathbf x_2, \mathbf y_2), \ldots, (\mathbf x_n, \mathbf y_n)\}$, our dataset
• $\mathbf x_i$ represents a set of input features to a model