“energy” ~ “scores” ~ “plausibility” → ranking things by how believable they are without invoking normalization for explicit probability/density

Conventional

Energy function → partition function → boltzmann distribution

<aside> 📖

Consider a single dependent variable $\mathbf x \in \mathcal X \subseteq \mathbb R^n$. The density given by an EBM is

$$ p_{\bm \theta}(\mathbf x) = \frac{\exp(-E_{\bm\theta}(\mathbf x))}{Z_{\bm \theta}} $$

With $Z_{\bm \theta} \in \mathbb R$ being a normalizing constant called the partition function, defined:

$$ Z_{\bm \theta} = \int_{\mathcal X} \exp(-E_{\bm \theta}(\mathbf x)) \mathrm d \mathbf x $$

Where:

$E_{\bm \theta} : \mathbb R^n \rightarrow \mathbb R$ is a functional parameterized by $\bm \theta$, called the energy </aside>

Note:

The density can be implicit, or explicit. e.g:
- Anything that uses softmax → explicit
- Every neural net trained with a loss (regularization included) trivially admits an energy function over its inputs

Sometimes, $Z_{\bm \theta}$ is intractable:

We have to integrate over the realm of all $\mathbf x$’s whatever they may be (parameters, hidden states, etc.)
If $\mathcal X$ is some continuum, this is especially hard.

LeCun Version