Table of Contents

The Setup

A relatively math-free introduction to the intuition behind the construction of much of the theory of Bayesian inference.

What does it mean to think Bayesian?

The frequentist definition of probability is based on asymptotic observations over a large number of trials. Suppose we have some event, $E$ and $n$ trials. Then the probability of our event, $P(E)$ under the frequentist framework is:

$$ P(E) = \lim_{n \rightarrow \infty} \frac{n_e}{n} $$

On the other hand, Bayesians view probability as a measure of belief (with some caveats).

This is an important distinction, because this means beliefs are tied to individuals, rather than observed frequencies in the world.
It’s also arguably closer to the way humans think of probability

Now recall the well-understood Bayes’ theorem:

<aside> 📖 For any 2 events, $A, X$, we can express $P(A|X)$ as:

$$ P(A|X) = \frac{P(A)|P(X|A)}{P(X)} $$

Similarly, for random variables $\bm \theta, \mathbf X$:

$$ p(\bm \theta| \mathbf X) = \frac{p(\mathbf X|\bm \theta) p(\bm \theta)}{p(\mathbf X)} $$

</aside>

You might say this is just a consequence of algebra, not explicitly connected to Bayesian inference.

…But we can do surprisingly a lot when we re-cast the factors under our Bayesian framework (We’ll also assign them some fancy names)

Expression	Name
$p(\bm \theta	\mathbf X)$
$p(\bm \theta)$	Prior
$p(\mathbf X	\bm \theta)$
$p(\mathbf X)$	Evidence

Now let’s break things down:

<aside> 📖 'beliefs' fall under the category of posterior and prior.

These belong to the individual. They are mutable, and subject to change in the face of evidence.
Prior = captures subjective beliefs before seeing data.
posterior = updated beliefs after observing data

PITFALL WARNING: Try not to interpret our prior literally as p(parameters); think more abstractly in what they represent. We are personally assigning mass/density to assumptions we find believable. Parameters are just how we implement these beliefs.

</aside>

<aside> 📖 Evidence and likelihood are a bit more rigid as they rely on our data.

likelihood tells us how reasonable our observations are under our set of beliefs. We can only change this by changing our beliefs, which is reasonable.
Evidence, on the other hand is immutable - Observations cannot be disputed. This also makes sense. </aside>

Sharp bits:

Priors represent more than just subjective beliefs: e.x We can incorporate objective data into our beliefs. Sometimes we even choose priors out of mathematical convenience (like conjugacy)