Frequentist parameter estimation

Likelihood Function

Maximum Likelihood Estimation

^ This is pretty important. There are a lot of setups in machine learning which reduce to MLE:

• Minimizing the MSE = finding extrema of the likelihood function if we assume gaussian response variables
  • Equivalent to MLE if our likelihood is convex. Otherwise, we only satisfy first order necessary conditions of the likelihood with minimization of MSE
• Cross entropy under multinomial targets is equivalent to the negative log likelihood

Bayesian Parameter Estimation

MAP Estimation