Laplace's approximation

Laplace's approximation or the quadratic approximation (QUAP) provides an analytical expression for a posterior probability distribution by fitting a Gaussian distribution with a mean equal to the MAP solution and precision equal to the observed Fisher information. The approximation is justified by the Bernstein–von Mises theorem, which states that, under regularity conditions, the error of the approximation tends to 0 as the number of data points tends to infinity. For example, consider a regression or classification model with data set { x n , y n } n = 1 , … , N {\displaystyle \{x_{n},y_{n}\}_{n=1,\ldots ,N}} comprising inputs x {\displaystyle x} and outputs y {\displaystyle y} with (unknown) parameter vector θ {\displaystyle \theta } of length D {\displaystyle D} . The likelihood is denoted p ( y | x , θ ) {\displaystyle p({\bf {y}}|{\bf {x}},\theta )} and the parameter prior p ( θ ) {\displaystyle p(\theta )} . Suppose one wants to approximate the joint density of outputs and parameters p ( y , θ | x ) {\displaystyle p({\bf {y}},\theta |{\bf {x}})} . Bayes' formula reads: p ( y , θ | x ) = p ( y | x , θ ) p ( θ | x ) = p ( y | x ) p ( θ | y , x ) ≃ q ~ ( θ ) = Z q ( θ ) . {\displaystyle p({\bf {y}},\theta |{\bf {x}})\;=\;p({\bf {y}}|{\bf {x}},\theta )p(\theta |{\bf {x}})\;=\;p({\bf {y}}|{\bf {x}})p(\theta |{\bf {y}},{\bf {x}})\;\simeq \;{\tilde {q}}(\theta )\;=\;Zq(\theta ).} The joint is equal to the product of the likelihood and the prior and by Bayes' rule, equal to the product of the marginal likelihood p ( y | x ) {\displaystyle p({\bf {y}}|{\bf {x}})} and posterior p ( θ | y , x ) {\displaystyle p(\theta |{\bf {y}},{\bf {x}})} ....