In Bayesian statistics, the maximum a posteriori (MAP) estimate of an unknown quantity is the mode of the posterior density. The MAP can be used to obtain a point estimate of an unobserved quantity on the basis of empirical data. It is closely related to the method of maximum likelihood (ML) estimation, but employs an augmented optimization objective which incorporates a prior density over the quantity one wants to estimate. MAP estimation is therefore a regularization of maximum likelihood estimation.
Description
Assume that we want to estimate an unobserved population parameter <math>\theta</math> on the basis of observations <math>x</math>. Let <math>f</math> be the sampling distribution of <math>x</math>, so that <math>f(x\mid\theta)</math> is the probability of <math>x</math> when the underlying population parameter is <math>\theta</math>. Then the function:
:<math>\theta \mapsto f(x \mid \theta) \!</math>
is known as the likelihood function and the estimate:
:<math>\hat{\theta}_{\mathrm{MLE(x) = \underset{\theta}{\operatorname{arg\,max \ f(x \mid \theta) \!</math>
is the maximum likelihood estimate of <math>\theta</math>.
Now assume that a prior distribution <math>g</math> over <math>\theta</math> exists. This allows us to treat <math>\theta</math> as a random variable as in Bayesian statistics. We can calculate the posterior density of <math>\theta</math> using Bayes' theorem:
:<math>\theta \mapsto f(\theta \mid x) = \frac{f(x \mid \theta) \, g(\theta)}{\displaystyle\int_{\Theta} f(x \mid \vartheta) \, g(\vartheta) \, d\vartheta} \!</math>
where <math>g</math> is density function of <math>\theta</math>, <math>\Theta</math> is the domain of <math>g</math>.
The method of maximum a posteriori estimation then estimates <math>\theta</math> as the mode of the posterior density of this random variable:
:<math>\begin{align}
\hat{\theta}_{\mathrm{MAP(x) &
= \underset{\theta}{\operatorname{arg\,max \ f(\theta \mid x) \\
& = \underset{\theta}{\operatorname{arg\,max \ \frac{f(x \mid \theta) \, g(\theta)}
{\displaystyle\int_{\Theta} f(x \mid \vartheta) \, g(\vartheta) \, d\vartheta} \\
& = \underset{\theta}{\operatorname{arg\,max \ f(x \mid \theta) \, g(\theta).
\end{align}
\!</math>
The denominator of the posterior density (the marginal likelihood of the model) is always positive and does not depend on <math>\theta</math> and therefore plays no role in the optimization. Observe that the MAP estimate of <math>\theta</math> coincides with the ML estimate when the prior <math>g</math> is uniform (i.e., <math>g</math> is a constant function), which occurs whenever the prior distribution is taken as the reference measure, as is typical in function-space applications.
In the context of Bayes estimators, the MAP can be recovered as the minimizer of the Bayes risk with risk function
:<math>
L(\theta, a) =
\begin{cases}
0, & \text{if } |a-\theta|<c, \\
1, & \text{otherwise}, \\
\end{cases}
</math>
in the limit as <math>c</math> goes to 0, provided that the distribution of <math>\theta</math> is quasi-concave. Generally, however, a MAP estimator is not a Bayes estimator unless <math>\theta</math> is discrete.
Computation
MAP estimates can be computed in several ways:
- Analytically, when the mode(s) of the posterior density can be given in closed form. This is the case when conjugate priors are used.
- Via numerical optimization such as the conjugate gradient method or Newton's method. This usually requires first or second derivatives, which have to be evaluated analytically or numerically.
- Via a modification of an expectation-maximization algorithm. This does not require derivatives of the posterior density.
- Via a Monte Carlo method using simulated annealing
Limitations
While only mild conditions are required for MAP estimation to be a limiting case of Bayes estimation (under the 0–1 loss function), In contrast, Bayesian posterior expectations are invariant under reparameterization.
As an example of the difference between Bayes estimators mentioned above (mean and median estimators) and using a MAP estimate, consider the case where there is a need to classify inputs <math>x</math> as either positive or negative (for example, loans as risky or safe). Suppose there are just three possible hypotheses about the correct method of classification <math>h_1</math>, <math>h_2</math> and <math>h_3</math> with posteriors 0.4, 0.3 and 0.3 respectively. Suppose given a new instance, <math>x</math>, <math>h_1</math> classifies it as positive, whereas the other two classify it as negative. Using the MAP estimate for the correct classifier <math>h_1</math>, <math>x</math> is classified as positive, whereas the Bayes estimators would average over all hypotheses and classify <math>x</math> as negative.
Example
Suppose that we are given a sequence <math>(x_1, \dots, x_n)</math> of IID <math>N(\mu,\sigma_v^2 )</math> random variables and a prior distribution of <math>\mu</math> is given by <math>N(\mu_0,\sigma_m^2 )</math>. We wish to find the MAP estimate of <math>\mu</math>. Note that the normal distribution is its own conjugate prior, so we will be able to find a closed-form solution analytically.
The function to be maximized is then given by
:<math>g(\mu) f(x \mid \mu)=\pi(\mu) L(\mu) = \frac{1}{\sqrt{2 \pi} \sigma_m} \exp\left(-\frac{1}{2} \left(\frac{\mu-\mu_0}{\sigma_m}\right)^2\right) \prod_{j=1}^n \frac{1}{\sqrt{2 \pi} \sigma_v} \exp\left(-\frac{1}{2} \left(\frac{x_j - \mu}{\sigma_v}\right)^2\right),</math>
which is equivalent to minimizing the following function of <math>\mu</math>:
:<math> \sum_{j=1}^n \left(\frac{x_j - \mu}{\sigma_v}\right)^2 + \left(\frac{\mu-\mu_0}{\sigma_m}\right)^2.</math>
Thus, we see that the MAP estimator for μ is given by