Mixture model

In statistics, a mixture model is a probabilistic model for representing the presence of subpopulations within an overall population, without requiring that an observed data set should identify the sub-population to which an individual observation belongs. Formally a mixture model corresponds to the mixture distribution that represents the probability distribution of observations in the overall population. However, while problems associated with "mixture distributions" relate to deriving the properties of the overall population from those of the sub-populations, "mixture models" are used to make statistical inferences about the properties of the sub-populations given only observations on the pooled population, without sub-population identity information. Mixture models are used for clustering, under the name model-based clustering, and also for density estimation.

Mixture models should not be confused with models for compositional data, i.e., data whose components are constrained to sum to a constant value (1, 100%, etc.). However, compositional models can be thought of as mixture models, where members of the population are sampled at random. Conversely, mixture models can be thought of as compositional models, where the total size reading population has been normalized to 1.

Structure

General mixture model

A typical finite-dimensional mixture model is a hierarchical model consisting of the following components:

• N random variables that are observed, each distributed according to a mixture of K components, with the components belonging to the same parametric family of distributions (e.g., all normal, all Zipfian, etc.) but with different parameters. However, it is also possible to have a finite mixture model where each component belongs to a different parametric family of distributions, for example, a mixture of a multivariate normal distribution and a generalized hyperbolic distribution.

• N random latent variables specifying the identity of the mixture component of each observation, each distributed according to a K-dimensional categorical distribution
• A set of K mixture weights, which are probabilities that sum to 1.
• A set of K parameters, each specifying the parameter of the corresponding mixture component. In many cases, each "parameter" is actually a set of parameters. For example, if the mixture components are Gaussian distributions, there will be a mean and variance for each component. If the mixture components are categorical distributions (e.g., when each observation is a token from a finite alphabet of size V), there will be a vector of V probabilities summing to 1.

In addition, in a Bayesian setting, the mixture weights and parameters will themselves be random variables, and prior distributions will be placed over the variables. In such a case, the weights are typically viewed as a K-dimensional random vector drawn from a Dirichlet distribution (the conjugate prior of the categorical distribution), and the parameters will be distributed according to their respective conjugate priors.

Mathematically, a basic parametric mixture model can be described as follows:

:<math>

\begin{array}{lcl}

K &=& \text{number of mixture components} \\

N &=& \text{number of observations} \\

\theta_{i=1 \dots K} &=& \text{parameter of distribution of observation associated with component } i \\

\phi_{i=1 \dots K} &=& \text{mixture weight, i.e., prior probability of a particular component } i \\

\boldsymbol\phi &=& K\text{-dimensional vector composed of all the individual } \phi_{1 \dots K} \text{; must sum to 1} \\

z_{i=1 \dots N} &=& \text{component of observation } i \\

x_{i=1 \dots N} &=& \text{observation } i \\

F(x|\theta) &=& \text{probability distribution of an observation, parametrized on } \theta \\

z_{i=1 \dots N} &\sim& \operatorname{Categorical}(\boldsymbol\phi) \\

x_{i=1 \dots N}|z_{i=1 \dots N} &\sim& F(\theta_{z_i})

\end{array}

</math>

In a Bayesian setting, all parameters are associated with random variables, as follows:

:<math>

\begin{array}{lcl}

K,N &=& \text{as above} \\

\theta_{i=1 \dots K}, \phi_{i=1 \dots K}, \boldsymbol\phi &=& \text{as above} \\

z_{i=1 \dots N}, x_{i=1 \dots N}, F(x|\theta) &=& \text{as above} \\

\alpha &=& \text{shared hyperparameter for component parameters} \\

\beta &=& \text{shared hyperparameter for mixture weights} \\

H(\theta|\alpha) &=& \text{prior probability distribution of component parameters, parametrized on } \alpha \\

\theta_{i=1 \dots K} &\sim& H(\theta|\alpha) \\

\boldsymbol\phi &\sim& \operatorname{Symmetric-Dirichlet}_K(\beta) \\

z_{i=1 \dots N}|\boldsymbol\phi &\sim& \operatorname{Categorical}(\boldsymbol\phi) \\

x_{i=1 \dots N}|z_{i=1 \dots N},\theta_{i=1 \dots K} &\sim& F(\theta_{z_i})

\end{array}

</math>

This characterization uses F and H to describe arbitrary distributions over observations and parameters, respectively. Typically H will be the conjugate prior of F. The two most common choices of F are Gaussian aka "normal" (for real-valued observations) and categorical (for discrete observations). Other common possibilities for the distribution of the mixture components are:

• Binomial distribution, for the number of "positive occurrences" (e.g., successes, yes votes, etc.) given a fixed number of total occurrences
• Multinomial distribution, similar to the binomial distribution, but for counts of multi-way occurrences (e.g., yes/no/maybe in a survey)
• Negative binomial distribution, for binomial-type observations but where the quantity of interest is the number of failures before a given number of successes occurs
• Poisson distribution, for the number of occurrences of an event in a given period of time, for an event that is characterized by a fixed rate of occurrence
• Exponential distribution, for the time before the next event occurs, for an event that is characterized by a fixed rate of occurrence
• Log-normal distribution, for positive real numbers that are assumed to grow exponentially, such as incomes or prices
• Multivariate normal distribution (aka multivariate Gaussian distribution), for vectors of correlated outcomes that are individually Gaussian-distributed
• Multivariate Student's t-distribution, for vectors of heavy-tailed correlated outcomes
• A vector of Bernoulli-distributed values, corresponding, e.g., to a black-and-white image, with each value representing a pixel; see the handwriting-recognition example below

Specific examples

Gaussian mixture model

right|250px|thumb|Non-Bayesian Gaussian mixture model using [[plate notation. Smaller squares indicate fixed parameters; larger circles indicate random variables. Filled-in shapes indicate known values. The indication [K] means a vector of size K.]]

A typical non-Bayesian Gaussian mixture model looks like this:

:<math>

\begin{array}{lcl}

K,N &=& \text{as above} \\

\phi_{i=1 \dots K}, \boldsymbol\phi &=& \text{as above} \\

z_{i=1 \dots N}, x_{i=1 \dots N} &=& \text{as above} \\

\theta_{i=1 \dots K} &=& \{ \mu_{i=1 \dots K}, \sigma^2_{i=1 \dots K} \} \\

\mu_{i=1 \dots K} &=& \text{mean of component } i \\

\sigma^2_{i=1 \dots K} &=& \text{variance of component } i \\

z_{i=1 \dots N} &\sim& \operatorname{Categorical}(\boldsymbol\phi) \\

x_{i=1 \dots N} &\sim& \mathcal{N}(\mu_{z_i}, \sigma^2_{z_i})

\end{array}

</math>

right|300px|thumb|Bayesian Gaussian mixture model using [[plate notation. Smaller squares indicate fixed parameters; larger circles indicate random variables. Filled-in shapes indicate known values. The indication [K] means a vector of size K.]]

A Bayesian version of a Gaussian mixture model is as follows:

:<math>

\begin{array}{lcl}

K,N &=& \text{as above} \\

\phi_{i=1 \dots K}, \boldsymbol\phi &=& \text{as above} \\

z_{i=1 \dots N}, x_{i=1 \dots N} &=& \text{as above} \\

\theta_{i=1 \dots K} &=& \{ \mu_{i=1 \dots K}, \sigma^2_{i=1 \dots K} \} \\

\mu_{i=1 \dots K} &=& \text{mean of component } i \\

\sigma^2_{i=1 \dots K} &=& \text{variance of component } i \\

\mu_0, \lambda, \nu, \sigma_0^2 &=& \text{shared hyperparameters} \\

\mu_{i=1 \dots K} &\sim& \mathcal{N}(\mu_0, \lambda\sigma_i^2) \\

\sigma_{i=1 \dots K}^2 &\sim& \operatorname{Inverse-Gamma}(\nu, \sigma_0^2) \\

\boldsymbol\phi &\sim& \operatorname{Symmetric-Dirichlet}_K(\beta) \\

z_{i=1 \dots N} &\sim& \operatorname{Categorical}(\boldsymbol\phi) \\

x_{i=1 \dots N} &\sim& \mathcal{N}(\mu_{z_i}, \sigma^2_{z_i})

\end{array}

</math><math></math>

thumb|end=49|Animation of the clustering process for one-dimensional data using a Bayesian Gaussian mixture model where normal distributions are drawn from a [[Dirichlet process. The histograms of the clusters are shown in different colours. During the parameter estimation process, new clusters are created and grow on the data. The legend shows the cluster colours and the number of datapoints assigned to each cluster.]]

Multivariate Gaussian mixture model

A Bayesian Gaussian mixture model is commonly extended to fit a vector of unknown parameters (denoted in bold), or multivariate normal distributions. In a multivariate distribution (i.e. one modelling a vector <math>\boldsymbol{x}</math> with N random variables) one may model a vector of parameters (such as several observations of a signal or patches within an image) using a Gaussian mixture model prior distribution on the vector of estimates given by

<math display="block">

p(\boldsymbol{\theta}) = \sum_{i=1}^K \phi_i \mathcal{N}(\boldsymbol{\mu}_i,\boldsymbol{\Sigma}_i)

</math>

where the i^th vector component is characterized by normal distributions with weights <math>\phi_i</math>, means <math>\boldsymbol{\mu}_i</math> and covariance matrices <math>\boldsymbol{\Sigma}_i</math>. To incorporate this prior into a Bayesian estimation, the prior is multiplied with the known distribution <math>p(\boldsymbol{x | \theta})</math> of the data <math>\boldsymbol{x}</math> conditioned on the parameters <math>\boldsymbol{\theta}</math> to be estimated. With this formulation, the posterior distribution <math>p(\boldsymbol{\theta | x})</math> is also a Gaussian mixture model of the form

<math display="block">

p(\boldsymbol{\theta | x}) = \sum_{i=1}^K \tilde{\phi}_i \mathcal{N}(\boldsymbol{\tilde{\mu}_i}, \boldsymbol{\tilde{\Sigma_i)

</math>

with new parameters <math>\tilde{\phi}_i, \boldsymbol{\tilde{\mu_i</math> and <math>\boldsymbol{\tilde{\Sigma_i</math> that are updated using the EM algorithm.

Although EM-based parameter updates are well-established, providing the initial estimates for these parameters is currently an area of active research. Note that this formulation yields a closed-form solution to the complete posterior distribution. Estimations of the random variable <math>\boldsymbol{\theta}</math> may be obtained via one of several estimators, such as the mean or maximum of the posterior distribution.

Such distributions are useful for assuming patch-wise shapes of images and clusters, for example. In the case of image representation, each Gaussian may be tilted, expanded, and warped according to the covariance matrices <math>\boldsymbol{\Sigma}_i</math>. One Gaussian distribution of the set is fit to each patch (usually of size 8×8 pixels) in the image. Notably, any distribution of points around a cluster (see k-means) may be accurately given enough Gaussian components, but scarcely over K=20 components are needed to accurately model a given image distribution or cluster of data.

Categorical mixture model

right|250px|thumb|Non-Bayesian categorical mixture model using [[plate notation. Smaller squares indicate fixed parameters; larger circles indicate random variables. Filled-in shapes indicate known values. The indication [K] means a vector of size K; likewise for [V].]]

A typical non-Bayesian mixture model with categorical observations looks like this:

• <math>K,N:</math> as above
• <math>\phi_{i=1 \dots K}, \boldsymbol\phi:</math> as above
• <math>z_{i=1 \dots N}, x_{i=1 \dots N}:</math> as above
• <math>V:</math> dimension of categorical observations, e.g., size of word vocabulary
• <math>\theta_{i=1 \dots K, j=1 \dots V}:</math> probability for component <math>i</math> of observing item <math>j</math>
• <math>\boldsymbol\theta_{i=1 \dots K}:</math> vector of dimension <math>V,</math> composed of <math>\theta_{i,1 \dots V};</math> must sum to 1

The random variables:

:<math>

\begin{array}{lcl}

z_{i=1 \dots N} &\sim& \operatorname{Categorical}(\boldsymbol\phi) \\

x_{i=1 \dots N} &\sim& \text{Categorical}(\boldsymbol\theta_{z_i})

\end{array}

</math>

<!--

The original version, all in LaTeX.

:<math>

\begin{array}{lcl}

K,N &=& \text{as above} \\

\phi_{i=1 \dots K}, \boldsymbol\phi &=& \text{as above} \\

z_{i=1 \dots N}, x_{i=1 \dots N} &=& \text{as above} \\

V &=& \text{dimension of categorical observations, e.g., size of word vocabulary} \\

\theta_{i=1 \dots K, j=1 \dots V} &=& \text{probability for component } i \text{ of observing the } j\text{th item} \\

\boldsymbol\theta_{i=1 \dots K} &=& V\text{-dimensional vector, composed of }\theta_{i,1 \dots V} \text{; must sum to 1} \\

z_{i=1 \dots N} &\sim& \operatorname{Categorical}(\boldsymbol\phi) \\

x_{i=1 \dots N} &\sim& \text{Categorical}(\boldsymbol\theta_{z_i})

\end{array}

</math>

-->

right|300px|thumb|Bayesian categorical mixture model using [[plate notation. Smaller squares indicate fixed parameters; larger circles indicate random variables. Filled-in shapes indicate known values. The indication [K] means a vector of size K; likewise for [V].]]

A typical Bayesian mixture model with categorical observations looks like this:

• <math>K,N:</math> as above
• <math>\phi_{i=1 \dots K}, \boldsymbol\phi:</math> as above
• <math>z_{i=1 \dots N}, x_{i=1 \dots N}:</math> as above
• <math>V:</math> dimension of categorical observations, e.g., size of word vocabulary
• <math>\theta_{i=1 \dots K, j=1 \dots V}:</math> probability for component <math>i</math> of observing item <math>j</math>
• <math>\boldsymbol\theta_{i=1 \dots K}:</math> vector of dimension <math>V,</math> composed of <math>\theta_{i,1 \dots V};</math> must sum to 1
• <math>\alpha:</math> shared concentration hyperparameter of <math>\boldsymbol\theta</math> for each component
• <math>\beta:</math> concentration hyperparameter of <math>\boldsymbol\phi</math>

The random variables:

:<math>

\begin{array}{lcl}

\boldsymbol\phi &\sim& \operatorname{Symmetric-Dirichlet}_K(\beta) \\

\boldsymbol\theta_{i=1 \dots K} &\sim& \text{Symmetric-Dirichlet}_V(\alpha) \\

z_{i=1 \dots N} &\sim& \operatorname{Categorical}(\boldsymbol\phi) \\

x_{i=1 \dots N} &\sim& \text{Categorical}(\boldsymbol\theta_{z_i})

\end{array}

</math>

<!--

The (beginning of) equivalent of below, using no LaTeX.

• K,N = as above
• &phi;_1,...,K, &phi; as above
• z_i=1...N, x_i=1...N = as above
• V = dimension of categorical observations, e.g., size of word vocabulary

-->

<!--

The equivalent using full LaTeX.

:<math>

\begin{array}{lcl}

K,N &=& \mbox{as above} \\

\phi_{i=1 \dots K}, \boldsymbol\phi &=& \text{as above} \\

z_{i=1 \dots N}, x_{i=1 \dots N} &=& \text{as above} \\

V &=& \text{dimension of categorical observations, e.g., size of word vocabulary} \\

\theta_{i=1 \dots K, j=1 \dots V} &=& \text{probability for component } i \text{ of observing the } j\text{th item} \\

\boldsymbol\theta_{i=1 \dots K} &=& V\text{-dimensional vector, composed of }\theta_{i,1 \dots V} \text{; must sum to 1} \\

\alpha &=& \text{shared concentration hyperparameter of } \boldsymbol\theta \text{ for each component} \\

\beta &=& \text{concentration hyperparameter of } \boldsymbol\phi \\

\boldsymbol\phi &\sim& \operatorname{Symmetric-Dirichlet}_K(\beta) \\

\boldsymbol\theta_{i=1 \dots K} &\sim& \text{Symmetric-Dirichlet}_V(\alpha) \\

z_{i=1 \dots N} &\sim& \operatorname{Categorical}(\boldsymbol\phi) \\

x_{i=1 \dots N} &\sim& \text{Categorical}(\boldsymbol\theta_{z_i})

\end{array}

</math>

-->

Examples

A financial model

thumb|250x250px|The [[normal distribution plotted with different means and variances]]

Financial returns often behave differently in normal situations and during crisis times. A mixture model for return data seems reasonable. Sometimes the model used is a jump-diffusion model, or as a mixture of two normal distributions. See and for further context.

House prices

Assume that we observe the prices of N different houses. Different types of houses in different neighborhoods will have vastly different prices, but the price of a particular type of house in a particular neighborhood (e.g., three-bedroom house in moderately upscale neighborhood) will tend to cluster fairly closely around the mean. One possible model of such prices would be to assume that the prices are accurately described by a mixture model with K different components, each distributed as a normal distribution with unknown mean and variance, with each component specifying a particular combination of house type/neighborhood. Fitting this model to observed prices, e.g., using the expectation-maximization algorithm, would tend to cluster the prices according to house type/neighborhood and reveal the spread of prices in each type/neighborhood. (Note that for values such as prices or incomes that are guaranteed to be positive and which tend to grow exponentially, a log-normal distribution might actually be a better model than a normal distribution.)

Topics in a document

Assume that a document is composed of N different words from a total vocabulary of size V, where each word corresponds to one of K possible topics. The distribution of such words could be modelled as a mixture of K different V-dimensional categorical distributions. A model of this sort is commonly termed a topic model. Note that expectation maximization applied to such a model will typically fail to produce realistic results, due (among other things) to the excessive number of parameters. Some sorts of additional assumptions are typically necessary to get good results. Typically two sorts of additional components are added to the model:

1. A prior distribution is placed over the parameters describing the topic distributions, using a Dirichlet distribution with a concentration parameter that is set significantly below 1, so as to encourage sparse distributions (where only a small number of words have significantly non-zero probabilities).
2. Some sort of additional constraint is placed over the topic identities of words, to take advantage of natural clustering.
3. For example, a Markov chain could be placed on the topic identities (i.e., the latent variables specifying the mixture component of each observation), corresponding to the fact that nearby words belong to similar topics. (This results in a hidden Markov model, specifically one where a prior distribution is placed over state transitions that favors transitions that stay in the same state.)
4. Another possibility is the latent Dirichlet allocation model, which divides up the words into D different documents and assumes that in each document only a small number of topics occur with any frequency.

Handwriting recognition

The following example is based on an example in Christopher M. Bishop, Pattern Recognition and Machine Learning.

Imagine that we are given an N×N black-and-white image that is known to be a scan of a hand-written digit between 0 and 9, but we don't know which digit is written. We can create a mixture model with <math>K=10</math> different components, where each component is a vector of size <math>N^2</math> of Bernoulli distributions (one per pixel). Such a model can be trained with the expectation-maximization algorithm on an unlabeled set of hand-written digits, and will effectively cluster the images according to the digit being written. The same model could then be used to recognize the digit of another image simply by holding the parameters constant, computing the probability of the new image for each possible digit (a trivial calculation), and returning the digit that generated the highest probability.

Assessing projectile accuracy (a.k.a. circular error probable, CEP)

Mixture models apply in the problem of directing multiple projectiles at a target (as in air, land, or sea defense applications), where the physical and/or statistical characteristics of the projectiles differ within the multiple projectiles. An example might be shots from multiple munitions types or shots from multiple locations directed at one target. The combination of projectile types may be characterized as a Gaussian mixture model. Further, a well-known measure of accuracy for a group of projectiles is the circular error probable (CEP), which is the number R such that, on average, half of the group of projectiles falls within the circle of radius R about the target point. The mixture model can be used to determine (or estimate) the value R. The mixture model properly captures the different types of projectiles.

Direct and indirect applications

The financial example above is one direct application of the mixture model, a situation in which we assume an underlying mechanism so that each observation belongs to one of some number of different sources or categories. This underlying mechanism may or may not, however, be observable. In this form of mixture, each of the sources is described by a component probability density function, and its mixture weight is the probability that an observation comes from this component.

In an indirect application of the mixture model we do not assume such a mechanism. The mixture model is simply used for its mathematical flexibilities. For example, a mixture of two normal distributions with different means may result in a density with two modes, which is not modeled by standard parametric distributions. Another example is given by the possibility of mixture distributions to model fatter tails than the basic Gaussian ones, so as to be a candidate for modeling more extreme events.

Predictive Maintenance

The mixture model-based clustering is also predominantly used in identifying the state of the machine in predictive maintenance. Density plots are used to analyze the density of high dimensional features. If multi-model densities are observed, then it is assumed that a finite set of densities are formed by a finite set of normal mixtures. A multivariate Gaussian mixture model is used to cluster the feature data into k number of groups where k represents each state of the machine. The machine state can be a normal state, power off state, or faulty state. Each formed cluster can be diagnosed using techniques such as spectral analysis. In the recent years, this has also been widely used in other areas such as early fault detection.

Fuzzy image segmentation

thumb|An example of Gaussian Mixture in image segmentation with grey histogram

In image processing and computer vision, traditional image segmentation models often assign to one pixel only one exclusive pattern. In fuzzy or soft segmentation, any pattern can have certain "ownership" over any single pixel. If the patterns are Gaussian, fuzzy segmentation naturally results in Gaussian mixtures. Combined with other analytic or geometric tools (e.g., phase transitions over diffusive boundaries), such spatially regularized mixture models could lead to more realistic and computationally efficient segmentation methods.

Point set registration

Probabilistic mixture models such as Gaussian mixture models (GMM) are used to resolve point set registration problems in image processing and computer vision fields. For pair-wise point set registration, one point set is regarded as the centroids of mixture models, and the other point set is regarded as data points (observations). State-of-the-art methods are e.g. coherent point drift (CPD)

and Student's t-distribution mixture models (TMM).

The result of recent research demonstrate the superiority of hybrid mixture models

(e.g. combining Student's t-distribution and Watson distribution/Bingham distribution to model spatial positions and axes orientations separately) compare to CPD and TMM, in terms of inherent robustness, accuracy and discriminative capacity.

Clustering in social science data

Mixture models are widely used in the social sciences to cluster observational data and identify latent group structure in complex, heterogeneous populations. In studies of armed conflict, unsupervised mixture-model-based clustering has been applied to conflict event data to group observations into empirically derived conflict types without relying on predefined categories. Such analyses reveal systematic differences across clusters in geographic, demographic, economic and infrastructural characteristics, corresponding to distinct conflict archetypes associated with different population and development profiles. This is one of the numerous examples of how mixture models can support data-driven classification in social science research.

Identifiability

Identifiability refers to the existence of a unique characterization for any one of the models in the class (family) being considered. Estimation procedures may not be well-defined and asymptotic theory may not hold if a model is not identifiable.

Example

Let be the class of all binomial distributions with . Then a mixture of two members of would have

<math display="block">\begin{align}

p_0 &= \pi {\left(1 - \theta_1\right)}^2 + \left(1 - \pi\right) {\left(1 - \theta_2\right)}^2 \\[1ex]

p_1 &= 2\pi \theta_1 \left(1 - \theta_1\right) + 2 \left(1 - \pi\right) \theta_2 \left(1 - \theta_2\right)

\end{align}</math>

and . Clearly, given and , it is not possible to determine the above mixture model uniquely, as there are three parameters to be determined.

Definition

Consider a mixture of parametric distributions of the same class. Let

<math display="block">J = \{f(\cdot ; \theta):\theta\in\Omega\}</math>

be the class of all component distributions. Then the convex hull of defines the class of all finite mixture of distributions in :

<math display="block">K = \left\{p(\cdot):p(\cdot)=\sum_{i=1}^n a_i f_i(\cdot ; \theta_i), a_i>0, \sum_{i=1}^n a_i=1, f_i(\cdot ; \theta_i)\in J\ \forall i,n\right\}</math>

is said to be identifiable if all its members are unique, that is, given two members and in K, being mixtures of distributions and distributions respectively in , we have if and only if, first of all, and secondly we can reorder the summations such that and for all .

Parameter estimation and system identification

Parametric mixture models are often used when we know the distribution Y and we can sample from X, but we would like to determine the a_i and θ_i values. Such situations can arise in studies in which we sample from a population that is composed of several distinct subpopulations.

It is common to think of probability mixture modeling as a missing data problem. One way to understand this is to assume that the data points under consideration have "membership" in one of the distributions we are using to model the data. When we start, this membership is unknown, or missing. The job of estimation is to devise appropriate parameters for the model functions we choose, with the connection to the data points being represented as their membership in the individual model distributions.

A variety of approaches to the problem of mixture decomposition have been proposed, many of which focus on maximum likelihood methods such as expectation maximization (EM) or maximum a posteriori estimation (MAP). Generally these methods consider separately the questions of system identification and parameter estimation; methods to determine the number and functional form of components within a mixture are distinguished from methods to estimate the corresponding parameter values. Some notable departures are the graphical methods as outlined in Tarter and Lock and more recently minimum message length (MML) techniques such as Figueiredo and Jain and to some extent the moment matching pattern analysis routines suggested by McWilliam and Loh (2009).

=== Expectation maximization (EM) === <!-- Linked from Expectation-maximization algorithm -->

Expectation maximization (EM) is seemingly the most popular technique used to determine the parameters of a mixture with an a priori given number of components. This is a particular way of implementing maximum likelihood estimation for this problem. EM is of particular appeal for finite normal mixtures where closed-form expressions are possible such as in the following iterative algorithm by Dempster et al. (1977)

:<math> w_s^{(j+1)} = \frac{1}{N} \sum_{t =1}^N h_s^{(j)}(t) </math>

:<math> \mu_s^{(j+1)} = \frac{\sum_{t =1}^N h_s^{(j)}(t) x^{(t){\sum_{t =1}^N h_s^{(j)}(t)} </math>

:<math> \Sigma_s^{(j+1)} = \frac{\sum_{t =1}^N h_s^{(j)}(t) [x^{(t)}-\mu_s^{(j+1)}][x^{(t)}-\mu_s^{(j+1)}]^{\top{\sum_{t =1}^N h_s^{(j)}(t)} </math>

with the posterior probabilities

:<math> h_s^{(j)}(t) = \frac{w_s^{(j)} p_s(x^{(t)}; \mu_s^{(j)},\Sigma_s^{(j)}) }{ \sum_{i = 1}^n w_i^{(j)} p_i(x^{(t)}; \mu_i^{(j)}, \Sigma_i^{(j)})}. </math>

Thus on the basis of the current estimate for the parameters, the conditional probability for a given observation x^(t) being generated from state s is determined for each ; N being the sample size. The parameters are then updated such that the new component weights correspond to the average conditional probability and each component mean and covariance is the component specific weighted average of the mean and covariance of the entire sample.

Dempster

Other common objections to the use of EM are that it has a propensity to spuriously identify local maxima, as well as displaying sensitivity to initial values. One may address these problems by evaluating EM at several initial points in the parameter space but this is computationally costly and other approaches, such as the annealing EM method of Udea and Nakano (1998) (in which the initial components are essentially forced to overlap, providing a less heterogeneous basis for initial guesses), may be preferable.

Figueiredo and Jain has indicated that such methods may be inefficient compared to EM. Nonetheless, there has been renewed interest in this method, e.g., Craigmile and Titterington (1998) and Wang.

McWilliam and Loh (2009) consider the characterisation of a hyper-cuboid normal mixture copula in large dimensional systems for which EM would be computationally prohibitive. Here a pattern analysis routine is used to generate multivariate tail-dependencies consistent with a set of univariate and (in some sense) bivariate moments. The performance of this method is then evaluated using equity log-return data with Kolmogorov–Smirnov test statistics suggesting a good descriptive fit.

Spectral method

Some problems in mixture model estimation can be solved using spectral methods.

In particular it becomes useful if data points x_i are points in high-dimensional real space, and the hidden distributions are known to be log-concave (such as Gaussian distribution or Exponential distribution).

Spectral methods of learning mixture models are based on the use of Singular Value Decomposition of a matrix which contains data points.

The idea is to consider the top k singular vectors, where k is the number of distributions to be learned. The projection

of each data point to a linear subspace spanned by those vectors groups points originating from the same distribution

very close together, while points from different distributions stay far apart.

One distinctive feature of the spectral method is that it allows us to prove that if

distributions satisfy certain separation condition (e.g., not too close), then the estimated mixture will be very close to the true one with high probability.

Graphical Methods

Tarter and Lock 2000) although common reference is made to the work of Karl Pearson (1894) as the first author to explicitly address the decomposition problem in characterising non-normal attributes of forehead to body length ratios in female shore crab populations. The motivation for this work was provided by the zoologist Walter Frank Raphael Weldon who had speculated in 1893 (in Tarter and Lock Since that time there has been a vast body of research on the subject spanning areas such as fisheries research, agriculture, botany, economics, medicine, genetics, psychology, palaeontology, electrophoresis, finance, geology and zoology.

See also

Mixture

• Mixture density
• Mixture (probability)
• Flexible Mixture Model (FMM)
• Subspace Gaussian mixture model
• Giry monad
• HELIOS Hybrid Evaluation of Lifecycle and Impact of Outstanding Science

Hierarchical models

• Graphical model
• Hierarchical Bayes model

Outlier detection

• RANSAC

References

Further reading

Books on mixture models

Application of Gaussian mixture models

External links

• The SOCR demonstrations of EM and Mixture Modeling
• Mixture modelling page (and the Snob program for Minimum Message Length (MML) applied to finite mixture models), maintained by D.L. Dowe.
• PyMix – Python Mixture Package, algorithms and data structures for a broad variety of mixture model based data mining applications in Python
• sklearn.mixture – A module from the scikit-learn Python library for learning Gaussian Mixture Models (and sampling from them), previously packaged with SciPy and now packaged as a SciKit
• GMM.m Matlab code for GMM Implementation
• GPUmix C++ implementation of Bayesian Mixture Models using EM and MCMC with 100x speed acceleration using GPGPU.
• [https://www.cs.ru.nl/~ali/index_files/EM.m] Matlab code for GMM Implementation using EM algorithm
• [https://vincentfpgarcia.github.com/jMEF/] jMEF: A Java open source library for learning and processing mixtures of exponential families (using duality with Bregman divergences). Includes a Matlab wrapper.
• Very Fast and clean C implementation of the Expectation Maximization (EM) algorithm for estimating Gaussian Mixture Models (GMMs).
• mclust is an R package for mixture modeling.
• dpgmm Pure Python Dirichlet process Gaussian mixture model implementation (variational).
• Gaussian Mixture Models Blog post on Gaussian Mixture Models trained via Expectation Maximization, with an implementation in Python.