Jump to content

User:Nmrenyi

From Wikipedia, the free encyclopedia

In machine learning and statistics, EM Algorithm is the abbreviation for Expectation Maximization Algorithm, which is used to handle latent variables. GMM Model means Gaussian mixture model. In this article, we'll have an introduction on how to use EM Algorithm to handle GMM model.

Background

[edit]

First let's warm up with a simple scenario. In the picture below, we have the Red Blood Cell Hemoglobin Concentration and the Red Blood Cell Volume data of two groups of people, the Anemia Group and the Control Group(i.e. the group of people without Anemia). It's clear that people with Anemia have lower red blood cell volume and lower red blood cell hemoglobin concentration than those without Anemia.

GMM model with labels

To make it simple, let be a random vector: And denote as the group where belongs.( when belongs to Anemia Group and when belongs to Control Group). And from medical knowledge, we believe that are normally distributed in each group, i.e. . Also , where (in this scenario, ). Now we'd like to estimate .

We can use maximum likelihood estimation on this question. The log likelihood function is shown below.

As we know the for each , the log likelihood function can be simplified as below:

Now we can maximize the likelihood function by making partial derivative over . Since this step only involves some simple algebra calculation, I'll directly show the result.

[1]

In the example above, we can see that if is known to us, the estimation of parameters can be quite simple with maximum likelihood estimation. But what if is unknown to us? It'll be hard to estimate the parameters. [2]

GMM without labels


In this case, we call a latent variable(i.e. not observed). With unlabled scenario, we need the Expectation Maximization Algorithm to estimate $z$ as well as other parameters. Generally, we would name the problem setting above as Gaussian Mixture Models(i.e. GMM) since the data in each group is normally distributed. [3]

In a general circumstance in machine learning, we can see the latent variable as some latent pattern lying under the data, which we cannot see very directly. And we can see as our data, as the parameter of the model. With EM algorithm, we may find some underlying pattern in the data , along with the estimation of parameters. The wide application of this circumstance in machine learning makes EM algorithm very important. [4]

EM Algorithm in GMM

[edit]

The Expectation Maximization Algorithm consists of two steps: the E-step and the M-step. Firstly, we can randomly initialize the value of our model parameters and the In the E-step, the algorithm tries to guess the value of based on the parameters. In the M-step, the algorithm updates the value of the model parameters based on the guess of in the E-step. These two steps will repeat until convergence. Let's see the algorithm in GMM first.

Repeat until convergence: {

   1. (E-step) For each , set 
   
   2. (M-step) Update the parameters
   
      
      

} [1]


We can take a closer look at the E-step. In fact, with Bayes Rule, we can get the following result:

According to GMM setting, we can have these following formulas:

In this way, we can clearly move on to the E-step and M-step according to the randomly initialized parameters.


References

[edit]
  1. ^ a b Ng, Andrew. "CS229 Lecture notes" (PDF).
  2. ^ Hui, Jonathan (13 October 2019). "Machine Learning —Expectation-Maximization Algorithm (EM)". Medium.
  3. ^ Tong, Y. L. (2 July 2020). "Multivariate normal distribution". Wikipedia.
  4. ^ Misra, Rishabh (7 June 2020). "Inference using EM algorithm". Medium.

Category:Machine learning