Skip to main content

Answer.md

Pre-required Knowledge​

  1. Log-Likelihood Function: From part (a), the log-likelihood is: ℓ(μ,Σ)=−Nd2log⁡(2π)−N2log⁡∣Σ∣−12∑i=1N(xi−μ)TΣ−1(xi−μ)\ell(\mu, \Sigma) = -\frac{Nd}{2}\log(2\pi) - \frac{N}{2}\log|\Sigma| - \frac{1}{2} \sum_{i=1}^N (x_i - \mu)^T \Sigma^{-1} (x_i - \mu)
  2. Trace Method: The scalar value aTBaa^T B a is equivalent to tr(aTBa)=tr(BaaT)\text{tr}(a^T B a) = \text{tr}(B a a^T). This is useful for moving vectors inside the summation into a matrix form.
  3. Matrix Derivatives (Given):
    • ∂∂Xlog⁡∣X∣=X−T\frac{\partial}{\partial X} \log |X| = X^{-T}. Since Σ\Sigma is symmetric, ∂∂Σlog⁡∣Σ∣=Σ−1\frac{\partial}{\partial \Sigma} \log |\Sigma| = \Sigma^{-1}.
    • ∂∂Xtr(X−1A)=−(X−TATX−T)\frac{\partial}{\partial X} \text{tr}(X^{-1}A) = -(X^{-T} A^T X^{-T}). Since Σ\Sigma and AA (which will be the scatter matrix) are symmetric, this simplifies to −Σ−1AΣ−1-\Sigma^{-1} A \Sigma^{-1}.

Step-by-Step Answer​

  1. Substitute the ML estimate of μ\mu: We substitute μ\mu with μ^ML=1N∑i=1Nxi\hat{\mu}_{ML} = \frac{1}{N} \sum_{i=1}^N x_i. Let S=∑i=1N(xi−μ^)(xi−μ^)TS = \sum_{i=1}^N (x_i - \hat{\mu})(x_i - \hat{\mu})^T be the scatter matrix.

  2. Rewrite the Log-Likelihood using Trace: The term in the summation is a scalar: (xi−μ)TΣ−1(xi−μ)=tr((xi−μ)TΣ−1(xi−μ))=tr(Σ−1(xi−μ)(xi−μ)T)(x_i - \mu)^T \Sigma^{-1} (x_i - \mu) = \text{tr}\left( (x_i - \mu)^T \Sigma^{-1} (x_i - \mu) \right) = \text{tr}\left( \Sigma^{-1} (x_i - \mu)(x_i - \mu)^T \right) Summing over ii: ∑i=1N(xi−μ)TΣ−1(xi−μ)=tr(Σ−1∑i=1N(xi−μ)(xi−μ)T)=tr(Σ−1S)\sum_{i=1}^N (x_i - \mu)^T \Sigma^{-1} (x_i - \mu) = \text{tr}\left( \Sigma^{-1} \sum_{i=1}^N (x_i - \mu)(x_i - \mu)^T \right) = \text{tr}(\Sigma^{-1} S)

    So the relevant part of the log-likelihood (ignoring constants) is: ℓ(Σ)∝−N2log⁡∣Σ∣−12tr(Σ−1S)\ell(\Sigma) \propto - \frac{N}{2}\log|\Sigma| - \frac{1}{2}\text{tr}(\Sigma^{-1} S)

  3. Differentiate with respect to Σ\Sigma: Using the provided identities:

    • ∂∂Σlog⁡∣Σ∣=Σ−T=Σ−1\frac{\partial}{\partial \Sigma} \log|\Sigma| = \Sigma^{-T} = \Sigma^{-1} (since symmetric).
    • ∂∂Σtr(Σ−1S)=−(Σ−TSTΣ−T)\frac{\partial}{\partial \Sigma} \text{tr}(\Sigma^{-1} S) = -(\Sigma^{-T} S^T \Sigma^{-T}). Since SS and Σ\Sigma are symmetric, this is −Σ−1SΣ−1-\Sigma^{-1} S \Sigma^{-1}.
    ∂ℓ∂Σ=−N2Σ−1−12(−Σ−1SΣ−1)=−N2Σ−1+12Σ−1SΣ−1\frac{\partial \ell}{\partial \Sigma} = -\frac{N}{2} \Sigma^{-1} - \frac{1}{2} (-\Sigma^{-1} S \Sigma^{-1}) = -\frac{N}{2} \Sigma^{-1} + \frac{1}{2} \Sigma^{-1} S \Sigma^{-1}

  4. Set derivative to zero and solve:

    −N2Σ−1+12Σ−1SΣ−1=0Σ−1SΣ−1=NΣ−1\begin{aligned} -\frac{N}{2} \Sigma^{-1} + \frac{1}{2} \Sigma^{-1} S \Sigma^{-1} &= 0 \\ \Sigma^{-1} S \Sigma^{-1} &= N \Sigma^{-1} \end{aligned}

    Multiply by Σ\Sigma on the left and right:

    Σ(Σ−1SΣ−1)Σ=Σ(NΣ−1)ΣS=NΣΣ=1NS\begin{aligned} \Sigma (\Sigma^{-1} S \Sigma^{-1}) \Sigma &= \Sigma (N \Sigma^{-1}) \Sigma \\ S &= N \Sigma \\ \Sigma &= \frac{1}{N} S \end{aligned}

    So, Σ^ML=1N∑i=1N(xi−μ^)(xi−μ^)T\hat{\Sigma}_{ML} = \frac{1}{N} \sum_{i=1}^N (x_i - \hat{\mu})(x_i - \hat{\mu})^T