Skip to main content

Answer.md

Pre-required Knowledge​

  1. Multivariate Gaussian Distribution PDF: The probability density function for a dd-dimensional Gaussian distribution with mean μ\mu and covariance matrix Σ\Sigma is: p(x∣μ,Σ)=1(2π)d/2∣Σ∣1/2exp⁡(−12(x−μ)TΣ−1(x−μ))p(x|\mu, \Sigma) = \frac{1}{(2\pi)^{d/2} |\Sigma|^{1/2}} \exp\left( -\frac{1}{2} (x-\mu)^T \Sigma^{-1} (x-\mu) \right)

  2. Likelihood Function: Assuming the samples {x1,…,xN}\{x_1, \dots, x_N\} are independent and identically distributed (i.i.d.), the likelihood function is: L(μ,Σ)=∏i=1Np(xi∣μ,Σ)L(\mu, \Sigma) = \prod_{i=1}^N p(x_i|\mu, \Sigma)

  3. Log-Likelihood Function: It is usually easier to maximize the log-likelihood: ℓ(μ,Σ)=log⁡L(μ,Σ)=∑i=1Nlog⁡p(xi∣μ,Σ)\ell(\mu, \Sigma) = \log L(\mu, \Sigma) = \sum_{i=1}^N \log p(x_i|\mu, \Sigma)

  4. Matrix/Vector Derivatives (Given in problem):

    • ∂∂xxTAx=(A+AT)x\frac{\partial}{\partial x} x^T A x = (A + A^T)x. Since Σ−1\Sigma^{-1} is symmetric, ∂∂xxTΣ−1x=2Σ−1x\frac{\partial}{\partial x} x^T \Sigma^{-1} x = 2\Sigma^{-1}x.

Step-by-Step Answer​

  1. Write down the Log-Likelihood Function:

    ℓ(μ,Σ)=∑i=1N[−d2log⁡(2π)−12log⁡∣Σ∣−12(xi−μ)TΣ−1(xi−μ)]=−Nd2log⁡(2π)−N2log⁡∣Σ∣−12∑i=1N(xi−μ)TΣ−1(xi−μ)\begin{aligned} \ell(\mu, \Sigma) &= \sum_{i=1}^N \left[ -\frac{d}{2}\log(2\pi) - \frac{1}{2}\log|\Sigma| - \frac{1}{2}(x_i - \mu)^T \Sigma^{-1} (x_i - \mu) \right] \\ &= -\frac{Nd}{2}\log(2\pi) - \frac{N}{2}\log|\Sigma| - \frac{1}{2} \sum_{i=1}^N (x_i - \mu)^T \Sigma^{-1} (x_i - \mu) \end{aligned}

  2. Differentiate with respect to μ\mu: We want to maximize ℓ(μ,Σ)\ell(\mu, \Sigma) w.r.t μ\mu. We can ignore terms that do not depend on μ\mu. Let J(μ)=−12∑i=1N(xi−μ)TΣ−1(xi−μ)J(\mu) = - \frac{1}{2} \sum_{i=1}^N (x_i - \mu)^T \Sigma^{-1} (x_i - \mu). Using the chain rule and the derivative ∂∂zzTAz=(A+AT)z\frac{\partial}{\partial z} z^T A z = (A + A^T)z: Let zi=xi−μz_i = x_i - \mu. Then ∂zi∂μ=−I\frac{\partial z_i}{\partial \mu} = -I. The term is of the form ziTΣ−1ziz_i^T \Sigma^{-1} z_i. Since Σ\Sigma is symmetric, Σ−1\Sigma^{-1} is symmetric. The derivative w.r.t ziz_i is 2Σ−1zi2\Sigma^{-1}z_i. So, ∂∂μ(xi−μ)TΣ−1(xi−μ)=2Σ−1(xi−μ)⋅(−1)=−2Σ−1(xi−μ)\frac{\partial}{\partial \mu} (x_i - \mu)^T \Sigma^{-1} (x_i - \mu) = 2\Sigma^{-1}(x_i - \mu) \cdot (-1) = -2\Sigma^{-1}(x_i - \mu).

    Therefore:

    ∂ℓ∂μ=−12∑i=1N[−2Σ−1(xi−μ)]=∑i=1NΣ−1(xi−μ)\frac{\partial \ell}{\partial \mu} = -\frac{1}{2} \sum_{i=1}^N \left[ -2\Sigma^{-1}(x_i - \mu) \right] = \sum_{i=1}^N \Sigma^{-1}(x_i - \mu)

  3. Set the derivative to zero and solve for μ^\hat{\mu}:

    ∑i=1NΣ−1(xi−μ^)=0Σ−1∑i=1N(xi−μ^)=0\begin{aligned} \sum_{i=1}^N \Sigma^{-1}(x_i - \hat{\mu}) &= 0 \\ \Sigma^{-1} \sum_{i=1}^N (x_i - \hat{\mu}) &= 0 \end{aligned}

    Assuming Σ\Sigma is positive definite (invertible), we can multiply by Σ\Sigma on the left:

    ∑i=1N(xi−μ^)=0∑i=1Nxi−∑i=1Nμ^=0∑i=1Nxi−Nμ^=0Nμ^=∑i=1Nxiμ^=1N∑i=1Nxi\begin{aligned} \sum_{i=1}^N (x_i - \hat{\mu}) &= 0 \\ \sum_{i=1}^N x_i - \sum_{i=1}^N \hat{\mu} &= 0 \\ \sum_{i=1}^N x_i - N\hat{\mu} &= 0 \\ N\hat{\mu} &= \sum_{i=1}^N x_i \\ \hat{\mu} &= \frac{1}{N} \sum_{i=1}^N x_i \end{aligned}

    Ideally, this is the sample mean.