Skip to main content

Answer

Pre-required Knowledge​

  1. Bayes' Theorem for Continuous Variables: p(θ∣D)=p(D∣θ)p(θ)p(D)∝p(D∣θ)p(θ)p(\theta | \mathcal{D}) = \frac{p(\mathcal{D} | \theta) p(\theta)}{p(\mathcal{D})} \propto p(\mathcal{D} | \theta) p(\theta)
  2. Multivariate Gaussian Distribution: N(x∣μ,Σ)∝exp⁡(−12(x−μ)TΣ−1(x−μ))\mathcal{N}(x | \mu, \Sigma) \propto \exp \left( -\frac{1}{2} (x - \mu)^T \Sigma^{-1} (x - \mu) \right)
  3. Completing the Square in Matrix Form: If a quadratic form in θ\theta looks like −12(θTAθ−2θTb)-\frac{1}{2} (\theta^T A \theta - 2 \theta^T b), it corresponds to a Gaussian with covariance A−1A^{-1} and mean A−1bA^{-1}b (ignoring constants).

Step-by-Step Answer​

  1. Identify the Likelihood function p(D∣θ)p(\mathcal{D}|\theta): From the model y=ΦTθ+ϵy = \Phi^T \theta + \epsilon, where ϵ∼N(0,Σ)\epsilon \sim \mathcal{N}(0, \Sigma), we have that yy given θ\theta follows a Gaussian distribution:

    p(y∣θ)=N(y∣ΦTθ,Σ)∝exp⁡(−12(y−ΦTθ)TΣ−1(y−ΦTθ))p(y | \theta) = \mathcal{N}(y | \Phi^T \theta, \Sigma) \propto \exp \left( -\frac{1}{2} (y - \Phi^T \theta)^T \Sigma^{-1} (y - \Phi^T \theta) \right)

    Here D=(X,y)\mathcal{D} = (X, y), but since XX is fixed (discriminative setting), we look at p(y∣θ)p(y|\theta).

  2. Identify the Prior distribution p(θ)p(\theta):

    p(θ)=N(θ∣0,Γ)∝exp⁡(−12θTΓ−1θ)p(\theta) = \mathcal{N}(\theta | 0, \Gamma) \propto \exp \left( -\frac{1}{2} \theta^T \Gamma^{-1} \theta \right)

  3. Formulate the Posterior p(θ∣D)p(\theta|\mathcal{D}): Using Bayes' rule, the posterior is proportional to the product of the likelihood and the prior:

    p(θ∣D)∝p(y∣θ)p(θ)p(\theta | \mathcal{D}) \propto p(y | \theta) p(\theta)

    Substitute the exponentials:

    ∝exp⁡(−12(y−ΦTθ)TΣ−1(y−ΦTθ))exp⁡(−12θTΓ−1θ)\propto \exp \left( -\frac{1}{2} (y - \Phi^T \theta)^T \Sigma^{-1} (y - \Phi^T \theta) \right) \exp \left( -\frac{1}{2} \theta^T \Gamma^{-1} \theta \right)

    Combine the exponents into a single expression EE:

    E=−12[(y−ΦTθ)TΣ−1(y−ΦTθ)+θTΓ−1θ]E = -\frac{1}{2} \left[ (y - \Phi^T \theta)^T \Sigma^{-1} (y - \Phi^T \theta) + \theta^T \Gamma^{-1} \theta \right]

  4. Expand and Group Terms w.r.t θ\theta: Expand the first term (note that (y−ΦTθ)T=yT−θTΦ(y - \Phi^T \theta)^T = y^T - \theta^T \Phi):

    (y−ΦTθ)TΣ−1(y−ΦTθ)=yTΣ−1y−yTΣ−1ΦTθ−θTΦΣ−1y+θTΦΣ−1ΦTθ(y - \Phi^T \theta)^T \Sigma^{-1} (y - \Phi^T \theta) = y^T \Sigma^{-1} y - y^T \Sigma^{-1} \Phi^T \theta - \theta^T \Phi \Sigma^{-1} y + \theta^T \Phi \Sigma^{-1} \Phi^T \theta

    Since the result is a scalar, yTΣ−1ΦTθ=(θTΦΣ−1y)T=θTΦΣ−1yy^T \Sigma^{-1} \Phi^T \theta = (\theta^T \Phi \Sigma^{-1} y)^T = \theta^T \Phi \Sigma^{-1} y (assuming Σ\Sigma is symmetric). So, the cross terms are −2θTΦΣ−1y-2 \theta^T \Phi \Sigma^{-1} y.

    Now substitute back into EE and group by powers of θ\theta:

    −2E=θTΦΣ−1ΦTθ−2θTΦΣ−1y+yTΣ−1y+θTΓ−1θ-2E = \theta^T \Phi \Sigma^{-1} \Phi^T \theta - 2 \theta^T \Phi \Sigma^{-1} y + y^T \Sigma^{-1} y + \theta^T \Gamma^{-1} \theta

    Group quadratic terms (θTAθ\theta^T A \theta) and linear terms (−2θTb-2 \theta^T b):

    −2E=θT(ΦΣ−1ΦT+Γ−1)θ−2θT(ΦΣ−1y)+const-2E = \theta^T (\Phi \Sigma^{-1} \Phi^T + \Gamma^{-1}) \theta - 2 \theta^T (\Phi \Sigma^{-1} y) + \text{const}

    where "const" involves terms independent of θ\theta (like yTΣ−1yy^T \Sigma^{-1} y).

  5. Complete the Square: We compare this to the exponent of a Gaussian N(θ∣μ^,Σ^)\mathcal{N}(\theta | \hat{\mu}, \hat{\Sigma}):

    −12(θ−μ^)TΣ^−1(θ−μ^)=−12[θTΣ^−1θ−2θTΣ^−1μ^+μ^TΣ^−1μ^]-\frac{1}{2} (\theta - \hat{\mu})^T \hat{\Sigma}^{-1} (\theta - \hat{\mu}) = -\frac{1}{2} [\theta^T \hat{\Sigma}^{-1} \theta - 2 \theta^T \hat{\Sigma}^{-1} \hat{\mu} + \hat{\mu}^T \hat{\Sigma}^{-1} \hat{\mu}]

    Comparing the quadratic term θT(… )θ\theta^T (\dots) \theta:

    Σ^θ−1=Γ−1+ΦΣ−1ΦT\hat{\Sigma}_\theta^{-1} = \Gamma^{-1} + \Phi \Sigma^{-1} \Phi^T

    So,

    Σ^θ=(Γ−1+ΦΣ−1ΦT)−1\hat{\Sigma}_\theta = (\Gamma^{-1} + \Phi \Sigma^{-1} \Phi^T)^{-1}

    Comparing the linear term −2θT(… )-2 \theta^T (\dots):

    Σ^θ−1μ^θ=ΦΣ−1y\hat{\Sigma}_\theta^{-1} \hat{\mu}_\theta = \Phi \Sigma^{-1} y μ^θ=Σ^θ(ΦΣ−1y)=(Γ−1+ΦΣ−1ΦT)−1ΦΣ−1y\hat{\mu}_\theta = \hat{\Sigma}_\theta (\Phi \Sigma^{-1} y) = (\Gamma^{-1} + \Phi \Sigma^{-1} \Phi^T)^{-1} \Phi \Sigma^{-1} y

  6. Conclusion: The posterior is indeed Gaussian with the derived mean and covariance:

    p(θ∣D)=N(θ∣μ^θ,Σ^θ)p(\theta|\mathcal{D}) = \mathcal{N}(\theta|\hat{\mu}_\theta, \hat{\Sigma}_\theta)