Skip to main content

Question

(e) Given a novel input x∗x_*, show that the predictive distribution of f∗=f(x∗,θ)f_* = f(x_*, \theta) is

p(f∗∣x∗,D)=N(f∗∣μ^∗,σ^∗2),μ^∗=ϕ(x∗)Tμ^θ,σ^∗2=ϕ(x∗)TΣ^θϕ(x∗).\begin{align} p(f_*|x_*, \mathcal{D}) &= \mathcal{N}(f_*|\hat{\mu}_*, \hat{\sigma}^2_*), \tag{3.50} \\ \hat{\mu}_* &= \phi(x_*)^T \hat{\mu}_\theta, \tag{3.51} \\ \hat{\sigma}^2_* &= \phi(x_*)^T \hat{\Sigma}_\theta \phi(x_*). \tag{3.52} \end{align}

(Hint: see Problem 1.1). Assuming the same observation noise σ2\sigma^2 as the training set, show that the predictive distribution of y∗y_* is

p(y∗∣x∗,D)=∫p(y∗∣x∗,θ)p(θ∣D)dθ=N(y∗∣μ^∗,σ2+σ^∗2).(3.53)p(y_*|x_*, \mathcal{D}) = \int p(y_*|x_*, \theta) p(\theta|\mathcal{D}) d\theta = \mathcal{N}(y_*|\hat{\mu}_*, \sigma^2 + \hat{\sigma}^2_*). \tag{3.53}

Hint: note that p(y∗∣x∗,θ)p(y_*|x_*, \theta) only depends on θ\theta through f∗=ϕ(x∗)Tθf_* = \phi(x_*)^T \theta. Hence, we can rewrite the integral over θ\theta with an integral over f∗f_*, while replacing p(θ∣D)p(\theta|\mathcal{D}) with p(f∗∣D)p(f_*|\mathcal{D}).

This is the linear version of Gaussian process regression. We will see how to derive the nonlinear (kernel) version in a later problem set.