Skip to main content

Answer

Prerequisites​

  • Lagrange Multipliers: A method for finding the local maxima and minima of a function subject to equality constraints. For maximizing f(x)f(x) subject to g(x)=0g(x)=0, we construct the Lagrangian L(x,λ)=f(x)+λg(x)L(x, \lambda) = f(x) + \lambda g(x) and set ∇L=0\nabla L = 0.
  • Derivatives: Specifically, the derivative of log⁡(x)\log(x).

Solution​

We want to maximize the objective function: f({πj})=∑j=1KNjlog⁡πjf(\{\pi_j\}) = \sum_{j=1}^K N_j \log \pi_j

Subject to the constraint: ∑j=1Kπj=1\sum_{j=1}^K \pi_j = 1

We can rewrite the equality constraint as g({πj})=∑j=1Kπj−1=0g(\{\pi_j\}) = \sum_{j=1}^K \pi_j - 1 = 0.

Step 1: Form the Lagrangian​

Construct the Lagrangian function L({πj},λ)L(\{\pi_j\}, \lambda): L({πj},λ)=∑j=1KNjlog⁡πj+λ(∑j=1Kπj−1)L(\{\pi_j\}, \lambda) = \sum_{j=1}^K N_j \log \pi_j + \lambda \left( \sum_{j=1}^K \pi_j - 1 \right)

Step 2: Take derivatives and set to zero​

Take the partial derivative with respect to πj\pi_j for any specific j∈{1,…,K}j \in \{1, \dots, K\}:

∂L∂πj=∂∂πj(Njlog⁡πj)+∂∂πj(λπj)\frac{\partial L}{\partial \pi_j} = \frac{\partial}{\partial \pi_j} (N_j \log \pi_j) + \frac{\partial}{\partial \pi_j} (\lambda \pi_j) ∂L∂πj=Njπj+λ\frac{\partial L}{\partial \pi_j} = \frac{N_j}{\pi_j} + \lambda

Set the derivative to zero to find the stationary point: Njπj+λ=0  ⟹  Nj=−λπj  ⟹  πj=−Njλ\frac{N_j}{\pi_j} + \lambda = 0 \implies N_j = -\lambda \pi_j \implies \pi_j = -\frac{N_j}{\lambda}

Step 3: Solve for the Lagrange multiplier λ\lambda​

Use the constraint ∑j=1Kπj=1\sum_{j=1}^K \pi_j = 1 by substituting the expression for πj\pi_j: ∑j=1K(−Njλ)=1\sum_{j=1}^K \left( -\frac{N_j}{\lambda} \right) = 1 −1λ∑j=1KNj=1-\frac{1}{\lambda} \sum_{j=1}^K N_j = 1

Solving for λ\lambda: λ=−∑j=1KNj\lambda = -\sum_{j=1}^K N_j

Let N=∑k=1KNkN = \sum_{k=1}^K N_k denote the total count. Then λ=−N\lambda = -N.

Step 4: Substitute λ\lambda back to find πj\pi_j​

Substitute λ=−∑k=1KNk\lambda = -\sum_{k=1}^K N_k back into the equation πj=−Njλ\pi_j = -\frac{N_j}{\lambda}: πj=−Nj−∑k=1KNk=Nj∑k=1KNk\pi_j = -\frac{N_j}{-\sum_{k=1}^K N_k} = \frac{N_j}{\sum_{k=1}^K N_k}

Thus, we have shown that the solution is: πj=Nj∑k=1KNk\pi_j = \frac{N_j}{\sum_{k=1}^K N_k}