5  Data 102 Discussion 05: Analyzing Estimators

5.0.1 Contact Information

Name Wesley Zheng
Pronouns He/him/his
Email wzheng0302@berkeley.edu
Discussion Wednesdays, 11–12 PM @ Wheeler 224
Office Hours Tuesdays, 9–10 AM & 11-12 PM @ Gateway B1040

Contact me by email at ease — I typically respond within a day or so!

5.1 Analyzing Estimators

Suppose that: \[ \Theta \sim \text{Normal}(\mu_{\theta},\sigma_{\theta}^2), \qquad \{X_i\}_{i=1}^n \mid \Theta = \theta \underset{\text{i.i.d}}{\sim} \text{Normal}(\theta,\sigma_x^2). \]

You can observe \(\{X_i\}_{i=1}^n\) and want to estimate \(\Theta\). Then the MLE is the sample average: \[ \hat{\theta}_{\text{MLE}}(\{x_i\}_{i=1}^n) = \bar{x}_n = \frac{1}{n} \sum_{i=1}^n x_i. \]

In this problem, you will compare the behavior of the MLE to the behavior of a posterior estimator (see lecture notes from week 5).

5.1.1 (a)

Show that: \[ \Theta \mid \{X_i = x_i\}_{i=1}^n \sim \text{Normal}\left( a, b \right) \] where: \[ a = (1 - \lambda) \hat{\theta}_{\text{MLE}}(\{x_i\}_{i=1}^n) + \lambda \mu_{\theta}, \qquad \lambda = \frac{1/\sigma_{\theta}^2}{n/\sigma_x^2 + 1/\sigma_{\theta}^2}, \] \[ \text{and} \qquad b = \frac{1}{n/\sigma_x^2 + 1/\sigma_{\theta}^2}. \]

Answer

Applying Bayes rule: \[ \begin{align*} f_{\Theta \mid \{X_i = x_i\}_{i=1}^n}(\theta) &\propto f_{\Theta}(\theta)\, f_{\{X_i\}_{i=1}^n \mid \Theta = \theta}(x_1,x_2,...,x_n) \\ &\propto \exp\left(-\frac{1}{2} \left( \frac{\sum_{i=1}^n (x_i - \theta)^2}{\sigma_x^2} + \frac{(\theta - \mu_{\theta})^2}{\sigma_{\theta}^2} \right)\right) \end{align*} \]

The prior is \(\Theta \sim \text{Normal}(\mu_\theta, \sigma_\theta^2)\), so \[ f_\Theta(\theta) = \frac{1}{\sqrt{2\pi\sigma_\theta^2}} \exp\left(-\frac{(\theta - \mu_\theta)^2}{2\sigma_\theta^2}\right) \propto \exp\left(-\frac{(\theta - \mu_\theta)^2}{2\sigma_\theta^2}\right). \] The front factor \(\frac{1}{\sqrt{2\pi\sigma_\theta^2}}\) has no \(\theta\) in it, so it gets dropped since it asks like a constant.

For the likelihood, each observation is \(X_i \mid \Theta = \theta \sim \text{Normal}(\theta, \sigma_x^2)\), i.e., the true \(\theta\) plus some noise. Because the observations are conditionally i.i.d., the joint density is a product of \(n\) individual densities: \[ f_{X \mid \Theta = \theta}(x_1, \ldots, x_n) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi\sigma_x^2}} \exp\left(-\frac{(x_i - \theta)^2}{2\sigma_x^2}\right). \] The constants multiply into \(\left(\frac{1}{\sqrt{2\pi\sigma_x^2}}\right)^n\), which again has no \(\theta\), so we drop it. Multiplying exponentials adds their exponents (\(e^{a_1} e^{a_2} \cdots e^{a_n} = e^{a_1 + \cdots + a_n}\)), so \[ f_{X \mid \Theta = \theta}(x_1, \ldots, x_n) \propto \exp\left(-\frac{1}{2\sigma_x^2} \sum_{i=1}^n (x_i - \theta)^2\right). \]

Multiplying the prior and the likelihood adds their exponents one more time, and pulling out the common \(-\frac{1}{2}\) gives exactly the second line of the solution.

The expression inside the parentheses is a quadratic function of \(\theta\), so \(\Theta \mid \{X_i = x_i\}_{i=1}^n\) is a real-valued, continuous random variable that is unbounded above and below whose log density is quadratic in \(\theta\). It follows that the posterior distribution must be normal.

To find the mean and variance, complete the square inside the exponential. Note, we can drop any additive terms that are constant in \(\theta\) since these are absorbed in the normalization of the density.

From here on, the whole job is to rearrange the exponent until it looks like the exponent of a Normal. A \(\text{Normal}(a, b)\) density is \[ f(\theta) \propto \exp\left(-\frac{(\theta - a)^2}{2b}\right), \] so if we can write our exponent as \(-\frac{1}{2} \cdot (\text{some number}) \cdot (\theta - a)^2\), we can read off the mean \(a\) and the variance \(b\).

Why is it fine to throw away constants inside the exponent? Because \(e^{(\text{stuff}) + c} = e^{c} \cdot e^{\text{stuff}}\), and \(e^c\) is just a multiplicative constant, which we already agreed doesn’t matter. The notation \(\underset{+c}{=}\) means “equal, up to an added constant that doesn’t depend on \(\theta\).”

First: \[ \begin{align*} \sum_{i=1}^n (x_i - \theta)^2 &= \sum_{i=1}^n (\theta - x_i)^2 \\ &\underset{+c}{=} n \theta^2 - 2 \left(\sum_{i=1}^n x_i\right) \theta \\ &= n\left(\theta^2 -2 \left(\frac{1}{n} \sum_{i=1}^n x_i \right) \theta \right) \\ &\underset{+c}{=} n \left( \theta - \frac{1}{n} \sum_{i=1}^n x_i \right)^2 = n(\theta - \bar{x}_n)^2. \end{align*} \]

Expand one square: \[ (x_i - \theta)^2 = x_i^2 - 2x_i\theta + \theta^2. \] Summing over \(i\): \[ \sum_{i=1}^n (x_i - \theta)^2 = \sum_{i=1}^n x_i^2 - 2\theta \sum_{i=1}^n x_i + n\theta^2. \] The term \(\sum_i x_i^2\) is just a number once the data are observed, so we drop it (that’s the first \(\underset{+c}{=}\)). Using \(\sum_i x_i = n\bar{x}_n\): \[ n\theta^2 - 2n\bar{x}_n\theta = n\left(\theta^2 - 2\bar{x}_n\theta\right). \] To complete the square, recall \((\theta - \bar{x}_n)^2 = \theta^2 - 2\bar{x}_n\theta + \bar{x}_n^2\), so \[ \theta^2 - 2\bar{x}_n\theta = (\theta - \bar{x}_n)^2 - \bar{x}_n^2. \] The leftover \(-\bar{x}_n^2\) is constant in \(\theta\), so we drop it too (the second \(\underset{+c}{=}\)).

Then: \[ \begin{align*} &\frac{\sum_{i=1}^n (x_i - \theta)^2}{\sigma_x^2} + \frac{(\theta - \mu_{\theta})^2}{\sigma_{\theta}^2} \\ &\underset{+c}{=} \left(\frac{n}{\sigma_x^2} + \frac{1}{\sigma_{\theta}^2} \right) \theta^2 - 2 \left( \frac{n}{\sigma_x^2} \bar{x}_n + \frac{1}{\sigma_{\theta}^2} \mu_{\theta} \right) \theta \\ &= \left(\frac{n}{\sigma_x^2} + \frac{1}{\sigma_{\theta}^2} \right) \left[ \theta^2 - 2 \left(\frac{\frac{n}{\sigma_x^2} \bar{x}_n + \frac{1}{\sigma_{\theta}^2} \mu_{\theta} }{ \frac{n}{\sigma_x^2} + \frac{1}{\sigma_{\theta}^2}} \right) \theta \right] \\ &\underset{+c}{=} \left(\frac{n}{\sigma_x^2} + \frac{1}{\sigma_{\theta}^2} \right) \left[ \theta - \left(\frac{\frac{n}{\sigma_x^2} \bar{x}_n + \frac{1}{\sigma_{\theta}^2} \mu_{\theta} }{ \frac{n}{\sigma_x^2} + \frac{1}{\sigma_{\theta}^2}} \right) \right]^2 \end{align*} \]

Plugging in the previous result, the exponent contains \[ \frac{n}{\sigma_x^2}(\theta - \bar{x}_n)^2 + \frac{1}{\sigma_\theta^2}(\theta - \mu_\theta)^2. \] Expand both squares: \[ \begin{align*} \frac{n}{\sigma_x^2}(\theta - \bar{x}_n)^2 &= \frac{n}{\sigma_x^2}\left(\theta^2 - 2\bar{x}_n\theta + \bar{x}_n^2\right), \\ \frac{1}{\sigma_\theta^2}(\theta - \mu_\theta)^2 &= \frac{1}{\sigma_\theta^2}\left(\theta^2 - 2\mu_\theta\theta + \mu_\theta^2\right). \end{align*} \] Drop the terms without \(\theta\) (\(\bar{x}_n^2\) and \(\mu_\theta^2\)), then collect the \(\theta^2\) terms and the \(\theta\) terms. That gives the second line of the solution.

The expression looks scary, so give the coefficients short names: \[ A = \frac{n}{\sigma_x^2} + \frac{1}{\sigma_\theta^2}, \qquad B = \frac{n}{\sigma_x^2}\bar{x}_n + \frac{1}{\sigma_\theta^2}\mu_\theta. \] Now the exponent is just \(A\theta^2 - 2B\theta\). Factor out \(A\) to get \(A\left(\theta^2 - 2\frac{B}{A}\theta\right)\), which is the third line. Then complete the square exactly as before, using \((\theta - c)^2 = \theta^2 - 2c\theta + c^2\) with \(c = \frac{B}{A}\): \[ A\theta^2 - 2B\theta \underset{+c}{=} A\left(\theta - \frac{B}{A}\right)^2. \] Substituting \(A\) and \(B\) back in gives the last line. So the posterior is \[ f_{\Theta \mid X}(\theta) \propto \exp\left[-\frac{1}{2} A \left(\theta - \frac{B}{A}\right)^2\right]. \]

We can now identify the mean and variance of the normal by recognizing their roles in the quadratic: \[ \begin{align*} a &= \frac{\frac{n}{\sigma_x^2} \bar{x}_n + \frac{1}{\sigma_{\theta}^2} \mu_{\theta} }{ \frac{n}{\sigma_x^2} + \frac{1}{\sigma_{\theta}^2}} = \frac{n/\sigma_x^2}{n/\sigma_x^2 + 1/\sigma_{\theta}^2} \bar{x}_n + \frac{1/\sigma_{\theta}^2}{n/\sigma_x^2 + 1/\sigma_{\theta}^2} \mu_{\theta} \\ b &= \left(\frac{n}{\sigma_x^2} + \frac{1}{\sigma_{\theta}^2} \right)^{-1} \end{align*} \]

Line our result up against the Normal template: \[ \begin{align*} \text{Normal}(\mu, \sigma^2): \quad & \exp\left[-\frac{1}{2} \cdot \frac{1}{\sigma^2} (y - \mu)^2\right] \\ \text{our posterior}: \quad & \exp\left[-\frac{1}{2} \left(\frac{n}{\sigma_x^2} + \frac{1}{\sigma_\theta^2}\right) (\theta - a)^2\right] \end{align*} \] Whatever is subtracted from \(\theta\) inside the square is the mean, so \(a = \frac{B}{A}\). The number multiplying the square is \(\frac{1}{\text{variance}}\), so \[ \frac{1}{b} = \frac{n}{\sigma_x^2} + \frac{1}{\sigma_\theta^2} \quad \Longrightarrow \quad b = \frac{1}{n/\sigma_x^2 + 1/\sigma_\theta^2}. \] This is the step people most often miss: the coefficient is one over the variance, not the variance itself.

To get the weighted-average form of \(a\), split the fraction into two pieces with the same denominator (the second expression for \(a\) above). The two coefficients add to \(1\), so if we call the second one \(\lambda\), the first one must be \(1 - \lambda\).

Substituting for \(\lambda\) completes the proposed statement.

The formula for \(a\) becomes much more intuitive once you think in terms of precision, which is just \(\frac{1}{\text{variance}}\). High variance means low confidence, which means low precision.

The prior has variance \(\sigma_\theta^2\), so its precision is \(\frac{1}{\sigma_\theta^2}\). The sample mean satisfies \(\bar{X}_n \mid \Theta = \theta \sim \text{Normal}\left(\theta, \frac{\sigma_x^2}{n}\right)\), so its precision is \(\frac{1}{\sigma_x^2 / n} = \frac{n}{\sigma_x^2}\). Those are exactly the two quantities that keep showing up: \[ \underbrace{\frac{n}{\sigma_x^2}}_{\text{data precision}} \qquad \text{and} \qquad \underbrace{\frac{1}{\sigma_\theta^2}}_{\text{prior precision}}. \] So the posterior mean is \[ a = \frac{(\text{data precision})(\bar{x}_n) + (\text{prior precision})(\mu_\theta)}{\text{data precision} + \text{prior precision}}, \] a weighted average where the more reliable source gets more weight. If \(\sigma_\theta^2\) is tiny, the prior precision is huge and the prior dominates. If \(n\) is huge, the data precision is huge and the data dominate. The posterior variance \(b\) is one over the total precision, so combining the two sources always makes you more confident than either one alone.

Putting the whole derivation in one chain: \[ \begin{align*} f_{\Theta \mid X}(\theta) &\propto f_\Theta(\theta) \prod_{i=1}^n f_{X_i \mid \Theta = \theta}(x_i) \\ &\propto \exp\left(-\frac{(\theta - \mu_\theta)^2}{2\sigma_\theta^2}\right) \exp\left(-\frac{\sum_i (x_i - \theta)^2}{2\sigma_x^2}\right) \\ &\propto \exp\left\{-\frac{1}{2}\left[\left(\frac{n}{\sigma_x^2} + \frac{1}{\sigma_\theta^2}\right)\theta^2 - 2\left(\frac{n}{\sigma_x^2}\bar{x}_n + \frac{1}{\sigma_\theta^2}\mu_\theta\right)\theta\right]\right\} \\ &\propto \exp\left[-\frac{1}{2}\left(\frac{n}{\sigma_x^2} + \frac{1}{\sigma_\theta^2}\right)(\theta - a)^2\right]. \end{align*} \] The two tricks doing all the work are dropping anything that doesn’t depend on \(\theta\), and completing the square so the exponent matches a Normal density.

5.1.2 (b)

Let \(\hat{\theta}_{\text{mean}}(\{x_i\}_{i=1}^n) = \mathbb{E}[\Theta \mid \{X_i = x_i\}_{i=1}^n] = a\) denote the posterior mean estimator.

5.1.2.1 (i)

Show that, for any fixed \(\hat{\theta}_{\text{MLE}} \neq \mu_{\theta}\), the posterior mean estimator lies between \(\hat{\theta}_{\text{MLE}}\) and \(\mu_{\theta}\) when \(\sigma_x\), \(n\), and \(\sigma_{\theta}\) are finite and nonzero.

Answer

The coefficient \(\lambda \in [0,1]\) so the posterior mean estimator is a convex combination of the MLE and the prior mean. Therefore, it must lie between the two.

5.1.2.2 (ii)

Show that, for any fixed \(\hat{\theta}_{\text{MLE}} \neq \mu_{\theta}\), the posterior mean estimator approaches \(\hat{\theta}_{\text{MLE}}\) if \(n \rightarrow \infty\). Justify this result conceptually.

Answer

If \(n \rightarrow \infty\) then \(\lambda \rightarrow 0\) and \(1 - \lambda \rightarrow 1\). Therefore, the posterior mean estimator converges to the MLE.

This is a sensible result since, as \(n\) increases we should pay more attention to our data, and less to our prior expectation.

5.1.2.3 (iii)

Show that the posterior mean estimator converges to the MLE in the limit as \(\sigma_{\theta}^2 \rightarrow \infty\). Justify this result conceptually.

Answer

If \(\sigma_{\theta} \rightarrow \infty\) then \(1/\sigma_{\theta}^2 \rightarrow 0\) so \(\lambda \rightarrow 0\) and \(1 - \lambda \rightarrow 1\).

In this case the result is sensible since:

  1. The mean of a normal distribution equals its mode, so the posterior mean estimator is also the MAP estimator.
  2. As \(\sigma_{\theta}\) diverges the prior distribution converges to a constant over any fixed interval. It follows that the posterior converges to a function proportional to the likelihood on any fixed interval, so the maximizer of the posterior should converge to the maximizer of the likelihood. Thus, the MAP estimator should converge to the MLE.

5.1.2.4 (iv)

Show that the posterior mean estimator converges to the prior mean, \(\mu_{\theta}\), in the limit as \(\sigma_{\theta}^2 \rightarrow 0\). Justify this result conceptually.

Answer

If \(\sigma_{\theta} \rightarrow 0\) then \(1/\sigma_{\theta}^2 \rightarrow \infty\) so \(\lambda \rightarrow 1\) and \(1 - \lambda \rightarrow 0\). Therefore, the posterior mean converges to the prior expectation.

This is also sensible. When \(\sigma_{\theta}\) is very small, the prior is very concentrated about its mean (we are very confident, a priori, that \(\Theta\) is very close to \(\mu_{\theta}\)).

5.1.3 (c)

Find the:

5.1.3.1 (i)

Frequentist and Bayesian bias in the MLE: \[ \mathbb{E}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{MLE}}(\{X_i\}_{i=1}^n) - \Theta] = ? \] \[ \mathbb{E}_{\Theta, \{X_i\}_{i=1}^n}[\hat{\theta}_{\text{MLE}}(\{X_i\}_{i=1}^n) - \Theta] = ? \]

Answer

The MLE is unbiased in both senses since sample averages are unbiased estimators for an unknown mean, no matter the mean.

Working step-by-step: \[ \begin{align*} \mathbb{E}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{MLE}}(\{X_i\}_{i=1}^n) - \Theta] &= \mathbb{E}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}\left[\frac{1}{n}\sum_{i=1}^n X_i \right] - \theta \\ &= \frac{1}{n} (n \theta) - \theta = 0. \end{align*} \]

Then: \[ \begin{align*} \mathbb{E}_{\Theta, \{X_i\}_{i=1}^n}[\hat{\theta}_{\text{MLE}}(\{X_i\}_{i=1}^n) - \Theta] &= \mathbb{E}_{\Theta}\left[\mathbb{E}_{\{X_i\}_{i=1}^n \mid \Theta}[\hat{\theta}_{\text{MLE}}(\{X_i\}_{i=1}^n) - \Theta] \right] \\ &= \mathbb{E}_{\Theta}[0] = 0. \end{align*} \]

Here we’re computing the bias of the MLE in two different senses. The MLE is \[ \hat\theta_{\text{MLE}} = \bar X_n = \frac{1}{n}\sum_{i=1}^n X_i, \] and the estimation error is \(\hat\theta_{\text{MLE}} - \Theta\).

For the frequentist bias, we treat the unknown parameter as fixed, \(\Theta = \theta\), so we average only over the randomness in the sampled data: \[ \mathbb{E}_{\{X_i\} \mid \Theta = \theta}[\hat\theta_{\text{MLE}} - \Theta]. \] Since \(\Theta = \theta\) is fixed inside this expectation, and then substituting the sample mean: \[ \begin{align*} \mathbb{E}_{\{X_i\} \mid \Theta = \theta}[\hat\theta_{\text{MLE}} - \Theta] &= \mathbb{E}_{\{X_i\} \mid \Theta = \theta}[\hat\theta_{\text{MLE}}] - \theta \\ &= \mathbb{E}\left[\frac{1}{n}\sum_{i=1}^n X_i \;\middle|\; \Theta = \theta\right] - \theta \\ &= \frac{1}{n}\sum_{i=1}^n \mathbb{E}[X_i \mid \Theta = \theta] - \theta && \text{(linearity of expectation)} \end{align*} \] From the model, \(X_i \mid \Theta = \theta \sim \text{Normal}(\theta, \sigma_x^2)\), so \(\mathbb{E}[X_i \mid \Theta = \theta] = \theta\). Therefore \[ \frac{1}{n}(n\theta) - \theta = \theta - \theta = \boxed{0}. \] So the MLE is frequentist-unbiased for every fixed \(\theta\).

For the Bayesian bias, instead of fixing \(\Theta\), we allow \(\Theta \sim \text{Normal}(\mu_\theta, \sigma_\theta^2)\) and average over both the random parameter and the random data. Using the law of iterated expectation: \[ \mathbb{E}_{\Theta, \{X_i\}}[\hat\theta_{\text{MLE}} - \Theta] = \mathbb{E}_\Theta\left[\mathbb{E}_{\{X_i\} \mid \Theta}[\hat\theta_{\text{MLE}} - \Theta]\right]. \] But we already showed that for every possible value of \(\Theta\), the inner expectation is zero: \[ \mathbb{E}_{\{X_i\} \mid \Theta}[\hat\theta_{\text{MLE}} - \Theta] = 0. \] Therefore \[ \mathbb{E}_{\Theta, \{X_i\}}[\hat\theta_{\text{MLE}} - \Theta] = \mathbb{E}_\Theta[0] = \boxed{0}, \] so the MLE is also Bayesian-unbiased.

The key distinction is: \[ \boxed{ \begin{array}{ll} \text{Frequentist bias:} & \Theta = \theta \text{ fixed; average over data} \\ \text{Bayesian bias:} & \Theta \text{ random; average over } \Theta \text{ and data} \end{array} } \]

5.1.3.2 (ii)

Frequentist and Bayesian bias in the posterior mean estimator: \[ \mathbb{E}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{mean}}(\{X_i\}_{i=1}^n) - \Theta] = ? \] \[ \mathbb{E}_{\Theta, \{X_i\}_{i=1}^n}[\hat{\theta}_{\text{mean}}(\{X_i\}_{i=1}^n) - \Theta] = ? \]

Answer

First, the Frequentist bias: \[ \begin{align*} \mathbb{E}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{mean}}(\{X_i\}_{i=1}^n) - \Theta] &= \mathbb{E}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{mean}}(\{X_i\}_{i=1}^n)] - \theta \\ &= (1 - \lambda) \mathbb{E}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}\left[\hat{\theta}_{\text{MLE}}(\{X_i\}_{i=1}^n) \right] + \lambda \mu_{\theta} - \theta \\ &= (1 - \lambda) \theta + \lambda \mu_{\theta} - \theta \\ &= \lambda (\mu_{\theta} - \theta). \end{align*} \]

Next, the Bayesian bias: \[ \begin{align*} \mathbb{E}_{\Theta, \{X_i\}_{i=1}^n}[\hat{\theta}_{\text{mean}}(\{X_i\}_{i=1}^n) - \Theta] &= \mathbb{E}_{\Theta}\left[\mathbb{E}_{\{X_i\}_{i=1}^n \mid \Theta}[\hat{\theta}_{\text{mean}}(\{X_i\}_{i=1}^n) - \Theta] \right] \\ &= \mathbb{E}_{\Theta}\left[\lambda (\mu_{\theta} - \Theta) \right] = \lambda (\mu_{\theta} - \mathbb{E}_{\Theta}[\Theta]) \\ &= \lambda (\mu_{\theta} - \mu_{\theta}) = 0. \end{align*} \]

This part asks for the bias of the posterior mean estimator in two different senses. Recall from part (a): \[ \hat{\theta}_{\text{mean}} = (1-\lambda)\hat{\theta}_{\text{MLE}} + \lambda\mu_\theta. \] Since \(\hat{\theta}_{\text{MLE}} = \bar X_n\), we can also write \[ \hat{\theta}_{\text{mean}} = (1-\lambda)\bar X_n + \lambda\mu_\theta. \]

For the frequentist bias, the quantity is \[ \mathbb{E}_{\{X_i\} \mid \Theta = \theta}\left[\hat\theta_{\text{mean}} - \Theta\right]. \] The important thing is that here we condition on \(\Theta = \theta\), so \(\Theta\) is treated as the fixed number \(\theta\). Then we substitute the posterior mean estimator, pull out the constants \(\lambda\) and \(\mu_\theta\), and use the fact from part (c)(i) that the MLE is conditionally unbiased, \(\mathbb{E}[\hat\theta_{\text{MLE}} \mid \Theta = \theta] = \theta\): \[ \begin{align*} \mathbb{E}_{\{X_i\} \mid \Theta = \theta}[\hat\theta_{\text{mean}} - \Theta] &= \mathbb{E}_{\{X_i\} \mid \Theta = \theta}[\hat\theta_{\text{mean}}] - \theta \\ &= \mathbb{E}\left[(1-\lambda)\hat\theta_{\text{MLE}} + \lambda\mu_\theta \mid \Theta = \theta\right] - \theta \\ &= (1-\lambda)\,\mathbb{E}[\hat\theta_{\text{MLE}} \mid \Theta = \theta] + \lambda\mu_\theta - \theta \\ &= (1-\lambda)\theta + \lambda\mu_\theta - \theta \\ &= \theta - \lambda\theta + \lambda\mu_\theta - \theta && \text{(expand)} \\ &= -\lambda\theta + \lambda\mu_\theta && \text{(the two } \theta\text{'s cancel)} \\ &= \boxed{\lambda(\mu_\theta - \theta)}. \end{align*} \]

What does that mean? Unless \(\theta = \mu_\theta\), this is generally not zero, because the posterior mean deliberately pulls the estimate toward the prior mean. For example, if \(\theta = 10\) and \(\mu_\theta = 4\), then \(\mu_\theta - \theta = -6\), so the bias is negative: \(-6\lambda\). That makes sense: the true value is \(10\), but the prior pulls your estimate downward toward \(4\). So the posterior mean is conditionally (frequentist) biased.

For the Bayesian bias, the question changes. We want \[ \mathbb{E}_{\Theta, \{X_i\}}[\hat\theta_{\text{mean}} - \Theta], \] and now \(\Theta\) itself is random, \(\Theta \sim \text{Normal}(\mu_\theta, \sigma_\theta^2)\). We already know that if we fix \(\Theta = \theta\), the conditional bias is \(\lambda(\mu_\theta - \theta)\). So if \(\Theta\) is random, replace \(\theta\) by \(\Theta\): \[ \mathbb{E}_{\{X_i\} \mid \Theta}[\hat\theta_{\text{mean}} - \Theta] = \lambda(\mu_\theta - \Theta). \] Now average that over \(\Theta\), pull out \(\lambda\), and use linearity. The prior says \(\mathbb{E}[\Theta] = \mu_\theta\), so \[ \begin{align*} \mathbb{E}_\Theta\left[\lambda(\mu_\theta - \Theta)\right] &= \lambda\,\mathbb{E}_\Theta[\mu_\theta - \Theta] \\ &= \lambda\left(\mu_\theta - \mathbb{E}[\Theta]\right) \\ &= \lambda(\mu_\theta - \mu_\theta) = \boxed{0}. \end{align*} \] So the posterior mean is Bayesian unbiased.

Why can it be biased in one sense but unbiased in the other? This is the important conceptual point. Suppose the true value happens to be above the prior mean, \(\theta > \mu_\theta\). Then the posterior mean gets pulled downward, so the bias is negative. If instead \(\theta < \mu_\theta\), the posterior mean gets pulled upward, so the bias is positive. Across the prior distribution, these effects balance out: since the prior is centered at \(\mu_\theta\), values above and below \(\mu_\theta\) average together so that \(\mathbb{E}_\Theta[\mu_\theta - \Theta] = 0\). So: \[ \boxed{\text{For fixed } \theta: \quad \text{bias} = \lambda(\mu_\theta - \theta)} \qquad \text{but} \qquad \boxed{\text{averaging over } \Theta: \quad \text{bias} = 0.} \] That is exactly why the solution says the posterior mean is conditionally biased but unbiased over the joint model.

5.1.3.3 (iii)

Identify which estimators are biased, and, when biased, under what sampling protocol (e.g. sampling data conditional on the unknown, or sampling jointly).

Answer The MLE is unbiased in both senses. The Bayesian estimator is conditionally biased, given \(\Theta = \theta\), with bias proportional to the difference between \(\theta\) and its prior expectation, weighted by the coefficient \(\lambda\) that determines how strongly the posterior mean is biased towards the prior mean. The posterior mean estimator is unbiased over the joint model.

5.1.4 (d)

Find the Frequentist variance in the errors for both estimators: \[ \text{Var}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{MLE}}(\{X_i\}_{i=1}^n) - \Theta] = ? \] \[ \text{Var}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{mean}}(\{X_i\}_{i=1}^n) - \Theta] = ? \] Which estimator is more stable (i.e. less variable)?

Answer

First, the MLE: \[ \begin{align*} \text{Var}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{MLE}}(\{X_i\}_{i=1}^n) - \Theta] &= \text{Var}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{MLE}}(\{X_i\}_{i=1}^n) - \theta] \\ &= \text{Var}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{MLE}}(\{X_i\}_{i=1}^n)] \\ &= \text{Var}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\bar{X}_n] = \frac{\sigma_x^2}{n} \end{align*} \] since the variance of a sample average of i.i.d. random variables is the variance in a single sample divided by the sample size.

Then: \[ \begin{align*} \text{Var}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{mean}}(\{X_i\}_{i=1}^n) - \Theta] &= \text{Var}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{mean}}(\{X_i\}_{i=1}^n) - \theta] \\ &= \text{Var}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{mean}}(\{X_i\}_{i=1}^n)] \\ &= \text{Var}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[(1 - \lambda) \bar{X}_n + \lambda \mu_{\theta}] \\ &= (1 - \lambda)^2 \frac{\sigma_x^2}{n} \end{align*} \]

The posterior mean estimator is less variable since the variance in the posterior mean estimator is \((1 - \lambda)^2\) the variance in the MLE and \(\lambda \in [0,1]\).

This is sensible. The posterior mean estimator is less sensitive to the data, so is less variable.

5.1.5 (e)

Using the law of total variance, find the Bayesian variance in the errors for both estimators: \[ \text{Var}_{\Theta, \{X_i\}_{i=1}^n}[\hat{\theta}_{\text{MLE}}(\{X_i\}_{i=1}^n) - \Theta] = ? \] \[ \text{Var}_{\Theta, \{X_i\}_{i=1}^n}[\hat{\theta}_{\text{mean}}(\{X_i\}_{i=1}^n) - \Theta] = ? \]

Answer

First, the MLE: \[ \begin{align*} \text{Var}_{\Theta,\{X_i\}_{i=1}^n}\left[\hat{\theta}_{\text{MLE}}(\{X_i\}_{i=1}^n) - \Theta \right] =\ & \mathbb{E}_{\Theta} \left[ \text{Var}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{MLE}}(\{X_i\}_{i=1}^n) - \Theta]\right] \\ & + \text{Var}_{\Theta} \left[ \mathbb{E}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{MLE}}(\{X_i\}_{i=1}^n) - \Theta]\right] \end{align*} \]

The second term is zero since the MLE is unbiased (see part (c)). Therefore, using our results from part (d): \[ \begin{align*} \text{Var}_{\Theta,\{X_i\}_{i=1}^n}\left[\hat{\theta}_{\text{MLE}}(\{X_i\}_{i=1}^n) - \Theta \right] &= \mathbb{E}_{\Theta} \left[ \text{Var}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{MLE}}(\{X_i\}_{i=1}^n) - \Theta]\right] \\ &= \mathbb{E}_{\Theta}\left[ \frac{\sigma_x^2}{n} \right] = \frac{\sigma_x^2}{n}. \end{align*} \]

Next, the posterior mean estimator: \[ \begin{align*} \text{Var}_{\Theta,\{X_i\}_{i=1}^n}\left[\hat{\theta}_{\text{mean}}(\{X_i\}_{i=1}^n) - \Theta \right] =\ & \mathbb{E}_{\Theta} \left[ \text{Var}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{mean}}(\{X_i\}_{i=1}^n) - \Theta]\right] \\ & + \text{Var}_{\Theta} \left[ \mathbb{E}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{mean}}(\{X_i\}_{i=1}^n) - \Theta]\right] \end{align*} \]

Now, as for the MLE, the conditional variance in the posterior mean estimator does not depend on \(\Theta\) so, by part (d): \[ \begin{align*} \mathbb{E}_{\Theta} \left[ \text{Var}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{mean}}(\{X_i\}_{i=1}^n) - \Theta]\right] &= \mathbb{E}_{\Theta}\left[ (1 - \lambda)^2 \frac{\sigma_x^2}{n}\right] \\ &= (1 - \lambda)^2 \frac{\sigma_x^2}{n}. \end{align*} \]

The variance in the conditional bias is, by part (c): \[ \begin{align*} \text{Var}_{\Theta} \left[ \mathbb{E}_{\{X_i\}_{i=1}^n \mid \Theta = \theta}[\hat{\theta}_{\text{mean}}(\{X_i\}_{i=1}^n) - \Theta]\right] &= \text{Var}_{\Theta}[\lambda (\mu_{\theta} - \Theta)] \\ &= \lambda^2 \text{Var}_{\Theta}[\Theta] = \lambda^2 \sigma_{\theta}^2. \end{align*} \]

Therefore: \[ \text{Var}_{\Theta,\{X_i\}_{i=1}^n}\left[\hat{\theta}_{\text{mean}}(\{X_i\}_{i=1}^n) - \Theta \right] = (1 - \lambda)^2 \frac{\sigma_x^2}{n} + \lambda^2 \sigma_{\theta}^2. \]

\[ \text{Var}(Y) = \mathbb{E}\left[\text{Var}(Y \mid X)\right] + \text{Var}\left(\mathbb{E}[Y \mid X]\right) \] Here, take \(Y = \hat\theta - \Theta\) (the estimation error), and condition on \(\Theta\). So: \[ \text{Var}_{\Theta, X}(\hat\theta - \Theta) = \mathbb{E}_\Theta\left[\text{Var}_{X \mid \Theta}(\hat\theta - \Theta)\right] + \text{Var}_\Theta\left[\mathbb{E}_{X \mid \Theta}(\hat\theta - \Theta)\right]. \]

The first term, \(\mathbb{E}_\Theta\left[\text{Var}_{X \mid \Theta}(Y)\right]\), measures: for a fixed \(\Theta\), how much does the error fluctuate because I happened to get different random datasets? Then we average that variability over possible \(\Theta\)’s.

The second term, \(\text{Var}_\Theta\left[\mathbb{E}_{X \mid \Theta}(Y)\right]\), measures: as \(\Theta\) itself changes, how much does the average error change? That second quantity is especially connected to the conditional bias.

For the posterior mean, from part (c), \[ \mathbb{E}_{X \mid \Theta = \theta}\left[\hat\theta_{\text{mean}} - \Theta\right] = \lambda(\mu_\theta - \theta). \] So the average error depends on which \(\theta\) happened to be chosen. Therefore that creates an additional source of variance: \[ \text{Var}_\Theta\left[\lambda(\mu_\theta - \Theta)\right] = \lambda^2\sigma_\theta^2. \] Meanwhile, even after fixing \(\Theta\), the sampled data still vary, giving \((1-\lambda)^2\frac{\sigma_x^2}{n}\). So together: \[ \text{Var}(\text{error}) = \underbrace{(1-\lambda)^2\frac{\sigma_x^2}{n}}_{\text{randomness from sampling data}} + \underbrace{\lambda^2\sigma_\theta^2}_{\text{variation in conditional bias as } \Theta \text{ changes}} \]

A good mental model is: \[ \boxed{\text{total variability} = \text{variability within each } \Theta + \text{variability between different } \Theta\text{'s}.} \] That is exactly what the law of total variance is decomposing.

5.1.6 (f)

Given an estimator \(\hat{\theta}\), let \(E = \hat{\theta}(\{X_i\}_{i=1}^n) - \Theta\) denote the error in the estimator, when applied to data \(\{X_i\}_{i=1}^n\) and when the true value of the unknown is \(\Theta\). Then, the Bayes risk for a squared-error loss can be expressed: \[ \begin{align*} R_B &= \mathbb{E}_{\Theta,\{X_i\}_{i=1}^n}[(\hat{\theta}(\{X_i\}_{i=1}^n) - \Theta)^2] \\ &= \mathbb{E}_{\Theta,\{X_i\}_{i=1}^n}[E]^2 + \text{Var}_{\Theta,\{X_i\}_{i=1}^n}[E] \end{align*} \] where \(\mathbb{E}_{\Theta,\{X_i\}_{i=1}^n}[E]\) is the bias in the estimator. Since both of our estimators are unbiased with respect to the joint model, the variances you computed in part (e) are their Bayes risk.

5.1.6.1 (i)

Relate the Bayes risk in the posterior mean estimator to the Bayes risk in the MLE. Your expression for the Bayes risk in the posterior mean should involve two terms representing separate sources of variance/error. Explain each term.

Answer

For the posterior mean estimator: \[ R_B = 0^2 + \lambda^2 \sigma_{\theta}^2 + (1 - \lambda)^2 \frac{\sigma_x^2}{n} = \lambda^2 \times \sigma_{\theta}^2 + (1 - \lambda)^2 \times \frac{\sigma_x^2}{n}. \]

For the MLE estimator: \[ R_B = 0^2 + \frac{\sigma_x^2}{n} = 0 \times \sigma_{\theta}^2 + 1 \times \frac{\sigma_x^2}{n}. \]

Both risks can be expanded into errors coming from variance in the prior (variance in the true value of the unknown) and variance in the sampled data. The MLE risk does not include any source associated with the prior, but is exposed to the full variance in the data. The risk for the posterior mean estimator includes both sources, but is less exposed to the variance in the data since it is less sensitive to the data. It does include a term associated with the variance in the prior since it biases its estimates towards the prior mean.

This part asks you to compare the Bayes risk of the posterior mean estimator with the Bayes risk of the MLE. Recall the Bayes risk under squared error is \[ R_B = \mathbb{E}_{\Theta, X}\left[(\hat\theta - \Theta)^2\right]. \] They define the error as \(E = \hat\theta - \Theta\), then use the identity \[ \mathbb{E}[E^2] = \text{Var}(E) + (\mathbb{E}[E])^2. \] So \[ R_B = \text{bias}^2 + \text{variance of error}. \] In this problem, both estimators have zero Bayesian bias, so the Bayes risk is just the Bayesian variance you computed in part (e). For the MLE: \[ R_B(\hat\theta_{\text{MLE}}) = \frac{\sigma_x^2}{n}. \] For the posterior mean: \[ R_B(\hat\theta_{\text{mean}}) = (1-\lambda)^2\frac{\sigma_x^2}{n} + \lambda^2\sigma_\theta^2. \]

The question wants you to interpret these two terms. The first term is \[ \boxed{(1-\lambda)^2\frac{\sigma_x^2}{n}} \] and this comes from the randomness in the data. Since the posterior mean only puts weight \(1-\lambda\) on the MLE/data, it is less sensitive to sample noise than the MLE. The MLE has full data variance \(\frac{\sigma_x^2}{n}\), while the posterior mean only has \((1-\lambda)^2\frac{\sigma_x^2}{n}\). So the posterior mean reduces the data-variance contribution.

The second term is \[ \boxed{\lambda^2\sigma_\theta^2} \] and this comes from the fact that the estimator shrinks toward the prior mean \(\mu_\theta\). Remember the conditional bias was \(\lambda(\mu_\theta - \Theta)\). As \(\Theta\) changes across the prior distribution, that conditional bias changes too. Its variance is \[ \text{Var}\left[\lambda(\mu_\theta - \Theta)\right] = \lambda^2\sigma_\theta^2. \] So the posterior mean introduces a second source of error from relying on the prior.

The comparison is therefore: \[ \boxed{R_B(\text{MLE}) = 0 \cdot \sigma_\theta^2 + 1 \cdot \frac{\sigma_x^2}{n}} \] while \[ \boxed{R_B(\text{posterior mean}) = \lambda^2\sigma_\theta^2 + (1-\lambda)^2\frac{\sigma_x^2}{n}.} \]

Conceptually:

  • The MLE ignores the prior, so it has no prior-related error term, but it suffers from the full sampling noise.
  • The posterior mean uses the prior, so it introduces some prior-related error, but reduces sensitivity to noisy data.

5.1.6.2 (ii)

Which estimator minimizes \(R_B\)?

Answer

The Bayes risk, with a square loss, is always minimized by the posterior mean estimator.

In this example, the posterior mean estimator minimizes the Bayes risk by biasing its estimates away from the MLE and towards the prior mean. This stabilizes the estimate enough to reduce the overall risk.

The question is: which estimator minimizes the Bayes risk? The answer is the posterior mean estimator. Under squared-error loss, \[ L(\hat\theta, \Theta) = (\hat\theta - \Theta)^2, \] the Bayes estimator is always \[ \boxed{\hat\theta_B(X) = \mathbb{E}[\Theta \mid X]} \] which is exactly the posterior mean.

The reason is easiest to see by conditioning on the observed data \(X\). Suppose after observing \(X = x\), you are choosing some estimate \(a\). Your posterior expected squared loss is \[ \mathbb{E}[(a - \Theta)^2 \mid X = x]. \] Now add and subtract the posterior mean \(m = \mathbb{E}[\Theta \mid X = x]\). Then \[ a - \Theta = (a - m) + (m - \Theta). \] Squaring, \[ (a - \Theta)^2 = (a - m)^2 + 2(a - m)(m - \Theta) + (m - \Theta)^2. \] Take the conditional expectation: \[ \begin{align*} \mathbb{E}[(a - \Theta)^2 \mid X = x] =\ & (a - m)^2 + 2(a - m)\,\mathbb{E}[m - \Theta \mid X = x] \\ & + \mathbb{E}[(m - \Theta)^2 \mid X = x]. \end{align*} \] But \[ \mathbb{E}[m - \Theta \mid X = x] = m - \mathbb{E}[\Theta \mid X = x] = m - m = 0. \] So the middle term disappears: \[ \mathbb{E}[(a - \Theta)^2 \mid X = x] = (a - m)^2 + \mathbb{E}[(m - \Theta)^2 \mid X = x]. \] The second term does not depend on \(a\). Therefore the only thing you can minimize is \((a - m)^2\), which is minimized at \(a = m\). Hence \[ \boxed{a = \mathbb{E}[\Theta \mid X].} \] So the posterior mean minimizes posterior expected squared error for every observed dataset, and therefore also minimizes the overall Bayes risk.

In this problem, \[ \boxed{\hat\theta_{\text{mean}} = (1-\lambda)\hat\theta_{\text{MLE}} + \lambda\mu_\theta} \] is therefore better than the MLE in terms of Bayes risk. The intuition is that the posterior mean accepts some conditional bias toward the prior mean, but that shrinkage reduces enough sampling variance that the overall expected squared error becomes smaller.