4 Data 102 Discussion 04: Conjugate Priors & Bayes Risk
4.0.1 Contact Information
| Name | Wesley Zheng |
| Pronouns | He/him/his |
| wzheng0302@berkeley.edu | |
| Discussion | Wednesdays, 11–12 PM @ Wheeler 224 |
| Office Hours | Tuesdays, 9–10 AM & 11-12 PM @ Gateway B1040 |
Contact me by email at ease — I typically respond within a day or so!
4.1 Conjugate Priors
A prior is conjugate for a likelihood if the posterior belongs to the same family as the prior (parameters may change).
Bayes’ rule for the posterior: \[ \begin{align*} p(\theta \mid x) &= \frac{p(x \mid \theta)\, p(\theta)}{p(x)} \\ &\propto p(x \mid \theta)\, p(\theta). \end{align*} \]
- \(p(\theta)\) = prior
- \(p(x \mid \theta)\) = likelihood
- \(p(\theta \mid x)\) = posterior
- \(p(x)\) is a normalizer that does not depend on \(\theta\), so it does not change the shape of the posterior — working with \(\propto\) is enough to recognize the family.
Classic pairs
- Beta prior + Binomial likelihood \(\Rightarrow\) Beta posterior
- Gamma prior + Exponential likelihood \(\Rightarrow\) Gamma posterior
In this question, we will investigate examples of conjugate priors: pairs of distributions (for the likelihood and the prior) such that the prior and posterior are from the same distribution, with possibly different parameters.
Recall that for observed data \(X\), and prior distribution \(p(\theta)\) on parameters \(\theta\), the posterior probability distribution on \(\theta\), after seeing the data, is given by1 \[ \begin{align*} p(\theta \mid x) &= \frac{p(x \mid \theta) \cdot p(\theta)}{p(x)} \\ &\propto p (x \mid \theta) \cdot p (\theta) \end{align*} \] where \(\propto\) denotes “proportional to.” Note here that \(p(x)\) is a normalization constant which allows the posterior distribution to sum to \(1\). However, it bears no influence on the shape of the posterior distribution because it doesn’t contain \(\theta\). Therefore, we can always work this proportionality to try to identify a posterior distribution.
4.1.1 (a)
Beta and Binomial
Say you’ve observed a sequence of coin flips, \(X_1, ..., X_n\), all using the same coin, which has some probability of landing heads, \(p_h\). Denote by \(H\) the total number of heads: \[ \begin{align*} H = \sum_{i=1}^n \mathbb{I}\{X_i = \textrm{heads} \} \end{align*} \] \(H\) follows a binomial distribution, with PDF \[ \begin{align*} p(H = k) = \binom{n}{k} p_h^k (1-p_h)^{n-k} \end{align*} \]
We didn’t make this coin, it was given to us. We’re willing to place a prior distribution on the probability of it landing heads and we’ll use the beta distribution to do so. The beta distribution is a suitable choice since it takes on values from [0,1], which can be used to model probabilities. The beta distribution PDF is parameterized by shape parameters \(\alpha > 0\) and \(\beta > 0\), and is given by \[ \begin{align*} f(z; \alpha, \beta) = \frac{(\alpha+ \beta-1)!}{(\alpha-1)! (\beta-1)!} z^{\alpha-1} (1-z)^{\beta-1}, \quad 0 < z < 1 \end{align*} \]
Show that the Beta distribution is a conjugate prior for the Binomial distribution. What are the parameters of the new Beta distribution?
Answer
\[ \begin{align*} f(p_h \mid H=k) &\propto (p_h)^{k + \alpha -1}(1-p_h)^{n-k+\beta -1} \end{align*} \] From this we conclude that the posterior distribution has a Beta(\(k+\alpha\), \(n-k+\beta\)) distribution.
\[ \begin{align*} f(k \mid p_h) \cdot f(p_h; \alpha, \beta) &= \binom{n}{k} p_h^k (1 - p_h)^{n-k} \cdot \frac{(\alpha+ \beta-1)!}{(\alpha-1)! (\beta-1)!} p_h^{\alpha-1} (1-p_h)^{\beta-1} \\ &= \frac{n!}{(n-k)! k!} \cdot \frac{(\alpha+ \beta-1)!}{(\alpha-1)! (\beta-1)!} (p_h)^{k + \alpha -1}(1-p_h)^{n-k+\beta -1} \end{align*} \] The second line follows from rearranging terms. In terms of \(p_h\), the first two fractions are constant, so we can write \[ \begin{align*} f(p_h \mid H=k) &\propto f(k \mid p_h) \cdot f(p_h; \alpha, \beta) \\ & \propto (p_h)^{k + \alpha -1}(1-p_h)^{n-k+\beta -1} \end{align*} \] From this we conclude that the posterior distribution has a Beta(\(k+\alpha\), \(n-k+\beta\)) distribution.
4.1.2 (b)
Gamma and Exponential
A Gamma distribution with parameters \(\alpha, \beta\) has density function \(p(x) = \frac{\beta^\alpha}{\Gamma(\alpha)} x^{\alpha-1} e^{-\beta x}\) where \(\Gamma(\alpha)\) is the Gamma function (see Wikipedia: Gamma distribution). Show that Gamma distribution is a conjugate prior for Exponential distribution for multiple measurements, i.e. if we have samples \(X_1, X_2, \cdots, X_n\) that are mutually independent given \(\lambda\), and each \(X_i \mid \lambda \sim \operatorname{Exp}(\lambda)\) and \(\lambda \sim \operatorname{Gamma}(\alpha, \beta)\), then \(\lambda \mid X_1, X_2,\cdots, X_n \sim \operatorname{Gamma}(\alpha^*, \beta^*)\) for some values \(\alpha^*, \beta^*\).
Answer
\[ \begin{align*} p(\lambda \mid X_1, X_2, \cdots, X_n) \propto \lambda^{n+\alpha - 1} e^{-(\sum_i X_i + \beta )\lambda} \end{align*} \] So \(\lambda \mid X_1, X_2,\cdots, X_n \sim \operatorname{Gamma}(\alpha^*, \beta^*)\) with \(\alpha^* = n+\alpha\), \(\beta^* =\sum_i X_i + \beta\).
We use the rate parameterization of the Exponential and Gamma:
- \(X_i \mid \lambda \sim \operatorname{Exp}(\lambda)\) means \(p(x_i \mid \lambda) = \lambda e^{-\lambda x_i}\) for \(x_i > 0\).
- \(\lambda \sim \operatorname{Gamma}(\alpha, \beta)\) means \[ p(\lambda) = \frac{\beta^\alpha}{\Gamma(\alpha)}\, \lambda^{\alpha-1} e^{-\beta \lambda}, \qquad \lambda > 0. \] Here \(\Gamma(\alpha)\) is the Gamma function (for positive integers, \(\Gamma(\alpha)=(\alpha-1)!\)). Importantly, \(\beta^\alpha / \Gamma(\alpha)\) does not depend on \(\lambda\), so when we work with \(\propto\) (as a function of \(\lambda\)), we can drop that constant and only keep \(\lambda^{\alpha-1} e^{-\beta\lambda}\).
Since the \(X_i\) are mutually independent given \(\lambda\), \[ \begin{align*} p(X_1,\ldots,X_n \mid \lambda) &= \prod_{i=1}^n p(X_i \mid \lambda) = \prod_{i=1}^n \lambda e^{-\lambda X_i} = \lambda^n \exp\!\left(-\lambda \sum_{i=1}^n X_i\right). \end{align*} \] Multiplying \(n\) copies of \(e^{-\lambda X_i}\) turns the product into a single exponential whose rate is the sum of the \(X_i\)’s — that is the key place where independence becomes additivity.
Then the posterior satisfies \[ \begin{align*} p(\lambda \mid X_1, X_2, \cdots, X_n) &\propto p(X_1, X_2,\cdots, X_n \mid \lambda )\, p(\lambda) \\ &\propto \underbrace{\lambda^n e^{-\lambda \sum_i X_i}}_{\text{likelihood}}\, \underbrace{\lambda^{\alpha-1} e^{-\beta\lambda}}_{\text{prior, constants dropped}} \\ &\propto \lambda^{n+\alpha - 1} e^{-(\sum_i X_i + \beta )\lambda}. \end{align*} \] Powers of \(\lambda\) multiply by adding exponents: \(n + (\alpha-1) = n+\alpha-1\). Exponentials in \(\lambda\) multiply by adding rates: \(\sum_i X_i + \beta\).
A \(\operatorname{Gamma}(\alpha^*,\beta^*)\) density looks like \[ p(\lambda) \propto \lambda^{\alpha^*-1} e^{-\beta^* \lambda}. \] Matching exponents gives \(\alpha^* - 1 = n+\alpha-1\) and \(\beta^* = \sum_i X_i + \beta\), so \[ \alpha^* = n+\alpha, \qquad \beta^* =\sum_i X_i + \beta. \] Thus \(\lambda \mid X_1, X_2,\cdots, X_n \sim \operatorname{Gamma}(\alpha^*, \beta^*)\) — same family as the prior, which is conjugacy.
4.2 Decision Theory: Computing and Minimizing the Bayes Risk
The Bayes risk of a decision rule \(\delta\) under loss \(\ell\) is \[ R^{*}(\delta) = \mathbb{E}_{\theta, X}[\ell(\theta, \delta(X))] = \mathbb{E}_{X} \big[\mathbb{E}_{\theta\mid X}[\ell(\theta, \delta(X)) \mid X]\big]. \]
We want \[ \delta^* = \operatorname*{argmin}_\delta R^{*}(\delta). \]
Pointwise minimization: for each fixed \(X=x\), choose the scalar \[ a^* = \operatorname*{argmin}_{a \in \mathbb{R}} \mathbb{E}_{\theta\mid X}[\ell(\theta, a) \mid X = x], \] then set \(\delta^*(x) = a^*\). Doing this for every \(x\) minimizes the Bayes risk.
Common losses (and the optimal \(\delta^*\))
- Squared-error \(\ell(\theta,a)=\tfrac12(\theta-a)^2\) \(\Rightarrow\) posterior mean \(\delta^*(X)=\mathbb{E}[\theta\mid X]\)
- Zero-one \(\ell(\theta,a)=\mathbf{1}[\theta\neq a]\) \(\Rightarrow\) posterior mode \(\delta^*(X)=\operatorname*{argmax}_\theta \mathbb{P}(\theta\mid X)\)
For the following two parts, derive the decision procedure \(\delta^*\) that minimizes the Bayes risk (not the same as the Bayesian posterior risk), for the given loss function. That is, provide an expression for \[ \delta^* = \operatorname*{argmin}_\delta R^{*}(\delta) \] where the Bayes risk \(R^{*}(\delta)\) can be written out as \[ R^{*}(\delta) = \mathbb{E}_{\theta, X}[\ell(\theta, \delta(X))] = \mathbb{E}_{X} [\mathbb{E}_{\theta\mid X}[\ell(\theta, \delta(X)) \mid X]]. \]
Hint: One strategy to find the decision rule that minimizes the Bayes risk is based on the following rationale. For any given value of the data, \(X = x\), the quantity \(\delta(x)\) is simply a scalar value. Suppose, for any given value of \(X = x\), we can find the scalar value \(\delta^*(x) = a^* \in \mathbb{R}\) such that \[ \begin{align*} a^* & = \operatorname*{argmin}_{a \in \mathbb{R}} \mathbb{E}_{\theta\mid X}[\ell(\theta, a) \mid X = x] \end{align*} \] (that is, \(a^*\) is the scalar value that minimizes the Bayes posterior risk for this particular value of \(X = x\)). Then, the rule given by this computation of \(\delta^*(x)\) (for each value of \(X = x\)) must also be the one that minimizes the Bayes risk, which just takes an expectation over all possible values of \(X\). This is sometimes referred to as a pointwise minimization strategy.
4.2.1 (a)
\(\ell(\theta, \delta(X)) = (1/2)(\theta - \delta(X))^2\) (squared-error loss)
Answer
Following the pointwise minimization strategy, for any particular value of \(X = x\) we find the value \(a^* = \delta(x)\) that solves \[ \begin{align*} a^* & = \operatorname*{argmin}_{a \in \mathbb{R}} \mathbb{E}_{\theta }[(1/2)(\theta - a)^2 \mid X = x]. \end{align*} \] To do this, we take the derivative with respect to \(a\) and set it to zero, since \(f(a) = \mathbb{E}_{\theta }[(1/2)(\theta - a)^2 \mid X = x]\) is a convex function in \(a\) (try to prove this as a quick exercise; see below for solution). Swapping the differentiation and expectation operators and applying the chain rule gives \[ f'(a) = \mathbb{E}_{\theta} [a - \theta \mid X = x] \] and setting the derivative to zero gives \[ f'(a) = 0 \implies a^* = \mathbb{E}[\theta \mid X = x]. \] That is, for any particular value of \(X = x\), we should take \(\delta^*(x) = \mathbb{E}[\theta \mid X = x]\). That means that the decision rule that minimizes the Bayes risk for the squared error loss is \(\delta^*(X) = \mathbb{E}[\theta \mid X]\), the posterior expectation (the expectation of the posterior distribution)!
To show that \(f(a)\) is a convex function, for any \(a_1, a_2 \in \mathbb{R}\) and \(t \in [0, 1]\), we have that \[ \begin{align*} f(ta_1 + (1 - t)a_2) & = \mathbb{E}_\theta[(1/2)(\theta - [ta_1 + (1 - t)a_2])^2 \mid X = x] \\ & \leq \mathbb{E}_\theta[(1/2)t(\theta - a_1)^2 + (1/2)(1 - t)(\theta - a_2)^2 \mid X = x] \\ & = t\mathbb{E}_{\theta }[(1/2)(\theta - a_1)^2 \mid X = x] + (1 - t)\mathbb{E}_{\theta }[(1/2)(\theta - a_2)^2 \mid X = x] \\ & = tf(a_1) + (1 - t)f(a_2) \end{align*} \] where the second line due to the convexity of the function \(g(a) = (\theta - a)^2\) and monotonicity of expectations, and the third line is due to linearity of expectations.
The derivative of an expectation is: \[ \begin{align*} \frac{d}{da} \mathbb{E}_\theta \left[ \ell(\theta, a) \mid X = x \right] &= \frac{d}{da} \int \ell(\theta, a)\, p(\theta \mid X = x)\, d\theta \\ &= \int \frac{d}{da} \ell(\theta, a)\, p(\theta \mid X = x)\, d\theta \\ &= \mathbb{E}_\theta \left[ \ell'(\theta, a) \mid X = x \right]. \end{align*} \] For squared-error loss, \[ \begin{align*} \mathbb{E}_{\theta} \left[ (1/2)(2)(-1)(\theta - a) \mid X = x \right] &= \mathbb{E}_{\theta} \left[ a - \theta \mid X = x \right]. \end{align*} \] Setting this to zero and solving for \(a\) gives \[ a = a^* = \mathbb{E}[\theta \mid X = x]. \]
Why does \(p(\theta \mid X = x)\) “disappear” when differentiating under the integral? It has no dependence on \(a\); it is constant with respect to \(\frac{d}{da}\).
4.2.2 (b)
\(\ell(\theta, \delta(X)) = \mathbf{1}[\theta \neq \delta(X)]\) (zero-one loss)
Answer
We use the same strategy as Part (a). For a given value \(X = x\), we assign \(\delta^*(x)\) to be the value \[ \begin{align*} \operatorname*{argmin}_a \mathbb{E}_{\theta \sim \mathbb{P}(\theta \mid X = x)}[\mathbf{1}[\theta \neq a]] & = \operatorname*{argmin}_a \mathbb{P}(\theta \neq a \mid X = x) \\ & = \operatorname*{argmin}_a( 1 - \mathbb{P}(\theta = a \mid X = x)) \\ & = \operatorname*{argmax}_a \mathbb{P}(\theta = a \mid X = x) \\ & = \operatorname*{argmax}_\theta \mathbb{P}(\theta \mid X = x). \end{align*} \] That is, the decision rule that minimizes the Bayes risk for the zero-one loss is \(\delta^*(X) = \operatorname*{argmax}_\theta \mathbb{P}(\theta \mid X)\) the posterior mode (the mode of the posterior distribution).
For any event \(A\), the indicator \(\mathbf{1}[A]\) is \(1\) when \(A\) happens and \(0\) otherwise, so its expectation is exactly the probability of \(A\): \[ \mathbb{E}[\mathbf{1}[A]] = 1\cdot\mathbb{P}(A) + 0\cdot\mathbb{P}(A^c) = \mathbb{P}(A). \] Conditioning on \(X=x\) is the same idea with the posterior: \[ \mathbb{E}_{\theta \mid X=x}\big[\mathbf{1}[\theta \neq a]\big] = \mathbb{P}(\theta \neq a \mid X = x). \] So zero-one posterior risk is just the posterior probability of being wrong if we guess \(a\).
From there, \[ \mathbb{P}(\theta \neq a \mid X = x) = 1 - \mathbb{P}(\theta = a \mid X = x). \] Minimizing \(1 - p(a)\) over \(a\) is the same as maximizing \(p(a)\) over \(a\): if \(g(a)\) is larger whenever \(f(a)\) is smaller for \(f(a)=1-g(a)\), then \[ \operatorname*{argmin}_a f(a) = \operatorname*{argmax}_a g(a). \] Here \(f(a)=\mathbb{P}(\theta\neq a\mid X=x)\) and \(g(a)=\mathbb{P}(\theta=a\mid X=x)\), so \[ \operatorname*{argmin}_a \mathbb{P}(\theta \neq a \mid X = x) = \operatorname*{argmax}_a \mathbb{P}(\theta = a \mid X = x) = \operatorname*{argmax}_\theta \mathbb{P}(\theta \mid X = x), \] i.e. pick the posterior mode.
The prior distribution on the parameters is given by \(p(\theta)\) and the likelihood \(p(x\mid\theta)\).↩︎