Exercises — Lesson 15: Discrete Random Variables and Expectation
Exercise 15.1. [Hand] Let \(X\) be the number of heads in 3 fair flips. Tabulate \(p_X\), compute \(\mathbf{E}[X]\) and \(\operatorname{var}(X)\) from the table, and confirm both against the Binomial\((3, \tfrac12)\) formulas.
Solution. Let \(X\) = number of heads in 3 fair flips. With \(0\) = “Tails” and \(1\) = “Heads”,
\[ \Omega = \{000,\, 001,\, 010,\, 011,\, 100,\, 101,\, 110,\, 111\}, \qquad |\Omega| = 8 \]
\[ \binom30 = 1, \qquad \binom33 = 1, \qquad \binom31 = \frac{3!}{1!\,2!} = 3, \qquad \binom32 = \frac{3!}{2!\,1!} = 3 \]
\[ p_X(x) = \begin{cases} 1/8, & x = 0 \text{ heads} \\[2pt] 3/8, & x = 1 \text{ head} \\[2pt] 3/8, & x = 2 \text{ heads} \\[2pt] 1/8, & x = 3 \text{ heads} \end{cases} \]
From the table:
\[ \begin{aligned} \mathbf{E}[X] &= 0\left(\tfrac18\right) + 1\left(\tfrac38\right) + 2\left(\tfrac38\right) + 3\left(\tfrac18\right) \\[6pt] &= \frac38 + \frac68 + \frac38 = \frac{12}{8} = \boxed{\;\frac32\;} \end{aligned} \]
\[ \begin{aligned} \operatorname{var}(X) &= \left(-\tfrac32\right)^2\left(\tfrac18\right) + \left(-\tfrac12\right)^2\left(\tfrac38\right) + \left(\tfrac12\right)^2\left(\tfrac38\right) + \left(\tfrac32\right)^2\left(\tfrac18\right) \\[6pt] &= \left(\tfrac94\right)\left(\tfrac18\right) + \left(\tfrac14\right)\left(\tfrac38\right) + \left(\tfrac14\right)\left(\tfrac38\right) + \left(\tfrac94\right)\left(\tfrac18\right) \\[6pt] &= \frac{9}{32} + \frac{3}{32} + \frac{3}{32} + \frac{9}{32} = \frac{24}{32} = \boxed{\;\frac34\;} \end{aligned} \]
Confirming against the Binomial formulas, \(X \sim \text{Binomial}(3, \tfrac12)\):
\[ \mathbf{E}[X] = np = 3\left(\tfrac12\right) = \frac32, \qquad \operatorname{var}(X) = np(1-p) = 3\left(\tfrac12\right)\left(\tfrac12\right) = \frac34 \]
Both agree with the table.
Exercise 15.2. [Hand] A quiz has 5 multiple-choice questions, 4 options each; you guess uniformly and independently. Let \(X\) = number correct. What are \(\mathbf{E}[X]\) and \(\operatorname{var}(X)\)? What is \(\mathbf{P}(X \ge 1)\)? (Complement trick.)
Solution. Let \(X\) = number correct. For \(i = 1, \dots, 5\) let
\[ X_i = \begin{cases} 1, & \text{with probability } 1/4 \\[2pt] 0, & \text{with probability } 3/4 \end{cases} \qquad X_i \sim \text{Bernoulli}\left(\tfrac14\right) \]
Then \(X = \sum_{i=1}^5 X_i\), and by independence \(X \sim \text{Binomial}(5, \tfrac14)\).
\[ \mathbf{E}[X] = \mathbf{E}\left[\sum_{i=1}^5 X_i\right] = \sum_{i=1}^5 \mathbf{E}[X_i] = \sum_{i=1}^5 \frac14 = \boxed{\;\frac54\;} \]
\[ \operatorname{var}(X) = np(1-p) = 5\left(\tfrac14\right)\left(\tfrac34\right) = \boxed{\;\frac{15}{16}\;} \]
\[ \begin{aligned} \mathbf{P}(X \ge 1) &= 1 - \mathbf{P}(X < 1) = 1 - \mathbf{P}(X = 0) = 1 - \left(\tfrac34\right)^5 \\[6pt] &= 1 - \frac{243}{1024} = \boxed{\;\frac{781}{1024} \approx 0.763\;} \end{aligned} \]
Exercise 15.3. [Proof, \(\star\)] Prove that \(\mathbf{E}[X]\) minimizes the mean squared error: for any constant \(c\), \(\mathbf{E}[(X-c)^2] = \operatorname{var}(X) + (c - \mathbf{E}[X])^2\), so the unique minimizing \(c\) is \(\mathbf{E}[X]\). (Expand; Proposition 15.6 does the rest — every term is a function of the single variable \(X\). Lesson 19 upgrades this from constants to functions of an observation — it becomes MMSE estimation.)
Proof. Solve \(\displaystyle \operatorname*{arg\,min}_{c} \mathbf{E}[(X-c)^2]\).
\[ \begin{aligned} \mathbf{E}[(X-c)^2] &= \mathbf{E}[X^2 - 2Xc + c^2] && \text{[expand]} \\[6pt] &= \mathbf{E}[X^2] - 2c\,\mathbf{E}[X] + c^2 && \text{[Proposition 15.6, linearity]} \\[6pt] &= \left(\mathbf{E}[X^2] - \mathbf{E}[X]^2\right) + \left(\mathbf{E}[X]^2 - 2c\,\mathbf{E}[X] + c^2\right) \\[6pt] &= \operatorname{var}(X) + (c - \mathbf{E}[X])^2 \end{aligned} \]
This is a convex function of \(c\), so it has a unique minimizer. Differentiating in \(c\) and setting equal to zero,
\[ \frac{d}{dc}\,\mathbf{E}[(X-c)^2] = 2c - 2\,\mathbf{E}[X] = 0 \;\implies\; 2\,\mathbf{E}[X] = 2c \;\implies\; c = \mathbf{E}[X] \]
\[ \frac{d^2}{dc^2}\,\mathbf{E}[(X-c)^2] = 2 > 0 \;\implies\; c = \mathbf{E}[X] \text{ is the unique global minimum.} \qquad \blacksquare \]
Exercise 15.4. [Proof] Derive \(\mathbf{E}[X] = 1/p\) for the geometric distribution using the memoryless recursion: condition on the first trial to get \(\mathbf{E}[X] = 1 + (1-p)\,\mathbf{E}[X]\), and justify the conditioning step. (This is the total expectation theorem of Lesson 16 in action; compare with the derivative trick of Example 15.9.)
Proof. Let \(X \sim \text{Geometric}(p)\). First, justify that \(\mathbf{E}[X] = 1 + (1-p)\,\mathbf{E}[X]\) comes from conditioning on the first trial.
If the first trial is a success, which happens with probability \(p\), then \(X = 1\).
If the first trial is a failure, which has probability \(1-p\), then you have already used one trial, but since each trial is independent (the memoryless property), the number of additional trials has the exact same probability distribution, so
\[ X = 1 + X_{\text{rest}}, \qquad X_{\text{rest}} \sim X \sim \text{Geometric}(p) \]
\[ \begin{aligned} \implies \mathbf{E}[X] &= p\,(1) + (1-p)\,\mathbf{E}[1 + X_{\text{rest}}] \\[6pt] &= p + (1-p)\left[\mathbf{E}[1] + \mathbf{E}[X_{\text{rest}}]\right] \\[6pt] &= p + (1-p)\left[1 + \mathbf{E}[X]\right] \\[6pt] &= p + (1-p) + (1-p)\,\mathbf{E}[X] \\[6pt] &= 1 + (1-p)\,\mathbf{E}[X] \end{aligned} \]
\[ \begin{aligned} &\implies \mathbf{E}[X]\left[1 - (1-p)\right] = 1 \\[6pt] &\implies \mathbf{E}[X]\,p = 1 \\[6pt] &\implies \mathbf{E}[X] = \frac1p \qquad \blacksquare \end{aligned} \]