Suppose \(X\) is a random variable with \(E[X^2] < \infty\). What is the constant \(c\) that minimizes \(E[(X - c)^2]\)?
As with most optimization problems, we will begin by taking the derivative of \(E[(X - c)^2]\) with respect to \(c\). To do so, we replace the expectation with its definition, letting \(f(x)\) represent the probability density function of \(X\).
\[\begin{split} \dfrac{d}{dc}E\left[(X - c)^2\right] &= \dfrac{d}{dc} \int_{-\infty}^{\infty} (x-c)^2f(x)dx \\ &= -2\int_{-\infty}^{\infty}(x-c)f(x)dx \\ &= -2\left[ \int_{-\infty}^{\infty}xf(x)dx - \int_{-\infty}^{\infty}cf(x)dx \right] \\ &= -2\left( E[X] - c \right) \\ \end{split}\]
Next, we set the derivative equal to \(0\) and solve for \(c\):
\[-2\left( E[X] - c \right) = 0\] \[c = E[X]\]
Thus, \(E[(X - c)^2]\) is minimized by \(c = E[X]\). We confirm this by noting \(\dfrac{d^2}{dc}E[(X - c)^2] = 2\), indicating upward concavity.
There is another, cleaner approach to finding the solution. We begin by utilizing the linearity of expectation property:
\[E[(X - c)^2] = E[X^2 - 2cX + c^2] = E[X^2] - 2cE[X] +c^2\]
Then, we take the derivative of this new formulation and set the result to \(0\):
\[\dfrac{d}{dc} \left[ E[X^2] - 2cE[X] +c^2 \right] = 0\] \[-2E[X] + 2c = 0\] \[c = E[X]\]