CS代考 MATH3821: Page 258 / 758 – cscodehelp代写
Slide 4.1 / 4.57
4 Classical estimation theory
4.1 Cramer-Rao inequality
4.2 Comments on applying the CR Inequality in the search of the UMVUE
Copyright By cscodehelp代写 加微信 cscodehelp
4.3 CRLB attainability examples
4.4 Which are the estimators that could attain the bound?
4.5 Rao-Blackwell theorem
4.6 Uniqueness of UMVUE
4.7 Completeness of a family of distributions
4.8 Theorem of Lehmann-Scheffe
4.9 Examples finding UMVUE using Lehmann-Scheffe theorem
Chapter 4 : Classical estimation theory MATH3821: Page 258 / 758
4.1 Cramer-Rao inequality Slide 4.2 / 4.57
4.1 Cramer-Rao inequality
Obtaining a point estimator of the parameter of interest is only the first step in inference.
Suppose X = (X1,X2,..,Xn) are i.i.d. from f(x,θ),θ ∈ R and we use a statistic Tn(X) to estimate θ.
If Eθ(Tn) = θ + bn(θ), then the quantity bn(θ) = Eθ(Tn) − θ
is called bias. Here and further, the notation Eθ(.) is used to denote the expected value when θ is the parameter of the distribution in the sample.
Chapter 4 : Classical estimation theory MATH3821: Page 259 / 758
4.1 Cramer-Rao inequality Slide 4.3 / 4.57
The bias generally depends on both θ and the sample size n although this dependence on the sample size may sometimes be suppressed in the notation.
We call an estimator unbiased if bn(θ) ≡ 0 for all θ ∈ Θ.
Interpretation of unbiasedness: when used repeatedly, an unbiased estimator, in the long run, will estimate the true value on average. This is why, in general, it is good to have estimators with zero bias for any θ ∈ Θ and any sample size n.
Remark 4.8
Caution: for some families an unbiased estimators may not exist or, even when they exist, may not be very useful.
Chapter 4 : Classical estimation theory MATH3821: Page 260 / 758
4.1 Cramer-Rao inequality
Slide 4.4 / 4.57
Example 4.29 (at lecture)
Geometric distribution
f(x,θ) = θ(1−θ)x−1,x = 1,2,… an unbiased estimator of θ, say, T(x) must satisfy
T(x)θ(1 − θ)x−1 = θ x=1
for all θ ∈ [0,1]. By a polynomial expansion, the estimator would satisfy
T(1) = 1, T(x) = 0 if x ≥ 2.
Having in mind the interpretation of θ, such an estimator is neither
reliable, nor useful.
Chapter 4 : Classical estimation theory MATH3821: Page 261 / 758
4.1 Cramer-Rao inequality Slide 4.5 / 4.57
Unbiasedness cannot be the only requirement for an estimator to be good. Note also that there may be many unbiased estimators for the same parameter and then we will be confronted with the problem to
“select the best”amongst them.
A quantity to look at when comparing quality of estimators, is the
mean squared error:
MSE(T)=E(T −θ)2. θnθn
Chapter 4 : Classical estimation theory
MATH3821: Page 262 / 758
4.1 Cramer-Rao inequality
Slide 4.6 / 4.57
The following property holds:
MS Eθ(Tn) = Varθ(Tn) + (bn(θ))2
=E(T −E(T)+E(T)−θ)2
= E (T −E (T ))2+E (E (T )−θ)2
θnθn θθn −2E (T −E (T ))(θ−E (T ))
θnθnθn = Var(Tn) + (bn(θ))2
Chapter 4 : Classical estimation theory MATH3821: Page 263 / 758
4.1 Cramer-Rao inequality Slide 4.7 / 4.57
Keeping the MSE as small as possible is more important than just to ask for unbiasedness. We would like to find an estimator that minimises the MSE.
Unfortunately, in the class of all estimators, an estimator that mini- mizes the MSE simultaneously for all θ values does not exist!
Chapter 4 : Classical estimation theory MATH3821: Page 264 / 758
4.1 Cramer-Rao inequality Slide 4.8 / 4.57
To see why, lets take any estimator θ ̃. Since the parameter θ ∈ Θ is unknown, there will be a certain value θ0 ∈ Θ for which MS Eθ0 (θ ̃) > 0.
Then we can consider as a competitor to θ ̃ the estimator θ∗ ≡ θ0. Note that θ∗ is not a very reasonable estimator (it does not even use the data!) but for the particular point θ0 we have MS Eθ0 (θ∗) = 0 and hence,
MS Eθ0 (θ ̃) > MS Eθ0 (θ∗).
Chapter 4 : Classical estimation theory
MATH3821: Page 265 / 758
4.1 Cramer-Rao inequality Slide 4.9 / 4.57
In other words, when considering the class of all estimators, there are so many estimators available to us that to find one that is uniformly better with respect to the MSE criterion, is just impossible. There are two ways out of this unpleasant situation:
We can restrict the class of estimators considered.
We could change the evaluation criterion when trying to define the best.
We deal with the first way out now (the other way was discussed in the Decision theory chapter: Bayes and minimax estimation).
Chapter 4 : Classical estimation theory MATH3821: Page 266 / 758
4.1 Cramer-Rao inequality Slide 4.10 / 4.57
The most common way to restrict the class of estimators is to impose the criterion of unbiasedness. This means to consider only the unbiased estimators when searching for the best one with the smallest MSE.
This simplifies, to a great extent, the task of minimising the MSE because then we only have to minimise the variance! In the smaller subset of unbiased estimators, one can very often find an estimator with the smallest MSE (variance) for all θ values.
It is called the uniformly minimum variance unbiased estimator (UMVUE). Let us first look at a well-known result that will help us in our search
of the UMVUE (the Cramer-Rao theorem).
Chapter 4 : Classical estimation theory MATH3821: Page 267 / 758
4.1 Cramer-Rao inequality Slide 4.11 / 4.57
Theorem 4.13
Let X = (X1, X2, . . . , Xn) have a distribution that depends on θ and
L(X, θ) be the joint density. Let τ(θ) be a smooth (i.e. differentiable)
function of θ that has to be estimated. Consider any unbiased estima-
tor W(X) of τ(θ), i.e.E W(X) = τ(θ). Suppose, in addition, that θ
L(X, θ) satisfies:
∂θ .. h(X)L(X,θ)dX1..dXn = .. h(X)∂θL(X,θ)dX1..dXn (∗) for any function h(X) with Eθ|h(X)| < ∞. Then:
for all θ holds.
Varθ(W(X)) ≥ ∂θ IX(θ)
( ∂ τ(θ))2
Chapter 4 : Classical estimation theory
MATH3821: Page 268 / 758
4.1 Cramer-Rao inequality Slide 4.12 / 4.57
The proof uses a clever application of the Cauchy-Schwartz inequality.
If Z and Y are two random variables with finite variances Var(Z) and Var(Y) then
[Cov(Z,Y)]2 = {E[(Z−E(Z))(Y−E(Y))]}2 ≤ Var(Z)Var(Y).
Chapter 4 : Classical estimation theory MATH3821: Page 269 / 758
4.1 Cramer-Rao inequality Slide 4.13 / 4.57
To prove the Cramer- , we choose W to be the Z-variable,
and the score V to be the Y-variable in the Cauchy- .
Since E V (X, θ) = 0 holds for the score, we have that θ
[Covθ(W, V)]2 = [Eθ(WV) − Eθ(W)Eθ(V)]2 = [Eθ(WV)]2
≤ Varθ(W)Varθ(V) Substituting the definition of the score, we get:
Covθ(W, V) = Eθ(WV) =
where dX = dX1dX2 . . . dXn is used as shorthand notation.
∂ L(X,θ) W(X) ∂θ
Chapter 4 : Classical estimation theory MATH3821: Page 270 / 758
4.1 Cramer-Rao inequality
Slide 4.14 / 4.57
Now if we utilise condition (*), we can continue to get:
Covθ(W, V) = ∂θ ... W(X)L(X, θ)dX
= ∂Eθ(W) ∂θ
= ∂ τ(θ) ∂θ
Then, Inequality (6) implies that:
∂ τ(θ)2 ≤ Varθ(W)IX(θ)
Varθ(W)≥ ∂θ . IX(θ)
( ∂ τ(θ))2
Chapter 4 : Classical estimation theory
MATH3821: Page 271 / 758
4.1 Cramer-Rao inequality Slide 4.15 / 4.57
We will skip the discussion on the multivariate version of this inequality which is applicable for the case of estimating a multidimensional parameter.
Remark 4.9
The Cramer-Rao (CR) Inequality was stated for continuous random variables. By an obvious modification of condition (*) requiring the ability to interchange differentiation and summation (instead of differ- entiation and integration) one can formulate this for discrete random variables, too. In this case, even though L(X, θ) may not be differ- entiable with respect to x, it has to be assumed to be differentiable with respect to θ.
Chapter 4 : Classical estimation theory MATH3821: Page 272 / 758
4.1 Cramer-Rao inequality
Slide 4.16 / 4.57
4.1.1 Corollary for i.i.d. case.
If X = (X1,X2,...,Xn) are i.i.d. with f(x,θ) then n
and the CR Inequality becomes:
Varθ(W(X)) ≥
f (Xi, θ) and IX(θ) = nIX1 (θ)
( ∂ τ(θ))2 ∂θ .
Chapter 4 : Classical estimation theory
MATH3821: Page 273 / 758
4.2 Comments on applying the CR Inequality in the search of the UMVUE Slide 4.17 / 4.57
4.2 Comments on applying the CR Inequality in the search of the UMVUE
a) Since the Inequality gives a lower bound on the variance of an unbiased estimator of the parameter τ(θ), it is obvious that, in the case where there exists an unbiased estimator of τ(θ) whose variance is equal to the lower bound given by CR Inequality, this will be the UMVUE of τ(θ).
Such a situation occurs often in the case of observations that arise from an exponential family.
Chapter 4 : Classical estimation theory MATH3821: Page 274 / 758
4.2 Comments on applying the CR Inequality in the search of the UMVUE Slide 4.18 / 4.57
Exercise 4.18 (at lecture)
Calculate the Cram ́er-Rao lower bound (CRLB) for the variance of an unbiased estimator of θ and find a statistic with variance equal to the bound when X1, X2, . . . , Xn are independent random variables each with a distribution from the exponential family of distributions.
i) Exponential (θ): f (x, θ) = 1θ e−x/θ, x > 0
ii) Bernoulli (θ):f(x,θ) = θx(1−θ)1−x, x ∈ {0,1}, θ ∈ (0,1)
1 −1(x−θ)2 iii)N(θ,1):f(x,θ)=√ e2 ,x∈R,θ∈R
iv) N(0,θ):f(x,θ)= √
e 2θ,x∈R,θ>0
Chapter 4 : Classical estimation theory
MATH3821: Page 275 / 758
4.2 Comments on applying the CR Inequality in the search of the UMVUE Slide 4.19 / 4.57
ii) Note the drawback related to the fact that condition (*) in the CR Theorem is a strong one and it often happens that it is not satisfied. A typical situation is when the range of the random variables Xi, i = 1, 2, .., n depends on θ, for example, in the case of a random sample from uniform [0, θ) observations.
According to the general Leibnitz′ rule for differentiation of parameter- dependent integrals:
∂θ f(x,θ)dx
= f(b(θ),θ)∂θb(θ)− f(a(θ),θ)∂θa(θ)+
holds and we see that, if a and b were genuine functions of θ, on the RHS there would be some additional non-zero terms included and condition (*) would not hold.
∂θ f(x,θ)dx
Chapter 4 : Classical estimation theory MATH3821: Page 276 / 758
4.2 Comments on applying the CR Inequality in the search of the UMVUE Slide 4.20 / 4.57
Example 4.30 (at lecture)
Assume that the right-hand limit θ of the interval [0,θ) is to be estimated and n i.i.d. observations from the uniform distribution [0, θ) are given.
Chapter 4 : Classical estimation theory MATH3821: Page 277 / 758
4.2 Comments on applying the CR Inequality in the search of the UMVUE
Slide 4.21 / 4.57
We can show that the density of Y = X(n) is given by n−1 n
ny /θ if0
