Sufficiency (statistics)
From Psychology Wiki
Community portal · Tasks to do · News · Help
Clinical · Educational · Ind&Org · Other fields · Professional · Transpersonal · World
Assessment | Biopsychology | Comparative | Cognitive | Developmental | Language | Personality | Philosophy | Research Methods | Social | Statistics
Statistics: Scientific method · Research methods · Experimental design · Undergraduate statistics courses · Statistical tests · Game theory · Decision theory
In statistics, sufficiency is the property possessed by a statistic, with respect to a parameter, "when no other statistic which can be calculated from the same sample provides any additional information as to the value of the parameter"[1]
This concept was due to Sir Ronald Fisher, and is equivalent to the most general statement of the above that, conditional on the value of a sufficient statistic, the distributions of samples drawn are independent of the underlying parameter(s) the statistic is sufficient for. Both the statistic and the underlying parameter can be vectors.
The concept has fallen out of favor in descriptive statistics because of the strong dependence on an assumption of the distributional form, but remains very important in theoretical work.[2]
Contents |
[edit] Mathematical definition
The concept is most general when defined as follows: a statistic T(X) is sufficient for underlying parameter θ precisely if the conditional probability distribution of the data X, given the statistic T(X), is independent of the parameter θ,[3] i.e.
or in shorthand
[edit] Example
As an example, the sample mean is sufficient for the mean (μ) of a normal distribution with known variance. Once the sample mean is known, no further information about μ can be obtained from the sample itself.
[edit] Fisher-Neyman factorization theorem
Fisher's factorization theorem or factorization criterion provides a convenient characterization of a sufficient statistic. If the probability density function is ƒθ(x), then T is sufficient for θ if and only if functions g and h can be found such that
i.e. the density ƒ can be factored into a product such that one factor, h, does not depend on θ and the other factor, which does depend on θ, depends on x only through T(x).
[edit] Interpretation
An implication of the theorem is that when using likelihood-based inference, two sets of data yielding the same value for the sufficient statistic T(X) will always yield the same inferences about θ. By the factorization criterion, the likelihood's dependence on θ is only in conjunction with T(X). As this is the same in both cases, the dependence on θ will be the same as well, leading to identical inferences.
[edit] Proof for the continuous case
Due to Hogg and Craig (ISBN 978-0023557224). Let X1, X2, ..., Xn, denote a random sample from a distribution having the pdf f(x,θ) for γ < θ < δ. Let Y = u(X1, X2, ..., Xn) be a statistic whose pdf is g(y;θ). Then Y = u(X1, X2, ..., Xn) is a sufficient statistic for θ if and only if, for some function H,
First, suppose that
We shall make the transformation yi = ui(x1, x2, ..., xn), for i = 1, ..., n, having inverse functions xi = wi(y1, y2, ..., yn), for i = 1, ..., n, and Jacobian J. Thus,
The left-hand member is the joint pdf g(y1, y2, ..., yn; θ) of Y1 = u1(X1, ..., Xn), ..., Yn = un(X1, ..., Xn). In the right-hand member, 
is the pdf of 
, so that 
is the quotient of and 
; that is, it is the conditional pdf 
of 
given 
.
But 
, and thus 
, was given not to depend upon 
. Since was not introduced in the transformation and accordingly not in the Jacobian 
, it follows that does not depend upon and that is a sufficient statistics for .
The converse is proven by taking:
where 
does not depend upon because 
depend only upon 
which are independent on 
when conditioned by , a sufficient statistics by hypothesis. Now divide both members by the absolute value of the non-vanishing Jacobian , and replace 
by the functions 
in 
. This yields
where 
is the Jacobian with 
replaced by their value in terms 
. The left-hand member is necessarily the joint pdf 
of 
. Since 
, and thus 
, does not depend upon , then
is a function that does not depend upon .
[edit] Proof for the discrete case
We use the shorthand notation to denote the joint probability of 
by 
. Since 
is a function of 
, we have 
and thus:
with the last equality being true by the definition of conditional probability distributions. Thus 
with 
and 
.
Reciprocally, if , we have
- WikiTeX: latex reported a failure, namely:
This is pdfeTeX, Version 3.141592-1.21a-2.2 (Web2C 7.5.4)
entering extended mode
(./5f50009ba96b278f0c0ae495272e0
LaTeX2e <2003/12/01>
Babel
! Package amsmath Error: Erroneous nesting of equation structures; (amsmath) trying to recover with `aligned'.
See the amsmath package documentation for explanation.
Type H
...
l.11 \end{align}
\end{equation*}
[1] (./5f50009ba96b278f0c0ae495272e0.aux) ) (see the transcript file for additional information) Output written on 5f50009ba96b278f0c0ae495272e0.dvi (1 page, 812 bytes).
Transcript written on 5f50009ba96b278f0c0ae495272e0.log.With the first equality by the definition of pdf for multiple variables, the second by the remark above, the third by hypothesis, and the fourth because the summation is not over 
.
Thus, the conditional probability distribution is:
- WikiTeX: latex reported a failure, namely:
This is pdfeTeX, Version 3.141592-1.21a-2.2 (Web2C 7.5.4)
entering extended mode
(./89ca7c36c6569ff69836a06db918a
LaTeX2e <2003/12/01>
Babel
! Package amsmath Error: Erroneous nesting of equation structures; (amsmath) trying to recover with `aligned'.
See the amsmath package documentation for explanation.
Type H
...
l.12 \end{align}
\end{equation*}
[1] (./89ca7c36c6569ff69836a06db918a.aux) ) (see the transcript file for additional information) Output written on 89ca7c36c6569ff69836a06db918a.dvi (1 page, 948 bytes).
Transcript written on 89ca7c36c6569ff69836a06db918a.log.With the first equality by definition of conditional probability density, the second by the remark above, the third by the equality proven above, and the fourth by simplification. This expression does not depend on and thus is a sufficient statistic.[4]
[edit] Minimal sufficiency
A sufficient statistic is minimal sufficient if it can be represented as a function of any other sufficient statistic. In other words, S(X) is minimal sufficient if and only if
- S(X) is sufficient, and
- if T(X) is sufficient, then there exists a function f such that S(X) = f(T(X)).
Intuitively, a minimal sufficient statistic most efficiently captures all possible information about the parameter θ.
A useful characterization of minimal sufficiency is that when the density fθ exists, S(X) is minimal sufficient if and only if
is independent of θ :
S(x) = S(y)
This follows as a direct consequence from the Fisher's factorization theorem stated above.
A sufficient and complete statistic is necessarily minimal sufficient. A minimal sufficient statistic always exists; a complete statistic need not exist.
[edit] Examples
[edit] Bernoulli distribution
If X1, ...., Xn are independent Bernoulli-distributed random variables with expected value p, then the sum T(X) = X1 + ... + Xn is a sufficient statistic for p (here 'success' corresponds to 
and 'failure' to 
; so T is the total number of successes)
This is seen by considering the joint probability distribution:
Because the observations are independent, this can be written as
and, collecting powers of p and 1 − p, gives
which satisfies the factorization criterion, with h(x)=1 being just a constant.
Note the crucial feature: the unknown parameter p interacts with the data x only via the statistic T(x) = Σ xi.
[edit] Uniform distribution
If X1, ...., Xn are independent and uniformly distributed on the interval [0,θ], then T(X) = max(X1, ...., Xn ) is sufficient for θ.
To see this, consider the joint probability distribution:
Because the observations are independent, this can be written as
where H(x) is the Heaviside step function. This may be written as
which can be viewed as a function of only θ and maxi(Xi) = T(X). This shows that the factorization criterion is satisfied, again where h(x)=1 is constant. Note that the parameter θ interacts with the data only through the data's maximum.
[edit] Poisson distribution
If X1, ...., Xn are independent and have a Poisson distribution with parameter λ, then the sum T(X) = X1 + ... + Xn is a sufficient statistic for λ.
To see this, consider the joint probability distribution:
Because the observations are independent, this can be written as
which may be written as
which shows that the factorization criterion is satisfied, where h(x) is the reciprocal of the product of the factorials. Note the parameter λ interacts with the data only through its sum T(X).
[edit] Rao-Blackwell theorem
Sufficiency finds a useful application in the Rao-Blackwell theorem. It states that if g(X) is any kind of estimator of θ, then typically the conditional expectation of g(X) given T(X) is a better estimator of θ, and is never worse. Sometimes one can very easily construct a very crude estimator g(X), and then evaluate that conditional expected value to get an estimator that is in various senses optimal.
[edit] See also
- Completeness of a statistic
- Basu's theorem on independence of complete sufficient and ancillary statistics
- Lehman-Scheffe theorem: a complete sufficient estimator is the best estimator of its expectation
[edit] References
- ↑ Fisher, Ronald (1922). On the Mathematical Foundations Of Theoretical Statistics.. Phil. Trans. R. Soc. Lond. A 222: 309-68.
- ↑ Stigler, Stephen (Dec. 1973). Studies in the History of Probability and Statistics. XXXII: Laplace, Fisher and the Discovery of the Concept of Sufficiency. Biometrika 60 (3): 439-445.
- ↑ Casella, George; Berger, Roger L. (2002). Statistical Inference, 2nd ed, Duxbury Press.
- ↑ The Fisher-Neyman Factorization Theorem.
| This page uses content from the English-language version of Wikipedia. The original article was at Sufficiency (statistics). The list of authors can be seen in the page history. As with Psychology Wiki, the text of Wikipedia is available under the GNU Free Documentation License. |
