Confidence interval

thumb|upright=1.3|Each row of points is a sample from the same normal distribution. The colored lines are 50% confidence intervals for the population mean μ. At the center of each interval is the sample mean <math display="inline">\bar{x}</math>, marked with a diamond. The blue intervals contain μ, and the red ones do not.

According to frequentist inference, a confidence interval (CI) is a range of values which is likely to contain (in repeated sampling) the true value of an unknown statistical parameter, such as a population mean. Rather than reporting a single point estimate (e.g. "the average screen time is 3 hours per day"), a confidence interval provides a range, such as 2 to 4 hours, along with a specified confidence level, typically 95%.

A 95% confidence level does not imply a 95% probability that the true parameter lies within a particular calculated interval, which is instead associated with the credible interval in Bayesian inference. The confidence level instead reflects the long-run reliability of the method used to generate the interval. In other words, if the same sampling procedure were repeated 100 times from the same population, approximately 95 of the resulting intervals would be expected to contain the true population mean. The frequentist approach sees the true population mean as a fixed unknown constant, while the confidence interval is calculated using data from a random sample. Because the sample is random, the interval endpoints are random variables.

Definition

Let <math>X</math> be a random sample from a probability distribution with statistical parameter <math>(\theta, \varphi)</math>. Here, <math>\theta</math> is the quantity to be estimated, while <math>\varphi</math> includes other parameters (if any) that determine the distribution. A confidence interval for the parameter <math>\theta</math>, with confidence level or coefficient <math>\gamma</math>, is an interval <math>(u(X), v(X))</math> determined by random variables <math>u(X)</math> and <math>v(X)</math> with the property:

The number <math>\gamma</math>, which is typically large (e.g. 0.95), is sometimes given in the form <math>1 - \alpha</math> (or as a percentage <math>100%\cdot(1 - \alpha)</math>), where <math>\alpha</math> is a small positive number, often 0.05. It means that the interval <math display="inline">(u(X), v(X))</math> has a probability <math display="inline">\gamma</math> of covering the value of <math display="inline">\theta</math> in repeated sampling.

In many applications, confidence intervals that have exactly the required confidence level are hard to construct, but approximate intervals can be computed. The rule for constructing the interval may be accepted if

to an acceptable level of approximation. Alternatively, some authors simply require that

When it is known that the coverage probability can be strictly larger than <math>\gamma</math> for some parameter values, the confidence interval is called conservative, i.e., it errs on the safe side; which also means that the interval can be wider than need be.

Methods of derivation

There are many ways of calculating confidence intervals, and the best method depends on the situation. Two widely applicable methods are bootstrapping and the central limit theorem. The latter method works only if the sample is large, since it entails calculating the sample mean <math>\bar{X}</math> and sample standard deviation <math>S</math> and using the asymptotically standard normal quantity

where <math display="inline">\mu</math> and <math>n</math> are the population mean and the sample size, respectively.

Example

thumb|200px|In this [[bar chart, the top ends of the brown bars indicate observed means and the red line segments ("error bars") represent the confidence intervals around them. Although the error bars are shown as symmetric around the means, that is not always the case. In most graphs, the error bars do not represent confidence intervals (e.g., they often represent standard errors or standard deviations).]]

Suppose <math>X_1, \ldots, X_n</math> is an independent sample from a normally distributed population with unknown parameters mean <math>\mu</math> and variance <math>\sigma^2.</math> Define the sample mean <math>\bar{X}</math> and unbiased sample variance <math>S^2</math> as

<math display="block">\begin{align}

\bar{X} &= \frac{1}{n} \left(X_1 + \cdots + X_n\right), \\

S^2 &= \frac{1}{n-1}\sum_{i=1}^n \left(X_i - \bar{X}\right)^2.

\end{align}</math>

Then the value

has a Student's t distribution with <math display="inline">n - 1</math> degrees of freedom. This value is useful because its distribution does not depend on the values of the unobservable parameters <math display="inline">\mu</math> and <math display="inline">\sigma^2</math>; i.e., it is a pivotal quantity.

Suppose we wanted to calculate a 95% confidence interval for <math display="inline">\mu.</math> First, let <math display="inline">c</math> be the 97.5th percentile of the distribution of <math display="inline">T</math>. Then there is a 2.5% chance that <math display="inline">T</math> will be less than <math display="inline">-c</math> and a 2.5% chance that it will be larger than <math display="inline">+c</math> (as the t distribution is symmetric about 0). In other words,

Consequently, by replacing <math display="inline">T</math> with <math>\frac{\bar{X} - \mu}{S/\sqrt{n</math> and re-arranging terms,

<math display="block">P_X {\left(\bar{X} - \frac{cS}{\sqrt{n \leq \mu \leq \bar{X} + \frac{cS}{\sqrt{n\right)} = 0.95</math>

where <math>P_X</math> is the probability measure for the sample <math>X_1, \ldots, X_n</math>.

It means that there is 95% probability with which this condition <math>\bar{X} - \frac} \leq \mu \leq \bar{X} + \frac}</math> occurs in repeated sampling. After observing a sample, we find values <math>\bar{x}</math> for <math>\bar{X}</math> and <math>s</math> for <math>S,</math> from which we compute the below interval, and we say it is a 95% confidence interval for the mean.

<math display="block">\left[\bar{x} - \frac{cs}{\sqrt{n, \bar{x} + \frac{cs}{\sqrt{n\right].</math>

Interpretation

Various interpretations of a confidence interval can be given (taking the 95% confidence interval as an example in the following).

The confidence interval can be expressed in terms of a long-run frequency in repeated samples (or in resampling): "Were this procedure to be repeated on numerous samples, the proportion of calculated 95% confidence intervals that encompassed the true value of the population parameter would tend toward 95%."

thumb|Interpretation of the 95% confidence interval in terms of statistical significance|369x369px|center

Common misunderstandings

thumb|A plot of 50 confidence intervals from 50 samples generated from a normal distribution

Confidence intervals and levels are frequently misunderstood, and published studies have shown that even professional scientists often misinterpret them.

Contrary to common misconceptions, a 95% confidence level does not mean that:

for a given realized interval there is a 95% probability that the population parameter lies within the interval;

For example, suppose a factory produces metal rods, and a random sample of 25 rods gives a 95% confidence interval of 36.8 to 39.0 mm for the population mean length.

It is incorrect to say that there is a 95% probability that the true population mean lies within this interval: the true mean is fixed, not random. The true mean could be 37 mm, which is within the confidence interval, or 40 mm, which is not; in any case, whether it falls between 36.8 and 39.0 mm is a matter of fact, not probability.
It is not necessarily true that the lengths of 95% of the sampled rods lie within this interval. In this case, it cannot be true: 95% of 25 is not an integer.
It is not generally true that there is a 95% probability that the sample mean length (an estimate of the population mean length) in a second sample would fall within this interval. In fact, if the true mean length is far from this specific confidence interval, it could be very unlikely that the next sample mean falls within the interval.

Instead, the 95% confidence level means that if we took 100 such samples, we would expect the true population mean to lie within approximately 95 of the calculated intervals. confidence intervals coincide with credible intervals under non-informative priors. In such cases, common misconceptions about confidence intervals (e.g. interpreting them as probability statements about the parameter) may yield practically correct conclusions.

Examples of how naïve interpretation of confidence intervals can be problematic

Confidence procedure for uniform location

thumb|Ten examples of the 50% Welch and Bayesian intervals are shown in contrasting white and gray rows. The examples are sorted top-to-bottom by decreasing distance between <math>X_1</math> and <math>X_2</math>.

Welch presented an example which clearly shows the difference between the theory of confidence intervals and other theories of interval estimation (including Fisher's fiducial intervals and objective Bayesian intervals). Robinson called this example "[p]ossibly the best known counterexample for Neyman's version of confidence interval theory." To Welch, it showed the superiority of confidence interval theory; to critics of the theory, it shows a deficiency. Here we present a simplified version.

Suppose that <math>X_1,X_2</math> are independent observations from a uniform <math>(\theta - 1/2, \theta + 1/2)</math> distribution. Then the optimal 50% confidence procedure for <math>\theta</math> is

<math display="block">\bar{X} \pm \begin{cases}

\dfrac{|X_1-X_2|}{2} & \text{if } |X_1-X_2| < 1/2 \\[8pt]

\dfrac{1-|X_1-X_2|}{2} &\text{if } |X_1-X_2| \geq 1/2.

\end{cases}

</math>

A fiducial or objective Bayesian argument can be used to derive the interval estimate

which is also a 50% confidence procedure. Welch showed that the first confidence procedure dominates the second, according to desiderata from confidence interval theory; for every <math>\theta_1\neq\theta</math>, the probability that the first procedure contains <math>\theta_1</math> is less than or equal to the probability that the second procedure contains <math>\theta_1</math>. The average width of the intervals from the first procedure is less than that of the second. Hence, the first procedure is preferred under classical confidence interval theory.

However, when <math>|X_1-X_2| \geq 1/2</math>, intervals from the first procedure are guaranteed to contain the true value <math>\theta</math>: Therefore, the nominal 50% confidence coefficient is unrelated to the uncertainty we should have that a specific interval contains the true value. The second procedure does not have this property.

Moreover, when the first procedure generates a very short interval, this indicates that <math>X_1,X_2</math> are very close together and hence only offer the information in a single data point. Yet the first interval will exclude almost all reasonable values of the parameter due to its short width. The second procedure does not have this property.

The two counter-intuitive properties of the first procedure – 100% coverage when <math>X_1,X_2</math> are far apart and almost 0% coverage when <math>X_1,X_2</math> are close together – balance out to yield 50% coverage on average. However, despite the first procedure being optimal, its intervals offer neither an assessment of the precision of the estimate nor an assessment of the uncertainty one should have that the interval contains the true value.

This example is used to argue against naïve interpretations of confidence intervals. If a confidence procedure is asserted to have properties beyond that of the nominal coverage (such as relation to precision, or a relationship with Bayesian inference), those properties must be proved; they do not follow from the fact that a procedure is a confidence procedure.

Confidence procedure for ω<sup>2</sup>

Steiger suggested a number of confidence procedures for common effect size measures in ANOVA. Morey et al. The main ideas of confidence intervals in general were developed in the early 1930s, and the first thorough and general account was given by Jerzy Neyman in 1937. It so happened that, somewhat earlier, Fisher published his first paper concerned with fiducial distributions and fiducial argument. Quite unexpectedly, while the conceptual framework of fiducial argument is entirely different from that of confidence intervals, the specific solutions of several particular problems coincided. Thus, in the first paper in which I presented the theory of confidence intervals, published in 1934, By 1988, medical journals were requiring the reporting of confidence intervals.