Law of large numbers

An [[illustration of the law of large numbers using a particular run of rolls of a single die. As the number of rolls in this run increases, the average of the values of all the results approaches 3.5. Although each run would show a distinctive shape over a small number of throws (at the left), over a large number of rolls (to the right) the shapes would be extremely similar.|thumb|upright=1.35]]

In probability theory, the law of large numbers is a mathematical law which states that the average of the results obtained from a large number of independent random samples converges to the true value, if it exists. More formally, the law of large numbers states that given a sample of independent and identically distributed values, the sample mean converges to the true mean.

The law of large numbers is important because it guarantees stable long-term results for the averages of some random events. For example, while a casino may lose money in a single spin of the roulette wheel, its earnings will tend towards a predictable percentage over a large number of spins. Any winning streak by a player will eventually be overcome by the parameters of the game. Importantly, the law applies (as the name indicates) only when a large number of observations are considered. There is no principle that a small number of observations will coincide with the expected value or that a streak of one value will immediately be "balanced" by the others .

Throughout its history, many mathematicians have refined this law. Today, the law of large numbers is used in many fields including statistics, probability theory, economics, and insurance.

Examples

A single roll of a six-sided dice produces one of the numbers 1, 2, 3, 4, 5, or 6, each with equal probability. Therefore, the expected value of the roll is:

According to the law of large numbers, if a large number of six-sided dice are rolled, the average of their values (sometimes called the sample mean) will approach 3.5, with the precision increasing as more dice are rolled.

It follows from the law of large numbers that the empirical probability of success in a series of Bernoulli trials will converge to the theoretical probability. For a Bernoulli random variable, the expected value is the theoretical probability of success, and the average of n such variables (assuming they are independent and identically distributed (i.i.d.)) is precisely the relative frequency.

thumb|295x295px| This image illustrates the convergence of relative frequencies to their theoretical probabilities. The probability of picking a red ball from a sack is 0.4 and black ball is 0.6. The left plot shows the relative frequency of picking a black ball, and the right plot shows the relative frequency of picking a red ball, both over 10,000 trials. As the number of trials increases, the relative frequencies approach their respective theoretical probabilities, demonstrating the law of large numbers.

For example, a fair coin toss is a Bernoulli trial. When a fair coin is flipped once, the theoretical probability that the outcome will be heads is equal to . Therefore, according to the law of large numbers, the proportion of heads in a "large" number of coin flips "should be" roughly . In particular, the proportion of heads after n flips will almost surely converge to as n approaches infinity.

Although the proportion of heads (and tails) approaches , almost surely the absolute difference in the number of heads and tails will become large as the number of flips becomes large. That is, the probability that the absolute difference is a small number approaches zero as the number of flips becomes large. Also, almost surely the ratio of the absolute difference to the number of flips will approach zero. Intuitively, the expected difference grows, but at a slower rate than the number of flips.

Another good example of the law of large numbers is the Monte Carlo method. These methods are a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results. The larger the number of repetitions, the better the approximation tends to be. The reason that this method is important is mainly that, sometimes, it is difficult or impossible to use other approaches.

Limitation

The average of the results obtained from a large number of trials may fail to converge in some cases. For instance, the average of n results taken from the Cauchy distribution or some Pareto distributions (α<1) will not converge as n becomes larger; the reason is heavy tails. The Cauchy distribution and the Pareto distribution represent two cases: the Cauchy distribution does not have an expectation, whereas the expectation of the Pareto distribution (α<1) is infinite. One way to generate the Cauchy-distributed example is where the random numbers equal the tangent of an angle uniformly distributed between −90° and +90°. The median is zero, but the expected value does not exist, and indeed the average of n such variables have the same distribution as one such variable. It does not converge in probability toward zero (or any other value) as n goes to infinity.

If the trials embed a selection bias, typical in human economic/rational behaviour, the law of large numbers does not help in solving the bias, even if the number of trials is increased the selection bias remains.

History

thumb|right|upright=1.15|[[Molecular diffusion|Diffusion is an example of the law of large numbers. Initially, there are solute molecules on the left side of a barrier (magenta line) and none on the right. The barrier is removed, and the solute diffuses to fill the whole container.]]

The Italian mathematician Gerolamo Cardano (1501–1576) stated without proof that the accuracies of empirical statistics tend to improve with the number of trials. Markov, Borel, Cantelli, Kolmogorov and Khinchin. These further studies have given rise to two prominent forms of the law of large numbers. One is called the "weak" law and the other the "strong" law, in reference to two different modes of convergence of the cumulative sample means to the expected value; in particular, as explained below, the strong form implies the weak.

Forms

There are two different versions of the law of large numbers that are described below. They are called the strong law of large numbers and the weak law of large numbers.

Mutual independence of the random variables can be replaced by pairwise independence or exchangeability in both versions of the law.

The difference between the strong and the weak version is concerned with the mode of convergence being asserted. For interpretation of these modes, see Convergence of random variables.

Weak law

The weak law of large numbers (also called Khinchin's law) states that given a collection of independent and identically distributed (iid) samples from a random variable with finite mean, the sample mean converges in probability to the expected value

That is, for any positive number ε,

\lim_{n\to\infty}\Pr\!\left(\,|\overline{X}_n-\mu| < \varepsilon\,\right) = 1.

</math>

Interpreting this result, the weak law states that for any nonzero margin specified (ε), no matter how small, with a sufficiently large sample there will be a very high probability that the average of the observations will be close to the expected value; that is, within the margin.

As mentioned earlier, the weak law applies in the case of i.i.d. random variables, but it also applies in some other cases. For example, the variance may be different for each random variable in the series, keeping the expected value constant. If the variances are bounded, then the law applies, as shown by Chebyshev as early as 1867. (If the expected values change during the series, then we can simply apply the law to the average deviation from the respective expected values. The law then states that this converges in probability to zero.) In fact, Chebyshev's proof works so long as the variance of the average of the first n values goes to zero as n goes to infinity.

{\longrightarrow}\ \mu \qquad\textrm{when}\ n \to \infty.

</math>|

That is,

\Pr\!\left( \lim_{n\to\infty}\overline{X}_n = \mu \right) = 1.

</math>

What this means is that, as the number of trials n goes to infinity, the probability that the average of the observations converges to the expected value, is equal to one. The modern proof of the strong law is more complex than that of the weak law, and relies on passing to an appropriate sub-sequence.

If the summands are independent but not identically distributed, then

{\longrightarrow}\ 0,

</math>|

provided that each Xk has a finite second moment and

\sum_{k=1}^{\infty} \frac{1}{k^2} \operatorname{Var}[X_k] < \infty.

</math>

This statement is known as Kolmogorov's strong law, see e.g. .

Differences between the weak law and the strong law

The weak law states that for a specified large n, the average <math style="vertical-align:-.35em">\overline{X}_n</math> is likely to be near μ. Thus, it leaves open the possibility that <math style="vertical-align:-.4em">|\overline{X}_n -\mu| > \varepsilon</math> happens an infinite number of times, although at infrequent intervals. (Not necessarily <math style="vertical-align:-.4em">|\overline{X}_n -\mu| \neq 0</math> for all n).

The strong law shows that this almost surely will not occur. I.e., with probability 1 for any the inequality <math style="vertical-align:-.4em">|\overline{X}_n -\mu| < \varepsilon</math> holds for all large enough n.

The strong law does not hold in the following cases, but the weak law does.

{\rightarrow} \ 0.

</math>

This result is useful to derive consistency of a large class of estimators (see Extremum estimator).

Borel's law of large numbers

Borel's law of large numbers, named after Émile Borel, states that if an experiment is repeated a large number of times, independently under identical conditions, then the proportion of times that any specified event is expected to occur approximately equals the probability of the event's occurrence on any particular trial; the larger the number of repetitions, the better the approximation tends to be. More precisely, if E denotes the event in question, p its probability of occurrence, and Nn(E) the number of times E occurs in the first n trials, then with probability one,

<math display="block"> \frac{N_n(E)}{n}\to p\text{ as }n\to\infty.</math>

This theorem makes rigorous the intuitive notion of probability as the expected long-run relative frequency of an event's occurrence. It is a special case of any of several more general laws of large numbers in probability theory.

Proof of the weak law

Given X1, X2, ... an infinite sequence of i.i.d. random variables with finite expected value <math>E(X_1)=E(X_2)=\cdots=\mu<\infty</math>, we are interested in the convergence of the sample average

<math display="block">\overline{X}_n=\tfrac1n(X_1+\cdots+X_n). </math>

The weak law of large numbers states:

Proof using Chebyshev's inequality assuming finite variance

This proof uses the assumption of finite variance <math> \operatorname{Var} (X_i)=\sigma^2 </math> (for all <math>i</math>). The independence of the random variables implies no correlation between them, and we have that

\operatorname{Var}(\overline{X}_n) = \operatorname{Var}(\tfrac1n(X_1+\cdots+X_n)) = \frac{1}{n^2} \operatorname{Var}(X_1+\cdots+X_n) = \frac{n\sigma^2}{n^2} = \frac{\sigma^2}{n}.

</math>

The common mean μ of the sequence is the mean of the sample average:

E(\overline{X}_n) = \mu.

</math>

Using Chebyshev's inequality on <math>\overline{X}_n </math> results in

\operatorname{P}( \left| \overline{X}_n-\mu \right| \geq \varepsilon) \leq \frac{\sigma^2}{n\varepsilon^2}.

</math>

This may be used to obtain the following:

\operatorname{P}( \left| \overline{X}_n-\mu \right| < \varepsilon) = 1 - \operatorname{P}( \left| \overline{X}_n-\mu \right| \geq \varepsilon) \geq 1 - \frac{\sigma^2}{n \varepsilon^2 }.

</math>

As n approaches infinity, the expression approaches 1. And by definition of convergence in probability, we have obtained

Proof using convergence of characteristic functions

By Taylor's theorem for complex functions, the characteristic function of any random variable, X, with finite mean μ, can be written as

<math display="block">\varphi_X(t) = 1 + it\mu + o(t), \quad t \rightarrow 0.</math>

All X1, X2, ... have the same characteristic function, so we will simply denote this φX.

Among the basic properties of characteristic functions there are

<math display="block">\varphi_{\frac 1 n X}(t)= \varphi_X(\tfrac t n) \quad \text{and} \quad

\varphi_{X+Y}(t) = \varphi_X(t) \varphi_Y(t) \quad </math> if X and Y are independent.

These rules can be used to calculate the characteristic function of <math>\overline{X}_n</math> in terms of φX:

<math display="block">\varphi_{\overline{X}_n}(t)= \left[\varphi_X\left({t \over n}\right)\right]^n = \left[1 + i\mu{t \over n} + o\left({t \over n}\right)\right]^n \, \rightarrow \, e^{it\mu}, \quad \text{as} \quad n \to \infty.</math>

The limit eitμ is the characteristic function of the constant random variable μ, and hence by the Lévy continuity theorem, <math> \overline{X}_n</math> converges in distribution to μ:

<math display="block">\overline{X}_n \, \overset{\mathcal D}{\rightarrow} \, \mu \qquad\text{for}\qquad n \to \infty.</math>

μ is a constant, which implies that convergence in distribution to μ and convergence in probability to μ are equivalent (see Convergence of random variables.) Therefore,

This shows that the sample mean converges in probability to the derivative of the characteristic function at the origin, as long as the latter exists.

Proof of the strong law

We give a relatively simple proof of the strong law under the assumptions that the <math>X_i</math> are iid, <math> {\mathbb E}[X_i] =: \mu < \infty </math>, <math> \operatorname{Var} (X_i)=\sigma^2 < \infty</math>, and <math> {\mathbb E}[X_i^4] =: \tau < \infty </math>.

Let us first note that without loss of generality we can assume that <math>\mu = 0</math> by centering. In this case, the strong law says that

\Pr\!\left( \lim_{n\to\infty}\overline{X}_n = 0 \right) = 1,

</math>

\Pr\left(\omega: \lim_{n\to\infty}\frac{S_n(\omega)}n = 0 \right) = 1.

</math>

It is equivalent to show that

\Pr\left(\omega: \lim_{n\to\infty}\frac{S_n(\omega)}n \neq 0 \right) = 0,

</math>

Note that

\lim_{n\to\infty}\frac{S_n(\omega)}n \neq 0 \iff \exists\epsilon>0, \left|\frac{S_n(\omega)}n\right| \ge \epsilon\ \mbox{infinitely often},

</math>

and thus to prove the strong law we need to show that for every <math>\epsilon > 0</math>, we have

\Pr\left(\omega: |S_n(\omega)| \ge n\epsilon \mbox{ infinitely often} \right) = 0.

</math>

Define the events <math> A_n = \{\omega : |S_n| \ge n\epsilon\}</math>, and if we can show that

\sum_{n=1}^\infty \Pr(A_n) <\infty,

</math>

then the Borel-Cantelli Lemma implies the result. So let us estimate <math>\Pr(A_n)</math>.

We compute

{\mathbb E}[S_n^4] = {\mathbb E}\left[\left(\sum_{i=1}^n X_i\right)^4\right] = {\mathbb E}\left[\sum_{1 \le i,j,k,l\le n} X_iX_jX_kX_l\right].

</math>

We first claim that every term of the form <math>X_i^3X_j, X_i^2X_jX_k, X_iX_jX_kX_l</math> where all subscripts are distinct, must have zero expectation. This is because <math>{\mathbb E}[X_i^3X_j] = {\mathbb E}[X_i^3]{\mathbb E}[X_j]</math> by independence, and the last term is zero—and similarly for the other terms. Therefore the only terms in the sum with nonzero expectation are <math>{\mathbb E}[X_i^4]</math> and <math>{\mathbb E}[X_i^2X_j^2]</math>. Since the <math>X_i</math> are identically distributed, all of these are the same, and moreover <math>{\mathbb E}[X_i^2X_j^2]=({\mathbb E}[X_i^2])^2</math>.

There are <math>n</math> terms of the form <math>{\mathbb E}[X_i^4]</math> and <math>3 n (n-1)</math> terms of the form <math>({\mathbb E}[X_i^2])^2</math>, and so

{\mathbb E}[S_n^4] = n \tau + 3n(n-1)\sigma^4.

</math>

Note that the right-hand side is a quadratic polynomial in <math>n</math>, and as such there exists a <math>C>0</math> such that <math> {\mathbb E}[S_n^4] \le Cn^2</math> for <math>n</math> sufficiently large. By Markov,

\Pr(|S_n| \ge n \epsilon) \le \frac1{(n\epsilon)^4}{\mathbb E}[S_n^4] \le \frac{C}{\epsilon^4 n^2},

</math>

for <math>n</math> sufficiently large, and therefore this series is summable. Since this holds for any <math>\epsilon > 0</math>, we have established the strong law of large numbers. The proof can be strengthened immensely by dropping all finiteness assumptions on the second and fourth moments. It can also be extended for example to discuss partial sums of distributions without any finite moments. Such proofs use more intricate arguments to prove the same Borel-Cantelli predicate, a strategy attributed to Kolmogorov to conceptually bring the limit inside the probability parentheses.

Consequences

The law of large numbers provides an expectation of an unknown distribution from a realization of the sequence, but also any feature of the probability distribution. Using the Monte Carlo method and the LLN, we can see that as the number of samples increases, the numerical value gets ever closer to 0.4180233.