In probability theory, conditional independence describes situations in which an observation is irrelevant or redundant when evaluating the certainty of a hypothesis. It is the opposite of conditional dependence. Conditional independence is usually formulated in terms of conditional probability, as a special case where the probability of the hypothesis given the uninformative observation is equal to the probability without. If <math>A</math> is the hypothesis, and <math>B</math> and <math>C</math> are observations, conditional independence can be stated as an equality:
:<math>P(A\mid B,C) = P(A \mid C)</math>
where <math>P(A \mid B, C)</math> is the probability of <math>A</math> given both <math>B</math> and <math>C</math>. Since the probability of <math>A</math> given <math>C</math> is the same as the probability of <math>A</math> given both <math>B</math> and <math>C</math>, this equality expresses that <math>B</math> contributes nothing to the certainty of <math>A</math>. In this case, <math>A</math> and <math>B</math> are said to be conditionally independent given <math>C</math>, written symbolically as: <math>(A \perp\!\!\!\perp B \mid C)</math>.
The concept of conditional independence is essential to graph-based theories of statistical inference, as it establishes a mathematical relation between a collection of conditional statements and a graphoid.
Conditional independence of events
Let <math>A</math>, <math>B</math>, and <math>C</math> be events. <math>A</math> and <math>B</math> are said to be conditionally independent given <math>C</math> if and only if <math>P(C) > 0</math> and<math>P(A \mid B, C) = P(A \mid C)</math>. This property is symmetric (more on this below) and often written as <math>(A \perp\!\!\!\perp B \mid C)</math>, which should be read as <math>((A \perp\!\!\!\perp B) \vert C)</math>.
Equivalently, conditional independence may be stated as <math>P(A,B|C) = P(A|C)P(B|C)</math> where <math>P(A,B|C)</math> is the joint probability of <math>A</math> and <math>B</math> given <math>C</math>. This alternate formulation states that <math>A</math> and <math>B</math> are independent events, <math>C</math>.
It demonstrates that <math>(A \perp\!\!\!\perp B \mid C)</math> is equivalent to <math>(B \perp\!\!\!\perp A \mid C)</math>.
Proof of the equivalent definition
:<math>\begin{align}
P(A, B \mid C) = P(A \mid C) P(B \mid C)
& \iff \frac{P(A, B, C)}{P(C)} = \left(\frac{P(A, C)}{P(C)}\right) \left(\frac{P(B, C)}{P(C)} \right) & \text{definition of conditional probability} \\
& \iff P(A, B, C) = \frac{P(A, C) P(B, C)}{P(C)} & \text{multiply both sides by P(C)} \\
& \iff \frac{P(A, B, C)}{P(B, C)}= \frac{P(A, C)}{P(C)} & \text{divide both sides by P(B, C)} \\
& \iff P(A \mid B, C) = P(A \mid C) &\text{definition of conditional probability}
\end{align}</math>
Examples
Coloured boxes
Each cell represents a possible outcome. The events <math>\color{red}R</math>, <math>\color{blue}B</math> and <math>\color{gold}Y</math> are represented by the areas shaded , and respectively. The overlap between the events <math>\color{red}R</math> and <math>\color{blue}B</math> is shaded .
450px|These are two examples illustrating conditional independence.
The probabilities of these events are shaded areas with respect to the total area. In both examples <math>\color{red}R</math> and <math>\color{blue}B</math> are conditionally independent given <math>\color{gold}Y</math> because:
:<math>\Pr({\color{red}R}, {\color{blue}B} \mid {\color{gold}Y}) = \Pr({\color{red}R} \mid {\color{gold}Y})\Pr({\color{blue}B} \mid {\color{gold}Y})</math>
but not conditionally independent given <math>\left[ \text{not }{\color{gold}Y}\right]</math> because:
:<math>\Pr({\color{red}R}, {\color{blue}B} \mid \text{not } {\color{gold}Y}) \not= \Pr({\color{red}R} \mid \text{not } {\color{gold}Y})\Pr({\color{blue}B} \mid \text{not } {\color{gold}Y})</math>
Proximity and delays
Let events A and B be defined as the probability that person A and person B will be home in time for dinner where both people are randomly sampled from the entire world. Events A and B can be assumed to be independent i.e. knowledge that A is late has minimal to no change on the probability that B will be late. However, if a third event is introduced, person A and person B live in the same neighborhood, the two events are now considered not conditionally independent. Traffic conditions and weather-related events that might delay person A, might delay person B as well. Given the third event and knowledge that person A was late, the probability that person B will be late does meaningfully change.
Dice rolling
Conditional independence depends on the nature of the third event. If you roll two dice, one may assume that the two dice behave independently of each other. Looking at the results of one die will not tell you about the result of the second die. (That is, the two dice are independent.) If, however, the 1st die's result is a 3, and someone tells you about a third event - that the sum of the two results is even - then this extra unit of information restricts the options for the 2nd result to an odd number. In other words, two events can be independent, but not conditionally independent. where <math>X</math> and <math>Y</math> take the values "brainy" and "sporty".
Conditional independence of random vectors
Two random vectors <math>\mathbf{X}=(X_1,\ldots,X_l)^{\mathrm T}</math> and <math>\mathbf{Y}=(Y_1,\ldots,Y_m)^{\mathrm T}</math> are conditionally independent given a third random vector <math>\mathbf{Z}=(Z_1,\ldots,Z_n)^{\mathrm T}</math> if and only if they are independent in their conditional cumulative distribution given <math>\mathbf{Z}</math>. Formally:
(\mathbf{x}) \cdot F_{\mathbf{Y}\,\mid\,\mathbf{Z}\,=\,\mathbf{z(\mathbf{y}) \quad \text{for all } \mathbf{x},\mathbf{y},\mathbf{z}</math>|
|cellpadding= 6
|border
|border colour = #0073CF
|background colour=#F5FFFA
where <math>\mathbf{x}=(x_1,\ldots,x_l)^{\mathrm T}</math>, <math>\mathbf{y}=(y_1,\ldots,y_m)^{\mathrm T}</math> and <math>\mathbf{z}=(z_1,\ldots,z_n)^{\mathrm T}</math> and the conditional cumulative distributions are defined as follows.
: <math>\begin{align}
F_{\mathbf{X},\mathbf{Y}\,\mid\,\mathbf{Z}\,=\,\mathbf{z(\mathbf{x},\mathbf{y}) &= \Pr(X_1 \leq x_1,\ldots,X_l \leq x_l, Y_1 \leq y_1,\ldots,Y_m \leq y_m \mid Z_1=z_1,\ldots,Z_n=z_n) \\[6pt]
F_{\mathbf{X}\,\mid\,\mathbf{Z}\,=\,\mathbf{z(\mathbf{x}) &= \Pr(X_1 \leq x_1,\ldots,X_l \leq x_l \mid Z_1=z_1,\ldots,Z_n=z_n) \\[6pt]
F_{\mathbf{Y}\,\mid\,\mathbf{Z}\,=\,\mathbf{z(\mathbf{y}) &= \Pr(Y_1 \leq y_1,\ldots,Y_m \leq y_m \mid Z_1=z_1,\ldots,Z_n=z_n)
\end{align}</math>
Uses in Bayesian inference
Let p be the proportion of voters who will vote "yes" in an upcoming referendum. In taking an opinion poll, one chooses n voters randomly from the population. For i = 1, ..., n, let X<sub>i</sub> = 1 or 0 corresponding, respectively, to whether or not the ith chosen voter will or will not vote "yes".
In a frequentist approach to statistical inference one would not attribute any probability distribution to p (unless the probabilities could be somehow interpreted as relative frequencies of occurrence of some event or as proportions of some population) and one would say that X<sub>1</sub>, ..., X<sub>n</sub> are independent random variables.
By contrast, in a Bayesian approach to statistical inference, one would assign a probability distribution to p regardless of the non-existence of any such "frequency" interpretation, and one would construe the probabilities as degrees of belief that p is in any interval to which a probability is assigned. In that model, the random variables X<sub>1</sub>, ..., X<sub>n</sub> are not independent, but they are conditionally independent given the value of p. In particular, if a large number of the Xs are observed to be equal to 1, that would imply a high conditional probability, given that observation, that p is near 1, and thus a high conditional probability, given that observation, that the next X to be observed will be equal to 1.
Rules of conditional independence
A set of rules governing statements of conditional independence have been derived from the basic definition.
These rules were termed "Graphoid Axioms"
by Pearl and Paz, because they hold in graphs, where <math>X \perp\!\!\!\perp A\mid B</math> is interpreted to mean: "All paths from X to A are intercepted by the set B".
Symmetry
: <math>
X \perp\!\!\!\perp Y \mid Z \quad
\Leftrightarrow
\quad
Y \perp\!\!\!\perp X \mid Z
</math>
Proof:
From the definition of conditional independence,
: <math>
X \perp\!\!\!\perp Y \mid Z \quad
\Leftrightarrow
\quad P(X, Y \mid Z) = P(X \mid Z) P(Y \mid Z) \quad
\Leftrightarrow
\quad
Y \perp\!\!\!\perp X \mid Z
</math>
Decomposition
: <math>
X \perp\!\!\!\perp Y \mid Z
\quad \Rightarrow \quad
h(X) \perp\!\!\!\perp Y \mid Z
</math>
Proof
From the definition of conditional independence, we seek to show that:
: <math>
X \perp\!\!\!\perp Y \mid Z
\quad \Rightarrow \quad
P(h(X), Y \mid Z) = P(h(X) \mid Z) P(Y \mid Z)
</math>
. The left side of this equality is:
: <math>
P(h(X)=a, Y=y \mid Z=z) = \sum_{X \colon h(X)=a} P(X=x, Y=y \mid Z=z)
</math>
, where the expression on the right side of this equality is the summation over <math>X</math> such that <math>h(X)=a</math> of the conditional probability of <math>X, Y</math> on <math>Z</math>.
Further decomposing,
: <math>
\begin{align}
\sum_{X \colon h(X)=a} P(X=x, Y=y \mid Z=z) =& \sum_{X \colon h(X)=a} P(X=x \mid Z=z) P(Y=y \mid Z=z) \\
=& P(Y=y \mid Z=z) \sum_{X \colon h(X)=a} P(X=x \mid Z=z) \\
=& P(Y \mid Z) P (h(X) \mid Z)
\end{align}
</math>
. Special cases of this property include
- <math>
(X, W) \perp\!\!\!\perp Y \mid Z
\quad \Rightarrow \quad
X \perp\!\!\!\perp Y \mid Z
</math>
- Proof: Let us define <math> A = (X,W) </math> and <math> h(\cdot) </math> be an 'extraction' function <math> h(X,W) = X</math>. Then:
: <math>\begin{align}
(X,W) \perp\!\!\!\perp Y \mid Z
\quad &\Leftrightarrow \quad
A \perp\!\!\!\perp Y \mid Z \\
&\Rightarrow \quad
h(A) \perp\!\!\!\perp Y \mid Z \quad &\text{Decomposition} \\
&\Leftrightarrow \quad
X \perp\!\!\!\perp Y \mid Z
\end{align}</math>
- <math>
X \perp\!\!\!\perp (Y, W) \mid Z
\quad \Rightarrow \quad
X \perp\!\!\!\perp Y \mid Z
</math>
- Proof: Let us define <math> V = (Y,W) </math> and <math> h(\cdot) </math> be again an 'extraction' function <math> h(Y,W) = Y</math>. Then:
: <math>
\begin{align}
X \perp\!\!\!\perp (Y,W) \mid Z
\quad &\Leftrightarrow \quad
X \perp\!\!\!\perp V \mid Z \\
&\Leftrightarrow \quad
V \perp\!\!\!\perp X \mid Z \quad &\text{Symmetry} \\
&\Rightarrow \quad
h(V) \perp\!\!\!\perp X \mid Z \quad &\text{Decomposition} \\
&\Leftrightarrow \quad
Y \perp\!\!\!\perp X \mid Z \\
&\Leftrightarrow \quad
X \perp\!\!\!\perp Y \mid Z \quad &\text{Symmetry}
\end{align}
</math>
Weak union
: <math>
X \perp\!\!\!\perp Y \mid Z
\quad \Rightarrow \quad
X \perp\!\!\!\perp Y \mid (Z, h(X))
</math>
Proof:
Given <math> X \perp\!\!\!\perp Y \mid Z </math>, we aim to show
: <math>
\begin{align}
X \perp\!\!\!\perp Y \mid (Z, h(X))
\quad &\Leftrightarrow \quad
X \perp\!\!\!\perp Y \mid U \quad &\text{where} \quad U = (Z, h(X)) \\
&\Leftrightarrow \quad
Y \perp\!\!\!\perp X \mid U \quad &\text{Symmetry} \\
&\Leftrightarrow \quad
P(Y\mid X, U) = P(Y\mid U) \\
&\Leftrightarrow \quad
P(Y \mid X, Z, h(X)) = P(Y \mid Z, h(X))
\end{align}
</math>
. We begin with the left side of the equation
: <math>
\begin{align}
P(Y \mid X, Z, h(X)) &= P(Y \mid X, Z) \\
&= P(Y \mid Z) &\text{Since by symmetry } Y \perp\!\!\!\perp X \mid Z
\end{align}
</math>
. From the given condition
: <math>
\begin{align}
X \perp\!\!\!\perp Y \mid Z
\quad &\Rightarrow \quad
h(X) \perp\!\!\!\perp Y \mid Z
\quad &\text{Decomposition} \\
&\Leftrightarrow \quad
Y \perp\!\!\!\perp h(X) \mid Z
\quad &\text{Symmetry} \\
&\Rightarrow \quad
P(Y \mid Z, h(X)) = P(Y \mid Z)
\end{align}
</math>
. Thus <math> P(Y \mid X, Z, h(X)) = P(Y \mid Z, h(X))
</math>, so we have shown that <math>
X \perp\!\!\!\perp Y \mid (Z, h(X))
</math>.
Special Cases:
Some textbooks present the property as
- <math> X \perp\!\!\!\perp (Y, W) \mid Z
\quad \Rightarrow \quad
X \perp\!\!\!\perp Y \mid (Z, W) </math>.
- <math> (X,W) \perp\!\!\!\perp Y \mid Z
\quad \Rightarrow \quad
X \perp\!\!\!\perp Y \mid (Z,W) </math>.
Both versions can be shown to follow from the weak union property given initially via the same method as in the decomposition section above.
Contraction
: <math>
\left.\begin{align}
X \perp\!\!\!\perp A \mid B \\
X \perp\!\!\!\perp B
\end{align}\right\}\text{ and }
\quad \Rightarrow \quad
X \perp\!\!\!\perp A,B
</math>
Proof
This property can be proved by noticing <math>\Pr(X\mid A,B) = \Pr(X\mid B) = \Pr(X)</math>, each equality of which is asserted by <math>X \perp\!\!\!\perp A \mid B</math> and <math>X \perp\!\!\!\perp B</math>, respectively.
Intersection
For strictly positive probability distributions,