In statistics, an effect size is a quantitative measure of the magnitude of a phenomenon. It can refer to the value of a statistic calculated from a sample of data, the value of one parameter for a hypothetical population, or the equation that operationalizes how statistics or parameters lead to the effect size value. Examples of effect sizes include the correlation between two variables, the regression coefficient in a regression, the mean difference, and the risk of a particular event (such as a heart attack). Effect sizes are a complementary tool for statistical hypothesis testing, and play an important role in statistical power analyses to assess the sample size required for new experiments. Effect size calculations are fundamental to meta-analysis, which aims to provide the combined effect size based on data from multiple studies. The group of data-analysis methods concerning effect sizes is referred to as estimation statistics.
Effect size is an essential component in the evaluation of the strength of a statistical claim, and it is the first item (magnitude) in the MAGIC criteria. The standard deviation of the effect size is of critical importance, as it indicates how much uncertainty is included in the observed measurement. A standard deviation that is too large will make the measurement nearly meaningless. In meta-analysis, which aims to summarize multiple effect sizes into a single estimate, the uncertainty in studies' effect sizes is used to weight the contribution of each study, so larger studies are considered more important than smaller ones. The uncertainty in the effect size is calculated differently for each type of effect size, but generally only requires knowing the study's sample size (N), or the number of observations (n) in each group.
Reporting effect sizes or estimates thereof (effect estimate [EE], estimate of effect) is considered good practice when presenting empirical research findings in many fields. The reporting of effect sizes facilitates the interpretation of the importance of a research result, in contrast to its statistical significance. Effect sizes are particularly significant in social science and medical research, with the latter emphasizing the importance of the magnitude of the average treatment effect.
Effect sizes may be measured in relative or absolute terms. In relative effect sizes, two groups are directly compared with each other, as in odds ratios and relative risks. A larger absolute value always indicates a stronger effect for absolute effect sizes. Many types of measurements can be expressed as either absolute or relative, and these can be used together because they convey different information. A prominent task force in the psychology research community made the following recommendation:
{\text{SS}_\text{total} + \text{MS}_\text{error .</math>
This form of the formula is limited to between-subjects analysis with equal sample sizes in all cells. In addition, methods to calculate partial ω<sup>2</sup> for individual factors and combined factors in designs with up to three independent variables have been published.
The <math>f^{2}</math> effect size measure for sequential multiple regression and also common for PLS modeling is defined as:
<math display="block">f^2 = {R^2_{AB} - R^2_A \over 1 - R^2_{AB</math>
where R<sup>2</sup><sub>A</sub> is the variance accounted for by a set of one or more independent variables A, and R<sup>2</sup><sub>AB</sub> is the combined variance accounted for by A and another set of one or more independent variables of interest B. By convention, f<sup>2</sup> effect sizes of <math>0.1^2</math>, <math>0.25^2</math>, and <math>0.4^2</math> are termed small, medium, and large, respectively.
<math display="block">\theta = \frac{\mu_1 - \mu_2} \sigma,</math>
where μ<sub>1</sub> is the mean for one population, μ<sub>2</sub> is the mean for the other population, and σ is a standard deviation based on either or both populations.
In the practical setting the population values are typically not known and must be estimated from sample statistics. The several versions of effect sizes based on means differ with respect to which statistics are used.
This form for the effect size resembles the computation for a t-test statistic, with the critical difference that the t-test statistic includes a factor of <math>\sqrt{n}</math>. This means that for a given effect size, the significance level increases with the sample size. Unlike the t-test statistic, the effect size aims to estimate a population parameter and is not affected by the sample size.
SMD values of 0.2 to 0.5 are considered small, 0.5 to 0.8 are considered medium, and greater than 0.8 are considered large.
Cohen's d
Cohen's d is defined as the difference between two means divided by a standard deviation for the data, i.e.
<math display="block">d = \frac{\bar{x}_1 - \bar{x}_2} s.</math>
Jacob Cohen defined s, the pooled standard deviation, as (for two independent samples):
<math display="block">s = \sqrt{\frac{(n_1-1)s^2_1 + (n_2-1)s^2_2}{n_1+n_2 - 2</math>
where the variance for one of the groups is defined as
<math display="block">s_1^2 = \frac 1 {n_1-1} \sum_{i=1}^{n_1} (x_{1,i} - \bar{x}_1)^2,</math>
and similarly for the other group.
Other authors choose a slightly different computation of the standard deviation when referring to "Cohen's d" where the denominator is without "-2"
<math display="block">s = \sqrt{\frac{(n_1-1)s^2_1 + (n_2-1)s^2_2}{n_1+n_2</math>
This definition of "Cohen's d" is termed the maximum likelihood estimator by Hedges and Olkin,
Cohen's d is frequently used in estimating sample sizes for statistical testing. A lower Cohen's d indicates the necessity of larger sample sizes, and vice versa, as can subsequently be determined together with the additional parameters of desired significance level and statistical power.
Glass' Δ
In 1976, Gene V. Glass proposed an estimator of the effect size that uses only the standard deviation of the second group
is like the other measures based on a standardized difference CRTs involve randomising clusters, such as schools or classrooms, to different conditions and are frequently used in education research.
Ψ, root-mean-square standardized effect
A similar effect size estimator for multiple comparisons (e.g., ANOVA) is the Ψ root-mean-square standardized effect:
:<math>\beta = \frac{\mu_1 - \mu_2}{\sqrt{\sigma_1^2 + \sigma_2^2 - 2\sigma_{12} .</math>
If the two groups are independent,
:<math>\beta = \frac{\mu_1 - \mu_2}{\sqrt{\sigma_1^2 + \sigma_2^2 .</math>
If the two independent groups have equal variances <math>\sigma^2</math>,
:<math>\beta = \frac{\mu_1 - \mu_2}{\sqrt{2}\sigma}.</math>
Other metrics
Mahalanobis distance (D) is a multivariate generalization of Cohen's d, which takes into account the relationships between the variables.
Subin's E (<math>E_{HBG}</math>) is a bounded effect size for paired repeated-assessments data. It standardizes mean gain using a reference spread, incorporates heterogeneity in individual gain scores, and applies a bounded arctangent transformation. It is intended for within-group paired comparisons across repeated assessment occasions.
<math display="block">
E_{HBG}=\frac{2}{\pi}\arctan\left(
\frac{J(\bar X_2-\bar X_1)}
{k\,S_{ref}\sqrt{1+\lambda\left(\frac{s_D}{S_{ref\right)^2
\right)
</math>
where
<math display="block">
S_{ref}=\sqrt{\frac{s_1^2+s_2^2}{2.
</math>
In the published calibration of the method, the recommended parameter values were <math>\lambda = 2.8759</math> and <math>k = 0.5069</math>. Cramér's V may be used with variables having more than two levels.
Phi can be computed by finding the square root of the chi-squared statistic divided by the sample size.
Similarly, Cramér's V is computed by taking the square root of the chi-squared statistic divided by the sample size and the length of the minimum dimension (k is the smaller of the number of rows r or columns c).
φ<sub>c</sub> is the intercorrelation of the two discrete variables and may be computed for any value of r or c. However, as chi-squared values tend to increase with the number of cells, the greater the difference between r and c, the more likely V will tend to 1 without strong evidence of a meaningful correlation.
Cohen's omega (ω)
Another measure of effect size used for chi-squared tests is Cohen's omega (<math> \omega</math>). This is defined as
<math display="block"> \omega = \sqrt{ \sum_{i=1}^m \frac{ (p_{1i} - p_{0i})^2 }{p_{0i } </math>
where p<sub>0i</sub> is the proportion of the i<sup>th</sup> cell under H<sub>0</sub>, p<sub>1i</sub> is the proportion of the i<sup>th</sup> cell under H<sub>1</sub> and m is the number of cells.
Odds ratio
The odds ratio (OR) is another useful effect size. It is appropriate when the research question focuses on the degree of association between two binary variables. For example, consider a study of spelling ability. In a control group, two students pass the class for every one who fails, so the odds of passing are two to one (or 2/1 = 2). In the treatment group, six students pass for every one who fails, so the odds of passing are six to one (or 6/1 = 6). The effect size can be computed by noting that the odds of passing in the treatment group are three times higher than in the control group (because 6 divided by 2 is 3). Therefore, the odds ratio is 3. Odds ratio statistics are on a different scale than Cohen's d, so this '3' is not comparable to a Cohen's d of 3.
Relative risk
The relative risk (RR), also called risk ratio, is simply the risk (probability) of an event relative to some independent variable. This measure of effect size differs from the odds ratio in that it compares probabilities instead of odds, but asymptotically approaches the latter for small probabilities. Using the example above, the probabilities for those in the control group and treatment group passing is 2/3 (or 0.67) and 6/7 (or 0.86), respectively. The effect size can be computed the same as above, but using the probabilities instead. Therefore, the relative risk is 1.28. Since rather large probabilities of passing were used, there is a large difference between relative risk and odds ratio. Had failure (a smaller probability) been used as the event (rather than passing), the difference between the two measures of effect size would not be so great.
While both measures are useful, they have different statistical uses. In medical research, the odds ratio is commonly used for case-control studies, as odds, but not probabilities, are usually estimated. Relative risk is commonly used in randomized controlled trials and cohort studies, but relative risk contributes to overestimations of the effectiveness of interventions.
Risk difference
The risk difference (RD), sometimes called absolute risk reduction, is simply the difference in risk (probability) of an event between two groups. It is a useful measure in experimental research, since RD tells you the extent to which an experimental interventions changes the probability of an event or outcome. Using the example above, the probabilities for those in the control group and treatment group passing is 2/3 (or 0.67) and 6/7 (or 0.86), respectively, and so the RD effect size is 0.86 − 0.67 = 0.19 (or 19%). RD is the superior measure for assessing effectiveness of interventions. They used the following example (about heights of men and women): "in any random pairing of young adult males and females, the probability of the male being taller than the female is .92, or in simpler terms yet, in 92 out of 100 blind dates among young adults, the male will be taller than the female", is a measure of how often the values in one distribution are larger than the values in a second distribution. Crucially, it does not require any assumptions about the shape or spread of the two distributions.
The sample estimate <math>d</math> is given by:
<math display="block">d = \frac{\sum_{i,j} [x_i > x_j] - [x_i < x_j]}{mn}</math>
where the two distributions are of size <math>n</math> and <math>m</math> with items <math>x_i</math> and <math>x_j</math>, respectively, and <math>[\cdot]</math> is the Iverson bracket, which is 1 when the contents are true and 0 when false.
<math>d</math> is linearly related to the Mann–Whitney U statistic; however, it captures the direction of the difference in its sign. Given the Mann–Whitney <math>U</math>, <math>d</math> is:
<math display="block">d = \frac{2U}{mn} - 1</math>
Cohen's g
One of simplest effect sizes for measuring how much a proportion differs from 50% is Cohen's g.