right|thumb|The result of fitting a set of data points with a quadratic function
thumb|Conic fitting a set of points using least-squares approximation
In regression analysis, least squares is a method to determine the best-fit model by minimizing the sum of the squared residuals—the differences between observed values and the values predicted by the model.
Least squares problems fall into two categories: linear or ordinary least squares and nonlinear least squares, depending on whether or not the model functions are linear in all unknowns. The linear least-squares problem occurs in statistical regression analysis; it has a closed-form solution. The nonlinear problem is usually solved by iterative refinement; at each iteration the system is approximated by a linear one, and thus the core calculation is similar in both cases.
Polynomial least squares describes the variance in a prediction of the dependent variable as a function of the independent variable and the deviations from the fitted curve.
When the observations come from an exponential family with identity as its natural sufficient statistics and mild-conditions are satisfied (e.g. for normal, exponential, Poisson and binomial distributions), standardized least-squares estimates and maximum-likelihood estimates are identical. The method of least squares can also be derived as a method of moments estimator.
History
The method was the culmination of several advances that took place during the course of the eighteenth century:
- The combination of different observations as being the best estimate of the true value; errors decrease with aggregation rather than increase, first appeared in Isaac Newton's work in 1671, though it went unpublished, and again in 1700. It was perhaps first expressed formally by Roger Cotes in 1722.
- The combination of different observations taken under the same conditions contrary to simply trying one's best to observe and record a single observation accurately. The approach was known as the method of averages. This approach was notably used by Newton while studying equinoxes in 1700, also writing down the first of the 'normal equations' known from ordinary least squares, Tobias Mayer while studying the librations of the Moon in 1750, and by Pierre-Simon Laplace in his work in explaining the differences in motion of Jupiter and Saturn in 1788.
- The combination of different observations taken under conditions. The method came to be known as the method of least absolute deviation. It was notably performed by Roger Joseph Boscovich in his work on the shape of the Earth in 1757 and by Pierre-Simon Laplace for the same problem in 1789 and 1799.
- The development of a criterion that can be evaluated to determine when the solution with the minimum error has been achieved. Laplace tried to specify a mathematical form of the probability density for the errors and define a method of estimation that minimizes the error of estimation. For this purpose, Laplace used a symmetric two-sided exponential distribution we now call Laplace distribution to model the error distribution, and used the sum of absolute deviation as error of estimation. He felt these to be the simplest assumptions he could make, and he had hoped to obtain the arithmetic mean as the best estimate. Instead, his estimator was the posterior median.
The method
thumb|upright=0.8|[[Carl Friedrich Gauss]]
The first clear and concise exposition of the method of least squares was published by Legendre in 1805. The technique is described as an algebraic procedure for fitting linear equations to data and Legendre demonstrates the new method by analyzing the same data as Laplace for the shape of the Earth. Within ten years after Legendre's publication, the method of least squares had been adopted as a standard tool in astronomy and geodesy in France, Italy, and Prussia, which constitutes an extraordinarily rapid acceptance of a scientific technique.
In 1809 Carl Friedrich Gauss published his method of calculating the orbits of celestial bodies. In that work he claimed to have been in possession of the method of least squares since 1795. This naturally led to a priority dispute with Legendre. However, to Gauss's credit, he went beyond Legendre and succeeded in connecting the method of least squares with the principles of probability and to the normal distribution. He had managed to complete Laplace's program of specifying a mathematical form of the probability density for the observations, depending on a finite number of unknown parameters, and define a method of estimation that minimizes the error of estimation. Gauss showed that the arithmetic mean is indeed the best estimate of the location parameter by changing both the probability density and the method of estimation. He then turned the problem around by asking what form the density should have and what method of estimation should be used to get the arithmetic mean as estimate of the location parameter. In this attempt, he invented the normal distribution.
An early demonstration of the strength of Gauss's method came when it was used to predict the future location of the newly discovered asteroid Ceres. On 1 January 1801, the Italian astronomer Giuseppe Piazzi discovered Ceres and was able to track its path for 40 days before it was lost in the glare of the Sun. Based on these data, astronomers desired to determine the location of Ceres after it emerged from behind the Sun without solving Kepler's complicated nonlinear equations of planetary motion. The only predictions that successfully allowed Hungarian astronomer Franz Xaver von Zach to relocate Ceres were those performed by the 24-year-old Gauss using least-squares analysis.
In 1810, after reading Gauss's work, Laplace, after proving the central limit theorem, used it to give a large sample justification for the method of least squares and the normal distribution. In 1822, Gauss was able to state that the least-squares approach to regression analysis is optimal in the sense that in a linear model where the errors have a mean of zero, are uncorrelated, normally distributed, and have equal variances, the best linear unbiased estimator of the coefficients is the least-squares estimator. An extended version of this result is known as the Gauss–Markov theorem.
The idea of least-squares analysis was also independently formulated by the American Robert Adrain in 1808. In the next two centuries workers in the theory of errors and in statistics found many different ways of implementing least squares.
Problem statement
The objective consists of adjusting the parameters of a model function to best fit a data set. A simple data set consists of n points (data pairs) <math>(x_i,y_i)\!</math>, i = 1, …, n, where <math>x_i\!</math> is an independent variable and <math>y_i\!</math> is a dependent variable whose value is found by observation. The model function has the form <math>f(x, \boldsymbol \beta)</math>, where m adjustable parameters are held in the vector <math>\boldsymbol \beta</math>. The goal is to find the parameter values for the model that "best" fits the data. The fit of a model to a data point is measured by its residual, defined as the difference between the observed value of the dependent variable and the value predicted by the model:
<math display="block">r_i = y_i - f(x_i, \boldsymbol \beta).</math>
thumb|The <math>y</math> values and residuals are plotted against corresponding <math>x</math> values. The random fluctuations about <math>r=0</math> indicate a linear model is appropriate.|250px
The least-squares method finds the optimal parameter values by minimizing the sum of squared residuals, <math>S</math>:
thumb|250px|The <math>y</math> values and residuals are plotted against the corresponding <math>x</math> values. The parabolic shape of the fluctuations about <math>r=0</math> indicates a parabolic model would be more appropriate.
If the residual points had some sort of a shape and were not randomly fluctuating, a linear model would not be appropriate. For example, if the residual plot had a parabolic shape as seen to the right, a parabolic model <math>(Y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + U_i)</math> would be appropriate for the data. The residuals for a parabolic model can be calculated via <math>r_i=y_i-\hat{\beta}_0-\hat{\beta}_1 x_i-\hat{\beta}_2 x_i^2</math>. An alternative approach is to fit a model by total least squares; this can be viewed as taking a pragmatic approach to balancing the effects of the different sources of error in formulating an objective function for use in model-fitting.
Solving the least squares problem
The minimum of the sum of squares is found by setting the gradient to zero. Since the model contains m parameters, there are m gradient equations:
<math display="block">\frac{\partial S}{\partial \beta_j}=2\sum_i r_i\frac{\partial r_i}{\partial \beta_j} = 0,\ j=1,\ldots,m,</math>
and since <math>r_i=y_i-f(x_i,\boldsymbol \beta)</math>, the gradient equations become
<math display="block">-2\sum_i r_i\frac{\partial f(x_i,\boldsymbol \beta)}{\partial \beta_j}=0,\ j=1,\ldots,m.</math>
The gradient equations apply to all least squares problems. Each particular problem requires particular expressions for the model and its partial derivatives.
Linear least squares
A regression model is a linear one when the model comprises a linear combination of the parameters, i.e.,
<math display="block"> f(x, \boldsymbol \beta) = \sum_{j = 1}^m \beta_j \phi_j(x),</math>
where the function <math>\phi_j</math> is a function of <math> x </math>.
<math display="block">L(D, \boldsymbol{\beta}) = \left\|Y - X\boldsymbol{\beta} \right\|^2
= (Y - X\boldsymbol{\beta})^\mathsf{T} (Y - X\boldsymbol{\beta})</math>
<math display="block">= Y^\mathsf{T}Y- 2Y^\mathsf{T}X\boldsymbol{\beta} + \boldsymbol{\beta}^\mathsf{T}X^\mathsf{T}X\boldsymbol{\beta}
</math>
The gradient of the loss is:
<math display="block">\frac{\partial L(D, \boldsymbol{\beta})}{\partial \boldsymbol{\beta
= \frac{\partial \left(Y^\mathsf{T}Y- 2Y^\mathsf{T}X\boldsymbol{\beta} + \boldsymbol{\beta}^\mathsf{T}X^\mathsf{T}X\boldsymbol{\beta}\right)}{\partial \boldsymbol{\beta
= -2X^\mathsf{T}Y + 2X^\mathsf{T}X\boldsymbol{\beta}</math>
Setting the gradient of the loss to zero and solving for <math>\boldsymbol{\beta}</math>, we get: Each experimental observation will contain some error, <math>\varepsilon</math>, and so we may specify an empirical model for our observations,
<math display="block"> y_i = kF_i + \varepsilon_i. </math>
There are many methods we might use to estimate the unknown parameter k. Since the n equations in the m variables in our data comprise an overdetermined system with one unknown and n equations, we estimate k using least squares. The sum of squares to be minimized is
- If the errors belong to a normal distribution, the least-squares estimators are also the maximum likelihood estimators in a linear model.
However, suppose the errors are not normally distributed. In that case, a central limit theorem often nonetheless implies that the parameter estimates will be approximately normally distributed so long as the sample is reasonably large. For this reason, given the important property that the error mean is independent of the independent variables, the distribution of the error term is not an important issue in regression analysis. Specifically, it is not typically important whether the error term follows a normal distribution.
Weighted least squares
thumb|250px|"Fanning out" effect of heteroscedasticity
A special case of generalized least squares called weighted least squares occurs when all the off-diagonal entries of Ω (the correlation matrix of the residuals) are null; the variances of the observations (along the covariance matrix diagonal) may still be unequal (heteroscedasticity). In simpler terms, heteroscedasticity is when the variance of <math>Y_i</math> depends on the value of <math>x_i</math> which causes the residual plot to create a "fanning out" effect towards larger or smaller <math>Y_i</math> values as seen in the residual plot to the right. On the other hand, homoscedasticity is assuming that the variance of <math>Y_i</math> and variance of <math>U_i</math> are equal.
Regularization
Tikhonov regularization
In some contexts, a regularized version of the least squares solution may be preferable. Tikhonov regularization (or ridge regression) adds a constraint that <math>\left\|\beta\right\|_2^2</math>, the squared <math>\ell_2</math>-norm of the parameter vector, is not greater than a given value to the least squares formulation, leading to a constrained minimization problem. This is equivalent to the unconstrained minimization problem where the objective function is the residual sum of squares plus a penalty term <math>\alpha \left\|\beta\right\|_2^2</math> and <math>\alpha</math> is a tuning parameter (this is the Lagrangian form of the constrained minimization problem).
In a Bayesian context, this is equivalent to placing a zero-mean normally distributed prior on the parameter vector.
Lasso method
An alternative regularized version of least squares is Lasso (least absolute shrinkage and selection operator), which uses the constraint that <math>\|\beta\|_1</math>, the L<sub>1</sub>-norm of the parameter vector, is no greater than a given value. (One can show like above using Lagrange multipliers that this is equivalent to an unconstrained minimization of the least-squares penalty with <math>\alpha\|\beta\|_1</math> added.) In a Bayesian context, this is equivalent to placing a zero-mean Laplace prior distribution on the parameter vector. The optimization problem may be solved using quadratic programming or more general convex optimization methods, as well as by specific algorithms such as the least angle regression algorithm.
One of the prime differences between Lasso and ridge regression is that in ridge regression, as the penalty is increased, all parameters are reduced while still remaining non-zero, while in Lasso, increasing the penalty will cause more and more of the parameters to be driven to zero. This is an advantage of Lasso over ridge regression, as driving parameters to zero deselects the features from the regression. Thus, Lasso automatically selects more relevant features and discards the others, whereas Ridge regression never fully discards any features. Some feature selection techniques are developed based on the LASSO including Bolasso which bootstraps samples, and FeaLect which analyzes the regression coefficients corresponding to different values of <math>\alpha</math> to score all the features.
The L<sup>1</sup>-regularized formulation is useful in some contexts due to its tendency to prefer solutions where more parameters are zero, which gives solutions that depend on fewer variables.