Linear Statistical Models: Regression

The Regression Zoo

There is more than one type of regression, listed below are of some of the types of regression that you are most likely to encounter:

• ordinary least squares regression (OLS) - The type of 'plain vanilla' regression that we have been doing in class in which parameters are estimated such that the sum of squared residuals is a minimum.

• anova - Anova can be thought of as linear regression with coded categorical independent variables.

• ancova - Ancova analyses combine continuous and categorical predictor variables. One or more of the continuous predictors is considered to be a nuisance variable.

• ati - Ati analyses also involve continuous and categorical predictor variables. In ati the continuous predictor variables are considered to be of direct interest.

• weighted least squares (WLS) - WLS encompases various schemes for weighting observations to reduce the effects of heteroscedasticity. WLS minimizes a weighted sum of squares of differences between the DV and its predicted value.

• maximum likelihood estimation (ML) - An estimation procedure, different from least squares, that iteratively estimates the parameters that make the observations most likely.

• robust regression - Methods for obtaining parameter estimates that are less sensitive to outliers and skewness than is OLS.

• logistic regression - A maximum likelihood procedure for a dichotomous dependent variable. The dependent variable is a 'true' qualitative dichotomous variable. Logistic regression makes use of the binomial distribution in making its parameter estimates. [see also logit]

• logit - The logit(P) = log(odds) which is used in logistic regression. [see also logistic regression]

• multinomial logistic regression - In many ways similar to logistic regression except that the dependent variable can take on more than two values. [same as polytomous logistic regression]

• ordinal logistic regression - A generalization of multinomial logistic regression for the case in which the categorical response variable is ordered (ordinal).

• probit - A maximum likelihood procedure for a dichotomous dependent variable. The dependent variable has an underlying quantitative dimension and makes use of the inverse normal cumulative probability distribution in making its parameter estimates.

• instrumental variable regression - Regression estimation method to deal with endogenous predictors.

• regression with censored values - [see also tobit regression]

• tobit regression - One type of regression with censored values. [see also regression with censored values]

• median regression - An estimation procedure that seeks to minimize the absolute residuals, rather than the sum of squares of the residuals as in ordinary regression. Median regression is less sensitive to outliers and skewness than is OLS. It is an instance of quantile regression in which the .5 quantile (med ian) is used. [see also quantile regression]

• poisson regression - A maximum likelihood procedure for use with nonnegative count variables. The procedure uses the poisson probability distribution in making its parameter estimates.

• negative binomial regression - A maximum likelihood procedure for use with nonnegative count variables. In this model, the count variable is believed to be generated by a poisson-like process but with greater variability, i.e., overdispersion. The procedure uses the negative binomial probability distribution in making its parameter estimates.

• two-stage least squares regression (2SLS) - (1) a stage in which new dependent or endogenous variables are created to substitute for the original ones, and (2) a stage in which the regression is computed in OLS fashion, but using the newly created variables. The purpose of the first stage is to create new dependent variables which do not violate OLS regression's recursivity assumption.

• polytomous logistic regression - In many ways similar to logistic regression except that the dependent variable can take on more than two values. [same as mutinomial logistic regression]

• curvilinear regression - Contrary to how it sounds, curvilinear regression uses a linear model to fit a curved line to data points. Curvilinear regression makes use of various transformations of variables to achieve its fit. An example of a curvilinear model is Y = b₀ + b₁X₁ + b₂X₂, where X₂ = X₁². See polynomial regression.

• polynomial regression - Polynomial regression is another name for curvilinear regression.

• nonlinear regression - Fits arbitrary nonlinear functions to the dependent variable. An example of a nonlinear model is Y = b₀(1 - e^-b₁X).

• quantile regression - An estimation procedure that seeks to minimize the absolute residuals, rather than the sum of squares of the residuals as in ordinary regression. Quantile regression is less sensitive to outliers and skewness than is OLS. Quantile regression is a generalization of median regression in which any quantile can be selected. [see also median regression]

• iteratively reweighted least squares (IRLS) - One of the techniques used to estimate parameters in robust regression [see also robust regression]

Linear Statistical Models Course