The term "robust regression" can be used to mean two slightly different things. The first usage should really be called regression with robust standard errors. In regression with robust standard errors the estimates of the regression coefficients are the same as in the standard OLS linear regression but the estimates of the standard errors are more robust to failure to meet assumptions concerning normality and homogeneity of variance of the residuals.
The second way the term robust regression is used involves both robust estimation of the regression coefficients and the standard errors. This approach is useful in situations where the are large outliers and observations with large leverage values.
OLS Example
This example uses Agresti & Finlay's crime data including Washington DC.
use http://www.philender.com/courses/data/crime, clear regress crime pctmetro pctwhite pcths poverty single Source | SS df MS Number of obs = 51 ---------+------------------------------ F( 5, 45) = 50.94 Model | 8267856.33 5 1653571.27 Prob > F = 0.0000 Residual | 1460618.42 45 32458.187 R-squared = 0.8499 ---------+------------------------------ Adj R-squared = 0.8332 Total | 9728474.75 50 194569.495 Root MSE = 180.16 ------------------------------------------------------------------------------ crime | Coef. Std. Err. t P>|t| [95% Conf. Interval] ---------+-------------------------------------------------------------------- pctmetro | 7.608732 1.295111 5.875 0.000 5.000243 10.21722 pctwhite | -4.486454 2.777134 -1.615 0.113 -10.07989 1.106981 pcths | 8.657511 7.826444 1.106 0.275 -7.105756 24.42078 poverty | 26.24974 11.08248 2.369 0.022 3.928483 48.571 single | 109.4519 20.35401 5.377 0.000 68.45678 150.4469 _cons | -1796.322 668.6139 -2.687 0.010 -3142.979 -449.6646 ------------------------------------------------------------------------------Regression with Robust Standard Errors
regress crime pctmetro pctwhite pcths poverty single, vce(robust) Regression with robust standard errors Number of obs = 51 F( 5, 45) = 26.52 Prob > F = 0.0000 R-squared = 0.8499 Root MSE = 180.16 ------------------------------------------------------------------------------ | Robust crime | Coef. Std. Err. t P>|t| [95% Conf. Interval] ---------+-------------------------------------------------------------------- pctmetro | 7.608732 1.64355 4.629 0.000 4.298451 10.91901 pctwhite | -4.486454 3.434026 -1.306 0.198 -11.40294 2.43003 pcths | 8.657511 8.08704 1.071 0.290 -7.630623 24.94565 poverty | 26.24974 11.46218 2.290 0.027 3.163728 49.33575 single | 109.4519 24.42208 4.482 0.000 60.26327 158.6404 _cons | -1796.322 750.7679 -2.393 0.021 -3308.446 -284.1978 ------------------------------------------------------------------------------
The vce(robust) option in Stata computes Huber-White robust estimates of the standard errors. Notice that:
Stata's rreg command estimates a robust regression using iteratively reweighted least squares. The procedure uses two kinds of weighting, Huber weights and Biweights originated by Tukey.
Robust Regression
rreg crime pctmetro pctwhite pcths poverty single, genwt(w) Huber iteration 1: maximum difference in weights = .45288397 Huber iteration 2: maximum difference in weights = .12866865 Huber iteration 3: maximum difference in weights = .03605028 Biweight iteration 4: maximum difference in weights = .22959899 Biweight iteration 5: maximum difference in weights = .10386227 Biweight iteration 6: maximum difference in weights = .07521054 Biweight iteration 7: maximum difference in weights = .07310603 Biweight iteration 8: maximum difference in weights = .01145342 Biweight iteration 9: maximum difference in weights = .00355889 Robust regression estimates Number of obs = 49 F( 5, 43) = 37.82 Prob > F = 0.0000 ------------------------------------------------------------------------------ crime | Coef. Std. Err. t P>|t| [95% Conf. Interval] ---------+-------------------------------------------------------------------- pctmetro | 5.075429 1.016701 4.992 0.000 3.025056 7.125802 pctwhite | -12.95398 3.747957 -3.456 0.001 -20.51245 -5.395501 pcths | 6.152996 6.637425 0.927 0.359 -7.232648 19.53864 poverty | 20.93053 8.687253 2.409 0.020 3.411011 38.45004 single | 68.82676 21.62407 3.183 0.003 25.21765 112.4359 _cons | -176.2423 586.6675 -0.300 0.765 -1359.37 1006.885 ------------------------------------------------------------------------------ predict leverage, leverage
This robust regression algorithm uses iteratively reweighted least squares to estimate both the regression coefficients and the standard errors. The procedure assigns weights to each of the observations. Those observations with high leverage or influence receive lower weights. The values for crime, leverage and weights for each of the observations are listed below. Note that Mississippi received a weight of zero and that both Hawaii and Washington DC were set to missing.
. list state crime leverage w state crime leverage w 1. ak 761 .2662776 .99961828 2. al 780 .0945071 .99999451 3. ar 593 .1086863 .9839592 4. az 715 .0667885 .99487205 5. ca 1078 .1113958 .96336903 6. co 567 .0956547 .99385151 7. ct 456 .0896476 .97898508 8. de 686 .0612597 .99990974 9. fl 1206 .1280566 .22272679 10. ga 723 .144481 .89110023 11. hi 261 .6788905 . 12. ia 326 .0632136 .78003713 13. id 282 .0911653 .89202312 14. il 960 .0315598 .75733757 15. in 489 .0443533 .99924468 16. ks 496 .0708015 .92429011 17. ky 463 .1473426 .99547118 18. la 1062 .1842347 .61529609 19. ma 805 .0782307 .77309056 20. md 998 .1033071 .96012278 21. me 126 .1107199 .99028947 22. mi 792 .0477468 .9766335 23. mn 327 .0646282 .97835866 24. mo 744 .0307904 .84930436 25. ms 434 .2337579 0 26. mt 178 .1331145 .9182153 27. nc 679 .0840172 .99994106 28. nd 82 .0773202 .99830127 29. ne 339 .0572846 .92786537 30. nh 138 .0575562 .97690802 31. nj 627 .1031666 .99805661 32. nm 930 .09128 .83527927 33. nv 875 .1066225 .78560769 34. ny 1074 .0509723 .93575196 35. oh 504 .0420508 .91836198 36. ok 635 .1136476 .9767223 37. or 503 .0496652 .99940287 38. pa 418 .0610071 .96063045 39. ri 402 .174163 .92255605 40. sc 1023 .0745018 .96795055 41. sd 208 .0866775 .99443631 42. tn 766 .0798104 .96640667 43. tx 762 .0547513 .99985684 44. ut 301 .1070068 .91502316 45. va 372 .093345 .74250873 46. vt 114 .1568013 .99276346 47. wa 515 .0862924 .88308747 48. wi 264 .0295518 .83338417 49. wv 208 .1868992 .94637392 50. wy 286 .1233388 .99740407 51. dc 2922 .5716585 .Bootstrap Standard Errors for Regression
Another way to obtain more robust standard errors is through the use of the bootstrap. The technique involves taking a large number of random samples (with replacement) from our dataset, estimating the regression model for each sample, saving the regression coefficients from each sample and then computing the standard deviation (standard error) for each coefficient.
use http://www.philender.com/courses/data/crime, clear drop if sid==51 regress murder poverty pctmetro pcths single Source | SS df MS Number of obs = 50 -------------+------------------------------ F( 4, 45) = 39.88 Model | 606.611746 4 151.652936 Prob > F = 0.0000 Residual | 171.137027 45 3.80304505 R-squared = 0.7800 -------------+------------------------------ Adj R-squared = 0.7604 Total | 777.748773 49 15.8724239 Root MSE = 1.9501 ------------------------------------------------------------------------------ murder | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------- poverty | .2237151 .1185554 1.89 0.066 -.0150679 .462498 pctmetro | .0534333 .013795 3.87 0.000 .0256488 .0812178 pcths | -.1938711 .0812756 -2.39 0.021 -.3575685 -.0301737 single | 1.388337 .2217525 6.26 0.000 .9417051 1.83497 _cons | -.0044014 7.478803 -0.00 1.000 -15.06748 15.05868 ------------------------------------------------------------------------------ regress murder poverty pctmetro pcths single, vce(robust) Regression with robust standard errors Number of obs = 50 F( 4, 45) = 46.13 Prob > F = 0.0000 R-squared = 0.7800 Root MSE = 1.9501 ------------------------------------------------------------------------------ | Robust murder | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------- poverty | .2237151 .130537 1.71 0.093 -.0392 .4866301 pctmetro | .0534333 .0114609 4.66 0.000 .0303498 .0765168 pcths | -.1938711 .0878619 -2.21 0.032 -.370834 -.0169082 single | 1.388337 .2013124 6.90 0.000 .9828735 1.793801 _cons | -.0044014 8.286424 -0.00 1.000 -16.69412 16.68531 ------------------------------------------------------------------------------ /* bootstrap standard errors */ regress murder poverty pctmetro pcths single, vce(bootstrap, reps(200)) (running regress on estimation sample) Bootstrap replications (200) ----+--- 1 ---+--- 2 ---+--- 3 ---+--- 4 ---+--- 5 .................................................. 50 .................................................. 100 .................................................. 150 .................................................. 200 Linear regression Number of obs = 50 Replications = 200 Wald chi2(4) = 193.86 Prob > chi2 = 0.0000 R-squared = 0.7800 Adj R-squared = 0.7604 Root MSE = 1.9501 ------------------------------------------------------------------------------ | Observed Bootstrap Normal-based murder | Coef. Std. Err. z P>|z| [95% Conf. Interval] -------------+---------------------------------------------------------------- poverty | .2237151 .1267078 1.77 0.077 -.0246277 .4720578 pctmetro | .0534333 .0140405 3.81 0.000 .0259144 .0809522 pcths | -.1938711 .0837606 -2.31 0.021 -.3580389 -.0297032 single | 1.388337 .2175473 6.38 0.000 .9619527 1.814722 _cons | -.0044014 7.794268 -0.00 1.000 -15.28089 15.27208 ------------------------------------------------------------------------------
Linear Statistical Models Course
Phil Ender, 17sep10, 14jan00