The term "robust regression" can be used to mean two slightly different things. The first usage should really be called regression with robust standard errors. In regression with robust standard errors the estimates of the regression coefficients are the same as in the standard OLS linear regression but the estimates of the standard errors are more robust to failure to meet assumptions concerning normality and homogeneity of variance of the residuals.
The second way the term robust regression is used involves both robust estimation of the regression coefficients and the standard errors. This approach is useful in situations where the are large outliers and observations with large leverage values.
OLS Example
This example uses Agresti & Finlay's crime data including Washington DC.
use http://www.philender.com/courses/data/crime, clear regress crime pctmetro pctwhite pcths poverty single Source | SS df MS Number of obs = 51 ---------+------------------------------ F( 5, 45) = 50.94 Model | 8267856.33 5 1653571.27 Prob > F = 0.0000 Residual | 1460618.42 45 32458.187 R-squared = 0.8499 ---------+------------------------------ Adj R-squared = 0.8332 Total | 9728474.75 50 194569.495 Root MSE = 180.16 ------------------------------------------------------------------------------ crime | Coef. Std. Err. t P>|t| [95% Conf. Interval] ---------+-------------------------------------------------------------------- pctmetro | 7.608732 1.295111 5.875 0.000 5.000243 10.21722 pctwhite | -4.486454 2.777134 -1.615 0.113 -10.07989 1.106981 pcths | 8.657511 7.826444 1.106 0.275 -7.105756 24.42078 poverty | 26.24974 11.08248 2.369 0.022 3.928483 48.571 single | 109.4519 20.35401 5.377 0.000 68.45678 150.4469 _cons | -1796.322 668.6139 -2.687 0.010 -3142.979 -449.6646 ------------------------------------------------------------------------------Regression with Robust Standard Errors
regress crime pctmetro pctwhite pcths poverty single, vce(robust)
Regression with robust standard errors Number of obs = 51
F( 5, 45) = 26.52
Prob > F = 0.0000
R-squared = 0.8499
Root MSE = 180.16
------------------------------------------------------------------------------
| Robust
crime | Coef. Std. Err. t P>|t| [95% Conf. Interval]
---------+--------------------------------------------------------------------
pctmetro | 7.608732 1.64355 4.629 0.000 4.298451 10.91901
pctwhite | -4.486454 3.434026 -1.306 0.198 -11.40294 2.43003
pcths | 8.657511 8.08704 1.071 0.290 -7.630623 24.94565
poverty | 26.24974 11.46218 2.290 0.027 3.163728 49.33575
single | 109.4519 24.42208 4.482 0.000 60.26327 158.6404
_cons | -1796.322 750.7679 -2.393 0.021 -3308.446 -284.1978
------------------------------------------------------------------------------The vce(robust) option in Stata computes Huber-White robust estimates of the standard errors. Notice that:
Stata's rreg command estimates a robust regression using iteratively reweighted least squares. The procedure uses two kinds of weighting, Huber weights and Biweights originated by Tukey.
Robust Regression
rreg crime pctmetro pctwhite pcths poverty single, genwt(w)
Huber iteration 1: maximum difference in weights = .45288397
Huber iteration 2: maximum difference in weights = .12866865
Huber iteration 3: maximum difference in weights = .03605028
Biweight iteration 4: maximum difference in weights = .22959899
Biweight iteration 5: maximum difference in weights = .10386227
Biweight iteration 6: maximum difference in weights = .07521054
Biweight iteration 7: maximum difference in weights = .07310603
Biweight iteration 8: maximum difference in weights = .01145342
Biweight iteration 9: maximum difference in weights = .00355889
Robust regression estimates Number of obs = 49
F( 5, 43) = 37.82
Prob > F = 0.0000
------------------------------------------------------------------------------
crime | Coef. Std. Err. t P>|t| [95% Conf. Interval]
---------+--------------------------------------------------------------------
pctmetro | 5.075429 1.016701 4.992 0.000 3.025056 7.125802
pctwhite | -12.95398 3.747957 -3.456 0.001 -20.51245 -5.395501
pcths | 6.152996 6.637425 0.927 0.359 -7.232648 19.53864
poverty | 20.93053 8.687253 2.409 0.020 3.411011 38.45004
single | 68.82676 21.62407 3.183 0.003 25.21765 112.4359
_cons | -176.2423 586.6675 -0.300 0.765 -1359.37 1006.885
------------------------------------------------------------------------------
predict leverage, leverageThis robust regression algorithm uses iteratively reweighted least squares to estimate both the regression coefficients and the standard errors. The procedure assigns weights to each of the observations. Those observations with high leverage or influence receive lower weights. The values for crime, leverage and weights for each of the observations are listed below. Note that Mississippi received a weight of zero and that both Hawaii and Washington DC were set to missing.
. list state crime leverage w
state crime leverage w
1. ak 761 .2662776 .99961828
2. al 780 .0945071 .99999451
3. ar 593 .1086863 .9839592
4. az 715 .0667885 .99487205
5. ca 1078 .1113958 .96336903
6. co 567 .0956547 .99385151
7. ct 456 .0896476 .97898508
8. de 686 .0612597 .99990974
9. fl 1206 .1280566 .22272679
10. ga 723 .144481 .89110023
11. hi 261 .6788905 .
12. ia 326 .0632136 .78003713
13. id 282 .0911653 .89202312
14. il 960 .0315598 .75733757
15. in 489 .0443533 .99924468
16. ks 496 .0708015 .92429011
17. ky 463 .1473426 .99547118
18. la 1062 .1842347 .61529609
19. ma 805 .0782307 .77309056
20. md 998 .1033071 .96012278
21. me 126 .1107199 .99028947
22. mi 792 .0477468 .9766335
23. mn 327 .0646282 .97835866
24. mo 744 .0307904 .84930436
25. ms 434 .2337579 0
26. mt 178 .1331145 .9182153
27. nc 679 .0840172 .99994106
28. nd 82 .0773202 .99830127
29. ne 339 .0572846 .92786537
30. nh 138 .0575562 .97690802
31. nj 627 .1031666 .99805661
32. nm 930 .09128 .83527927
33. nv 875 .1066225 .78560769
34. ny 1074 .0509723 .93575196
35. oh 504 .0420508 .91836198
36. ok 635 .1136476 .9767223
37. or 503 .0496652 .99940287
38. pa 418 .0610071 .96063045
39. ri 402 .174163 .92255605
40. sc 1023 .0745018 .96795055
41. sd 208 .0866775 .99443631
42. tn 766 .0798104 .96640667
43. tx 762 .0547513 .99985684
44. ut 301 .1070068 .91502316
45. va 372 .093345 .74250873
46. vt 114 .1568013 .99276346
47. wa 515 .0862924 .88308747
48. wi 264 .0295518 .83338417
49. wv 208 .1868992 .94637392
50. wy 286 .1233388 .99740407
51. dc 2922 .5716585 .
Bootstrap Standard Errors for RegressionAnother way to obtain more robust standard errors is through the use of the bootstrap. The technique involves taking a large number of random samples (with replacement) from our dataset, estimating the regression model for each sample, saving the regression coefficients from each sample and then computing the standard deviation (standard error) for each coefficient.
use http://www.philender.com/courses/data/crime, clear
drop if sid==51
regress murder poverty pctmetro pcths single
Source | SS df MS Number of obs = 50
-------------+------------------------------ F( 4, 45) = 39.88
Model | 606.611746 4 151.652936 Prob > F = 0.0000
Residual | 171.137027 45 3.80304505 R-squared = 0.7800
-------------+------------------------------ Adj R-squared = 0.7604
Total | 777.748773 49 15.8724239 Root MSE = 1.9501
------------------------------------------------------------------------------
murder | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
poverty | .2237151 .1185554 1.89 0.066 -.0150679 .462498
pctmetro | .0534333 .013795 3.87 0.000 .0256488 .0812178
pcths | -.1938711 .0812756 -2.39 0.021 -.3575685 -.0301737
single | 1.388337 .2217525 6.26 0.000 .9417051 1.83497
_cons | -.0044014 7.478803 -0.00 1.000 -15.06748 15.05868
------------------------------------------------------------------------------
regress murder poverty pctmetro pcths single, vce(robust)
Regression with robust standard errors Number of obs = 50
F( 4, 45) = 46.13
Prob > F = 0.0000
R-squared = 0.7800
Root MSE = 1.9501
------------------------------------------------------------------------------
| Robust
murder | Coef. Std. Err. t P>|t| [95% Conf. Interval]
-------------+----------------------------------------------------------------
poverty | .2237151 .130537 1.71 0.093 -.0392 .4866301
pctmetro | .0534333 .0114609 4.66 0.000 .0303498 .0765168
pcths | -.1938711 .0878619 -2.21 0.032 -.370834 -.0169082
single | 1.388337 .2013124 6.90 0.000 .9828735 1.793801
_cons | -.0044014 8.286424 -0.00 1.000 -16.69412 16.68531
------------------------------------------------------------------------------
/* bootstrap standard errors */
regress murder poverty pctmetro pcths single, vce(bootstrap, reps(200))
(running regress on estimation sample)
Bootstrap replications (200)
----+--- 1 ---+--- 2 ---+--- 3 ---+--- 4 ---+--- 5
.................................................. 50
.................................................. 100
.................................................. 150
.................................................. 200
Linear regression Number of obs = 50
Replications = 200
Wald chi2(4) = 193.86
Prob > chi2 = 0.0000
R-squared = 0.7800
Adj R-squared = 0.7604
Root MSE = 1.9501
------------------------------------------------------------------------------
| Observed Bootstrap Normal-based
murder | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
poverty | .2237151 .1267078 1.77 0.077 -.0246277 .4720578
pctmetro | .0534333 .0140405 3.81 0.000 .0259144 .0809522
pcths | -.1938711 .0837606 -2.31 0.021 -.3580389 -.0297032
single | 1.388337 .2175473 6.38 0.000 .9619527 1.814722
_cons | -.0044014 7.794268 -0.00 1.000 -15.28089 15.27208
------------------------------------------------------------------------------
Linear Statistical Models Course
Phil Ender, 17sep10, 14jan00