Linear Statistical Models: Regression

Robust Regression


The term "robust regression" can be used to mean two slightly different things. The first usage should really be called regression with robust standard errors. In regression with robust standard errors the estimates of the regression coefficients are the same as in the standard OLS linear regression but the estimates of the standard errors are more robust to failure to meet assumptions concerning normality and homogeneity of variance of the residuals.

The second way the term robust regression is used involves both robust estimation of the regression coefficients and the standard errors. This approach is useful in situations where the are large outliers and observations with large leverage values.

OLS Example

This example uses Agresti & Finlay's crime data including Washington DC.

use http://www.philender.com/courses/data/crime, clear

regress crime pctmetro pctwhite pcths poverty single

  Source |       SS       df       MS                  Number of obs =      51
---------+------------------------------               F(  5,    45) =   50.94
   Model |  8267856.33     5  1653571.27               Prob > F      =  0.0000
Residual |  1460618.42    45   32458.187               R-squared     =  0.8499
---------+------------------------------               Adj R-squared =  0.8332
   Total |  9728474.75    50  194569.495               Root MSE      =  180.16

------------------------------------------------------------------------------
   crime |      Coef.   Std. Err.       t     P>|t|       [95% Conf. Interval]
---------+--------------------------------------------------------------------
pctmetro |   7.608732   1.295111      5.875   0.000       5.000243    10.21722
pctwhite |  -4.486454   2.777134     -1.615   0.113      -10.07989    1.106981
   pcths |   8.657511   7.826444      1.106   0.275      -7.105756    24.42078
 poverty |   26.24974   11.08248      2.369   0.022       3.928483      48.571
  single |   109.4519   20.35401      5.377   0.000       68.45678    150.4469
   _cons |  -1796.322   668.6139     -2.687   0.010      -3142.979   -449.6646
------------------------------------------------------------------------------
Regression with Robust Standard Errors

regress crime pctmetro pctwhite pcths poverty single, vce(robust)

Regression with robust standard errors                 Number of obs =      51
                                                       F(  5,    45) =   26.52
                                                       Prob > F      =  0.0000
                                                       R-squared     =  0.8499
                                                       Root MSE      =  180.16

------------------------------------------------------------------------------
         |               Robust
   crime |      Coef.   Std. Err.       t     P>|t|       [95% Conf. Interval]
---------+--------------------------------------------------------------------
pctmetro |   7.608732    1.64355      4.629   0.000       4.298451    10.91901
pctwhite |  -4.486454   3.434026     -1.306   0.198      -11.40294     2.43003
   pcths |   8.657511    8.08704      1.071   0.290      -7.630623    24.94565
 poverty |   26.24974   11.46218      2.290   0.027       3.163728    49.33575
  single |   109.4519   24.42208      4.482   0.000       60.26327    158.6404
   _cons |  -1796.322   750.7679     -2.393   0.021      -3308.446   -284.1978
------------------------------------------------------------------------------

The vce(robust) option in Stata computes Huber-White robust estimates of the standard errors. Notice that:

  • the estimates of the regression coefficients are the same as in the first analysis,
  • the overall F is much lower than in the first analysis,
  • and the individual standard errors are larger than the first analysis.

    Stata's rreg command estimates a robust regression using iteratively reweighted least squares. The procedure uses two kinds of weighting, Huber weights and Biweights originated by Tukey.

    Robust Regression

    rreg crime pctmetro pctwhite pcths poverty single, genwt(w)
    
       Huber iteration 1:  maximum difference in weights = .45288397
       Huber iteration 2:  maximum difference in weights = .12866865
       Huber iteration 3:  maximum difference in weights = .03605028
    Biweight iteration 4:  maximum difference in weights = .22959899
    Biweight iteration 5:  maximum difference in weights = .10386227
    Biweight iteration 6:  maximum difference in weights = .07521054
    Biweight iteration 7:  maximum difference in weights = .07310603
    Biweight iteration 8:  maximum difference in weights = .01145342
    Biweight iteration 9:  maximum difference in weights = .00355889
    
    Robust regression estimates                            Number of obs =      49
                                                           F(  5,    43) =   37.82
                                                           Prob > F      =  0.0000
    
    ------------------------------------------------------------------------------
       crime |      Coef.   Std. Err.       t     P>|t|       [95% Conf. Interval]
    ---------+--------------------------------------------------------------------
    pctmetro |   5.075429   1.016701      4.992   0.000       3.025056    7.125802
    pctwhite |  -12.95398   3.747957     -3.456   0.001      -20.51245   -5.395501
       pcths |   6.152996   6.637425      0.927   0.359      -7.232648    19.53864
     poverty |   20.93053   8.687253      2.409   0.020       3.411011    38.45004
      single |   68.82676   21.62407      3.183   0.003       25.21765    112.4359
       _cons |  -176.2423   586.6675     -0.300   0.765       -1359.37    1006.885
    ------------------------------------------------------------------------------
    
    predict leverage, leverage

    This robust regression algorithm uses iteratively reweighted least squares to estimate both the regression coefficients and the standard errors. The procedure assigns weights to each of the observations. Those observations with high leverage or influence receive lower weights. The values for crime, leverage and weights for each of the observations are listed below. Note that Mississippi received a weight of zero and that both Hawaii and Washington DC were set to missing.

    . list state crime leverage w
    
             state     crime   leverage           w 
      1.        ak       761   .2662776   .99961828  
      2.        al       780   .0945071   .99999451  
      3.        ar       593   .1086863    .9839592  
      4.        az       715   .0667885   .99487205  
      5.        ca      1078   .1113958   .96336903  
      6.        co       567   .0956547   .99385151  
      7.        ct       456   .0896476   .97898508  
      8.        de       686   .0612597   .99990974  
      9.        fl      1206   .1280566   .22272679  
     10.        ga       723    .144481   .89110023  
     11.        hi       261   .6788905           .  
     12.        ia       326   .0632136   .78003713  
     13.        id       282   .0911653   .89202312  
     14.        il       960   .0315598   .75733757  
     15.        in       489   .0443533   .99924468  
     16.        ks       496   .0708015   .92429011  
     17.        ky       463   .1473426   .99547118  
     18.        la      1062   .1842347   .61529609  
     19.        ma       805   .0782307   .77309056  
     20.        md       998   .1033071   .96012278  
     21.        me       126   .1107199   .99028947  
     22.        mi       792   .0477468    .9766335  
     23.        mn       327   .0646282   .97835866  
     24.        mo       744   .0307904   .84930436  
     25.        ms       434   .2337579           0  
     26.        mt       178   .1331145    .9182153  
     27.        nc       679   .0840172   .99994106  
     28.        nd        82   .0773202   .99830127  
     29.        ne       339   .0572846   .92786537  
     30.        nh       138   .0575562   .97690802  
     31.        nj       627   .1031666   .99805661  
     32.        nm       930     .09128   .83527927  
     33.        nv       875   .1066225   .78560769  
     34.        ny      1074   .0509723   .93575196  
     35.        oh       504   .0420508   .91836198  
     36.        ok       635   .1136476    .9767223  
     37.        or       503   .0496652   .99940287  
     38.        pa       418   .0610071   .96063045  
     39.        ri       402    .174163   .92255605  
     40.        sc      1023   .0745018   .96795055  
     41.        sd       208   .0866775   .99443631  
     42.        tn       766   .0798104   .96640667  
     43.        tx       762   .0547513   .99985684  
     44.        ut       301   .1070068   .91502316  
     45.        va       372    .093345   .74250873  
     46.        vt       114   .1568013   .99276346  
     47.        wa       515   .0862924   .88308747  
     48.        wi       264   .0295518   .83338417  
     49.        wv       208   .1868992   .94637392  
     50.        wy       286   .1233388   .99740407  
     51.        dc      2922   .5716585           .
    Bootstrap Standard Errors for Regression

    Another way to obtain more robust standard errors is through the use of the bootstrap. The technique involves taking a large number of random samples (with replacement) from our dataset, estimating the regression model for each sample, saving the regression coefficients from each sample and then computing the standard deviation (standard error) for each coefficient.


    Linear Statistical Models Course

    Phil Ender, 17sep10, 14jan00