Permutation Tests

Introduction to Research Design and Statistics

Permutation Tests

For most hypothesis tests, we start with the assumptions and work forward to derive the sampling distribution of the test statistic under the null hypothesis. For permutation tests we will reverse the procedure, since the sampling distribution involves the permutations which give the procedure its name and are the key theoretical issue in understanding the test.

In mathematics, a permutation is a reordering of the numbers 1, ..., n. For example,

are all permutations of the numbers 1 through 6 (note that this includes the standard order in first line). There are n! (n factorial) permutations of n objects. In this case, 6! = 720, so you can see why they aren't all written out here.

The term permutation tests refers to rearrangements of the data. The null hypothesis of the test specifies that the permutations are all equally likely. A concise way to say this is that the distribution of the data under the null hypothesis satisfies exchangeability.

The sampling distribution of the test statistic under the null hypothesis is computed by forming all or many of the permutations, calculating the test statistic for each and considering these values all equally likely.

Consider the following two group example.

Group 1: 55 58 60
Group 2: 12 22 34

Clearly a t-test will not work with this small example, however a permutation test will do the job.

Here are the steps we will follow to use a permutation test to analyze the differences between the two groups.

Analyze the problem
Choose a test statistic
Resample and recompute the test statistic
Reject or fail to reject the null hypothesis

For the original order the sum for Group 1 is 173. In this example, if the groups were truly equal then ramdomly moving the observations among the groups would make no difference in the sum for Group 1. Some of the sums would be a little larger than the original sum and some would be a bit smaller. For the six observations there are 720 permutations of which there are 20 distinct combinations for which we can compute the sum of Group 1.

Order | Group 1  | Group 2  | Sum 
  1   | 55 58 60 | 12 22 34 | 173
  2   | 55 58 12 | 60 22 34 | 125
  3   | 55 58 22 | 12 60 34 | 135
  4   | 55 58 34 | 12 22 60 | 148
  5   | 55 12 60 | 58 22 34 | 127
  6   | 55 22 60 | 12 58 34 | 137
  7   | 55 34 60 | 12 22 58 | 149
  8   | 12 58 60 | 55 22 34 | 130
  9   | 22 58 60 | 12 55 34 | 140
 10   | 34 58 60 | 12 22 55 | 152
 11   | 12 22 60 | 55 58 34 |  94
 12   | 12 58 22 | 55 60 34 |  92
 13   | 55 12 22 | 12 55 58 |  89
 14   | 12 34 60 | 55 22 58 | 106
 15   | 12 58 34 | 55 22 60 | 104
 16   | 55 12 34 | 58 22 60 | 101
 17   | 22 34 60 | 12 55 58 | 116
 18   | 22 58 34 | 12 55 60 | 114
 19   | 55 22 34 | 12 58 60 | 111
 20   | 12 22 34 | 55 58 60 |  68

Of these 20 different orderings only one has a Group 1 sum that greater than or equal to the Group 1 sum from our original ordering. Therefor the probability that a sum this large or larger would occur by chance alone is 1/20 = 0.05 and can be considered to be statistically significant.

In this case the permutation yielded an exact test because we were able to enumerate all of the possible combinations. We will also demonstrate this using the Stata permute command using the data from above. The permute command randomly places observations into groups so we will use 200 replications to insure that all combinations have a chance to be present.

input y grp
55  1
58  1
60  1
12  0
22  0
34  0
end

permute grp r(sum) if grp, reps(200) nodots nodrop nowarn: summarize y

Monte Carlo permutation results                   Number of obs   =          6

      command:  summarize y if grp
        _pm_1:  r(sum)
  permute var:  grp

------------------------------------------------------------------------------
T            |     T(obs)       c       n   p=c/n   SE(p) [95% Conf. Interval]
-------------+----------------------------------------------------------------
       _pm_1 |        173      10     200  0.0500  0.0154  .0242342   .0900275
------------------------------------------------------------------------------
Note:  Confidence interval is with respect to p=c/n.
Note:  c = #{|T| >= |T(obs)|}

In larger examples it won't be possible to enumerate every permutaion so we will have to take numerous random orderings. We will demonstrate this using the Stata permute command with the hsb2 dataset with 1,000 replications.

use http://www.philender.com/courses/data/hsb2, clear
  
ttest read, by(female)

Two-sample t test with equal variances

------------------------------------------------------------------------------
   Group |     Obs        Mean    Std. Err.   Std. Dev.   [95% Conf. Interval]
---------+--------------------------------------------------------------------
    male |      91    52.82418    1.101403    10.50671    50.63605     55.0123
  female |     109    51.73394    .9633659    10.05783    49.82439     53.6435
---------+--------------------------------------------------------------------
combined |     200       52.23    .7249921    10.25294    50.80035    53.65965
---------+--------------------------------------------------------------------
    diff |            1.090231    1.457507               -1.783997    3.964459
------------------------------------------------------------------------------
Degrees of freedom: 198

                  Ho: mean(male) - mean(female) = diff = 0

     Ha: diff < 0               Ha: diff ~= 0              Ha: diff > 0
       t =   0.7480                t =   0.7480              t =   0.7480
   P < t =   0.7723          P > |t| =   0.4553          P > t =   0.2277
  
/* use ~female because males have the larger mean and therefore larger sum */
  
permute female sum=r(sum), reps(1000) nodots nodrop nowarn: summarize read if ~female

      command:  summarize read if ~female
          sum:  r(sum)
  permute var:  female

Monte Carlo permutation statistics                Number of obs    =       200
                                                  Replications     =      1000

------------------------------------------------------------------------------
T            |     T(obs)       c       n   p=c/n   SE(p) [95% Conf. Interval]
-------------+----------------------------------------------------------------
sum          |       4807     255    1000  0.2550  0.0138  .2282394   .2832112 
------------------------------------------------------------------------------
Note:  confidence interval is with respect to p=c/n
Note:  c = #{|T| >= |T(obs)|}


  
ttest science, by(female)

Two-sample t test with equal variances

------------------------------------------------------------------------------
   Group |     Obs        Mean    Std. Err.   Std. Dev.   [95% Conf. Interval]
---------+--------------------------------------------------------------------
    male |      91    53.23077    1.125037    10.73217    50.99569    55.46585
  female |     109    50.69725    .8657315    9.038503    48.98122    52.41328
---------+--------------------------------------------------------------------
combined |     200       51.85    .7000987    9.900891    50.46944    53.23056
---------+--------------------------------------------------------------------
    diff |            2.533522    1.397901                -.223164    5.290207
------------------------------------------------------------------------------
Degrees of freedom: 198

                  Ho: mean(male) - mean(female) = diff = 0

     Ha: diff < 0               Ha: diff ~= 0              Ha: diff > 0
       t =   1.8124                t =   1.8124              t =   1.8124
   P < t =   0.9643          P > |t| =   0.0714          P > t =   0.0357
  
/* use ~female because males have the larger mean and therefore larger sum */
  
permute female sum=r(sum), reps(1000) nodots nodrop nowarn: summarize science if ~female

      command:  summarize science if ~female
          sum:  r(sum)
  permute var:  female

Monte Carlo permutation statistics                Number of obs    =       200
                                                  Replications     =      1000

------------------------------------------------------------------------------
T            |     T(obs)       c       n   p=c/n   SE(p) [95% Conf. Interval]
-------------+----------------------------------------------------------------
sum          |       4844      43    1000  0.0430  0.0064  .0312912   .0574863 
------------------------------------------------------------------------------
Note:  confidence interval is with respect to p=c/n
Note:  c = #{|T| >= |T(obs)|}

tabstat write, by(female) stat(median)

Summary for variables: write
     by categories of: female 

female |       p50
-------+----------
  male |        52
female |        57
-------+----------
 Total |        54
------------------

median write, by(female)

Median test

   Greater |
  than the |        female
    median |      male     female |     Total
-----------+----------------------+----------
        no |        56         51 |       107 
       yes |        35         58 |        93 
-----------+----------------------+----------
     Total |        91        109 |       200 

          Pearson chi2(1) =   4.3369   Pr = 0.037

   Continuity corrected:
          Pearson chi2(1) =   3.7643   Pr = 0.052

/* use female since females have higher median */

permute female median=r(p50), reps(1000) nodots nodrop nowarn: summarize write if female, detail

      command:  summarize write if female, detail
       median:  r(p50)
  permute var:  female

Monte Carlo permutation statistics                Number of obs    =       200
                                                  Replications     =      1000

------------------------------------------------------------------------------
T            |     T(obs)       c       n   p=c/n   SE(p) [95% Conf. Interval]
-------------+----------------------------------------------------------------
median       |         57      66    1000  0.0660  0.0079  .0514086   .0832061 
------------------------------------------------------------------------------
Note:  confidence interval is with respect to p=c/n
Note:  c = #{|T| >= |T(obs)|}

Intro Home Page

Phil Ender, 23mar06, 9Jun04, 30Jun98