Multivariate Analysis
Multivariate Data: The Long and the Wide of It


There are two primary orientations for multivariate data; wide and long. Many researchers are familiar with the wide data orientation in which each row contains all of the variables for a single observation or subject. In the long orientation the response variables are stacked on top of each other in a single variable. The long orientation requires variables identifying subject and the response variable.

Here are two simple examples using the hsb2 dataset. We will use write (y0) and read (y1) as response variables. We will begin with a simple two-group manova followed by a multivariate regression with a single continuous predictor. We start with data in the wide orientation.

use hsb2, clear

rename write   y0
rename science y1
drop race ses schtyp prog socst

/* categorical predictor */

manova y0 y1 = female

                           Number of obs =     200

                           W = Wilks' lambda      L = Lawley-Hotelling trace
                           P = Pillai's trace     R = Roy's largest root

                  Source |  Statistic     df   F(df1,    df2) =   F   Prob>F
              -----------+--------------------------------------------------
                  female | W   0.8229      1     2.0   197.0    21.20 0.0000 e
                         | P   0.1771            2.0   197.0    21.20 0.0000 e
                         | L   0.2153            2.0   197.0    21.20 0.0000 e
                         | R   0.2153            2.0   197.0    21.20 0.0000 e
                         |--------------------------------------------------
                Residual |               198
              -----------+--------------------------------------------------
                   Total |               199
              --------------------------------------------------------------
                           e = exact, a = approximate, u = upper bound on F

/* convert to approximate chi-square */

display 2*21.2
42.4

/* do the manova using mvreg */

mvreg y0 y1 = female

Equation          Obs  Parms        RMSE    "R-sq"          F        P
----------------------------------------------------------------------
y0                200      2      9.1846    0.0658   13.94331   0.0002
y1                200      2     9.84454    0.0163   3.284704   0.0714

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
y0           |
      female |   4.869947   1.304191     3.73   0.000     2.298059    7.441835
       _cons |   50.12088   .9628077    52.06   0.000     48.22221    52.01955
-------------+----------------------------------------------------------------
y1           |
      female |  -2.533522   1.397901    -1.81   0.071    -5.290207     .223164
       _cons |   53.23077   1.031988    51.58   0.000     51.19567    55.26587
------------------------------------------------------------------------------

test [y0]female = [y1]female

 ( 1)  [y0]female - [y1]female = 0

       F(  1,   198) =   40.28
            Prob > F =    0.0000

/* continuous predictor */

manova y0 y1 = read, continuous(read)

                           Number of obs =     200

                           W = Wilks' lambda      L = Lawley-Hotelling trace
                           P = Pillai's trace     R = Roy's largest root

                  Source |  Statistic     df   F(df1,    df2) =   F   Prob>F
              -----------+--------------------------------------------------
                    read | W   0.5194      1     2.0   197.0    91.13 0.0000 e
                         | P   0.4806            2.0   197.0    91.13 0.0000 e
                         | L   0.9252            2.0   197.0    91.13 0.0000 e
                         | R   0.9252            2.0   197.0    91.13 0.0000 e
                         |--------------------------------------------------
                Residual |               198
              -----------+--------------------------------------------------
                   Total |               199
              --------------------------------------------------------------
                           e = exact, a = approximate, u = upper bound on F


mvreg y0 y1 = read

Equation          Obs  Parms        RMSE    "R-sq"          F        P
----------------------------------------------------------------------
y0                200      2    7.624871    0.3561   109.5213   0.0000
y1                200      2    7.707104    0.3971   130.4121   0.0000

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
y0           |
        read |   .5517051   .0527178    10.47   0.000     .4477445    .6556656
       _cons |   23.95944   2.805744     8.54   0.000     18.42647    29.49242
-------------+----------------------------------------------------------------
y1           |
        read |   .6085207   .0532864    11.42   0.000      .503439    .7136024
       _cons |   20.06696   2.836003     7.08   0.000     14.47432    25.65961
------------------------------------------------------------------------------

/* test that slope is equal for y0 & y1 */

test [y0]read = [y1]read

 ( 1)  [y0]read - [y1]read = 0

       F(  1,   198) =    0.84
            Prob > F =    0.3619
Next, we do the same analyses after converting to the long orientation.
/* convert to long */

reshape long y, i(id) j(v)
generate vf = v*female
generate vr = v*read

/* categorical predictor */

xtmixed y female v vf || id:

Mixed-effects REML regression                   Number of obs      =       400
Group variable: id                              Number of groups   =       200

                                                Obs per group: min =         2
                                                               avg =       2.0
                                                               max =         2


                                                Wald chi2(3)       =     43.74
Log restricted-likelihood = -1413.8646          Prob > chi2        =    0.0000

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      female |   4.869947   1.351858     3.60   0.000     2.220353     7.51954
           v |    3.10989   .8611413     3.61   0.000     1.422084    4.797696
          vf |  -7.403468   1.166477    -6.35   0.000    -9.689721   -5.117215
       _cons |   50.12088   .9979975    50.22   0.000     48.16484    52.07692
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
id: Identity                 |
                   sd(_cons) |    7.54286   .5041268      6.616769    8.598567
-----------------------------+------------------------------------------------
                sd(Residual) |   5.808716    .291899      5.263876     6.40995
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =    99.19 Prob >= chibar2 = 0.0000

/* test whether coefficient for female is the same for y0 & y1 */

test vf

 ( 1)  [y]vf = 0

           chi2(  1) =   40.28
         Prob > chi2 =    0.0000
         
/* continuous predictor */         

xtmixed y read v vr || id:

Mixed-effects REML regression                   Number of obs      =       400
Group variable: id                              Number of groups   =       200

                                                Obs per group: min =         2
                                                               avg =       2.0
                                                               max =         2


                                                Wald chi2(3)       =    185.56
Log restricted-likelihood = -1373.5858          Prob > chi2        =    0.0000

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        read |   .5517051   .0530029    10.41   0.000     .4478213    .6555888
           v |   -3.89248   3.309103    -1.18   0.239     -10.3782    2.593242
          vr |   .0568156   .0621756     0.91   0.361    -.0650463    .1786776
       _cons |   23.95944   2.820914     8.49   0.000     18.43055    29.48833
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
id: Identity                 |
                   sd(_cons) |   4.281807   .5108883       3.38895    5.409898
-----------------------------+------------------------------------------------
                sd(Residual) |   6.358867   .3195449      5.762425    7.017044
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =    20.27 Prob >= chibar2 = 0.0000

test vr

 ( 1)  [y]vr = 0

           chi2(  1) =    0.84
         Prob > chi2 =    0.3608

/*  compute the coefficient and standard errors for the y1 equation */

lincom _cons + v

 ( 1)  [y]v + [y]_cons = 0

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |   20.06696   2.820914     7.11   0.000     14.53807    25.59585
------------------------------------------------------------------------------

lincom read + vr

 ( 1)  [y]read + [y]vr = 0

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |   .6085207   .0530029    11.48   0.000     .5046369    .7124045
------------------------------------------------------------------------------
The z-test for the interaction in the last model above is a test that the slope for y0 regressed on read is different from the slope for y1 regressed on read. It isn't.

There is another way we can test this and that is to compare, using a likelihood ratio test, whether the model with interaction is significantly different from the model without the interaction. We will use the mle option so that the likelihood ratio test will be valid with different fixed effects.

The logic behind this is that the model without the interaction forces (constrains) the slopes to be equal where as the model with the interaction doesn't.

/* model 1 -- no interaction */

xtmixed y read v || id: , mle

Mixed-effects ML regression                     Number of obs      =       400
Group variable: id                              Number of groups   =       200

                                                Obs per group: min =         2
                                                               avg =       2.0
                                                               max =         2


                                                Wald chi2(2)       =    186.59
Log likelihood = -1370.4704                     Prob > chi2        =    0.0000

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        read |   .5801129   .0427133    13.58   0.000     .4963963    .6638294
           v |      -.925    .634032    -1.46   0.145     -2.16768      .31768
       _cons |    22.4757   2.295281     9.79   0.000     17.97704    26.97437
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
id: Identity                 |
                   sd(_cons) |   4.250426   .5074216      3.363682    5.370937
-----------------------------+------------------------------------------------
                sd(Residual) |    6.34032   .3170158      5.748455    6.993125
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =    20.22 Prob >= chibar2 = 0.0000

estimates store M1

/* model 2 -- with interaction */

xtmixed y read v vr || id: , mle

Mixed-effects ML regression                     Number of obs      =       400
Group variable: id                              Number of groups   =       200

                                                Obs per group: min =         2
                                                               avg =       2.0
                                                               max =         2


                                                Wald chi2(3)       =    187.44
Log likelihood = -1370.0495                     Prob > chi2        =    0.0000

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        read |   .5517051   .0527372    10.46   0.000      .448342    .6550681
           v |   -3.89248   3.292516    -1.18   0.237    -10.34569    2.560733
          vr |   .0568156   .0618639     0.92   0.358    -.0644355    .1780667
       _cons |   23.95944   2.806774     8.54   0.000     18.45827    29.46062
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
id: Identity                 |
                   sd(_cons) |   4.260343   .5057796      3.375916    5.376474
-----------------------------+------------------------------------------------
                sd(Residual) |   6.326994   .3163496      5.736372    6.978426
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =    20.48 Prob >= chibar2 = 0.0000

estimates store M2

lrtest M1 M2

Likelihood-ratio test                                  LR chi2(1)  =      0.84
(Assumption: M1 nested in M2)                          Prob > chi2 =    0.3589
Now, you're probably thinking, this is all fine and good, but how is this useful to me. The answer lies in multivariate analysis with missing data. Say that y0 was missing for subject 167 (observation 21). With data in the wide orientation, we would have to discard the entire observation, including the non-missing value for y1.

When the data are in the long orientation (and the data are missing at random) we can keep the non-missing value for y1 and estimate the appropriate statistics and coefficients. That is, we can do this as long as we are using one of the maximum likelihood estimation techniques, such as found in xtmixed.

We will illustrate this by deleting six of the values in our dataset and repeating the xtmixed commands used earlier. This is, of course, a fairly trivial example but it demonstrates the principal involved.

/* convert back to wide and create missing values */

drop vr vf
reshape wide
replace y0 = . in  21
replace y0 = . in  89
replace y0 = . in 162
replace y1 = . in  35
replace y1 = . in 114
replace y1 = . in 187

/* back to long once again */

reshape long
generate vf = v*female
generate vr = v*read

/* categorical predictor */

xtmixed y female v vf || id:

Mixed-effects REML regression                   Number of obs      =       394
Group variable: id                              Number of groups   =       200

                                                Obs per group: min =         1
                                                               avg =       2.0
                                                               max =         2


                                                Wald chi2(3)       =     51.55
Log restricted-likelihood =  -1389.246          Prob > chi2        =    0.0000

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      female |   5.106562   1.358938     3.76   0.000     2.443093    7.770032
           v |    3.15318   .8444089     3.73   0.000     1.498169    4.808191
          vf |  -7.855456   1.147064    -6.85   0.000    -10.10366   -5.607252
       _cons |   50.16495   1.002231    50.05   0.000     48.20062    52.12929
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
id: Identity                 |
                   sd(_cons) |   7.679059   .5042653      6.751677    8.733822
-----------------------------+------------------------------------------------
                sd(Residual) |   5.643999   .2879057      5.107005    6.237456
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =   105.07 Prob >= chibar2 = 0.0000

/* test whether coefficient for female is the same for y0 & y1 */

test vf

 ( 1)  [y]vf = 0

           chi2(  1) =   46.90
         Prob > chi2 =    0.0000

/* continuous predictor */

xtmixed y read v vr || id:

Mixed-effects REML regression                   Number of obs      =       394
Group variable: id                              Number of groups   =       200

                                                Obs per group: min =         1
                                                               avg =       2.0
                                                               max =         2


                                                Wald chi2(3)       =    179.92
Log restricted-likelihood = -1353.2521          Prob > chi2        =    0.0000

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        read |   .5411034   .0536441    10.09   0.000     .4359628     .646244
           v |  -4.874603   3.312018    -1.47   0.141    -11.36604    1.616833
          vr |   .0742502   .0622633     1.19   0.233    -.0477837    .1962841
       _cons |   24.62704   2.859805     8.61   0.000     19.02193    30.23216
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
id: Identity                 |
                   sd(_cons) |   4.425277   .5073333       3.53472    5.540207
-----------------------------+------------------------------------------------
                sd(Residual) |   6.294144   .3207963      5.695779     6.95537
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =    22.21 Prob >= chibar2 = 0.0000

/*  compute the coefficient and standard errors for the y1 equation */

lincom _cons + v

 ( 1)  [y]v + [y]_cons = 0

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |   19.75244    2.84493     6.94   0.000     14.17648     25.3284
------------------------------------------------------------------------------

lincom read + vr

 ( 1)  [y]read + [y]vr = 0

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |   .6153536   .0535817    11.48   0.000     .5103354    .7203719
------------------------------------------------------------------------------

/* likelihood ratio test using mle option */

/* model 1 -- no interaction */

xtmixed y read v || id: , mle

Mixed-effects ML regression                     Number of obs      =       394
Group variable: id                              Number of groups   =       200

                                                Obs per group: min =         1
                                                               avg =       2.0
                                                               max =         2


                                                Wald chi2(2)       =    180.40
Log likelihood = -1350.4648                     Prob > chi2        =    0.0000

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        read |   .5783108   .0434167    13.32   0.000     .4932157     .663406
           v |   -.997793   .6350524    -1.57   0.116    -2.242473    .2468868
       _cons |   22.67819   2.335366     9.71   0.000     18.10096    27.25542
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
id: Identity                 |
                   sd(_cons) |   4.383084   .5049358        3.4972    5.493373
-----------------------------+------------------------------------------------
                sd(Residual) |   6.286582   .3187162      5.691942    6.943344
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =    21.92 Prob >= chibar2 = 0.0000


estimates store M1

/* model 2 -- with interaction */

xtmixed y read v vr || id: , mle

Mixed-effects ML regression                     Number of obs      =       394
Group variable: id                              Number of groups   =       200

                                                Obs per group: min =         1
                                                               avg =       2.0
                                                               max =         2


                                                Wald chi2(3)       =    181.76
Log likelihood = -1349.7494                     Prob > chi2        =    0.0000

------------------------------------------------------------------------------
           y |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
        read |    .541103   .0533716    10.14   0.000     .4364967    .6457094
           v |  -4.874626   3.295078    -1.48   0.139    -11.33286    1.583609
          vr |   .0742505   .0619449     1.20   0.231    -.0471592    .1956602
       _cons |   24.62707   2.845274     8.66   0.000     19.05043     30.2037
------------------------------------------------------------------------------

------------------------------------------------------------------------------
  Random-effects Parameters  |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
id: Identity                 |
                   sd(_cons) |     4.4031   .5021979      3.521067    5.506084
-----------------------------+------------------------------------------------
                sd(Residual) |    6.26195   .3175345      5.669522    6.916284
------------------------------------------------------------------------------
LR test vs. linear regression: chibar2(01) =    22.44 Prob >= chibar2 = 0.0000


estimates store M2

lrtest M1 M2

Likelihood-ratio test                                  LR chi2(1)  =      1.43
(Assumption: M1 nested in M2)                          Prob > chi2 =    0.2316


/* compare with z-squared */

1.4161


Multivariate Course Page

Phil Ender, 8may05