# Re: re: st: permutation, strata() for empirical statistic distributions

 From David Airey To statalist@hsphsun2.harvard.edu Subject Re: re: st: permutation, strata() for empirical statistic distributions Date Thu, 6 Nov 2003 10:27:30 -0600

```Thank you for this useful tutorial. More thanks below.

```
nested ANOVA:

> Here's a concrete example of what I want to do using permute. I have a
> simple data set with 2 groups, 5 subjects per group, and 10 replicate
> measures per subject. The anova statement for this data is:
>
> . anova y group / subject|group /
>
> I'd like to use permute in the same way as the example on page 157 of
> R[N-R], "permute", to get a distribution of statistics for my data
> permuted within subject within group. Does "permute" handle nested
> models? I'm assuming the option strata(subject group) does this?
>
> How would one write a program equivalent to the one given on page 157
> for the model above? The program on page 157 for the anova model "anova
> y treatment subject" is:
>
> program panova
> version 8.0
> args response fac_intrst fac_other
> anova `response' `fac_intrst' `fac_other'
> test `fac_intrst'
> end
>
> This program is called by the permute statement:
>
> . permute treatment "panova y treatment subject" treatmentF=r(F)
> modelF=e(F), reps(1000) strata(subject) saving(permanova)
>
> My problem is that the examples don't tell me what to do with the / |
> characters in my anova statement.

For David's particular analysis, the -panova- program can be modified to
accommodate the call to -anova-, but I'm not sure what statistic David had an
interest in (of its permutation distribution). Assuming David is only
interested in the permutation distribution of the F statistic of -group- on
-subject|group-, then David could use the following:

program panova_nest
version 8.0
args response group subject
anova `response' `group' / `subject'|`group' /
test `group' / `subject'|`group'
end

Unfortunately, using -panova_nest- with -permute- and supplying the option
-strata(subject group)- will NOT have the expected result. The -anova- and
-test- results will be the same for each permutation because the effect of the
following command

. permute y "panova_nest y group subject" groupF=r(F),
reps(5) strata(group subject)

is to permute the response values within each subject of each group, thus the
sum of squares from each permutation will all be the same, so the F test will
take on the same value as well.
After I found this out first hand, I knew my example was not well thought out. My apologies. But here I see how to handle the slashes and pipes in the command--they go outside the quotes. Thank you!

The example in [R] permute on page 157 works out because of 2 things:

1. The hypothesis of interest was clearly identified.

The hypothesis associated with the F statistic for the treatment
effect.

2. There is a way to permute the data that is consistent with the hypothesis
of interest.

The null hypothesis for the example on page 157 is that there is no
treatment effect, thus randomly permuting the treatment identifiers
among the observations should result in values of the F statistics
that are similar to the one we observed.
I see. The treatment identifiers are moved around, the response stays still. This helps too.

Note that we permute the
treatment ids within subject, because we do not want to confound our
result with a possible subject effect (which is not part of the
hypothesis we are interested in).

If David is interested in the permutation distribution of

test group / subject|group

then a little more programming is required because -permute- is not able to
permute groups of observations, that is, -permute- does not have a -cluster()-
option.
This brings up my need to investigate what the difference between strata() and cluster() is, since permute has a strata() option.

If David's groups were unbalanced, I would have to end the discussion here
given that a solution is beyond the scope of this little email.

Fortunately, David's since data is balanced, I can keep going. Since the
groups are balanced (in both the number of subjects but also the number of
replicates per subject), we would do the following:

1. reshape the data from long to wide, making sure that the subject ids are
unique between group (it also helps to have a replicate id within subject
before we reshape)

2. modify the program that -permute- calls to reshape the data back to long,
run the -anova-, run the -test-, post the results of -test- to -r()-, and
reshape back to wide.

Here is a modification of -panova_nest_ from above that accounts for
reshaping the data.

program panova_nest_reshape, rclass
version 8.0
args response group subject

reshape long

capture noisily {
anova `response' `group' / `subject'|`group' /
test `group' / `subject'|`group'
}
local rc = c(rc)

reshape wide

exit `rc'
end

Here is what I did to reshape the data when testing this idea:

. egen id = group(group subject)
. bysort id : gen obs = _n
. reshape wide y, i(id) j(obs)

Now just use -permute- with the wide data and new program:

. permute group "panova_nest_reshape y group subject"
groupF=r(F), reps(5)

--Jeff
Thank you Jeff.

I'll make an unrelated public statement. While making the tough choice to learn R to fill in what is missing in StataSE 8.2 (general mixed model procedures like NLME/LME in R), I also decided to send my money to Stata for software upgrades, as the service and quality is so high. At the moment I use:

Data Desk 6, for EDA graphics
R, for general mixed model procedures not possible in StataSE
StataSE, for everything else.

(Two typos on page 157, at the top of the page the number of treatments is 10, not five, and at the bottom of the page the word expected should be expect).

-Dave

*
* For searches and help try:
* http://www.stata.com/support/faqs/res/findit.html
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/