[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: RE: Pearson chi square and Rao and Scott correction validity

From   Steven Samuels <>
Subject   Re: st: RE: Pearson chi square and Rao and Scott correction validity
Date   Fri, 7 Nov 2008 11:33:23 -0500


Chamber and Skinner also suggest the model-based approach in Chapter 7. "Many hypotheses of interests may be expressed as a hypothesis in a log-linear model." It turns out that the Chi Square Pearson or Likelihood Ratio tests in an I x J table are equivalent to simultaneous tests that (I-!)(J-1) odds ratios are all equal to 1. In fact, a major competitor to the adjusted Chi square tests, with good properties in simulations, is doing Bonferroni adjusted tests of individual OR's.

I don't think that the logistic and multinomial logistic models are "cumbersome". The command set-up is quite simple( e.g. xi: -mlogit- y i.x ) The test of independence is just the test that the "x" coefficients in all equations are zero.

With the Chi-Square tests you are left only with a single "reject" or "cannot reject" answer. If the test rejects, then why? Modeling can give the answer. In the IxJ case, if there is any structure in either factor, you can assess models that are intermediate between complete saturation (IxJ parameters) and independence (I + J -1 parameters). And, you can ask whether other factors modify the degree of association. Stata can fit loglinear models not only with - mlogit- (a special case), but also with -glm- and -poisson-. In addition, the contributed commands -ipf- and -mclest- can be downloaded from SSC.

The original rules of thumb for Chi square tests were criteria for insuring that the test statistic indeed had a chi square distribution. Asymptotics depended on sample size. For the recommended tests in Skinner and Chamber's section 7.3.2, a major determinant of the accuracy of tests (level under the null hypothesis, power) is the number of clusters.


On Nov 7, 2008, at 8:35 AM, Ángel Rodríguez Laso wrote:

Reading back Korn and Graubard's (1999) Analysis of Health Surveys,
Wiley, in page 78 they recommend for testing lack of association in a
2 x J contingency table a logistic regression with the
presence/absence of the condition as the dependent variable, and for I
x J contingency tables, a multinomial logit regression. I find this
cumbersome. They consider chi-square statistics inappropriate in
complex surveys, but they do not talk about the Rao and Scott

I will try to contact Korn or Graubard to see if they can add more light.

Many thanks,

Angel Rodriguez-Laso

2008/11/7 Steven Samuels <>:
I see that in my previous post I confused two issues 1) the sample
size requirements for validity of the survey-adjusted chi square
tests in Stata; 2) sample size requirements for estimates of cell
totals or proportions, with small counts.  Ángel asked about the
first issue. Bottom line: I really don't have an answer.


On Nov 6, 2008, at 5:04 PM, Steven Samuels wrote:

I've looked though Chapters 6-7 of Chamber's and Skinner's book
Analysis of Survey Data, Wiley, 2003, but I have no definitive
answer. I do have some thoughts:

* "Expected" count is not a guide in the survey setting--it is a
sum of weights of sample observations in the table cell.

* The accuracy of the second-order Rao-Scott statistic chi square,
probably the best test in -svy: tab-, is apt to depend on the
number of clusters, on the crude counts, and on the distribution of
the observations across clusters. The rule of thumb of 5
observations (or 1) in a cell is based on theory of  i.i.d.
observations that does not hold in the complex survey setting.

* With a small number of events, I ordinarily display only
unweighted numbers and do not reported weighted estimates or
confidence intervals. When I have wanted to infer something about a
proportion based on small outcome count, I've resorted to the
methods on pp. 64-68 of Korn and Graubard (1999) Analysis of Health
Surveys, Wiley.

A quick Google search turned up one survey which would not report a
cell with fewer than 25 observations (
showsrvy.cfm?srvy_CatID=5&srvy_Seri=16) and another in which the
minimum cell size was 4,000! (

So a guess for Ángel is that not even five observations in table
cell is enough.


On Nov 6, 2008, at 7:33 AM, Nick Cox wrote:

There is no need to invoke belief! My -tabchi- and -tabchii-
(programs) from the -tab_chi- package on SSC do indeed give
warnings. (There is no Stata program called tab-chi.)

But these old warnings are very conservative. Many writers now
advise that chi-square works fine so long as all expected
frequencies are above about 1. In any case, the point can be
explored by simulations or bootstrapping. Often it is better to
use Fisher's exact test.

I can't advise on the main issue, which is for svy-savvy people,
but in general very low expected frequencies could be problematic
for any method.


Ángel Rodríguez Laso

I've been reviewing the manuals and statalist archives and I've
confirmed that Stata does not give any automatic warning message when
requirements for a valid chi-square test are not met (i.e. no more
than 20% of the expected values in a table are less than 5 and none
are less than 1), what I think is a nuisance. I suppose this can be
only worked out by writing the option 'expected' after tabulate and
checking oneself if the requirements are met. I believe Cox's tab-chi
package does give a warning when requirements are not met.

I wonder also if the Rao and Scott correction of Pearson chi-square
that is recommended for survey designs needs the same requirements.
The problem then would be that -svy:tab- doesn't support the
'expected' option neither tab-chi is suitable for survey analysis.

*   For searches and help try:

Steven Samuels
18 Cantine's Island
Saugerties, NY 12477
EFax: 208-498-7441

*   For searches and help try:

*   For searches and help try:

*   For searches and help try:

© Copyright 1996–2022 StataCorp LLC   |   Terms of use   |   Privacy   |   Contact us   |   What's new   |   Site index