# Re: st: composite kappa & intercenter variability in kappas

 From Constantine Daskalakis To statalist@hsphsun2.harvard.edu Subject Re: st: composite kappa & intercenter variability in kappas Date Wed, 17 Aug 2005 15:30:50 -0400

```At 01:50 PM 8/17/2005, Richard_Lenhardt@rush.edu wrote:
```
```Hello

We submitted a manuscript comparing two observers measuring 10 dicotomous
variables.  We measured kappa, to assess concordance between observers.  A
reviewer asked that we calculate an overall value (composite) of the
kappas, for all 10 variables.   (Fleiss, Stat Methods for Rates and
Proportions, 2nd or 3rd edition).

I couldn't find such a function in stata (overall kappa).  Would there be
another way to calculate this other than by hand?

Also, the study was completed at 5 different hospitals.  The reviewer
asked that we take into account hospital to hospital variation when
calculating kappas.  I wasn't aware that kappas could be adjusted for the
5 hospitals, other than generating a large matrix of kappas corresponding
to the different hospitals.  Would there be a way to take into account the
variation in agreement among the hospitals when estimating the kappas?

Thank you!

Richard Lenhardt, MD, MPH
Assistant Professor of Medicine
Division of Pulmonary and Critical Care Medicine
Rush University Medical Center
Chicago, IL
```

Hospital can be used as a stratification factor.
You say you have 10 dichotomous variables that are being rated. This is what you can do.

Take variable Y1 first. Compute stratum-specific kappas (for each hospital separately). Then, compute a weighted-average combined kappa. Whether that's reasonable depends on whether you are estimating the same kappa parameter in all the strata -- eyeballing and a homogeneity test might help here. In flavor, all this is like stratification and computation of an adjusted MH odds ratio. I don't have Fleiss in front of me but the specifics are in there.

Then do the same thing for the rest of your variables, Y2, ..., Y10.

You now have 10 "adjusted" kappas (controlling for hospital).

Should you compute an overall adjusted kappa across all 10 ratings? Maybe yes, maybe no.

Could you do it? In principle, you could use the same procedure to get a weighted kappa from all those 10 kappas. The trouble is that the individual kappas are correlated beyond the fact that you are using the same raters -- you are also probably using the same patients. I think the only feasible way to get a handle on such a beast is through the bootstrap.

Hope this helps.

CD

The documents accompanying this transmission may contain confidential health or business information. This information is intended for the use of the individual or entity named above. If you have received this information in error, please notify the sender immediately and arrange for the return or destruction of these documents.

________________________________________________________________
Assistant Professor,
Thomas Jefferson University, Division of Biostatistics,
211 S. 9th St., Suite 602, Philadelphia, PA 19107
Tel: 215-955-5695
Fax: 215-503-3804