re: Re: st: RE: MCNEMAR test or Average treatment effects in matched data.

["Ariel Linden" <ariel.linden@gmail.com>]

First off, I suggest you adhere to a couple of important items noted on the
Statalist FAQ, (1) use your real name so that I know who I am actually
addressing, and (2) if you cite a paper, provide the entire reference, not
something incoherent as "RubinandThomas, 1996"

As for where you got this quote from, I see it was taken directly from
Elizabeth Stuart's paper. However, if you would have read the following few
sentences in that paper, you would have seen where my point was made: 

" When matching using propensity scores, detailed below, there is little
cost to including variables that are
actually unassociated with treatment assignment, as they will be of little
influence in the propensity score
model. Including variables that are actually unassociated with the outcome
can yield slight increases in variance.
However, excluding a potentially important confounder can be very costly in
terms of increased bias.
Researchers should thus be liberal in terms of including variables that may
be associated with treatment assignment and/or the outcomes. Some examples
of matching have 50 or even 100 covariates included in the procedure (e.g.,
Rubin, 2001)."

So, no, the issue is not under debate as you claim.

More importantly it addresses the crux of the matter here, and that is that
you should utilize all the variables you have in generating the propensity
score, and then present them, as I have suggested, in a typical "table 1"
format. 

I am not sure where you are having difficulty in understanding the point
about not using P values for determining covariate balance? I gave the
reasoning in my prior post, yet you are asking the same question again: 

"How should should I show these results? Should i include them in table 1
before and after the match? But without showing the p values? How can they
know that there are not statistical differences?"

At this point, I suggest that you spend time reading Elizabeth Stuart's
paper, as well as the following references. I don't think there is anything
else for me to add to this thread.

Ariel

References:

Stuart EA. Matching methods for causal inference: a review and a look
forward. Statistical Science 2010;25(1):1?21.

Caliendo M, Kopeinig S. Some practical guidance for the implementation of
propensity score matching. Journal of Economic Surveys 2008;22:31-72.

Austin PC. Balance diagnostics for comparing the distribution of baseline
covariates between treatment groups in propensity-score matched samples.
Statistics in Medicine 2009;28:3083-3107.

Linden A, Samuels SJ. Using balance statistics to determine the optimal
number of controls in matching studies. Journal of Evaluation in Clinical
Practice 2013;19(5):968?975.

Linden A. Identifying spin in health management evaluations. Journal of
Evaluation in Clinical Practice. 2011;17:1223-1230.

From: mccali mccalister <mccali85@hotmail.com>
Date: 8 January 2014 18:01
Subject: Re: st: RE: MCNEMAR test or Average treatment effects in matched
data.
To: "statalist@hsphsun2.harvard.edu" <statalist@hsphsun2.harvard.edu>


Dear Ariel,
About the first question. I got it clear. As I said in my first thread I
checked the balance of the covariates using ps test, I meant using
standarised mean differences. About the variables included, I only included
variables known to be related to both treatment assignment and the
outcome(RubinandThomas,1996;Heckman,IchimuraandTodd,1998;Glazerman,LevyandMy
ers,2003;
Hill,ReiterandZanutto,2004). But  i guess this issue still in debate.
So i still have don't know how to do, to show all these variables that are
not related to the outcome nor  to the treatment assignment. These may be
interesting to the readers to know. How should should I show these results?
Should i include them in table 1 before and after the match? But without
showing the p values? How can they know that there are not statistical
differences?

I may be new in this issue but that does not mean that  I just run stata. I
am aware that  it's me who does the research and stata only calculate. I
only want to make sure that what I do is correct.

Best regards


Enviado desde mi iPad

> El 08/01/2014, a las 17:02, "Ariel Linden" <ariel.linden@gmail.com>
escribió:
>
> Let me weigh in here, since I was the poster of that previous thread 
> on this subject 
> (http://www.stata.com/statalist/archive/2012-08/msg00985.html)
>
> The questions that are being asked here are somewhat difficult to 
> follow, so I'll attempt to answer them, given what I understand:
>
> " So if I understood well, you simply would use the log.regression to 
> see the effect of the my treatment. What about the average treatment
effects?
> Are they usefull? In my case?(the incidence of mediastinitis is the 
> outcome
> ) Maybe the interpretaion in medical issues is not so clear?So should 
> I avoid it?"
>
> If your outcome is binary, then yes, you can use a logistic regression.
> Which type of treatment effect you'll be estimating depends on how you 
> do your matching/weighting. In general, matching untreated individuals 
> to treated individuals provides an average treatment effect on the 
> treated (ATT). Why, because you are choosing untreated units that have 
> the same characteristics as the treated. You can use weighting 
> strategies instead of matching, and choose your treatment effect 
> estimator, if you are not happy with the matching approach.
>
> " It is very frequent that many paper using propensity score (not 
> necessary should  be right)  they show the new propensity score groups 
> that they are balanaced using parametric tests or non parametric 
> test(even then they use standarizes mean differences). What would you 
> advice me to use if I want to compare baseline caracteristic of my 
> population  before and after the propensity score ? Note that I want 
> to show all the baseline variables and many of them are not included 
> in the propensity score model. Should. I simply use parametric tests
weighted by psmatch2?"
>
> As common practice, you should show a "table 1" before and after matching.
> This will provide "evidence" that matching resulted in balance on 
> observed pre-intervention characteristics. As per my previous thread, 
> you should NOT be using t-tests to assess balance. This is not 
> inferential statistics, and thus we are not attempting to draw 
> assumptions to a population. Moreover, sample size will influence the 
> statistics, which we hope to avoid. You should use standardized 
> differences for calculating balance, with the intent of reducing the 
> value as far as possible (down to zero). You should also review Q-Q 
> plots, box plots, density plots, etc. to visually inspect the
distributions of each covariate.
>
> Lastly, you write that you want to show baseline variables that are 
> not included in the propensity score model? Why would you not include 
> them in the model? If you read the Elizabeth Stuart article that Joe 
> referenced, you will see that the rule of thumb is to include as many 
> covariates as available into the propensity score. This increases the 
> likelihood that you've balanced all observable characteristics, 
> thereby reducing the possibility of bias from omitted variables.
>
> In summary, you should spend some time reading about these issues. 
> Blindly running statistics without a firm understanding of the 
> underlying principles is not a recommended practice for researchers.
>
> I hope this helps
>
> Ariel
>
>
> Date: Tue, 7 Jan 2014 17:28:37 +0000
> From: Joe Canner <jcanner1@jhmi.edu>
> Subject: RE: st: RE: MCNEMAR test or Average treatment effects in 
> matched data.
>
> Dr. Ayaon,
>
> I am not an expert on propensity score matching; I have used it 
> occasionally and was sharing a few insights and experiences since no 
> one else had replied.  I would recommend that you familiarize yourself 
> with the recent literature on the subject.  A good start would be:
Stuart, E.A. (2010).
> Matching Methods for Causal Inference: A review and a look forward.
> Statistical Science 25(1): 1-21.
>
> In a recent thread on this subject, someone recommended against 
> checking for balance using methods like t-tests which are dependent on 
> sample size.  This is certainly a useful caution, depending on the 
> size of your study.  If balance is determined by not rejecting the 
> null hypothesis of no difference, you need to be careful that this 
> result is due to the lack of difference between the groups rather than 
> to insufficient sample size.  If sample size is an issue for you in 
> this regard, then perhaps you can try a non-parametric test.
>
> Regards,
> Joe
> ________________________________________
> From: owner-statalist@hsphsun2.harvard.edu
> [owner-statalist@hsphsun2.harvard.edu] on behalf of mccali mccalister 
> [mccali85@hotmail.com]
> Sent: Tuesday, January 07, 2014 8:17 AM
> To: statalist@hsphsun2.harvard.edu
> Subject: Re: st: RE: MCNEMAR test or Average treatment effects in 
> matched data.
>
> Dear Dr.Canner,
>
> I really apreciate your replay. So if I understood well, you simply 
> would use the log.regression to see the effect of the my treatment. 
> What about the average treatment effects? Are they usefull? In my 
> case?(the incidence of mediastinitis is the outcome ) Maybe the 
> interpretaion in medical issues is not so clear?So should I avoid it?
>
> It is very frequent that many paper using propensity score (not 
> necessary should  be right)  they show the new propensity score groups 
> that they are balanaced using parametric tests or non parametric 
> test(even then they use standarizes mean differences). What would you 
> advice me to use if I want to compare baseline caracteristic of my 
> population  before and after the propensity score ? Note that I want 
> to show all the baseline variables and many of them are not included 
> in the propensity score model. Should. I simply use parametric tests
weighted by psmatch2?
>
> Best regards
> Dr. Ayaon
> Cardiovascular resident
>
> Enviado desde mi iPad
>
>> El 06/01/2014, a las 23:12, "Joe Canner" <jcanner1@jhmi.edu> escribió:
>>
>> Dear Dr. Ayaon,
>>
>> According to a previous thread on this subject
> (http://www.stata.com/statalist/archive/2012-08/msg00985.html) it is 
> not necessary to used a matched analysis in a 1:k propensity score 
> matched analysis.  In fact, I'm not even sure how one would do a 
> McNemar test for
> 1:5 matching (although with a little work it might be possible for a 
> 1:1 match).  More to the point, as mentioned in the previous thread, I 
> would question whether propensity score matching truly qualifies as a 
> matched analysis for McNemar purposes.  Two people can have the same 
> (or similar) propensity score even if they have a quite different set of
characteristics.
>>
>> As noted in the previous thread, it should be sufficient (if not
> preferable) to do a logistic regression of your outcome variable 
> versus the treatment group, weighted using the _weight variable 
> provided by -psmatch2-, and including all of the matching variables 
> (and any other relevant
> variables) as covariates.
>>
>> Regards,
>> Joe Canner
>> Johns Hopkins University School of Medicine
>


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/