Re: st: Bootstrapped Standard Errors

[Danny Dan <danny2011dan@gmail.com>]

Thank you so much Professor Kolenikov for answering my question in
details. Your points really make sense. I will certainly try to
contact Professor King on this issue. Thank you so much for pointing
out the uncertainty in using the cem_weights. I will thoroughly
contemplate and try to solve this issue. If I get any solution, I will
certainly update my post. In the meantime if any other member has any
solution on this please let me know.

My sincere thanks to Professor Kolenikov and Professor Millimet for
their most helpful suggestions.

Best,

Dan

On Thu, Mar 22, 2012 at 1:23 PM, Stas Kolenikov <skolenik@gmail.com> wrote:
> This gets very complicated, in my opinion. Conceptually speaking, your
> standard errors should reflect all sources of uncertainty about the
> final treatment effect estimate you obtained. If by virtue of your
> sample design, the controls were oversampled, then it would have been
> relatively easier for you to find the controls for your treated, thus
> reducing your bias and mean squared error. The procedure to get the
> standard errors should take into account the fact that the original
> data were obtained by random sampling; replacing the sampling weights
> with CEM weights eliminates this source of uncertainty, and probably
> biases your resulting estimate. If the bootstrap is to mimic the
> sampling process, then it should remove some of the observations in
> both the control and the treatment group, thus making it more
> difficult to find matches (may be as far as having to drop the whole
> CEM stratum if it does not have matches, which has a drastic effect on
> the estimate and its degrees of freedom), and increasing your bias/MSE
> within the bootstrap subsamples. I am not entirely sure I see how to
> handle all of these issues and incorporate all different sources of
> uncertainty. Following Don Rubin's tradition, Gary King does not
> address the question of variance estimation, not even in the JASA
> paper, let alone the Stata Journal paper. I guess you would want to
> write to Gary King and ask about his advice on your data situation.
> You can utilize -bsweights/bs4rw- with the CEM weights as the
> baseline, but how far off your resulting standard errors would be from
> the true sampling variability is beyond me. If I were you, I would set
> up a simulation to see if the standard errors are at least
> approximately correct; you would have to make some smart choices
> regarding the sample design and the decisions within CEM steps,
> though. In the best case, you will see 95% coverage of your confidence
> intervals (rarely happened to me in my simulations). In the worst
> case, you will have material for a paper titled "Failures of CEM
> method: how on earth does one get the standard errors right???".
>
> On Thu, Mar 22, 2012 at 12:22 PM, Danny Dan <danny2011dan@gmail.com> wrote:
>> Hello Professor Kolenikov,
>>
>> I have seen your article in the Stata journal ("Resampling variance
>> estimation for complex survey data") and was also trying to use it in
>> my work. However, I am little confused about the use of the
>> appropriate weights in using your tool as because I am not using the
>> usual sampling weights as available in the raw data but using weights
>> generated after implementing a matching method.
>>
>> I am trying to use weights that are generated after coarsened exact
>> matching (CEM). After running CEM matching it gives both CEM_STRATA
>> and CEM_WEIGHTS (where wt=1 for treated and some positive values for
>> the controls). My question is can I use CEM_WEIGHTS and CEM_STRATA in
>> using your tool?
>>
>> Please let me know whether this would be appropriate.
>>
>> Also for your information, I also have weights from my raw data
>> (Primary sampling unit and strata for variance estimation (VARPSU and
>> VARSTR)).
>>
>> How shall I set the svy (-svyset-)? Shall I do the following as shown
>> in your example:
>>
>> egen upsu = group(strata psu)
>> . svyset upsu [pw=finalwgt], strata(cstrata)
>>
>> In my case, shall I replace finalwt with cem_weights and cstrata with
>> cem_strata and psu with varpsu.
>>
>> Please let me know.
>>
>> Thank you for your reply.
>>
>> Best ,
>>
>> Dan
>>
>> On Wed, Mar 21, 2012 at 8:41 PM, Danny Dan <danny2011dan@gmail.com> wrote:
>>> Hello All,
>>>
>>> I have a question:
>>>
>>> I know that bootstrapping cannot be applied with weights, however, is
>>> there anyway in STATA that after doing a weighted regression (like
>>> regress `depvar' `indepvar' [weight=wt],  or, probit  `depvar'
>>> `indepvar' [weight=wt]) I can use bootstrapping option separately to
>>> generate the standard errors?
>>>
>>> I am asking this because I need to generate bootstrapped standard
>>> errors because of its unknown structure in my model. To be more
>>> precise, I am doing a 2-stage estimation and trying to use
>>> bootstrapping to generate standard errors in the 2nd stage.
>>>
>>> Please let me know if I am not clear with my question then I will try
>>> to clarify it further for the ease of comprehensibility.
>>>
>>> Please help.
>>>
>>> Thank you.
>>>
>>> Best,
>>>
>>> Dan
>>> *
>>> *   For searches and help try:
>>> *   http://www.stata.com/help.cgi?search
>>> *   http://www.stata.com/support/statalist/faq
>>> *   http://www.ats.ucla.edu/stat/stata/
>>
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/statalist/faq
>> *   http://www.ats.ucla.edu/stat/stata/
>
>
>
> --
> Stas Kolenikov, also found at http://stas.kolenikov.name
> Small print: I use this email account for mailing lists only.
>
> *
> *   For searches and help try:
> *   http://www.stata.com/help.cgi?search
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/