st: Computing differences in probability of type-one error for different samples

[George Murray <george.murray16@gmail.com>]

Statalisters,

Suppose I run a certain model ~1000 times, but a different sample is
used each type the model is run. The statistical significance of one
of the variables is tested for *all* of the ~1000 models. I am aware
that the (of course, arbitrarily chosen) significance level can be
used to find the probability that a type I error has been made but
obviously, the probability of type-I error will not be uniform across
each of the models where the null is rejected. Is it then possible to
use Stata to compute the probability of that a type I error has been
made (or is the only solution Bayesian techniques?)

Is it blasphemous to use the p-value (such that the differing
probabilities of a type-I error is adjusted for) as an approximation,
given that I have no a priori information? Is anyone aware of any
applied papers which have (rightly or wrongly) used this to
approximate that a type-I error has been made? Since the same model is
used, is it possible that the p-value is inversely related to the
probability of a type-I error. (Apologies if this question sounds
really silly).

Regards,

George.
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/