st: Reproducing results - was managing updates

["Allan Reese (Cefas)" <allan.reese@cefas.co.uk>]

>let's say i'm developing a model using Stata. if i download the
>updates, it can change the results, the model coefficients etc. how
>would i be able to verify the results later? suppose on Jan 1st 2011 i
>updated Stata, regressed the model, got the coefficients, and used to
>report financial results for Q4 2010. Now 2 years later, the internal
>audit wants to verify how I got the coefficients. How would I be able
>to reproduce the state of Stata software as of Jan 2011?

This appears to connect with a current debate in Science and Nature
magazines.  For a general introduction see blog
http://wattsupwiththat.com/2011/02/26/the-code-of-nature-making-authors-
part-with-their-programs/

If you have a command log file started automatically from profile.do,
keep an output log file of critical runs, and keep versions of data
files (rather than -replace- them) you are well ahead of most people.
-update query- would insert in your output the dates of last update.
Obviously you need to keep any locally-written code files.  If an
analysis run in 2011 gives answer A but "the same" analysis run in 2013
gives answer B, the issues are (1) is either A or B provably or credibly
correct, and (2) if A is undermined, is anyone suggesting bad faith
rather than bad luck?
  
The blog links to a Nature editorial
http://www.nature.com/nature/journal/v470/n7334/full/470305b.html that
includes:
"Edward McCabe, then at the California NanoSystems Institute at the
University of California, Los Angeles, was so perturbed when different
versions of the same bioinformatics software gave wildly different
results that he published a paper on it (N. K. Henderson-Maclennan et
al. Mol. Genet. Metab. 101, 134-140; 2010). Reviewers resisted its
publication, asking what was new about the findings, as it was already
common knowledge that different software versions could dramatically
affect analyses."

That's the bizarre attitude in science! Every other aspect of scientific
work is scrutinised, calibrated and checked.  Every other product comes
with guarantee of quality.  Only IT is allowed to be sold as "buyer
beware".

  
R Allan Reese
Senior statistician, Cefas
The Nothe, Weymouth DT4 8UB 

Tel: +44 (0)1305 206614 -direct
Fax: +44 (0)1305 206601 

www.cefas.co.uk 



***********************************************************************************
This email and any attachments are intended for the named recipient only.  Its unauthorised use, distribution, disclosure, storage or copying is not permitted.  If you have received it in error, please destroy all copies and notify the sender.  In messages of a non-business nature, the views and opinions expressed are the author's own and do not necessarily reflect those of the organisation from which it is sent.  All emails may be subject to monitoring.
***********************************************************************************


*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/