Home  /  Resources & support  /  Certification results

Diehard test results on Stata's random-number generator for Stata 14

  1. Explanation
  2. Summary
  3. Command and log files
  4. Diehard test p-values
  5. References

1. Explanation

The default pseudorandom-number generator introduced in Stata 14 is the 64-bit Mersenne Twister (Matsumoto and Nishimura 1998). This pseudorandom-number generator was subjected to the Diehard suite of tests, which were developed by George Marsaglia and provide extensive testing of pseudorandom-number generators. The results of those tests are presented here.

2. Summary

The distribution of the p-values from the Diehard suite of tests is shown below and indicates that Stata's random-number generator is doing a great job.

                        Observed   "Expected"
        p-value range   Percent     Percent
        -------------   --------   ----------
         0.0 -- 0.1        11         10
         0.1 -- 0.2         5         10
         0.2 -- 0.3        11         10
         0.3 -- 0.4         8         10
         0.4 -- 0.5        11         10
         0.5 -- 0.6        12         10
         0.6 -- 0.7        11         10
         0.7 -- 0.8         9         10
         0.8 -- 0.9        13         10
         0.9 -- 1.0         7         10

When interpreting the p-values produced by Diehard, be aware of the following note at the top of the Diehard test log file—randnumb_mt64.out.

Most of the tests in Diehard return a p-value, which should be uniform on [0,1) if the input file contains truly independent random bits. Those p-values are obtained by p=F(X), where F is the assumed distribution, often normal, of the sample random variable X. But that assumed F is just an asymptotic approximation, for which the fit will be worst in the tails. Thus you should not be surprised with occasional p-values near 0 or 1, such as .0012 or .9983. When a bit stream really FAILS BIG, you will get p's of 0 or 1 to six or more places. By all means, do not, as a statistician might, think that a p < .025 or p > .975 means that the RNG has "failed the test at the .05 level". Such p's happen among the hundreds that Diehard produces, even with good RNGs. So keep in mind that "p happens".

The smallest p-value produced was .000094, and the largest was .999713, which seems reasonable given the quote above.

3. Command and log files

The web address for the Diehard suite of tests is http://stat.fsu.edu/pub/diehard/.

The Diehard suite of tests require as input a specially formatted binary file containing 3 million 32-bit integers. This specially formatted binary file was first produced, and then the Diehard tests were performed on the resulting file. Here are the steps that generated the results:

  1. The randnumb_mt64.do do-file was run within Stata Version 14 on March 22, 2015. This produced the ASCII file randnumb_mt64.raw containing 3 million 32-bit integers. randnumb_mt64.log is the corresponding Stata log file. (The same numbers are produced by Stata regardless of which operating system is used.)
  2. The program raw2bin.c was run on an Intel i7-based computer running Linux (CentOS 6.6) to convert randnumb_mt64.raw into randnumb_mt64.bin, which is a specially formatted binary file for Diehard. The program called "diehard" was run on an Intel i7-based computer running Linux (CentOS 6.6) with randnumb_mt64.bin as input. The program produced a log file. That file, randnumb_mt64.out, was then copied to this website.

4. Diehard test p-values

The following table shows the p-values produced by the Diehard suite of tests. Click here to see the full Diehard test log file from which these numbers were extracted. A summary of the distribution of these p-values is shown above, along with a note on how to interpret these p-values.

        +---------------------------------------------------------------------------+
        | Birthday Spacing | OPSO             | Count 1s Stream  | Overlapping Sum  |
        | ---------------- | ---------------- | ---------------- | ---------------- |
        | .534258          | .847383          | .665422          | .003240          |
        | .777841          | .570246          |                  | .003961          |
        | .636896          | .413110          |                  | .215563          |
        | .088078          | .407747          |                  | .460988          |
        | .014736          | .399734          | Count 1s Bytes   | .210620          |
        | .263990          | .456519          | ---------------- | .570424          |
        | .844720          | .874095          | .724951          | .550444          |
        | .637550          | .038257          | .610315          | .686698          |
        | .016412          | .507332          | .568201          | .512566          |
        | ---------------- | .689432          | .712203          | .047878          |
        | .276060 KS test  | .366837          | .491990          | ---------------- |
        |                  | .747504          | .220968          | .027433 KS test  |
        |                  | .796370          | .710812          |                  |
        | OPERM5           | .977311          | .686541          |                  |
        | ---------------- | .669694          | .260480          | Runs             |
        | .967568          | .179806          | .259194          | ---------------- |
        | .505140          | .655841          | .927859          | .047776          |
        |                  | .615713          | .469000          | .103935          |
        |                  | .239098          | .630028          | .827530          |
        | Bin. Rank 31x31  | .327388          | .215518          | .175675          |
        | ---------------- | .373348          | .746112          |                  |
        | .254             | .441512          | .010439          |                  |
        |                  | .850614          | .379785          | Craps Wins       |
        |                  |                  | .625812          | ---------------- |
        | Bin. Rank 32x32  |                  | .574355          | .21021           |
        | ---------------- | OQSO             | .778313          |                  |
        | .635             | ---------------- | .553488          |                  |
        |                  | .205365          | .649636          | Craps Throws     |
        |                  | .213164          | .574815          | ---------------- |
        | Bin. Rank 6x8    | .905406          | .494218          | .210209          |
        | ---------------- | .870748          | .800135          | .226782          |
        | .158369          | .073567          |                  |                  |
        | .459508          | .829872          |                  |                  |
        | .836005          | .766247          | CDPARK           |                  |
        | .621290          | .353773          | ---------------- |                  |
        | .725663          | .518021          | .928018          |                  |
        | .795746          | .087738          | .192812          |                  |
        | .939979          | .000094          | .205562          |                  |
        | .930465          | .827285          | .232514          |                  |
        | .125577          | .427818          | .001109          |                  |
        | .060232          | .892217          | .481790          |                  |
        | .032242          | .570399          | .767486          |                  |
        | .849561          | .986136          | .427537          |                  |
        | .871583          | .274626          | .374623          |                  |
        | .052149          | .291834          | .659449          |                  |
        | .499620          | .212180          | ---------------- |                  |
        | .138093          | .322795          | .523303 KS test  |                  |
        | .765582          | .691259          |                  |                  |
        | .346380          | .675573          |                  |                  |
        | .230276          | .842433          | Minimum Dist.    |                  |
        | .723265          | .089919          | ---------------- |                  |
        | .830596          | .499094          | .448147          |                  |
        | .536723          | .912617          |                  |                  |
        | .629361          | .596844          |                  |                  |
        | .548481          | .357562          | 3D Spheres       |                  |
        | .643121          |                  | ---------------- |                  |
        | ---------------- |                  | .978315          |                  |
        | .634144 KS test  | DNA              | .794868          |                  |
        |                  | ---------------- | .062266          |                  |
        |                  | .864610          | .587930          |                  |
        | Overlap 20       | .028606          | .057155          |                  |
        | ---------------- | .854749          | .605296          |                  |
        | .450105          | .728619          | .814708          |                  |
        | .581752          | .851357          | .534597          |                  |
        | .880203          | .846520          | .500675          |                  |
        | .973383          | .741181          | .606762          |                  |
        | .832324          | .451085          | .270515          |                  |
        | .646891          | .818464          | .132134          |                  |
        | .486329          | .817685          | .926987          |                  |
        | .186890          | .084317          | .999713          |                  |
        | .090066          | .100922          | .238046          |                  |
        | .439949          | .895734          | .456988          |                  |
        | .082709          | .800842          | .404913          |                  |
        | .484466          | .732517          | .928175          |                  |
        | .361531          | .108412          | .517729          |                  |
        | .730807          | .842292          | .911754          |                  |
        | .349349          | .845821          | ---------------- |                  |
        | .460293          | .124574          | .314681 KS test  |                  |
        | .317085          | .357627          |                  |                  |
        | .889726          | .249983          |                  |                  |
        | .749739          | .575253          |                  |                  |
        | .385417          | .047594          |                  |                  |
        |                  | .985405          |                  |                  |
        |                  | .473341          |                  |                  |
        |                  | .409390          |                  |                  |
        |                  | .559012          |                  |                  |
        |                  | .528607          |                  |                  |
        |                  | .290937          |                  |                  |
        |                  | .320929          |                  |                  |
        |                  | .556684          |                  |                  |
        +---------------------------------------------------------------------------+

5. References

Matsumoto, M., and T. Nishimura. 1998. Mersenne Twister: A 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Transactions on Modeling and Computer Simulation 8: 3-30.

Diehard test results on Stata's random-number generator for Stata 13 (and earlier)