Re: st: Alternative to coefficient of variation (CV)/relative standard error as a measure of estimate's reliability/stability

[David Radwin <dradwin@mprinc.com>]

Nick, 

Thank you for your thoughtful contribution. I personally agree with your criticism of a one-size-fits-all solution, but that is the sort of statistic I am seeking nonetheless. I would very much appreciate suggestions, however imperfect they may be, from other Statalisters.

Incidentally, an even more thorny situation is when the estimate is zero or close to zero, perhaps because it has been mean-deviated (demeaned) or is a measure of change when there is no change. Then the CV is undefined or essentially infinity!

David
--
David Radwin
Senior Research Associate
MPR Associates, Inc.
2150 Shattuck Ave., Suite 800
Berkeley, CA 94704
Phone: 510-849-4942
Fax: 510-849-0794

www.mprinc.com

----- Original Message -----
From: "Nick Cox" <njcoxstata@gmail.com>
To: statalist@hsphsun2.harvard.edu
Sent: Tuesday, June 12, 2012 5:57:44 PM
Subject: Re: st: Alternative to coefficient of variation (CV)/relative standard error as a measure of estimate's reliability/stability

A simple-minded but I think reasonable first stab at this issue is
that the CV is a fair criterion if SD (specifically SE) is indeed
proportional to mean (generally estimate), so that SD / mean is
approximately constant in a given kind of situation.

If that is not true as a kind of first approximation then information
is needed on the relation between variability and estimate to make a
better judgement.

One example that I studied in excruciating detail several years ago
was the variability of annual rainfall in which many people had been
led from the  observation that variability of rainfall as measured by
SD tends to increase with the mean to conclude that the CV was in some
sense a better summary. However, in many cases it is more nearly true
that SD varies as a power around 0.7 of the mean, in which case CV
over-compensates. CV is, as you imply, even less appropriate for
proportions.

I'm not sure this will help you at all. I became sensitive to
inappropriate uses of the CV years ago when I encountered examples
even worse than those with annual rainfall and my best summary is that
when the CV is not appropriate there is not usually a simpler or even
different alternative that works well at all; rather analysis has to
look directly at the structure of variability.

Nick

On Wed, Jun 13, 2012 at 12:37 AM, David Radwin <dradwin@mprinc.com> wrote:

> I hope you will indulge me this statistics question that is not strictly
> Stata-specific. I am looking for a statistic measuring an estimate's
> reliability or stability as an alternative to the coefficient of variation
> (CV), also known as the relative standard error. The CV is the standard
> error of an estimate (proportion, mean, regression coefficient, etc.)
> divided by the estimate itself, usually expressed as a percentage. For
> example, if a survey finds 15% unemployment with a 6% standard error, the
> CV is .06/.15 = .4 = 40%.
>
> Some US government agencies flag or suppress as unreliable any estimate
> with a CV over a certain threshold such as 30% or 50%. But this standard
> can be arbitrary (for example, 85% employment would have a much lower CV
> of .06/.85 = 7%) and has other limitations.
>
> Can anyone suggest an alternative measure of stability or reliability?
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/