Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down at the end of May, and its replacement, statalist.org is already up and running.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: Identifying differences in partial duplicates


From   Neil Shephard <nshephard@gmail.com>
To   statalist@hsphsun2.harvard.edu
Subject   Re: st: Identifying differences in partial duplicates
Date   Fri, 17 Dec 2010 16:56:45 +0000

On Fri, Dec 17, 2010 at 4:01 PM, David Puddicombe
<david.puddicombe@npeu.ox.ac.uk> wrote:
> I am working with a dataset which contains 80,000 observations and 100 variables.  There are a few hundred partial duplicates which need to be investigated to determine whether they are duplicate observations with slightly different data or unique observations.
>
> I can identify the partial duplicates but not which variables have different values for the very similar observations.
>
> dups b1 b5 b6 c1 d2_dt d3 d4 t2 d5_dt d6 d7 d13 d14, key(id lcm_id tcode xcode ppob) verbose
>
> Can anyone suggest an efficient method of highlighting the difference(s) between the potentially duplicated observations.

You don't state which version of Stata you are using nor the author of
the -dups- command (which is Nick Cox and  Thomas Steichen).

Perhaps consider using the official -duplicates- command with either
the -list- or more likely to the -tag- option to first tag your
duplicates and then e.g.

duplicates tag b1 b5 b6 c1 d2_dt d3 d4 t2 d5_dt d6 d7 d13 d14, gen(poss_dup)
list id lcm_id tcode xcode ppob if(poss_dup != 0)

Without resorting to -outsheet-ing possible duplicates to separate
files (see useful HowTo on flagging duplicates in this manner at
http://www.stata.com/support/faqs/data/distinct.html) and then using
*NIX tools of 'diff' I can only think that listing and eye-balling the
possible duplicates is the best way.


Neil



-- 
"Our civilization would be pitifully immature without the intellectual
revolution led by Darwin" - Motoo Kimura, The Neutral Theory of
Molecular Evolution

Email - nshephard@gmail.com
Website - http://kimura-no-ip.org/
Photos - http://www.flickr.com/photos/slackline/

*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index