Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: st: Identifying differences in partial duplicates
From
Neil Shephard <[email protected]>
To
[email protected]
Subject
Re: st: Identifying differences in partial duplicates
Date
Fri, 17 Dec 2010 16:56:45 +0000
On Fri, Dec 17, 2010 at 4:01 PM, David Puddicombe
<[email protected]> wrote:
> I am working with a dataset which contains 80,000 observations and 100 variables. There are a few hundred partial duplicates which need to be investigated to determine whether they are duplicate observations with slightly different data or unique observations.
>
> I can identify the partial duplicates but not which variables have different values for the very similar observations.
>
> dups b1 b5 b6 c1 d2_dt d3 d4 t2 d5_dt d6 d7 d13 d14, key(id lcm_id tcode xcode ppob) verbose
>
> Can anyone suggest an efficient method of highlighting the difference(s) between the potentially duplicated observations.
You don't state which version of Stata you are using nor the author of
the -dups- command (which is Nick Cox and Thomas Steichen).
Perhaps consider using the official -duplicates- command with either
the -list- or more likely to the -tag- option to first tag your
duplicates and then e.g.
duplicates tag b1 b5 b6 c1 d2_dt d3 d4 t2 d5_dt d6 d7 d13 d14, gen(poss_dup)
list id lcm_id tcode xcode ppob if(poss_dup != 0)
Without resorting to -outsheet-ing possible duplicates to separate
files (see useful HowTo on flagging duplicates in this manner at
http://www.stata.com/support/faqs/data/distinct.html) and then using
*NIX tools of 'diff' I can only think that listing and eye-balling the
possible duplicates is the best way.
Neil
--
"Our civilization would be pitifully immature without the intellectual
revolution led by Darwin" - Motoo Kimura, The Neutral Theory of
Molecular Evolution
Email - [email protected]
Website - http://kimura-no-ip.org/
Photos - http://www.flickr.com/photos/slackline/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/statalist/faq
* http://www.ats.ucla.edu/stat/stata/