Bookmark and Share

Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: Identifying differences in partial duplicates

From   Neil Shephard <>
Subject   Re: st: Identifying differences in partial duplicates
Date   Fri, 17 Dec 2010 16:56:45 +0000

On Fri, Dec 17, 2010 at 4:01 PM, David Puddicombe
<> wrote:
> I am working with a dataset which contains 80,000 observations and 100 variables.  There are a few hundred partial duplicates which need to be investigated to determine whether they are duplicate observations with slightly different data or unique observations.
> I can identify the partial duplicates but not which variables have different values for the very similar observations.
> dups b1 b5 b6 c1 d2_dt d3 d4 t2 d5_dt d6 d7 d13 d14, key(id lcm_id tcode xcode ppob) verbose
> Can anyone suggest an efficient method of highlighting the difference(s) between the potentially duplicated observations.

You don't state which version of Stata you are using nor the author of
the -dups- command (which is Nick Cox and  Thomas Steichen).

Perhaps consider using the official -duplicates- command with either
the -list- or more likely to the -tag- option to first tag your
duplicates and then e.g.

duplicates tag b1 b5 b6 c1 d2_dt d3 d4 t2 d5_dt d6 d7 d13 d14, gen(poss_dup)
list id lcm_id tcode xcode ppob if(poss_dup != 0)

Without resorting to -outsheet-ing possible duplicates to separate
files (see useful HowTo on flagging duplicates in this manner at and then using
*NIX tools of 'diff' I can only think that listing and eye-balling the
possible duplicates is the best way.


"Our civilization would be pitifully immature without the intellectual
revolution led by Darwin" - Motoo Kimura, The Neutral Theory of
Molecular Evolution

Email -
Website -
Photos -

*   For searches and help try:

© Copyright 1996–2018 StataCorp LLC   |   Terms of use   |   Privacy   |   Contact us   |   Site index