Notice: On April 23, 2014, Statalist moved from an email list to a forum, based at statalist.org.
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: pairing unpaired data [was: Re: st: any idea?]
From
"Y.R.E. Retamal" <[email protected]>
To
[email protected]
Subject
Re: pairing unpaired data [was: Re: st: any idea?]
Date
Sat, 11 Jan 2014 12:52:16 +0000
Dear all
Many thanks to Nick(s), Sarah and Fernando for your response and sorry
for the delay in my response, I have been off the last week.
I have checked your suggestions and it seems Sarah advice has been the
most useful until now. I cannot find the teffects command in STATA 11,
so I could not check it.
The method of osteometric sorting that I want to perform is described by
John Byrd in the Chapter 10 "Models and Methods for Osteometric Sorting"
from the book "Recovery, Analysis, and Identification of Commingled
Human Remains", Bradley J. Adams and John E. Byrd, eds. Humana Press,
2008. It can be found online relatively easy.
In this chapter, Byrd explain the methods of osteometric sorting:
"The basic principle underlying osteometric sorting is that two bones
that are of sizes more disparate than observed in most humans are likely
to be commingled".
Models for comparison of right and left paired bones were developed that
emphasize shape, taking the general form D = Σ(ai − bi), where a is the
right side bone measurement i, and b is the left side bone measurement
i for each of the measurements included in the comparison. The null
hypothesis of no difference is tested by comparing the value of D
against “0” (no difference) and using the reference data standard
deviation of D. The deviation from “0” divided by the reference data
standard deviation is evaluated against the t-distribution with two
tails to obtain a p-value. A low p-value provides a measure of the
strength of evidence against the null, which can also be taken as
evidence for how atypical the case specimens are assuming they originate
in the same individual.
Best wishes
Rodrigo
On 2014-01-08 16:39, Nick Winter wrote:
Could nearest-neighbor matching from the land of treatment effects
estimation be repurposed here?
Using the data input below, something like this:
encode side, gen(nside)
gen junkoutcome = uniform()
teffects nnmatch (junkoutcome length) (nside), generate(match)
list id type side length match*, clean
id type side length match1
1. 1 femur left 18 11
2. 2 femur left 65.85 12
3. 3 femur left 69.1 12
4. 4 femur left 130 16
5. 5 femur left 131.2 16
6. 6 femur left 143 18
7. 7 femur left 145 18
8. 8 femur left 160 19
9. 9 femur left 183 20
10. 10 femur left 200 20
11. 11 femur right 28 1
12. 12 femur right 80 3
13. 13 femur right 96.5 3
14. 14 femur right 126 4
15. 15 femur right 127 4
16. 16 femur right 128 4
17. 17 femur right 138 6
18. 18 femur right 146 7
19. 19 femur right 148 7
20. 20 femur right 200 10
Nick Winter
On 1/7/2014 3:36 PM, Sarah Edgington wrote:
Rodrigo,
This is a complicated problem because it requires doing a calculation
for
each possible pair of left/right bones. Depending on how many bones
you
have, this could turn out to be quite cumbersome.
The near matching method Fernando suggests could work, but the fact
that
you'll ultimately need to match on more than one dimension seems like
it
might create problems.
How many bones of each type do you actually have? If it's a
relatively
small number (for example a few hundred each of left and right for
each type
of bone) you may be able to just use a brute force method by creating
a
dataset with each possible combination of left and right bones.
You'd want
to do this separately by bone type.
For example, you might create a dataset of left femur measurements and
a
dataset of right femur measurements. You could then use joinby to
create
all the possible combinations between the two.
This might look something like the code below (note that I've only
input the
femur data here, but this code assumes you have other types as well).
Keep
in mind that this creates a dataset that has NrightXNleft
observations. For
large datasets this likely won't be possible.
clear
input id str10 type str5 side length
1 femur left 18
2 femur left 65.85
3 femur left 69.1
4 femur left 130
5 femur left 131.2
6 femur left 143
7 femur left 145
8 femur left 160
9 femur left 183
10 femur left 200
11 femur right 28
12 femur right 80
13 femur right 96.5
14 femur right 126
15 femur right 127
16 femur right 128
17 femur right 138
18 femur right 146
19 femur right 148
20 femur right 200
end
keep if type=="femur"
preserve
keep if side=="left"
rename length left_length
rename id left_id
drop side
tempfile leftfemur
save `leftfemur'
restore
keep if side=="right"
rename length right_length
rename id right_id
drop side
joinby type using `leftfemur'
**you now have every possible pair of measurements
gen lengthdiff=abs(right_length-left_length)
At this point you'll need very exact rules about what constitutes a
match.
Once you've done that, that is still not the end of the task. From
there
you'll have to see how often you have bones that match multiple other
bones.
Again, to do this you'll need to specify the exact rules about what is
"close enough" to consider it a possible match. Then you'll need to
come up
with rules for disambiguation.
This is not an elegant solution and if you have a lot of data it may
not
work. However, if you have few enough cases for this to work it has
the
advantage of making it pretty easy to specify matching rules for
multiple
measurements.
-Sarah -----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Fernando
Rios
Avila
Sent: Tuesday, January 07, 2014 11:38 AM
To: [email protected]
Subject: Re: pairing unpaired data [was: Re: st: any idea?]
Rodrigo,
Perhaps a direction you could follow is by using a near matching
method.
Since you can separate the information in two datasets (namely left
and
right), you can do so, and then "merge" them using the user written
program
-nearmrg-.
That will give you a start point to match up your data, but you might
need
to make further revisions to ensure that there are no duplicate
matching.
Best
On Tue, Jan 7, 2014 at 2:27 PM, Nick Cox <[email protected]> wrote:
Thanks for the details of your problem. I can't see that you have a
method that is translatable into Stata code: your procedure is too
vaguely specified. That need not stop other people suggesting
methods.
Nick
[email protected]
On 7 January 2014 19:20, Y.R.E. Retamal <[email protected]> wrote:
Dear Nick
Thanks a lot for your soon response. The method is no more than
showed. I have to add other variables like width and height for the
same bone. So, if three variables match, probably both bones would
be
from the same skeleton.
I would expect that many bones would not match between them, so I
could discard them being from the same skeleton. Problems would
appear if e.g. a right bone matches with more than one left bone.
But
at least I could simplify the work and after I could focus on
problematic
cases.
Rodrigo
On 2014-01-07 18:49, Nick Cox wrote:
I changed the thread title, which was not informative.
You need a method. Some predictable pitfalls are that for some
bones
there is no acceptable match and that others there could be two or
more acceptable matches. I don't think there is a canned solution
independent of your spelling out what the method is.
Nick
[email protected]
On 7 January 2014 18:20, Y.R.E. Retamal <[email protected]> wrote:
Thank you very much Eric and Nick for the advices.
I will try to give a clearer idea of what want to do:
For example I have the following database of human bones. I
removed
missing values of length for a better understanding:
id type side length id type side
length
1 femur left 18 21 humerus left 13
2 femur left 65.85 22 humerus left 56
3 femur left 69.1 23 humerus left 92
4 femur left 130 24 humerus left
126
5 femur left 131.2 25 humerus left
154
6 femur left 143 26 humerus left
170
7 femur left 145 27 humerus left
198
8 femur left 160 28 humerus left
228
9 femur left 183 29 humerus left
230
10 femur left 200 30 humerus left
232
11 femur right 28 31 humerus right
238
12 femur right 80 32 humerus right 10
13 femur right 96.5 33 humerus right 66
14 femur right 126 34 humerus right
123
15 femur right 127 35 humerus right
128
16 femur right 128 36 humerus right
143
17 femur right 138 37 humerus right
200
18 femur right 146 38 humerus right
228
19 femur right 148 39 humerus right
230
20 femur right 200 40 humerus right
241
These data belong to a commingled skeletal collection and some
right bones (femurs and humerus respectively) should match with a
left bone, but I do not know which bones match. Following the idea
that a right bone from a same skeleton should have the same length
(approximately) with its respective left bone, I want to subtract
each right femur to each left femur, with the aim to find which
right femur matches with a left femur, i.e. have the same or
almost
the same length, so the subtraction would be zero or near zero.
The same proceeding with the humerus (and other bones).
If you have any idea to perform this, please let me know.
Rodrigo
Best wishes
Rodrigo
On 2014-01-05 23:54, Nick Cox wrote:
<>
Eric Booth gives very good advice.
Your problem with the link to the Stata Journal file you were
directed to me may be just that you didn't step past the standard
material bundled with every reprint file.
Nick
[email protected]
On 5 January 2014 21:03, Eric Booth <[email protected]>
wrote:
<>
The Stata Journal link you mention that Nick sent you works for
me.
The
title of the article is "Stata tip 71: The problem of split
identity, or how to group dyads" by Nick J. Cox, so maybe you
can
google that title if your browser isn't navigating to it
properly.
Your example dataset doesn't align with your desired dataset.
How do we know what is x and what is j in the first 20 obs of
your example data (see below) (also note the Statalist FAQ about
not sending
attachments) ?
You need some kind of identifier that ties, for example, obs or
id 1 (even though it's missing) to the other right side femur
observation of interest (is it id 7 or id 9 or ??).
**your example data:
id type side length
1 femur right
2 femur left
3 femur right
4 femur left
5 femur right 373
6 femur left 416
7 femur right 138
8 femur left
9 femur right 270
10 femur left
11 femur left
12 femur right
13 femur left
14 femur right
15 femur left 281
16 femur right
17 femur left 160
18 femur left
19 femur right
20 femur left
We can't just sort by 'type' and 'side' to get a dataset of the
same structure as you presented initially, so I think you need
to
provide more information about this. (also, if the rule is, as
you imply, to sort by type and side and then subtract every
third
observation from each other then what do we do with missing
'length' and missing 'side'?)
If the rule is that id 1 and id 2 are a pair then whey does the
left/right ordering suddenly change starting around id 17?
- Eric
On Jan 5, 2014, at 2:46 PM, Y.R.E. Retamal <[email protected]>
wrote:
Dear Guys
Some weeks ago, Red Owl and Nick helped me with some loops for
my work.
I have tried to run some suggestion in my dataset, but I had
some difficulties.
I give you the basic structure of my dataset and my question:
I want to create some new variables containing the difference
between the length of two individuals from different groups:
id side length newvar1 newvar2 newvar3
1 right x x-j x-k x-l
2 right y y-j y-k y-l
3 right z z-j z-k z-l
4 left j j-x j-y j-z
5 left k k-x k-y k-z
6 left l l-x l-y l-z
Red Owl suggested me following this example:
*** BEGIN CODE ***
* Build demo data set.
clear
* Length is capitalized to distinguish from length().
input id str5(side) Length
1 right 10
2 right 15
3 right 11
4 left 13
5 left 10
6 left 12
end
gen byte newvar1 = .
forval i = 1/3 {
replace newvar1 = Length[`i'] - Length[4] in `i'
}
forval i = 4/6 {
replace newvar1 = Length[`i'] - Length[1] in `i'
}
gen byte newvar2 = .
forval i = 1/3 {
replace newvar2 = Length[`i'] - Length[5] in `i'
}
forval i = 4/6 {
replace newvar2 = Length[`i'] - Length[2] in `i'
}
gen byte newvar3 = .
forval i = 1/3 {
replace newvar3 = Length[`i'] - Length[6] in `i'
}
forval i = 4/6 {
replace newvar3 = Length[`i'] - Length[3] in `i'
}
list, noobs sep(0)
*** END CODE ***
However, my dataset is much more longer and is difficult to
perform it.
I hope you can help me giving me more ideas.
I send you an extract of my dataset in .xlsx format Also, the
webpage suggested by Nick to review the discussion about the
topic
(http://www.stata-journal.com/sjpdf.html?articlenum=dm0043)
redirects
me to a non-sense file to download. Please give me the number
of
the journal to read the discussion.
Happy new year to all of you
Rodrigo
On 2013-12-15 22:39, Y.R.E. Retamal wrote:
Dear Red Owl and Nick
Thank you very much for your response. The code works
perfectly, just as I need.
Best wishes
Rodrigo
On 2013-12-14 22:31, Nick Cox wrote:
In addition to Red's helpful suggestions, note that technique
for such paired data was discussed in
http://www.stata-journal.com/sjpdf.html?articlenum=dm0043
which is publicly accessible. The problem is that the
identifiers in Rodrigo's example appear to make little sense.
How is Stata expected to know that 1 and 4, 2 and 5, 3 and 6
are paired? Perhaps the structure of the dataset is clearer
in
practice. If so, basic calculations are just a couple of
lines or
so.
Nick
[email protected]
On 14 December 2013 15:33, Red Owl <[email protected]> wrote:
Rodrigo,
The following code demonstrates an approach with basic
loops.
It could be made more efficient with a different loop
structure, but this approach may be more informative.
*** BEGIN CODE ***
* Build demo data set.
clear
* Length is capitalized to distinguish from length().
input id str5(side) Length
1 right 10
2 right 15
3 right 11
4 left 13
5 left 10
6 left 12
end
gen byte newvar1 = .
forval i = 1/3 {
replace newvar1 = Length[`i'] - Length[4] in `i'
}
forval i = 4/6 {
replace newvar1 = Length[`i'] - Length[1] in `i'
}
gen byte newvar2 = .
forval i = 1/3 {
replace newvar2 = Length[`i'] - Length[5] in `i'
}
forval i = 4/6 {
replace newvar2 = Length[`i'] - Length[2] in `i'
}
gen byte newvar3 = .
forval i = 1/3 {
replace newvar3 = Length[`i'] - Length[6] in `i'
}
forval i = 4/6 {
replace newvar3 = Length[`i'] - Length[3] in `i'
}
list, noobs sep(0)
*** END CODE ***
Good luck.
Red Owl
[email protected]
Y.R.E. Retamal" <[email protected]> Sat, 14 Dec 2013
12:08:42:
Dear list
I am very complicated trying to perform an analysis using
STATA and I
cannot find the way. Maybe you could help me. I want to
create
some
new
variables containing the difference between the length of
two
individuals from different groups:
id side length newvar1 newvar2
newvar3
1 right x x-j x-k
x-l
2 right y y-j y-k
y-l
3 right z z-j z-k
z-l
4 left j j-x j-y
j-z
5 left k k-x k-y
k-z
6 left l l-x l-y
l-z
I do not know if I do explain myself clearly, the
individuals
are
bones (clavicles, for example), so it is possible that some
right
clavicles pair-match with left clavicles, following the idea
that
an
individual has bone of similar length.
Any help could bring me a light!
Best wishes
Rodrigo
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
*
http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
*
http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/
<example.xlsx>
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/
*
* For searches and help try:
* http://www.stata.com/help.cgi?search
* http://www.stata.com/support/faqs/resources/statalist-faq/
* http://www.ats.ucla.edu/stat/stata/