[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]
Re: st: Memory during a merge
Fred Wolfe <email@example.com> replied to my desciption of
how -merge, nokeep()- works,
> So if I understand you correctly, the observations are brought in and then
> the non keep variables are deleted? If that is the case the maximum memory
> use with keep() would be no different than from the memory use without keep.
In Stata, you think of bringing in the entire dataset. Inside the C
code, we have the ability to bring in the dataset an observation at a time,
and to throw those observations away as we go.
Consider a dataset of 10,000 observations, each 5,000 bytes long. The dataset
is then 10,000x5,000 = 50,000,000 bytes. Let's say that's the dataset on disk
when we -merge- and that we -keep()- only one of the variables: a 2-byte one.
We do not bring in the 50,000,000 bytes and then reduce that to 2*10,000 =
We bring in 5,000 bytes. We then decide whether that observation merges.
If it does, we copy 2 bytes from the 5,000 bytes to the appropriate place, and
then throw the 5,000 bytes away. Then we do it again. And again.
Fred mentioned -merge-'s -keep()- option (keep a subset of the variables), but
he did not mention the -nokeep- option (keep only observations that merge).
Perhaps Fred wants to specify both options.
* For searches and help try: