Stata The Stata listserver
[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]

st: RE: thanks again, more Stata data manipulation tips, or alternative statistical software?


From   "Nichols, Austin" <ANichols@ui.urban.org>
To   "'statalist@hsphsun2.harvard.edu'" <statalist@hsphsun2.harvard.edu>
Subject   st: RE: thanks again, more Stata data manipulation tips, or alternative statistical software?
Date   Mon, 23 Aug 2004 17:00:24 -0400

Stata 8 works fine as long at you don't -set mem- to 
more than about 65% of your system's free physical RAM
when using a MS Windows platform.  Windows seems often 
to eat up about 30% of the RAM, and more as you leave 
your CPU running, so if you have 512MB, just -set mem 
350m- or -set mem 400m- at most and reshape the data 
in halves, or upgrade your RAM.  

It's unclear to me why StataCorp has never integrated 
better data management tools in the executable (e.g. 
why not zip and unzip all data on the fly?)

Maybe someone on the list can recommend a freeware data 
re-shaper.  Most folks I know use SAS to make the data, 
and Stata to analyze the data, but this is clearly a 
second-best and expensive solution.

-----Original Message-----
From: yumin sheng [mailto:shengyumin@yahoo.com]
Sent: Monday, August 23, 2004 3:16 PM
To: statalist@hsphsun2.harvard.edu
Subject: st: thanks again, more Stata data manipulation tips, or
alternative statistical software?


Dear all,

  Thanks so much again to Dev for recommending
-reshape-, to Toyoto for venturing an illustrative
example, and especially to Professor Nick Cox for
kindly writing the immensely helpful code to "reshape"
the WDI data. Sorry that I haven't been able to follow
up on this until now; it took my computer many days to
figure out that Stata might not be able to easily
"manipulate" dataset of this large size. 

  Indeed, although with 2 or 3 countries' worth of
data from WDI each time, Stata 8 was able to perform
Nick's powerful "reshape" code quite smoothly and
speedily, it seemed very slow when the entire dataset
of around 200 countries (with 200-500 variables each,
for 43 years) is included. Nick's first-line -reshape-
command took about 24 hours to execute, the "fillin"
command took much longer and could never finish
completely after about 60 hours before Stata finally
crashed.

  The original WDI data is 62.5 MB in Stata, and the
size of the transformed data after the first command
"reshape long v , i(country var) j(year)" grows to 540
MB. My computer has a memory of about 512 MB DDR SDRAM
at 333MHz. The hard drive has 80 GB. Maybe it is
because I am just using the intercooled version of
Stata 8, not the SE, but I also set the Stata memory
to 1000 MB.

  Is it true that Stata 8 is no longer suitable for
data manipulation of this kind? If not, how should I
proceed next so that Nick's code could be run to the
finishing line? If yes, could anyone kindly suggest
alternative statistical software that might do the
job?

  Terribly sorry to trouble you all again with this,
but your wise suggestion will be tremendously
appreciated, as always!

Best wishes,
Yumin

--- Nick Cox <n.j.cox@durham.ac.uk> wrote:

> Yumin Sheng asked [slight editing here,  combining 
> two postings] 
> 
> ---------------------
> I have been trying to save data from World
> Development
> Indicators (WDI, online) into a cross-section
> time-series format. The original data look like the
> following
> 
> Country	Variable Name	1960	1961	...	2002
> Afghanistan	Aid	X	X	...	X
> Afghanistan	Population	X	X	...X 
> ...	...	...	...	...	...
> ...	...	...	...	...	...
> Afghanistan	Wage	X	X	...	X
> Albania	 Aid		X	X        ...	X
> Albania	Population	X	X	...	X
> ...	...	...	...	...	...
> ...	...	...	...	...	...
> Albania	Wage	X	X	...	X
> 
> The ideal format I would like to have would be:
> 
> Country	Aid	Population	...	...	Wage
> Afghanistan	1960	X	X	...	X
> Afghanistan	1961	X	X	...	X
> ...	...	...	...	...	...
> ...	...	...	...	...	...
> Afghanistan	2002	X	X	...	X
> Albania	1960	X	X	...	X
> Albania	1961	X	X	...	X
> ...	...	...	...	...	...
> ...	...	...	...	...	...
> Albania	2002	X	X	...	X
> ...	...	...	...	...	...
> China	1960	X	X	...	X
> China	1961	X	X	...	X
> ...	...	...	...	...	...
> ...	...	...	...	...	...
> China	2002	X	X	...	X
> 
> World Development Indicators have about 200
> variables
> for each country. 
> 
> But (1) sometimes many words are contained in
> one variable name such as "Export share in GDP, %"
> and (2) some countries have data on more
> variables than do other countries. 
> --------------------
> 
> Dev Vencappa recommended -reshape- and Toyoto Iwata 
> posted code for a example dataset. Toyoto's code 
> doesn't extend to coping with the first problem 
> above and I am not sure that it would cope with the 
> second problem above. 
> 
> Here is a sketch of slightly more general code:
> the dataset is just an example. (I guess that 
> there is a shorter and still general solution
> but I can't find it.) 
> 
> . l 
> 
>      +------------------------------------------+
>      |     country     variable   v1960   v1961 |
>      |------------------------------------------|
>   1. | Afghanistan          Aid       1       6 |
>   2. | Afghanistan   Population       2       7 |
>   3. |     Albania          Aid       3       8 |
>   4. |     Albania      Fishing       4       9 |
>   5. |     Albania   Population       5      10 |
>      +------------------------------------------+
> 
> -reshape- to long and then rectangularise. 
> -fillin- takes care of problem (2). 
> 
> . reshape long v , i(country var) j(year) 
> . fillin country var year 
> . drop _fillin 
> 
> We know from problem (1) that the values in 
> -variable- aren't all legal variable names, 
> so we can't use them directly. One trick is 
> 
> . egen group = group(variable) 
> 
> That gives ways of counting variables and 
> years, so our code is more general: 
> 
> . su group , meanonly 
> . local nvars = r(max) 
> . qui count if country == country[1] & group ==
> group[1]
> . local nyears = r(N) 
> 
> Now we split the composite -v- into variables
> and assign variable labels, picking them up 
> from -variable-: 
> 
> . gen order = _n 
> 
> . qui forval i = 1/`nvars' { 
> .	gen v`i' = v if group == `i' 
> .	su order if group == `i', meanonly 
> .	label var v`i' `"`=variable[`r(min)']'"' 
> . }
> 
> . drop v variable order 
> 
> . l 
> 
>      +-------------------------------------------+
>      |     country   year   group   v1   v2   v3 |
>      |-------------------------------------------|
>   1. | Afghanistan   1960       1    1    .    . |
>   2. | Afghanistan   1961       1    6    .    . |
>   3. | Afghanistan   1960       2    .    .    . |
>   4. | Afghanistan   1961       2    .    .    . |
>   5. | Afghanistan   1960       3    .    .    2 |
>      |-------------------------------------------|
>   6. | Afghanistan   1961       3    .    .    7 |
>   7. |     Albania   1960       1    3    .    . |
>   8. |     Albania   1961       1    8    .    . |
>   9. |     Albania   1960       2    .    4    . |
>  10. |     Albania   1961       2    .    9    . |
>      |-------------------------------------------|
>  11. |     Albania   1960       3    .    .    5 |
>  12. |     Albania   1961       3    .    .   10 |
>      +-------------------------------------------+
> 
> We need to pull values for most variables from 
> later in order: 
> 
> . qui forval j = 2/`nvars' { 
> . 	replace v`j' = v`j'[_n+`nyears' * (`j' - 1)] 
> . }
> 
> . keep if group == 1
> . drop group 
> 
> . l 
> 
>      +-----------------------------------+
>      |     country   year   v1   v2   v3 |
>      |-----------------------------------|
>   1. | Afghanistan   1960    1    .    2 |
>   2. | Afghanistan   1961    6    .    7 |
>   3. |     Albania   1960    3    4    5 |
>   4. |     Albania   1961    8    9   10 |
>      +-----------------------------------+
> 
> I also looked at a solution centred on -xpose-, 
> but the preparation and clean-up were not 
> easier than this. 
> 
> Nick 
> n.j.cox@durham.ac.uk 
> 
> *
> *   For searches and help try:
> *  
> http://www.stata.com/support/faqs/res/findit.html
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
> 



		
_______________________________
Do you Yahoo!?
Win 1 of 4,000 free domain names from Yahoo! Enter now.
http://promotions.yahoo.com/goldrush
*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/



© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   What's new   |   Site index