[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]

From |
"Lina Jonsson" <lina.jonsson@vti.se> |

To |
<statalist@hsphsun2.harvard.edu> |

Subject |
SV: st: SV: Re: Memory problems when creating a spatial weight matrix |

Date |
Wed, 26 Sep 2007 12:59:47 +0200 |

Hi again. Thank you for the quick reply Sergiy! There is no problem running the code you gave me. I belive that the spatwmat-command is creating something temporarily that takes a lot of memory. I have only been using Stata for a couple of months so I have never looked into a source code before - how do I do that? Does anyone else have experience in using the spatwmat-command on "large" (more than 800 observations) datasets? Best Regards /Lina Jonsson > -----Ursprungligt meddelande----- > Från: owner-statalist@hsphsun2.harvard.edu > [mailto:owner-statalist@hsphsun2.harvard.edu] För Sergiy Radyakin > Skickat: den 26 september 2007 12:34 > Till: statalist@hsphsun2.harvard.edu > Ämne: Re: st: SV: Re: Memory problems when creating a spatial > weight matrix > > Hi Lina, > > may be I am missing something, but a matrix 500x500 at 4 > bytes per cell occupies 1MB of memory. At 8 bytes per cell > 2MB respectively. How does this relate to the 500/900MB of > memory limit? As I understand, your dataset is 800 > observations and -spatwmat- produces a matrix 800x800. > > Could you run the following and tell us what happens? > > clear > set mem 100m <-- or whatever your maximum is > set matsize 800 > matrix Z=J(800,800,0) > matrix dir > matrix A=J(800,800,0) > matrix dir > matrix B=J(800,800,0) > matrix dir > matrix C=J(800,800,0) > matrix dir > use "mydata" <--- whatever your dataset is > > If it works fine, than it is not a memory issue, but rather > you'd have to look into the source code of -spatwmat- to find > if the memory is used there efficiently (e.g. it does not > generate 800 temporary matrices). > > From the first look, it might be relatively easy to port the > code into matlab. However then you need matlab, why not Stata > SE? If you don't have constraints with software -- buy Stata > 10 SE and this might resolve the issue. If you do have > constraints, than what are they (which software is available > to you)? There might be just a handful of people working with > specialized commands like this, so please explain us what > this command does in plain English. > > Best regards, > Sergiy Radyakin > > > > > > On 9/26/07, Lina Jonsson <lina.jonsson@vti.se> wrote: > > Dear statalister. > > > > Sorry I haven´t had time to follow up my question until now. > > > > The memory problem occurs only when using the command for > creating the spatial weight matrix (spatwmat). The dataset > I´m using consists of only 28 variables - all numerical and > dummies (0,1). According to an earlier question concering > memory problems when using the spatwmat-command; "-spatwmat- > will reset the number of observations to (N/2)*N-1" which > means about 320 000 if I´m using a dataset with 800 > observations. The advice that is given in the former question > is to set memory to a higher value. That is the advice that > also is given for my error message "no room to add more > variables". But the problem is that I cannot set memory to > more then 940m. > > > > I am using Windows XP and have just increased my RAM-memory > to 2 GB which had no effect at all on the maximum memory I > can set within Stata. As I understand it; the hotfix Michel > mentioned is only for Stata 10 and I`m using Intercooled Stata 9.2. > > > > Is anyone having any suggestions on how this problem can be > solved or is it not possible to use the spatwmat-command on > datasets with 800 observation using "a normal computer". In > fact I whould like to use up to 1738 observations but for > that I whould need to switch to Stata SE instead of > Intercooled Stata due to the fact that Intercooled Stata can > set matsize to a maximum of 800. But there is no point in > changing to Stata SE if the available memory sets the limit > to less than 800 observations... > > > > If Stata can´t handle spatial weight matrices for large > datasets, do any of you have any suggestions of other > statistical software that can? > > > > Best regards. > > > > /Lina Jonsson > > > > > > > > > > > Från: owner-statalist@hsphsun2.harvard.edu > > > [mailto:owner-statalist@hsphsun2.harvard.edu] För Michael Blasnik > > > Skickat: den 21 september 2007 14:49 > > > Till: statalist@hsphsun2.harvard.edu > > > Ämne: st: Re: Memory problems when creating a spatial > weight matrix > > > > > > You need to give us more information about your dataset, > operating > > > system, etc. > > > The 940m should be plenty of room for a 500 observation > dataset and > > > an 800x800 matrix. Can you describe your dataset in more > detail? > > > Do you have hundreds of long string variables? Is there > a way for > > > you to encode any long strings and compress other variables? In > > > terms of accessing more memory -- what OS are you using? If MS > > > Windows XP, have you tried the memory hotfix described here > > > http://www.stata.com/support/faqs/win/winmemory.html? > > > > > > Michael Blasnik > > > > > > ----- Original Message ----- > > > From: "Lina Jonsson" <lina.jonsson@vti.se> > > > To: <statalist@hsphsun2.harvard.edu> > > > Sent: Friday, September 21, 2007 5:25 AM > > > Subject: st: Memory problems when creating a spatial weight matrix > > > > > > > > > > Dear statalisters. > > > > > > > > I am working with a hedonic price study where I whould like > > > to check for > > > > and correct for spatial autocorrelation. For this I > whould like to > > > > create a spatial weight matrix based on the x- and > y-coordinates > > > > for each house using the spatwmat-command in spdep. Since I > > > have Intercooled > > > > Stata the maximum matsize that I can set is 800, which > > > means that I can > > > > use no more than 800 observations. But since I cannot set > > > memory to more > > > > than 940m, the actual numbers of observations that I can use is > > > > less than 500. This is to few. > > > > > > > > After reading some old questions concerning memory problems > > > I realize > > > > that this might be a problem that is hard to solve. > > > > Does any of you have any suggestions on how I can increase > > > the memory in > > > > Stata. Can increasing the RAM-memory on my computer > help? Or is it > > > > unrealistic to think that it will be possible to create a > > > matrix whith > > > > the spatwmat-command that will be almost 8000x800 on a > "standard" > > > > computer? Should I change to some other software and what > > > in that case? > > > > > > > > I am greatful for any suggestions! > > > > > > > > /Lina Jonsson > > > > > > * > > > * For searches and help try: > > > * http://www.stata.com/support/faqs/res/findit.html > > > * http://www.stata.com/support/statalist/faq > > > * http://www.ats.ucla.edu/stat/stata/ > > > > > > > * > > * For searches and help try: > > * http://www.stata.com/support/faqs/res/findit.html > > * http://www.stata.com/support/statalist/faq > > * http://www.ats.ucla.edu/stat/stata/ > > > > * > * For searches and help try: > * http://www.stata.com/support/faqs/res/findit.html > * http://www.stata.com/support/statalist/faq > * http://www.ats.ucla.edu/stat/stata/ > * * For searches and help try: * http://www.stata.com/support/faqs/res/findit.html * http://www.stata.com/support/statalist/faq * http://www.ats.ucla.edu/stat/stata/

**Follow-Ups**:**Re: st: SV: Re: Memory problems when creating a spatial weight matrix***From:*"Sergiy Radyakin" <serjradyakin@gmail.com>

**References**:**st: bug in ivtobit***From:*"David Roodman (DRoodman@cgdev.org)" <DRoodman@CGDEV.ORG>

**st: Memory problems when creating a spatial weight matrix***From:*"Lina Jonsson" <lina.jonsson@vti.se>

**st: Re: Memory problems when creating a spatial weight matrix***From:*"Michael Blasnik" <michael.blasnik@verizon.net>

**st: SV: Re: Memory problems when creating a spatial weight matrix***From:*"Lina Jonsson" <lina.jonsson@vti.se>

**Re: st: SV: Re: Memory problems when creating a spatial weight matrix***From:*"Sergiy Radyakin" <serjradyakin@gmail.com>

- Prev by Date:
**[no subject]** - Next by Date:
**Re: st: SV: Re: Memory problems when creating a spatial weight matrix** - Previous by thread:
**Re: st: SV: Re: Memory problems when creating a spatial weight matrix** - Next by thread:
**Re: st: SV: Re: Memory problems when creating a spatial weight matrix** - Index(es):

© Copyright 1996–2016 StataCorp LP | Terms of use | Privacy | Contact us | What's new | Site index |