Statalist


[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]

SV: st: SV: Re: Memory problems when creating a spatial weight matrix


From   "Lina Jonsson" <lina.jonsson@vti.se>
To   <statalist@hsphsun2.harvard.edu>
Subject   SV: st: SV: Re: Memory problems when creating a spatial weight matrix
Date   Wed, 26 Sep 2007 12:59:47 +0200

Hi again.


Thank you for the quick reply Sergiy! 

There is no problem running the code you gave me. I belive that the spatwmat-command is creating something temporarily that takes a lot of memory. I have only been using Stata for a couple of months so I have never looked into a source code before - how do I do that? 

Does anyone else have experience in using the spatwmat-command on "large" (more than 800 observations) datasets?

Best Regards

/Lina Jonsson

> -----Ursprungligt meddelande-----
> Från: owner-statalist@hsphsun2.harvard.edu 
> [mailto:owner-statalist@hsphsun2.harvard.edu] För Sergiy Radyakin
> Skickat: den 26 september 2007 12:34
> Till: statalist@hsphsun2.harvard.edu
> Ämne: Re: st: SV: Re: Memory problems when creating a spatial 
> weight matrix
> 
> Hi Lina,
> 
> may be I am missing something, but  a matrix 500x500 at 4 
> bytes per cell occupies 1MB of memory. At 8 bytes per cell 
> 2MB respectively. How does this relate to the 500/900MB of 
> memory limit? As I understand, your dataset is 800 
> observations and -spatwmat- produces a matrix 800x800.
> 
> Could you run the following and tell us what happens?
> 
> clear
> set mem 100m   <-- or whatever your maximum is
> set matsize 800
> matrix Z=J(800,800,0)
> matrix dir
> matrix A=J(800,800,0)
> matrix dir
> matrix B=J(800,800,0)
> matrix dir
> matrix C=J(800,800,0)
> matrix dir
> use "mydata"   <--- whatever your dataset is
> 
> If it works fine, than it is not a memory issue, but rather 
> you'd have to look into the source code of -spatwmat- to find 
> if the memory is used there efficiently (e.g. it does not 
> generate 800 temporary matrices).
> 
> From the first look, it might be relatively easy to port the 
> code into matlab. However then you need matlab, why not Stata 
> SE? If you don't have constraints with software -- buy Stata 
> 10 SE and this might resolve the issue. If you do have 
> constraints, than what are they (which software is available 
> to you)? There might be just a handful of people working with 
> specialized commands like this, so please explain us what 
> this command does in plain English.
> 
> Best regards,
>    Sergiy Radyakin
> 
> 
> 
> 
> 
> On 9/26/07, Lina Jonsson <lina.jonsson@vti.se> wrote:
> > Dear statalister.
> >
> > Sorry I haven´t had time to follow up my question until now.
> >
> > The memory problem occurs only when using the command for 
> creating the spatial weight matrix (spatwmat). The dataset 
> I´m using consists of only 28 variables - all numerical and 
> dummies (0,1). According to an earlier question concering 
> memory problems when using the spatwmat-command; "-spatwmat- 
> will reset the number of observations to (N/2)*N-1" which 
> means about 320 000 if I´m using a dataset with 800 
> observations. The advice that is given in the former question 
> is to set memory to a higher value. That is the advice that 
> also is given for my error message "no room to add more 
> variables". But the problem is that I cannot set memory to 
> more then 940m.
> >
> > I am using Windows XP and have just increased my RAM-memory 
> to 2 GB which had no effect at all on the maximum memory I 
> can set within Stata. As I understand it; the hotfix Michel 
> mentioned is only for Stata 10 and I`m using Intercooled Stata 9.2.
> >
> > Is anyone having any suggestions on how this problem can be 
> solved or is it not possible to use the spatwmat-command on 
> datasets with 800 observation using "a normal computer". In 
> fact I whould like to use up to 1738 observations but for 
> that I whould need to switch to Stata SE instead of 
> Intercooled Stata due to the fact that Intercooled Stata can 
> set matsize to a maximum of 800. But there is no point in 
> changing to Stata SE if the available memory sets the limit 
> to less than 800 observations...
> >
> > If Stata can´t handle spatial weight matrices for large 
> datasets, do any of you have any suggestions of other 
> statistical software that can?
> >
> > Best regards.
> >
> > /Lina Jonsson
> >
> >
> >
> >
> > > Från: owner-statalist@hsphsun2.harvard.edu
> > > [mailto:owner-statalist@hsphsun2.harvard.edu] För Michael Blasnik
> > > Skickat: den 21 september 2007 14:49
> > > Till: statalist@hsphsun2.harvard.edu
> > > Ämne: st: Re: Memory problems when creating a spatial 
> weight matrix
> > >
> > > You need to give us more information about your dataset, 
> operating 
> > > system, etc.
> > > The 940m should be plenty of room for a 500 observation 
> dataset and 
> > > an 800x800 matrix.  Can you describe your dataset in more 
> detail?  
> > > Do you have hundreds of long string variables?  Is there 
> a way for 
> > > you to encode any long strings and compress other variables?  In 
> > > terms of accessing more memory -- what OS are you using?  If MS 
> > > Windows XP, have you tried the memory hotfix described here 
> > > http://www.stata.com/support/faqs/win/winmemory.html?
> > >
> > > Michael Blasnik
> > >
> > > ----- Original Message -----
> > > From: "Lina Jonsson" <lina.jonsson@vti.se>
> > > To: <statalist@hsphsun2.harvard.edu>
> > > Sent: Friday, September 21, 2007 5:25 AM
> > > Subject: st: Memory problems when creating a spatial weight matrix
> > >
> > >
> > > > Dear statalisters.
> > > >
> > > > I am working with a hedonic price study where I whould like
> > > to check for
> > > > and correct for spatial autocorrelation. For this I 
> whould like to 
> > > > create a spatial weight matrix based on the x- and 
> y-coordinates 
> > > > for each house using the spatwmat-command in spdep. Since I
> > > have Intercooled
> > > > Stata the maximum matsize that I can set is 800, which
> > > means that I can
> > > > use no more than 800 observations. But since I cannot set
> > > memory to more
> > > > than 940m, the actual numbers of observations that I can use is 
> > > > less than 500. This is to few.
> > > >
> > > > After reading some old questions concerning memory problems
> > > I realize
> > > > that this might be a problem that is hard to solve.
> > > > Does any of you have any suggestions on how I can increase
> > > the memory in
> > > > Stata. Can increasing the RAM-memory on my computer 
> help? Or is it 
> > > > unrealistic to think that it will be possible to create a
> > > matrix whith
> > > > the spatwmat-command that will be almost 8000x800 on a 
> "standard"
> > > > computer? Should I change to some other software and what
> > > in that case?
> > > >
> > > > I am greatful for any suggestions!
> > > >
> > > > /Lina Jonsson
> > >
> > > *
> > > *   For searches and help try:
> > > *   http://www.stata.com/support/faqs/res/findit.html
> > > *   http://www.stata.com/support/statalist/faq
> > > *   http://www.ats.ucla.edu/stat/stata/
> > >
> >
> > *
> > *   For searches and help try:
> > *   http://www.stata.com/support/faqs/res/findit.html
> > *   http://www.stata.com/support/statalist/faq
> > *   http://www.ats.ucla.edu/stat/stata/
> >
> 
> *
> *   For searches and help try:
> *   http://www.stata.com/support/faqs/res/findit.html
> *   http://www.stata.com/support/statalist/faq
> *   http://www.ats.ucla.edu/stat/stata/
> 

*
*   For searches and help try:
*   http://www.stata.com/support/faqs/res/findit.html
*   http://www.stata.com/support/statalist/faq
*   http://www.ats.ucla.edu/stat/stata/



© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   What's new   |   Site index