Stata The Stata listserver
[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]

Re: Re: st: multiply two data sets

From   Natalie Karavarsamis <>
To   "''" <>
Subject   Re: Re: st: multiply two data sets
Date   Mon, 30 Jun 2003 10:06:00 +1000

Thank you very much for your feedback!

Date: Tue, 24 Jun 2003 10:01:33 -0500
From: (Jeff Pitblado, Stata Corp.)
Subject: Re: st: multiply two data sets

Natalie Karavarsamis <> asks about multiplying two
datasets, treating them as if they were matrices, where one of the datasets
too big to convert to a matrix:

> I have two data sets; a data file, call this A, which is 41000 rows x 130
> columns, and another file,call this B, 130 rows by 50 columns.
> I want to multiply A and B (C=AxB). It would be ideal to treat A and B as
> matrices and use matrix multiplication but the maximum matrix size is
> x 11000 (we run Stata 7.0 SE). Is there a way around this? If not, are
> any suggestions of how else to do this? I don't want to cut matrix A (or
> into smaller data sets (matrices).  

If you have enough memory to hold both A and C, which I estimate to be just
under 60m, I would suggest using -matrix score-.  -matrix score- will
a new variable from the linear combination of elements in a row vector and
the variables in memory.  See [P] matrix score.

To illustrate, the following do-file generates two datasets, a.dta and
according to the sizes Natalie indicates:

***** BEGIN:
* generate some data
set mem 50m
set obs 41000
forval i = 1/130 {
	di as txt "generating a`i'"
	gen double a`i' = uniform()
save a, replace

set obs 130
forval i = 1/50 {
	di as txt "generating b`i'"
	gen double b`i' = uniform()
save b, replace
***** END:

In, prepare for the product by setting the memory to be large
Then put the data from b.dta into a matrix -b- using -mkmat- (notice the
I use to get a list of all the variable names into the -`varlist'- macro).

Use the data in a.dta and loop over the columns of matrix -b-, generating
new column of the new dataset/matrix C using -matrix score-.  Note that when
you grab each column of the matrix -b-, turn it into a row vector and put
variable names from dataset a.dta as its column names.  Then -matrix score-
does all the work of multiplying.

***** BEGIN:
* take matrix product of datasets a.dta and b.dta
* make the matrix from b.dta (the smaller dataset)
set mem 60m
use b
local 0
syntax [varlist]
mkmat `varlist', matrix(b)

* use -matrix score- to compute the linear combinations of the variables in
* a.dta, where the coefficients are from the columns of b.dta
use a, clear
local 0
syntax [varlist]
local k = colsof(b)
forval i = 1/`k' {
	matrix bi = b[1...,`i']'
	matrix colnames bi = `varlist'
	matrix score double c`i' = bi
	di as txt "generating c`i'"
keep c*
save c, replace
***** END:

I tested the above do-files using Stata/SE 7.0 and Stata/SE 8.0.

- --Jeff

Natalie Karavarsamis

Cancer Epidemiology Centre
The Cancer Council Victoria
100 Drummond Street
Carlton VIC 3053
ph: (03) 9635 5159
fax: (03) 9635 5330
*   For searches and help try:

© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   What's new   |   Site index