Statalist The Stata Listserver

[Date Prev][Date Next][Thread Prev][Thread Next][Date index][Thread index]

st: Rolling a better toke(n)

From   "David Elliott" <>
Subject   st: Rolling a better toke(n)
Date   Wed, 18 Oct 2006 10:57:07 -0300

One doesn't have to do much programming without coming across
situation where it is necessary to parse a list into individual tokens
for subsequent use, usually in some form of looping structure.
However, Stata's built in tokenize  requires additional manipulation
of the tokens, especially if a non space parsing character is used.

On Oct 04/2006 I posted a question regarding tokenize
( ) and
received several excellent responses including a Mata routine and
direction to Nick Cox's tknz command.  The latter was more to my
purpose but still included any non space parsing character as one of
the tokens. I took tknz and made some modifications (with Nick's
permission) which are now available on SSC, specifically:

(1) causing its no-options default to behave exactly as tokenize does
(2) allowing an option to drop non space parsing characters to get a
"clean" list
(3) returning the number of parsed tokens in s(items)


tknz "fe,fi,fo,fum" , s(v) p(,) nochar


local choices = "fe,fi,fo,fum"
tknz `"`choices'"' , s(v) p(,) nochar

After tknz, one can directly process the list with:

   forval i=1/`s(items)' {
       di as res "Token `i' is `v`i''"

The modified routine can be accessed with:
-net describe tknz, from(

I hope this proves useful to some of you.

David Elliott
*   For searches and help try:

© Copyright 1996–2021 StataCorp LLC   |   Terms of use   |   Privacy   |   Contact us   |   What's new   |   Site index