Bookmark and Share

Notice: On March 31, it was announced that Statalist is moving from an email list to a forum. The old list will shut down on April 23, and its replacement, statalist.org is already up and running.


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: st: reading HTML source in Chinese but get a messy code


From   Sergiy Radyakin <serjradyakin@gmail.com>
To   "statalist@hsphsun2.harvard.edu" <statalist@hsphsun2.harvard.edu>
Subject   Re: st: reading HTML source in Chinese but get a messy code
Date   Mon, 10 Jun 2013 02:14:04 -0400

following statement in the documentation:"str# variables require #
bytes per observation".Sergiy.

On Mon, Jun 10, 2013 at 1:06 AM, Sergiy Radyakin <serjradyakin@gmail.com> wrote:
> Tony,
>
> if your choice of package is based solely on whether it supports
> unicode or not, I would probably recommend Microsoft's Excel or
> OpenOffice's Calc. However since you are in this forum, you probably
> intend to do some statistical processing of that information. In that
> case what is that analysis? In many cases you actually don't need to
> see the text, but just rely on the package to handle it. If you don't
> want to specify, consider SPSS or SAS which (according to the
> manufacturers) both support unicode. I have also asked if new Stata 13
> supports unicode and hope for the best. If you want to harvest
> information about the user profiles, you will need to check with the
> site owner whether this would be permitted, and if you have valid
> scientific needs to do it, perhaps, the owner might simply pass that
> information to you in an organized way. From what it appears on this
> page you can pull some information like ID, age, gender, phone, and
> province, and the rest (name, address) is hardly of any value for
> statistical processing.
>
> Best, Sergiy
>
> On Sat, Jun 8, 2013 at 9:53 PM, Li Chuntao (Tony) <leechtcn@gmail.com> wrote:
>> Dear Sergiy,
>>
>>     Thank you for your advice. Actually i need the whole lines of
>> information from Line 2~9. Maybe Stata just cannot handle it because
>> of the unicode problem. If you know any package can do it, please
>> advice.
>>
>> thanks again
>>
>> Tony
>>
>>
>> On Sun, Jun 9, 2013 at 12:09 AM, Sergiy Radyakin <serjradyakin@gmail.com> wrote:
>>> Tony, after visiting the link I see in lines 2-9 characters in
>>> Chinese. Stata will not show you these characters because Stata does
>>> not work with unicode. To see the file through Stata's eyes, go to the
>>> link you posted in FireFox, then go to the menu View-->Character
>>> Encoding-->More Encodings-->West European-->Western(Windows-1252).
>>> This is what you can import into Stata and, yes, it does look messy.
>>> This is the best you can get with it. The good thing is that if you
>>> process your data in Stata and then output the same messy text you
>>> will end up with a very readable text, but readable elsewhere (e.g. in
>>> notepad or a browser). To cut it short, if your analysis requires e.g.
>>> search of a substring in a text - you might do it by searching for
>>> byte sequences, and those sequences would not look intuitive at all.
>>> But if it is something more involved then you might want to rethink
>>> the choice of a package to do it. Perhaps if you describe the broad
>>> goal of what you are doing it would be easier to advise.
>>> Best, Sergiy
>>>
>>> On Sat, Jun 8, 2013 at 11:20 AM, Li Chuntao (Tony) <leechtcn@gmail.com> wrote:
>>>> Dear Prof. Nick,
>>>>
>>>>    Line 2 to 9 are what i want, from the page of
>>>> http://html2text.theinfo.org/?url=++http%3A%2F%2Fqq.ico.la%2Fqq459322464.html
>>>>
>>>> thanks
>>>>
>>>> Tony
>>>>
>>>>
>>>> On Sat, Jun 8, 2013 at 11:08 PM, Nick Cox <njcoxstata@gmail.com> wrote:
>>>>> OK.
>>>>>
>>>>> Looking at the file in a text editor shows that alternate lines are
>>>>> blank. I don't know which lines are data for you.
>>>>> Nick
>>>>> njcoxstata@gmail.com
>>>>>
>>>>>
>>>>> On 8 June 2013 16:04, Li Chuntao (Tony) <leechtcn@gmail.com> wrote:
>>>>>> Yes, of course i understand my own code. Here i just want to display
>>>>>> the first two lines to show that there is a messay output and seeking
>>>>>> helps.
>>>>>>
>>>>>> Thank you, Nick, for your always kind help helpfulness
>>>>>>
>>>>>> Tony
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Jun 8, 2013 at 10:59 PM, Nick Cox <njcoxstata@gmail.com> wrote:
>>>>>>> Your own code doesn't seem well matched to the input. In your first
>>>>>>> post you were looping over the lines of the file, reading them one by
>>>>>>> one and then processing them. You have abandoned that here. Do you
>>>>>>> understand what the original Mata code does?
>>>>>>> Nick
>>>>>>> njcoxstata@gmail.com
>>>>>>>
>>>>>>>
>>>>>>> On 8 June 2013 15:51, Li Chuntao (Tony) <leechtcn@gmail.com> wrote:
>>>>>>>> Well, it still does not work, as can be seen from the output of the
>>>>>>>> following codes:
>>>>>>>>
>>>>>>>> I mean, the output from http://html2text.theinfo.org seems quite
>>>>>>>> clean, but it turns to a messay when i tried to read it into Stata,
>>>>>>>> weather by insheet using or by the Mata code followed.
>>>>>>>>
>>>>>>>> Do anyone have such an experience?
>>>>>>>>
>>>>>>>>
>>>>>>>> thanks
>>>>>>>>
>>>>>>>> Chuntao
>>>>>>>>
>>>>>>>>
>>>>>>>> copy "http://html2text.theinfo.org/?url=++http%3A%2F%2Fqq.ico.la%2Fqq459322464.html";
>>>>>>>> d:\temp.txt, replace
>>>>>>>> mata:
>>>>>>>>         fh = fopen("d:\temp.txt", "r")
>>>>>>>>         junk=fget(fh)
>>>>>>>>                 junk
>>>>>>>>         junk=fget(fh)
>>>>>>>>                 junk
>>>>>>>>
>>>>>>>>         }
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jun 7, 2013 at 2:36 AM, Sergiy Radyakin <serjradyakin@gmail.com> wrote:
>>>>>>>>> Chuntao,
>>>>>>>>>
>>>>>>>>> adding to Nick's comments, you don't have to parse HTML code yourself
>>>>>>>>> as this is a pretty standard task. For your purposes the following
>>>>>>>>> should yield a pretty clean file:
>>>>>>>>> http://html2text.theinfo.org/?url=http%3A%2F%2Fqq.ico.la%2Fqq459322466.html
>>>>>>>>>
>>>>>>>>> where you supply your URL as a parameter.
>>>>>>>>>
>>>>>>>>> Best, Sergiy Radyakin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jun 6, 2013 at 12:58 PM, Nick Cox <njcoxstata@gmail.com> wrote:
>>>>>>>>>> If a file contains junk in lines 1 to 31, don't skip lines 1 to 34!
>>>>>>>>>>
>>>>>>>>>> A more fundamental point is that this is HTML:
>>>>>>>>>>
>>>>>>>>>> 1. So, lines will necessarily include HTML markup code in many if not
>>>>>>>>>> all lines. You will need to strip those too, or interpret them.
>>>>>>>>>>
>>>>>>>>>> 2. Mark-up code won't necessarily be interpretable if you ignore previous lines.
>>>>>>>>>>
>>>>>>>>>> In this particular case, there are many references to yet other files,
>>>>>>>>>> perhaps not of concern to you.
>>>>>>>>>>
>>>>>>>>>> I can't read Chinese, so that is far as I go.
>>>>>>>>>>
>>>>>>>>>> Nick
>>>>>>>>>> njcoxstata@gmail.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 6 June 2013 14:36, Li Chuntao (Tony) <leechtcn@gmail.com> wrote:
>>>>>>>>>>> Dear Listers,
>>>>>>>>>>>
>>>>>>>>>>>        I want to import the following HTML source files:
>>>>>>>>>>>
>>>>>>>>>>>         http://qq.ico.la/qq459322466.html
>>>>>>>>>>>
>>>>>>>>>>>         The source file contains some information in Chinese, which is
>>>>>>>>>>> located in line 32 to 73.
>>>>>>>>>>>
>>>>>>>>>>>          i tried to import the information by using the following code:
>>>>>>>>>>>
>>>>>>>>>>> clear all
>>>>>>>>>>> set obs 500
>>>>>>>>>>> copy  "http://qq.ico.la/qq459322466.html"; d:\qq.txt, replace
>>>>>>>>>>>
>>>>>>>>>>> mata:
>>>>>>>>>>>         fh = fopen("d:\qq.txt", "r")
>>>>>>>>>>>         for(i=1; i<=34; i++) {
>>>>>>>>>>>         junk=fget(fh)
>>>>>>>>>>>         }
>>>>>>>>>>>         for(i=; i<=20; i++) {
>>>>>>>>>>>         junk=fget(fh)
>>>>>>>>>>>         junk
>>>>>>>>>>>         }
>>>>>>>>>>>
>>>>>>>>>>> end
>>>>>>>>>>>
>>>>>>>>>>> but the result data in memory is only a messy.
>>>>>>>>>>>
>>>>>>>>>>> Similar code has been used for other webpage, thanks to Prof. Kit
>>>>>>>>>>> Baum, as can be seen following:
>>>>>>>>>>>
>>>>>>>>>>> clear all
>>>>>>>>>>> set obs 500
>>>>>>>>>>> local stkcd="000002"
>>>>>>>>>>> gen str20 date="2012.12.31"
>>>>>>>>>>> copy "http://stockdata.stock.hexun.com/2008/lr.aspx?stockid=`stkcd'&accountdate=2012.12.31"
>>>>>>>>>>>  d:\date.txt, replace
>>>>>>>>>>> mata:
>>>>>>>>>>>         fh = fopen("d:\date.txt", "r")
>>>>>>>>>>>         for(i=1; i<=444; i++) {
>>>>>>>>>>>         junk=fget(fh)
>>>>>>>>>>>         }
>>>>>>>>>>>
>>>>>>>>>>> Can someone familiar with Chinese encoding give me some hits?
>>>>>>>>>>>
>>>>>>>>>>> Best
>>>>>>>>>>>
>>>>>>>>>>> Chuntao
>>>>>>>>>>> *
>>>>>>>>>>> *   For searches and help try:
>>>>>>>>>>> *   http://www.stata.com/help.cgi?search
>>>>>>>>>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>>>>>>>>>> *   http://www.ats.ucla.edu/stat/stata/
>>>>>>>>>> *
>>>>>>>>>> *   For searches and help try:
>>>>>>>>>> *   http://www.stata.com/help.cgi?search
>>>>>>>>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>>>>>>>>> *   http://www.ats.ucla.edu/stat/stata/
>>>>>>>>> *
>>>>>>>>> *   For searches and help try:
>>>>>>>>> *   http://www.stata.com/help.cgi?search
>>>>>>>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>>>>>>>> *   http://www.ats.ucla.edu/stat/stata/
>>>>>>>> *
>>>>>>>> *   For searches and help try:
>>>>>>>> *   http://www.stata.com/help.cgi?search
>>>>>>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>>>>>>> *   http://www.ats.ucla.edu/stat/stata/
>>>>>>> *
>>>>>>> *   For searches and help try:
>>>>>>> *   http://www.stata.com/help.cgi?search
>>>>>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>>>>>> *   http://www.ats.ucla.edu/stat/stata/
>>>>>> *
>>>>>> *   For searches and help try:
>>>>>> *   http://www.stata.com/help.cgi?search
>>>>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>>>>> *   http://www.ats.ucla.edu/stat/stata/
>>>>> *
>>>>> *   For searches and help try:
>>>>> *   http://www.stata.com/help.cgi?search
>>>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>>>> *   http://www.ats.ucla.edu/stat/stata/
>>>> *
>>>> *   For searches and help try:
>>>> *   http://www.stata.com/help.cgi?search
>>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>>> *   http://www.ats.ucla.edu/stat/stata/
>>> *
>>> *   For searches and help try:
>>> *   http://www.stata.com/help.cgi?search
>>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>>> *   http://www.ats.ucla.edu/stat/stata/
>> *
>> *   For searches and help try:
>> *   http://www.stata.com/help.cgi?search
>> *   http://www.stata.com/support/faqs/resources/statalist-faq/
>> *   http://www.ats.ucla.edu/stat/stata/
*
*   For searches and help try:
*   http://www.stata.com/help.cgi?search
*   http://www.stata.com/support/faqs/resources/statalist-faq/
*   http://www.ats.ucla.edu/stat/stata/


© Copyright 1996–2014 StataCorp LP   |   Terms of use   |   Privacy   |   Contact us   |   Site index