StreamTokenizer Behavior for division operator "/"

Question

Take a look at the demo code for StreamTokenizer here. It doesn't seems to work properly when there is / in string(Just add / in between string in StringReader). Here is the code from mentioned link,

StreamTokenizer tokenizer = new StreamTokenizer(
    new StringReader("Mary had 1 little lamb..."));

    while(tokenizer.nextToken() != StreamTokenizer.TT_EOF){

        if(tokenizer.ttype == StreamTokenizer.TT_WORD) {
            System.out.println(tokenizer.sval);
        } else if(tokenizer.ttype == StreamTokenizer.TT_NUMBER) {
            System.out.println(tokenizer.nval);
        } else if(tokenizer.ttype == StreamTokenizer.TT_EOL) {
            System.out.println();
        }
    }

For example, for string "Mary had 1 little lamb...", output is

Mary
had
1.0
little
lamb...

For string, "Mary had 1 /little lamb...", output is

Mary
had
1.0

Does / work as EOF token? If so, why?
Is there any way to distinguish / as a different token other than EOF.

Seems OK to me. Can you share a code snippet, the output you're getting and the output you expect? — Mureinik, Apr 24 '15 at 12:07
What exactly doesn't work? Note: [`StreamTokenizer`](https://docs.oracle.com/javase/8/docs/api/java/io/StreamTokenizer.html) is deprecated. — Maroun, Apr 24 '15 at 12:07

Mureinik · Accepted Answer · 2015-04-24T12:34:58.763

3

As per the documentation, / is the comment character in a StreamTokenizer. Everything that comes after it, until an EOL or EOF will be ignored and won't be tokenized.

E.g., to continue the example you've given, if the string is "Mary had 1 / 2\n little lamb...", 2 is commented out and won't be tokenized, and the tokenization will resume after the \n. So the output would be:

Mary
had
1.0
little
lamb...

edited Apr 24 '15 at 12:34

answered Apr 24 '15 at 12:18

Mureinik

252,575
45
248
283

@Mureinik That is very good answer buddy :) . May be downvoters expected more (i'm not sure what) – Ankur Anand Apr 24 '15 at 12:45
@Mureinik: What is preferred way in which I can parse and print `/` as a separate token? One way is to replace `/` with some string which we are sure will not going to come in original string, and in if-else condition check and replace that particular string back with `/`. But this doesn't seem elegant way. Is there any better way? – user148865 Apr 24 '15 at 13:20

score 1 · Answer 2 · answered Apr 24 '15 at 12:43

Just Adding to above answer .Since / is a comment character in StreamTokenizer and continue until the end of the line. Hence you have to add either a new line \n or \r in the String So "Mary had 1 / \n little /\nlamb..." or "Mary had 1 / \r little/\rlamb..." both will work for you

StreamTokenizer Behavior for division operator "/"

2 Answers2