How do you parse a file into a vector of tokens?

Question

So what I want to do is to create a c++ function which reads in a file and converts that file's text to a vector of tokens.

Now the text file I have requires many delimiters including periods, quotation marks, etc. so I thought strtok would work better than sstream to read in tokens. However when traversing through my vector i notice there is nothing in it. The code comes up blank. What am I doing wrong?

Please help me!

My code is here:

void getTokenFreq(string inFile_name) {
ifstream inFile;
int n = 0;
char *token;
vector<string> result(1);

inFile.open(inFile_name);

if (inFile.fail()){
    cout << "Fail to open the file tmp.txt.\n";
    exit(-1);
}

while(inFile.good()) {
    getline(inFile, s);
    char *str = new char[s.length() + 1];
    strcpy(str, s.c_str());
    token = strtok(str, " ’—\",;.:?“”");
    while (token != NULL) {
        result.push_back();
        token = strtok (NULL, " ’—\",;.:?“”");
        n++;
    }
}

for(int i = 0; i < n; i++) {
    cout << result[i];
}

inFile.close();

}

"The code comes up blank". What do you mean? There are some errors here so it shouldn't even compile. — xinaiz, Feb 22 '17 at 23:52
Can you please post a complete example program that can be compiled as-is? Would make it much easier to help... — zett42, Feb 23 '17 at 00:46
`while(inFile.good()) {` That's all well and good, but what you really want to know is the state of the stream after `getline(inFile, s);`. `while(getline(inFile, s)) {` gives you the best of both worlds. More details here: http://stackoverflow.com/questions/5605125/why-is-iostreameof-inside-a-loop-condition-considered-wrong — user4581301, Feb 23 '17 at 00:47

score -1 · Accepted Answer · answered Feb 23 '17 at 00:10

-1

You are good, except some errors:

First:

vector<string> result(1);

Why would you have one empty std::string at the start? That would break your vector, and the result would be like: "", "one", "two", "three", .... Change it to:

vector<string> result;

Second:

result.push_back();

You can't call push_back with no arguments. With logic of your program, you should push_back the token, so change it to:

result.push_back(token);

Third:

You don't free allocated memory. After inner while, call:

delete [] str;

Also std::string s is not visible in your function, but I assume it is global.

answered Feb 23 '17 at 00:10

xinaiz

6,915
3
26
67

Wouldn't mind knowing what earned this the downvotification. Yes this can be done with regex and some deeper stream parsing with a stringstream, but really – user4581301 Feb 23 '17 at 00:51
Actually this helped me! My program works perfectly. I have no idea why someone would down vote this because this is actually the solution. Thank you for your help. – Wes Feb 23 '17 at 05:58

How do you parse a file into a vector of tokens?

1 Answers1