How to break up a string into a vector fast?

Question

I am processing CSV and using the following code to process a single line.

play with code

std::vector<std::string> string_to_vector(const std::string& s, const char delimiter, const char escape) {
  std::stringstream sstr{s};
  std::vector<std::string> result;
  while (sstr.good()) {
    std::string substr;
    getline(sstr, substr, delimiter);
    while (substr.back() == escape) {
      std::string tmp;
      getline(sstr, tmp, delimiter);
      substr += "," + tmp;
    }
    result.emplace_back(substr);
  }
  return result;
}

What it does: Function breaks up string s based on delimiter. If the delimiter is escaped with escape the delimiter will be ignored.

This code works but is super slow. How can I speed it up?

Do you know any existing csv processing implementation that does exactly this and which I could use?

Answer 1

The fastest way to do something is to not do it at all.

If you can ensure that your source string s will outlive the use of the returned vector, you could replace your std::vector<std::string> with std::vector<char*> which would point to the beginning of each substring. You then replace your identified delimiters with zeroes.

[EDIT] I have not moved up to C++17, so no string_view for me :)

NOTE: typical CSV is different from what you imply; it doesn't use escape for the comma, but surrounds entries with comma in it with double quotes. But I assume you know your data.

Implementation:

#include <iostream>
#include <vector>
#include <string>

std::vector<char*> string_to_vector(std::string& s, 
                                    const char delimiter, const char escape) 
{
  size_t prev(0), pos(0), from(0);
  std::vector<char*> v;
  while ((pos = s.find(delimiter, from)) != s.npos)
  {
    if (pos == 0 || s[pos - 1] != escape)
    {
      s[pos] = 0;
      v.push_back(&s[prev]);
      prev = pos + 1;
    }
    from = pos + 1;
  }
  v.push_back(&s[prev]);
  return v;
}

int main() {
  std::string test("this,is,a\\,test");
  std::vector<char*> v = string_to_vector(test, ',', '\\');

  for (auto& s : v)
    std::cout << s << " ";
}