How can i create a istream from a uint8_t vector?

Question

I am in the process of adding the ability to get data over the network to code that used to only read local files. The network library I am using sends and receives data in the form of a vector<uint8_t>. I'd like to be able to reuse the code that processes the data after reading the file, but that code expects a std::istream, is there a way to have an istream read the vector data? It's the same data so I feel like there should be a way, but i haven't been able to find or figure out code for how to do it.

current code:

    std::ifstream stream("data.img", std::ios::in | std::ios::binary | std::ios::ate);

    if (!stream.is_open())
    {
        throw std::invalid_argument("Could not open file.");
    }
    // the arg for processData is std::istream
    processData(stream);

network framework:

    vector<uint8_t> data = networkMessage.data;

    // need some way to create istream from data
    std::istream stream = ?

    processData(stream);
    stream.close();

Is there a way to do this, or am I barking up the wrong tree?

Answer 1

std::basic_istream gets its data from an associated std::basic_streambuf derived class. The STL provides such classes for file I/O and string I/O, but not for memory I/O or network I/O.

You could easily write (or find a 3rd party) memory-based streambuf class that uses the std::vector as its underlying buffer, and then you can construct an std::istream that uses that memory streambuf. For example (using the imemstream class from this answer):

std::vector<uint8_t> &data = networkMessage.data;
imemstream stream(reinterpret_cast<const char*>(data.data()), data.size());
processData(stream);

Answer 2

Well C++ does actually have a class for this - istrstream, and you could use it like this:

    vector<uint8_t> data = ...;

    // need some way to create istream from data
    std::istrstream stream(reinterpret_cast<const char*>(data.data()), data.size());

    processData(stream);

As far as I can tell this doesn't copy the data, unlike the other answers. However it was also deprecated in C++98 because it's hard to know who is responsible for freeing the buffer, so you may want to write your own equivalent.

Answer 3

1. The istream is a reference of raw data. It doesn't hold the data, and just a visitor, by keeping some char* pointers of the begin and end of data memory address.

2. The storage in vector<> is continuous, but by using push_back(), the storage address may changed, (copied inner vector)

3. So it's possible to make an istream to const vector

The reference

https://en.cppreference.com/w/cpp/io/basic_istream https://www.cplusplus.com/reference/streambuf/streambuf/

The shortest example

class vectorbuf : public std::streambuf {
public:
    vectorbuf(std::vector<uint8_t> &v){
        setg((char*)v.data(), (char*)v.data(), (char*)(v.data() + v.size()));
    }
    ~vectorbuf() {}
};

//Usage:
vector<uint8_t>  arr{11,12,13,14,15,16};
vectorbuf vbuf(arr);
std::istream is(&vbuf);

The full WRONG sample code

#include <streambuf>
#include <iostream>
#include <iomanip>
#include <vector>

using namespace std;


template<typename T>
class vectorbuf : public std::streambuf {
public:
    vectorbuf(std::vector<T> &v) : _value(v) {
        char *bptr = (char*)_value.data();
        char *eptr = (char*)(_value.data() + _value.size());
        setg(bptr, bptr, eptr);
        cout<<"Setg: "<<(void*)bptr<<" "<<(void*)eptr<<endl;
    }
    ~vectorbuf() {}

//Zone start ---
//Note: this zone of code can be commented since the virtual function in base class do same 
protected:
    virtual int underflow() {
        char *bptr = (char*)_value.data();
        char *new_eptr = (char*)(_value.data() + _value.size());
        cout<<"[underflow() when gptr()="<<(void*)gptr()
            <<", now_bptr="<<(void*)bptr<<" now_eptr="<<(void*)new_eptr<<"]";

        return traits_type::eof();
        //since the vector& must not modified, the code below is unnecessary.
        if (new_eptr == egptr())
            return traits_type::eof();
        setg(bptr, gptr(), new_eptr);
        return *gptr();
    }
//Zone end ---

private:
    std::vector<T> &_value;
};


int main() {
    vector<int>  arr{'a',12,13,14,15};
    cout<<"The array: ";
    for (int i=0; i<arr.size(); i++)
        cout<<arr[i]<<" ";
    cout<<endl;
    cout<<"  storage: ";
    for (int i=0; i<arr.size()*sizeof(int); i++) {
        char *ptr = (char*)arr.data();
        cout<<static_cast<int>(ptr[i])<<" ";
    }
    cout<<endl;

    vectorbuf<int> vbuf(arr);
    std::istream is(&vbuf);
    arr.push_back(16); //!!! wrong code here !!!

    //the size of arr is 6*4 == 24, with sizeof(int)==4
    for (int i=0; i<26; i++) {
        cout<<"good?"<<is.good()
            <<", fail?"<<is.fail()
            <<", bad?"<<is.bad()
            <<", eof?"<<is.eof()
            <<", tellg="<<is.tellg();

        //Note there must be char
        //'int a' would not accepted and make is.fail() to true
        //and std::noskipws is also importanted
        char a;
        is>>std::noskipws>>a;
        int out = a;
        cout<<", Read from arr: "<<out<<endl;
    }
    return 0;
}

Answer 4

You can do this via assigning the data to a std::string and use a std::istringstream bound to that (leaving aside the unsigned char to signed char conversion issues):

std::string s((char*)networkMessage.data(),networkMessage.size());
std::istringstream iss(s);

std::istream& stream = iss;
         // ^ Note the reference here.

processData(stream);
stream.close();

Answer 5

This will work with vector of any type, not just uint8_t:

std

template <class T>
auto make_istringstream_std_1(const std::vector<T>& v) -> std::istringstream
{
    using namespace std::string_literals;
    std::string str;

    for (auto& e : v)
    {
        str += std::to_string(e) + " "s;
    }
    // the trailing space is not an issue

    return std::istringstream{str};
}

std algorithm

template <class T>
auto make_istringstream_std_2(const std::vector<T>& v) -> std::istringstream
{
    std::stringstream ss;
    std::copy(v.begin(), v.end(), std::ostream_iterator<int>(ss, " "));
    // the trailing space is not an issue

    return std::istringstream{ss.str()};
}

boost

template <class T>
auto make_istringstream_boost(const std::vector<T>& v) -> std::istringstream
{
    using boost::adaptors::transformed;
    using boost::algorithm::join;

    return std::istringstream{
        join(v | transformed([](int a) { return std::to_string(a); }), " ")};
}

---

attribution:

How to transform a vector<int> into a string?

A good example for boost::algorithm::join