0

I am currently having issue with trying to read and manipulate data from 9 .csv files, i was able to do it with 1 however i don't know how to do it with 9 data file. Below is the code on how i did it with 1 data file and i find it impractical to repeat it 9 times. Could anyone advise me?

There are 9 different file names:

  • Jan20071toDec31abcdefghijklmnopq.csv
  • Jan20081toDec31abcdefghijklmnopq.csv
  • Jan20091toDec31abcdefghijklmnopq.csv
  • MetData_Jan01-2010-Jan01-2011-ALL.csv
  • MetData_Jan01-2011-Jan01-2012-ALL.csv
  • MetData_Jan01-2012-Jan01-2013-ALL.csv
  • MetData_Jan01-2013-Jan01-2014-ALL.csv
  • MetData_Jan01-2014-Jan01-2015-ALL.csv
  • MetData_Jan01-2015-Jan01-2016-ALL.csv

        ifstream infile("Jan20071toDec31abcdefghijklmnopq.csv");
    
        while (!infile.eof())
        {
            infile.ignore(50, ' ');
    
            getline(infile, day, '/');
    
            vecDay = atoi(day.c_str());
    
            getline(infile, month, '/');
            vecMonth = atoi(month.c_str());
    
            getline(infile, year, ' ');
            vecYear = atoi(year.c_str());
    
            getline(infile, hour, ':');
            vecHour = atoi(hour.c_str());
    
            getline(infile, minutes, ',');
            vecMinutes = atoi(minutes.c_str());
    
            getline(infile, dp, ',');
            vecDP = atoi(dp.c_str());
    
            getline(infile, dta, ',');
            vecDTA = atoi(dta.c_str());
    
            getline(infile, dts, ',');
            vecDTS = atoi(dts.c_str());
    
            getline(infile, ev, ',');
            vecEV = atoi(ev.c_str());
    
            getline(infile, qfe, ',');
            vecQFE = atoi(qfe.c_str());
    
            getline(infile, qff, ',');
            vecQFF = atoi(qff.c_str());
    
            getline(infile, qnh, ',');
            vecQNH = atoi(qnh.c_str());
    
            getline(infile, rf, ',');
            vecRF = atoi(rf.c_str());
    
            getline(infile, rh, ',');
            vecRH = atoi(rh.c_str());
    
            getline(infile, s, ',');
            vecS = atoi(s.c_str());
    
            getline(infile, sr, ',');
            vecSR = atoi(sr.c_str());
    
            getline(infile, st1, ',');
            vecST1 = atoi(st1.c_str());
    
            getline(infile, st2, ',');
            vecST2 = atoi(st2.c_str());
    
            getline(infile, st3, ',');
            vecST3 = atoi(st3.c_str());
    
            getline(infile, st4, ',');
            vecST4 = atoi(st4.c_str());
    
            getline(infile, sx, ',');
            vecSX = atoi(sx.c_str());
    
            infile >> t;
          }
    

EDIT (Add on) So far i have manage to store the file names in the vector but i can't seems to read it.

string fileNames;
ifstream infile;

vector <string> vecFileNames
{
    "Jan20071toDec31abcdefghijklmnopq.csv",
    "Jan20081toDec31abcdefghijklmnopq.csv",
    "Jan20091toDec31abcdefghijklmnopq.csv",
    "MetData_Jan01-2010-Jan01-2011-ALL.csv",
    "MetData_Jan01-2011-Jan01-2012-ALL.csv",
    "MetData_Jan01-2012-Jan01-2013-ALL.csv",
    "MetData_Jan01-2013-Jan01-2014-ALL.csv",
    "MetData_Jan01-2014-Jan01-2015-ALL.csv",
    "MetData_Jan01-2015-Jan01-2016-ALL.csv"
};

for (unsigned i = 0; i < vecFileNames.size(); i++)
{
    fileNames = vecFileNames[i];

    cout << fileNames << endl;
    infile(fileNames); // <- Having problems
}

Add on (This is how the data was formatted)

UTC Dp Dta Dts EV QFE QFF QNH RF RH S SR ST1 ST2 ST3 ST4 Sx T

31/12/2006 1:00 9.3 50 16 934.6 1009 1012.4 1012.6 0 32.1 9 657 25.4 28.7 28.1 26 13 27.44

EDIT

Here is what i've tried so far, i created a date and time class which i have to used so i've edited the output and input a little. I've created a csv file with 1 record, however after i compile there was no error but no record was showing.

#include <iostream>
#include <fstream>
#include <sstream>
#include <string>
#include <map>
#include <iterator>
#include <utility>
#include <vector>
#include "binaryTreeType.h"
#include "bSearchTreeType.h"
#include "Data.h"
#include "Date.h"
#include "Time.h"

std::istream &operator>>(std::istream &is, char const *delim) {
if (is.flags() & std::ios::skipws) {
    while (isspace((unsigned char)is.peek()))
        is.ignore(1);
}

while (*delim && *delim == is.peek()) {
    ++delim;
    is.ignore(1);
}
if (*delim)
    is.setstate(std::ios::failbit);
return is;
}

struct record 
{
int day, month, year;
int hour, minutes;
double dp, ev, qfe, qff, qnh, rh, st[4], t;
int dta, dts, rf, s, sr, sx;

friend std::istream &operator>>(std::istream &is, record &r) {

    is >> r.day >> "/" >> r.month >> "/" >> r.year >> " " >> r.hour >> ":" 
>> r.minutes >> "," >> r.dp >> "," >> r.dta >> "," >> r.dts >> "," >> r.ev 
>> "," >> r.qfe >> "," >> r.qff >> "," >> r.qnh >> "," >> r.rf >> "," >> 
r.rh >> "," >> r.s >> "," >> r.sr;

    for (int i = 0; i < 4; i++)
    {
        is >> "," >> r.st[i];
    }
          is >> r.sx >> "," >> r.t;
    return is;
}

friend std::ostream &operator<<(std::ostream &os, record const &r) {

    os << r.day << "/" << r.month << "/" << r.year << " " << r.hour << ":" 
<< r.minutes << "," << r.dp << "," << r.dta << "," << r.dts << "," << r.ev 
<< "," << r.qfe << "," << r.qff << "," << r.qnh << "," << r.rf << ","      
<< r.rh << "," << r.s << "," << r.sr;

    for (int i = 0; i < 4; i++)
    {
        os << "," << r.st[i];
    }
          os << r.sx << "," << r.t;
    return os;
}
};

template <class Container>
void read_data(std::string const &name, Container &c) {
std::ifstream in(name);
record temp;
while (in >> temp)
{
    c.push_back(temp);
}
}

using namespace std;

int main()
{
   vector <string> filenames
  {
    "Test.csv"
    //"Jan20071toDec31abcdefghijklmnopq.csv",
    //"Jan20081toDec31abcdefghijklmnopq.csv",
    //"Jan20091toDec31abcdefghijklmnopq.csv",
    //"MetData_Jan01-2010-Jan01-2011-ALL.csv",
    //"MetData_Jan01-2011-Jan01-2012-ALL.csv",
    //"MetData_Jan01-2012-Jan01-2013-ALL.csv",
    //"MetData_Jan01-2013-Jan01-2014-ALL.csv",
    //"MetData_Mar01-2014-Mar01-2015-ALL.csv",
    //"MetData_Mar01-2015-Mar01-2016-ALL.csv"
    };

   vector <record> data;

for (auto && s : filenames)
{
    read_data(s, data);
}

for (auto const &r : data)
{
    cout << r << "\n";
}
NoobCoder
  • 3
  • 3
  • 2
    This code looks extremely brittle and even the slightest change in the CSV file is bound to cause catastrophic problems. Why not use a [C++ CSV library](https://github.com/ben-strasser/fast-cpp-csv-parser)? – tadman Jul 11 '17 at 03:38
  • https://stackoverflow.com/questions/5605125/why-is-iostreameof-inside-a-loop-condition-considered-wrong – Retired Ninja Jul 11 '17 at 03:39
  • Thank you tadman and Retired Ninja the link you posted is very informative! – NoobCoder Jul 11 '17 at 04:07
  • You could read from `stdin` in your program and then send all the CSV files to `stdin` when you run it, like this `yourProgram < *.csv` – Mark Setchell Jul 13 '17 at 07:51

1 Answers1

0

Put the code in a function, and pass the name of the file to the function as a parameter:

void read_data(std::string const &filename) {
    ifstream infile(filename);

    // code to read dates
}

Then you can call that nine times:

std::vector<std::string> filenames { 
    "Jan20071toDec31abcdefghijklmnopq.csv",
    "Jan20081toDec31abcdefghijklmnopq.csv",
    "Jan20091toDec31abcdefghijklmnopq.csv",
    "MetData_Jan01-2010-Jan01-2011-ALL.csv",
    "MetData_Jan01-2011-Jan01-2012-ALL.csv",
    "MetData_Jan01-2012-Jan01-2013-ALL.csv",
    "MetData_Jan01-2013-Jan01-2014-ALL.csv",
    "MetData_Jan01-2014-Jan01-2015-ALL.csv",
    "MetData_Jan01-2015-Jan01-2016-ALL.csv"
};

for (auto && s : filenames)
    read_data(s);

Then, you probably want to look up std::get_time. It'll let you read the dates a lot more easily. Oh, and using while (!whatever.eof()) is almost always a bug (including the code here).

Just for grins, I created a couple of test files with somewhat abbreviated forms of the data you seem to way, formatted like this:

4/4/2017 1:40, 54321, 5432, 543, 54, 5 
17/4/2017 12:47, 1, 12, 123, 1234, 12345 

I put one of those lines in each of two files, then ran the following code:

#include <vector>
#include <string>
#include <ctime>
#include <iomanip>
#include <iostream>
#include <fstream>

std::istream &operator>>(std::istream &is, char const *delim) {
    if (is.flags() & std::ios::skipws) {
        while (isspace((unsigned char)is.peek()))
            is.ignore(1);
    }

    while (*delim && *delim == is.peek()) {
        ++delim;
        is.ignore(1);
    }
    if (*delim)
        is.setstate(std::ios::failbit);
    return is;
}

struct record {
    tm date;
    int dp, dta, dts, ev, qfe; // , qff, qnh, rf, rh, s, sr, st[4], sx, t;

    friend std::istream &operator>>(std::istream &is, record &r) {
        is >> std::get_time(&r.date, "%d / %m / %Y %H : %M");
        is >> "," >> r.dp >> "," >> r.dta >> "," >> r.dts >> "," >> r.ev >> "," >> r.qfe;
//      >> "," >> r.qff >> "," >> r.qnh >> "," >> r.rf << "," >> r.rh >> "," >> r.s >> "," >> r.sr;
//      for (int i = 0; i < 4; i++)
//          is >> "," >> r.st[i];
//      is >> r.sx >> "," >> r.t;
        return is;
    }

    friend std::ostream &operator<<(std::ostream &os, record const &r) {
        os << std::put_time(&r.date, "%Y/%m/%d %H:%M");
        os << "," << r.dp << "," << r.dta << "," << r.dts << "," << r.ev << "," << r.qfe;
        //      << "," << r.qff << "," << r.qnh << "," << r.rf << "," << r.rh << "," << r.s << "," << r.sr;
        //      for (int i = 0; i < 4; i++)
        //          os << "," << r.st[i];
        //      os << r.sx << "," << r.t;
        return os;
    }


};

template <class Container>
void read_data(std::string const &name, Container &c) {
    std::ifstream in(name);
    record temp;
    while (in >> temp)
        c.push_back(temp);
}

int main() {

    std::vector<std::string> filenames{
        "Jan20071toDec31abcdefghijklmnopq.csv",
//      "Jan20081toDec31abcdefghijklmnopq.csv",
//      "Jan20091toDec31abcdefghijklmnopq.csv",
//      "MetData_Jan01-2010-Jan01-2011-ALL.csv",
//      "MetData_Jan01-2011-Jan01-2012-ALL.csv",
//      "MetData_Jan01-2012-Jan01-2013-ALL.csv",
//      "MetData_Jan01-2013-Jan01-2014-ALL.csv",
//      "MetData_Jan01-2014-Jan01-2015-ALL.csv",
        "MetData_Jan01-2015-Jan01-2016-ALL.csv"
    };

    std::vector<record> data;

    for (auto && s : filenames)
        read_data(s, data);

    for (auto const &r : data)
        std::cout << r << "\n";
}

The result was this:

2017/04/17 12:47,1,12,123,1234,12345
2017/04/04 01:40,54321,5432,543,54,5

Note that I've re-formatted the output data a little (e.g., changed it from day/month/year to year/month/day). Making it work with more data fields should be mostly more repetitions of similar code (and likewise, making it work with more files should be mostly a matter of un-commenting the file names).

Jerry Coffin
  • 437,173
  • 71
  • 570
  • 1,035
  • Hi Jerry Coffin, I have trouble understanding this statement "for (auto && s : filenames)". Could you explain to me what does this for loop do? Thank you for your advise in creating a function i'll try it! – NoobCoder Jul 11 '17 at 04:08
  • @G.Kenneth: It steps through the vector from beginning to end. On each iteration of the loop, `s` refers to an item from the vector. – Jerry Coffin Jul 11 '17 at 04:25
  • Thank you!! I understand it now. – NoobCoder Jul 11 '17 at 04:35
  • how do i open the file since it's a vector? the "infile.open" function doesn't work on a vector. – NoobCoder Jul 11 '17 at 07:52
  • You give it an individual element of the vector, as shown. – Jerry Coffin Jul 11 '17 at 08:03
  • i am terribly sorry could you elaborate it further because i tried it out i realize i am lost. Is all the data stored in the vector or just the file name? How do i retrieve the data and perform calculations? – NoobCoder Jul 11 '17 at 09:48
  • The vector that's shown contains only the filenames. The data certainly can be read into a vector as well (but you probably want to use a second vector). If you want advice about how to store the data from the file, the obvious starting point would be to tell us how the data in the file is formatted. I could try to infer that from your code, but that's error prone at best. – Jerry Coffin Jul 12 '17 at 14:43
  • @G.Kenneth: I meant how the data in the data file(s) you're reading is formatted. – Jerry Coffin Jul 13 '17 at 02:46
  • i tried storing the files names in a vector but i am having problem reading it – NoobCoder Jul 13 '17 at 02:47
  • thank you for your help however i am still having issue. I was able to compile without any issue however no records was showing, – NoobCoder Jul 13 '17 at 07:29
  • @G.Kenneth: I'm afraid I've already done about as much as I can to help you out. – Jerry Coffin Jul 13 '17 at 07:34