3

i have a project which use the same data, which in my c++ code it need 17sec to train 100 data, meanwhile in javascript code from this project

https://github.com/CodingTrain/Toy-Neural-Network-JS it running only about 10sec to train 2400data please someone help me whats wrong, and i need to complete my project for my undergraduate thesis.

ive already build 2 project, which one of them(this) is the same neuralnetwork in c++ from that javascript code(kinda), but still giving the same results

NeuralNetwork::NeuralNetwork(int a,int b,int c)
{
    this->numInput = a;
    this->numHidden = b;
    this->numOutput = c;
    std::vector<double> vec(a, 0.1);
    for (int i = 0; i < b; ++i) {
        this->weightIH.push_back(vec);
    }
    std::vector<double> vec2(b, 0.1);
    for (int i = 0; i < c; ++i) {
        this->weightHO.push_back(vec2);
    }

}


NeuralNetwork::~NeuralNetwork()
{
}

std::vector<double> NeuralNetwork::tambahbias(std::vector<double> a) {
    int size = a.size();
    for (int i = 0; i < size; ++i) {
        a[i] = a[i] + 1;
    }

    return a;
}

std::vector<double> NeuralNetwork::activate(std::vector<double> a) {
    int size = a.size();
    for (int i = 0; i < size; ++i) {
        a[i] = a[i] / (1 + abs(a[i]));
    }
    return a;
}

std::vector<double> NeuralNetwork::derivation(std::vector<double> a) {
    int size = a.size();
    for (int i = 0; i < size; ++i) {
        a[i] = a[i] * (1 - a[i]);
    }
    return a;
}

std::vector<double> NeuralNetwork::hitungError(std::vector<double> a, std::vector<double> b) {
    int size = a.size();
    for (int i = 0; i < size; ++i) {
        a[i] = b[i] - a[i];
    }

    return a;
}


    void NeuralNetwork::train(std::vector<double> a, std::vector<double> target) {
        std::vector<double> hidden(numHidden);
        for (int i = 0; i < numHidden; ++i) {
            for (int j = 0; j < numInput; ++j) {
                hidden[i] += a[j] * weightIH[i][j];
            }
        }
        hidden = tambahbias(hidden);
        hidden = activate(hidden);
        std::vector<double> output(numOutput);
        for (int i = 0; i < numOutput; ++i) {
            for (int j = 0; j < numHidden; ++j) {
                output[i] += hidden[j] * weightHO[i][j];
            }
        }
        output = tambahbias(output);
        output = activate(output);
        std::vector<double> errorO(numOutput);
        errorO = hitungError(output, target);
        std::vector<double> gradO(numOutput);
        gradO = derivation(output);
        for (int i = 0; i < numOutput; ++i) {
            gradO[i] = gradO[i] * errorO[i] * 0.1;
        }
        for (int i = 0; i < numOutput; ++i) {
            for (int j = 0; j < numHidden; ++j) {
                weightHO[i][j] += (gradO[i] * hidden[j]);
            }
        }
        std::vector<double> gradH(numHidden);
        std::vector<double> derH(numHidden);
        derH = derivation(hidden);
        for (int i = 0; i < numHidden; ++i) {
            for (int j = 0; j < numOutput; ++j) {
                gradH[i] = gradO[j] * weightHO[j][i];
            }
            gradH[i] = gradH[i] * derH[i] * 0.1;
        }
        for (int i = 0; i < numHidden; ++i) {
            for (int j = 0; j < numInput; ++j) {
                weightIH[i][j] += (gradH[i] * a[j]);
            }
        }


    }
Nurjamil
  • 21
  • 5
  • 5
    How did you do your benchmarking, which compiler optimization levels did you set when compiling the code, debug / release compilaiton? Many questions to clarify before a concise explanation can be given for your code. – πάντα ῥεῖ May 18 '19 at 09:05
  • Can I see how you are compiling this? – dyz May 18 '19 at 09:05
  • I think the biggest problem is that you're not using vectors and matrices, which would be way faster than using `std::vector`. See link for some libraries you can use: https://stackoverflow.com/questions/1380371/what-are-the-most-widely-used-c-vector-matrix-math-linear-algebra-libraries-a – dyz May 18 '19 at 09:09
  • 3
    Id consider using fixes arrays to avoid reallocating memory for the vectors, as well as making sure to pass your args by ref. Depending on whether you're using c++17 or have rvo turned on, returning vectors from functions could be costly. Id say the two main things to get speed up are make sure you're using c++17 and have the full level of optimisation enabled for your compiler. – George May 18 '19 at 09:13
  • @dyz i don't see how using an arbrirtrary vector/matrix library would be way faster than using std::vector here? – George May 18 '19 at 09:19
  • i only using clock for the benchmark, about compiler optimization i dont know much abt it but, i only enable /MP in my visual studio project, can u elaborate that to me ? @πάνταῥεῖ – Nurjamil May 18 '19 at 09:23
  • Vector/matrices leverage hardware for (huge) acceleration. I suggest you Google this for further explanation. It would actually be kind of embarrassing to have a Neural Net implementation not using vector/matrices. If you're against a rewrite, an easy change would be to use OpenMP to autoparallelize this. But vector/matrices is the way to go. You can also compile with optimization flags. – dyz May 18 '19 at 09:25
  • this is really mindblowing for me, when i tried compiled both of them, it using almost the same percent of my cpu. – Nurjamil May 18 '19 at 09:25
  • the thing is, that code from javascript using array which is kidna the same with vector / matrices, am i wrong? – Nurjamil May 18 '19 at 09:26
  • There are no `tambahbias`, `activate`, `derivation` implementation in your code, it is hard to guess how you implemented them. You should only compare time with "Release" build, make sure you are not benchmarking "Debug" one. Also, `std::vector a` as function parameter (most likely) makes copy of caller's matrix. You should prefer `const td::vector& a` here – R2RT May 18 '19 at 09:44
  • is the only solution here is using javascript instead of c++ ? – Nurjamil May 18 '19 at 09:44
  • @R2RTthere you go, i edit it, but i dont think reference do 35x more performance in this case like that javascript code, and actually i already did that in my other project, and literally no imrpovement so i dont think const reference really effect performance to that degree – Nurjamil May 18 '19 at 09:49
  • i only copy that code half right about the neuralnetwork, so i just wanna know what affect the performance – Nurjamil May 18 '19 at 09:50
  • @Nurjamil Everything is already in comments. Passing vectors by value do. Optimization flags do. "about compiler optimization i dont know much abt it but," -> For sure you can find good answer to this question already on SO, but you have to search for it first. – R2RT May 18 '19 at 10:23
  • 3
    Also, I've experienced up to 1000x speed up when switching from Debug to Release configuration in similar algorithms. – R2RT May 18 '19 at 10:27
  • it turns out im just an idiot, who doesnt know about debug / release. thank you @R2RT – Nurjamil May 18 '19 at 10:50

2 Answers2

0

You're copying all your std::vectors into functions:

void NeuralNetwork::train(std::vector<double> a, std::vector<double> target)

use references instead:

void NeuralNetwork::train(const std::vector<double>& a, const std::vector<double>& target)

Copying a vector is an O(n) operation in both space and time, using a reference is O(1) in both.

A const std::vector reference can't be modified, when you're copying the vector in and out again after modifying it:

std::vector<double> NeuralNetwork::derivation(std::vector<double> a)

use a non-const reference instead:

void NeuralNetwork::derivation(std::vector<double>& a)
Paul Evans
  • 26,111
  • 3
  • 30
  • 50
0

it turns out im just an idiot, who doesnt know about debug / release, make this program to release just solve the problem, thank you everyone for your help

Nurjamil
  • 21
  • 5
  • Ahh, yes. You did not include _how_ you compiled your code as part of the question. Someone would have pointed out that your application had not had the compiler perform optimization. – Eljay May 18 '19 at 13:19