Statistical sampling of code

Question

This may be a very open ended question. I have to quickly measure time of some section of code. I'm using the std::chrono::high_resolution_clock functionality. I have to run this code for many iterations and measure the duration.

So here is the problem: I can measure minimum and maximum duration values, and calculate average using the number of samples count. In this case, I only need to store 4 values. But I would also like to know how the data is distributed. Calculation of the standard deviation or histogram requires that all data points be stored. However, this will require either one giant initial data structure or dynamically growing data structure - both of which will affect the measured code on my embedded system. Is there a way to calculate standard deviation for this sample using the standard deviation of the previous sample?

Answer 1

Calculation of the standard deviation or histogram requires that all data points be stored

That's trivially false. You can calculate a running standard deviation with Welford's algorithm, which just requires one extra variable besides the running mean and the current count of elements.

As for the histogram, you don't need to keep all the data - you just need to keep the counts for each bin, and increment the right bin each time you have a new sample. Of course for this simple approach to pay out you need to know in advance an expected range and number of bins. If this isn't possible, you can always start with small bins over a small range and scale the bins size (merging the adjacent bins) whenever you meet an element outside of the current range. Again, all this requires just a fixed quantity of memory (one integer for each bin and two values for the range).