0

I have data signal which is generated at constant time interval of 500ms. It is a real world measurement and thus exhibits some noise in its baseline. The noise is fairly consistent.

I obtain the measurement at a regular time interval and stash it inside a circular rolling buffer of sample data. This rolling buffer is an array of values which are located at positions (0,1,2,3,4,5... -> n). Once the nth position in this array is reached, my buffer rolls over and overwrites the value at 0th array position followed by the value at the 1st array position and so on until the value at the nth position is overwritten (then rollover again). This process repeats over and over. I use the rolling sample buffer to calculate a moving average and a std dev and perform a linear extrapolation calculation using the data within the buffer.

Now for my issue:

The real world signal has the potential to be in 3 states. Either increasing over time at an unknown rate or decreasing over time at an unknown rate or it can also remain fairly constant over time until an increase or decrease is detected. (I need to detect the increase or decrease from a fairly stable baseline) Such an increase of decrease can happen at any point.

I want to detect whether the signal is Increasing over time, Decreasing over time, or is remaining Constant over time with a good degree of certainty. Graphically speaking, I expect my data to fluctuate at some baseline value and remain constant until some time t1 at which point the signal will either increase or decrease and continue to increase or decrease until some time t2 at which the state will change. What sort of algorithm would effectively notice this?

I was thinking maybe using the derivative calculated from the data points in the array and if the slope of the tangent line is positive then the graph is increasing over time, if negative then decreasing and if 0 then constant? Not sure how to implement that in Java on my rolling array or if it would work.

I think that a fairly reliable increasing/decreasing signal would be if my data is 3*stdDev above or below the moving average but I am not sure if that is the case.

Circular array:

  private final static int NUM_SAMPLES = 100;
            double[] samples = new double[NUM_SAMPLES];
            int sampleIndex;

        public boolean isBufPopulated() {
                synchronized(this) {
                    return this.isBufPopulated;
                }
            }

            public void addSample(double val) {
                synchronized(this) {
                    samples[sampleIndex++] = val;
                    if(sampleIndex == NUM_SAMPLES) {
                        sampleIndex = 0;
                        isBufPopulated = true;
                    }
                    this.avg = calcAvg();
                }
            }

My real world baseline looks like this:

data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -9    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -9    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -9    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -9    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -9    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -9    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -9    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -9    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 3    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 3    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 3    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 3    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 3    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 3    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 3    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -9    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -9    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -9    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -9    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -9    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -9    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -9    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -9    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -11    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -11    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -11    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -11    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -11    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -11    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -11    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 1    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -2    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -4    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= -7    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 3    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 3    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 3    avgData= -3    stdDevData= 3    extrapolatedData= -3
data= 3    avgData= -3    stdDevData= 3    extrapolatedData= -3
Thornack
  • 41
  • 5

2 Answers2

0

I would make this a comment, because it's just some advice and a reference, but is too long.

Statistics are tricky, easy to calculate and difficult to interpret.

If this is a mathematical question, I guess I can give you suggestions but not answers.

Much depend of how noisy is your signal and even the shape of the noise, including the fact that you could have (or not) outliers (read which can be very away from the average and or from the actual measurable).

Then there are also the requirements your application must be met. How much delay you can have between the measure of a value and the actual detection of a change?

What I can tell you is that the idea of a circular buffer is very good. You should really avoid using new allocated objects as much as you can, because GC can be your enemy here.

500ms give you some leeway but you still need to keep your memory in check to avoid long GC stop the world pauses.

You can calculate running average and even a running standard deviation, without re-reading all your buffer.

Check this answer to have some computational advice.

I would compare 2 sub-sample and check if one is greater than the other by comparing their average divided by the first sub-sample sigma, trying to find a good compromise between missed-changes and and false-positive by tweaking the size of the samples, the offset/overlap and the sigma needed to trigger a change.

Also I would consider using median instead of average to be less sensitive to outliers if needed.

minus
  • 2,611
  • 13
  • 14
0

I too would leave a comment but its too long.

I calculate my std dev, and average using the following code:

Note* I have a java swing worker which handles my multi threading stuff which is why my methods have to be synchronized.

    public double getLatestSample() {
        synchronized(this) {
            return samples[getCircularIndex(sampleIndex-1,NUM_SAMPLES)];
        }
    }

    public double getAverage() {
        synchronized(this) {
            return this.avg;
        }
    }

    public double getStdDev() {
        synchronized(this) {
            double sum = 0;
            for(int i = 0; i < samples.length; i++) {
                double x = samples[i];
                sum += (x-avg)*(x-avg);
            }
            return Math.sqrt(sum / (samples.length-1));
        }
    }

    public double getExtrapolatedValue() {
        synchronized(this) {
            double sum = 0;
            for(int i = 0; i < samples.length; i++) {
                sum += i * (samples[getCircularIndex(i-samples.length, samples.length)] - avg);
            }
            double m = sum/samples.length;
            double b = avg - m;
            return m+b;
        }
    }

    private double calcAvg() {
        synchronized(this) {
            double sum = 0;
            for(int i = 0; i < samples.length; i++) {
                sum += samples[i];
            }
            return sum/samples.length;
        }
    }

    private int getCircularIndex(int i, int N) {
        return (i + N) % N;
    }

the sub sample idea is pretty decent, Im curious if also taking the derivative would tell me a more accurate rate of change. If I can obtain the slope of the tangent line to the curve that is represented by all data points stored in my buffer then I can establish whether or not the overall signal is increasing or decreasing or is constant (I think) through whether the slope is positive, negative or approx 0.

How would you calculate the derivative from the samples in my rolling buffer?

Thornack
  • 41
  • 5