Does reactive extensions support rolling buffers?

Question

I'm using reactive extensions to collate data into buffers of 100ms:

this.subscription = this.dataService
    .Where(x => !string.Equals("FOO", x.Key.Source))
    .Buffer(TimeSpan.FromMilliseconds(100))
    .ObserveOn(this.dispatcherService)
    .Where(x => x.Count != 0)
    .Subscribe(this.OnBufferReceived);

This works fine. However, I want slightly different behavior than that provided by the Buffer operation. Essentially, I want to reset the timer if another data item is received. Only when no data has been received for the entire 100ms do I want to handle it. This opens up the possibility of never handling the data, so I should also be able to specify a maximum count. I would imagine something along the lines of:

.SlidingBuffer(TimeSpan.FromMilliseconds(100), 10000)

I've had a look around and haven't been able to find anything like this in Rx? Can anyone confirm/deny this?

I'm sure I saw this behaviour in one of the tutorial videos on Rx but I'm afraid I can't remember what or exactly where. :( — Chris, Sep 29 '11 at 13:11
Ah, throttle (http://msdn.microsoft.com/en-us/library/hh229298%28v=vs.103%29.aspx) is what I was thinking of but I don't think that does what you want on its own. Not sure if there might be some way to combine it to do what is wanted... — Chris, Sep 29 '11 at 13:16

Colonel Panic · Answer 1 · 2012-10-29T10:49:06.493

This is possible by combining the built-in Window and Throttle methods of Observable. First, let's solve the simpler problem where we ignore the maximum count condition:

public static IObservable<IList<T>> BufferUntilInactive<T>(this IObservable<T> stream, TimeSpan delay)
{
    var closes = stream.Throttle(delay);
    return stream.Window(() => closes).SelectMany(window => window.ToList());
}

The powerful Window method did the heavy lifting. Now it's easy enough to see how to add a maximum count:

public static IObservable<IList<T>> BufferUntilInactive<T>(this IObservable<T> stream, TimeSpan delay, Int32? max=null)
{
    var closes = stream.Throttle(delay);
    if (max != null)
    {
        var overflows = stream.Where((x,index) => index+1>=max);
        closes = closes.Merge(overflows);
    }
    return stream.Window(() => closes).SelectMany(window => window.ToList());
}

I'll write a post explaining this on my blog. https://gist.github.com/2244036

Documentation for the Window method:

With the above BufferUntilInactive scenario - If the subscriber is slow than the producer, you may see a scenario where the next set of windowed items will be buffered and will not be pushed to subscriber unless an item is generated... — Rohit Sharma, Jan 25 '14 at 04:25
i've attached a sample http://snipt.org/Bhao0. In visual studio (1) open the output window (2) Check the hang button (3) Click on the button (4) wait for it to print "Click now" on the console. (5) press button three times, you will see those three clicks are missed. — Rohit Sharma, Jan 25 '14 at 06:14

score 16 · Accepted Answer · answered Sep 30 '11 at 00:32

I wrote an extension to do most of what you're after - BufferWithInactivity.

Here it is:

public static IObservable<IEnumerable<T>> BufferWithInactivity<T>(
    this IObservable<T> source,
    TimeSpan inactivity,
    int maximumBufferSize)
{
    return Observable.Create<IEnumerable<T>>(o =>
    {
        var gate = new object();
        var buffer = new List<T>();
        var mutable = new SerialDisposable();
        var subscription = (IDisposable)null;
        var scheduler = Scheduler.ThreadPool;

        Action dump = () =>
        {
            var bts = buffer.ToArray();
            buffer = new List<T>();
            if (o != null)
            {
                o.OnNext(bts);
            }
        };

        Action dispose = () =>
        {
            if (subscription != null)
            {
                subscription.Dispose();
            }
            mutable.Dispose();
        };

        Action<Action<IObserver<IEnumerable<T>>>> onErrorOrCompleted =
            onAction =>
            {
                lock (gate)
                {
                    dispose();
                    dump();
                    if (o != null)
                    {
                        onAction(o);
                    }
                }
            };

        Action<Exception> onError = ex =>
            onErrorOrCompleted(x => x.OnError(ex));

        Action onCompleted = () => onErrorOrCompleted(x => x.OnCompleted());

        Action<T> onNext = t =>
        {
            lock (gate)
            {
                buffer.Add(t);
                if (buffer.Count == maximumBufferSize)
                {
                    dump();
                    mutable.Disposable = Disposable.Empty;
                }
                else
                {
                    mutable.Disposable = scheduler.Schedule(inactivity, () =>
                    {
                        lock (gate)
                        {
                            dump();
                        }
                    });
                }
            }
        };

        subscription =
            source
                .ObserveOn(scheduler)
                .Subscribe(onNext, onError, onCompleted);

        return () =>
        {
            lock (gate)
            {
                o = null;
                dispose();
            }
        };
    });
}

+1 thanks. Did you write this just for this question or for yourself? Has it been used in production code? — Kent Boogaart, Sep 30 '11 at 06:37
@KentBoogaart - I wrote it months ago, but it's not yet in production code. It's still a WIP. — Enigmativity, Sep 30 '11 at 09:00

score 2 · Answer 3 · answered Mar 19 '15 at 20:06

2

With Rx Extensions 2.0, your can answer both requirements with a new Buffer overload accepting a timeout and a size:

this.subscription = this.dataService
    .Where(x => !string.Equals("FOO", x.Key.Source))
    .Buffer(TimeSpan.FromMilliseconds(100), 1)
    .ObserveOn(this.dispatcherService)
    .Where(x => x.Count != 0)
    .Subscribe(this.OnBufferReceived);

See https://msdn.microsoft.com/en-us/library/hh229200(v=vs.103).aspx for the documentation.

answered Mar 19 '15 at 20:06

Sébastien Lorion

80
6

But this won't have a sliding window, with the kind of 'debounce' behaviour that was requested? – Cocowalla Mar 29 '17 at 19:43
@Cocowalla I reread the original question and the code I provided does fulfill all the requirements. I have used this in production code with great success. – Sébastien Lorion Apr 10 '17 at 15:42
Sorry, I meant specifically the debounce behaviour: "I want to reset the timer if another data item is received" - I don't see that your code does this? AFAICS, your code will always push the buffer to the subscriber every 100ms (as long as it's not empty) – Cocowalla Apr 10 '17 at 20:07
I now see what you mean by debounce, thought my understanding of that term is more like http://reactivex.io/documentation/operators/debounce.html, which is `Observable.Throttle`. What you ask is more complicated but I guess it can be done with `Observable.Window`. In any case, unless I am missing something, my answer does exactly the same thing as the accepted answer on this question. – Sébastien Lorion Apr 27 '17 at 16:00
Nope, this answer has a different behavior than the accepted answer. The accepted answer correctly (according to the requirements) postpones emitting the buffer in case the source observable is continuously active. This answer simply emits the buffer every 100 milliseconds. – Theodor Zoulias Dec 28 '20 at 07:41

Ankur · Answer 4 · 2011-10-01T08:56:54.183

I guess this can be implemented on top of Buffer method as shown below:

public static IObservable<IList<T>> SlidingBuffer<T>(this IObservable<T> obs, TimeSpan span, int max)
        {
            return Observable.CreateWithDisposable<IList<T>>(cl =>
            {
                var acc = new List<T>();
                return obs.Buffer(span)
                        .Subscribe(next =>
                        {
                            if (next.Count == 0) //no activity in time span
                            {
                                cl.OnNext(acc);
                                acc.Clear();
                            }
                            else
                            {
                                acc.AddRange(next);
                                if (acc.Count >= max) //max items collected
                                {
                                    cl.OnNext(acc);
                                    acc.Clear();
                                }
                            }
                        }, err => cl.OnError(err), () => { cl.OnNext(acc); cl.OnCompleted(); });
            });
        }

NOTE: I haven't tested it, but I hope it gives you the idea.

score -1 · Answer 5 · answered Dec 27 '20 at 11:30

Here is another implementation of the BufferUntilInactive operator:

/// <summary>
/// Projects each element of an observable sequence into a buffer that's sent out
/// when either a given inactivity timespan has elapsed, or it's full.
/// </summary>
public static IObservable<IList<T>> BufferUntilInactive<T>(
    this IObservable<T> source, TimeSpan dueTime, int maxCount)
{
    return source.Publish(published =>
    {
        var windowReplay = new ReplaySubject<IObservable<T>>(1);
        var overflow = windowReplay
            .SelectMany(window => window.Take(maxCount).Count())
            .Where(windowCount => windowCount == maxCount)
            .Select(_ => Unit.Default);
        return published
            .Window(() => published
                .Throttle(dueTime)
                .Select(_ => Unit.Default)
                .Merge(overflow)
            )
            .Do(window => windowReplay.OnNext(window))
            .SelectMany(window => window.ToList());
    });
}

Does reactive extensions support rolling buffers?

5 Answers5

Linked

Related