Can I get my WPF application to play an image sequence at 120 FPS without using so much memory?

Question

I'm writing an application to play back an image sequence of varying size (possibly 1000+ images) at variable FPS (up to 120), and while my current solution works for a small amount of images, it eats up memory very quickly (playing ~109mb of images takes ~526mb of Process Memory, and it seems to increase exponentially) and if given a large amount of images will produce an OutOfMemoryException.

My MainWindow and VideoViewer classes look like this -

MainWindow:

public partial class MainWindow : Window
{
    OpenFileDialog openFileDialog1 = new OpenFileDialog();
    VideoViewer vV = new VideoViewer();
    DispatcherTimer timer = new DispatcherTimer();

    public MainWindow()
    {
        InitializeComponent();

        this.DataContext = vV;
    }

    private void PlayButton_Click(object sender, RoutedEventArgs e)
    {
        timer = new DispatcherTimer { Interval = TimeSpan.FromSeconds(1f/120f) };
        timer.Tick += TimerTick;
        timer.Start();
    }

    private void PauseButton_Click(object sender, RoutedEventArgs e)
    {
        timer.Stop();
    }

    private void TimerTick(object sender, EventArgs e)
    {
        vV.NextFrame();
    }

    private void BrowseButton_Click(object sender, RoutedEventArgs e)
    {
        // Set the file dialog to filter for graphics files.
        this.openFileDialog1.Filter =
            "Images (*.BMP;*.JPG;*.GIF;*.PNG)|*.BMP;*.JPG;*.GIF;*.PNG|" +
            "All files (*.*)|*.*";

        // Allow the user to select multiple images.
        this.openFileDialog1.Multiselect = true;
        this.openFileDialog1.Title = "My Image Browser";

        DialogResult dr = this.openFileDialog1.ShowDialog();
        if (dr == System.Windows.Forms.DialogResult.OK)
        {
            if (dr == System.Windows.Forms.DialogResult.OK)
            {
                List<ImageSource> images = new List<ImageSource>();

                // Read the files
                foreach (String file in openFileDialog1.FileNames)
                {            
                    var fullFilePath = @file;

                    //Create a new bitmap to assign our image to
                    BitmapImage bitmap = new BitmapImage();
                    bitmap.BeginInit();
                    bitmap.CreateOptions = BitmapCreateOptions.IgnoreImageCache;
                    bitmap.CacheOption = BitmapCacheOption.None;
                    bitmap.UriSource = new Uri(fullFilePath, UriKind.Absolute);
                    bitmap.EndInit();

                    images.Add(bitmap);
                }

                vV.imagesArray = images;
            }
        }
    }
}

VideoViewer:

public class VideoViewer : INotifyPropertyChanged
{
    private ImageSource image;
    private List<ImageSource> imageLocs = new List<ImageSource>();
    private int imageIndex = 0;

    public event PropertyChangedEventHandler PropertyChanged;

    public ImageSource imageSource
    {
        get
        {
            return image;
        }
        set
        {
            image = value;
            OnPropertyChanged("imageSource");
        }
    }

    public List<ImageSource> imagesArray
    {
        get
        {
            return imageLocs;
        }
        set
        {
            imageLocs = value;
            OnPropertyChanged("imagesArray");
        }
    }

    public void NextFrame()
    {
        //If not on first or last frame
        if(imageIndex < (imageLocs.Count - 1))
        {
            imageIndex += 1;
            imageSource = imagesArray[imageIndex];
            OnPropertyChanged("imageSource");
        }
        else if(imageIndex == (imageLocs.Count - 1))
        {
            imageIndex = 0;
            imageSource = imagesArray[imageIndex];
            OnPropertyChanged("imageSource");
        }
    }

    // Create the OnPropertyChanged method to raise the event
    protected void OnPropertyChanged(string name)
    {
        PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(name));
    }
}

When playing the sequence, the first loop plays back slowly, and this is where the memory is taken up. After the first loop, it plays back at the desired speed and the memory sits at a constant value, which leads me to think it's something to do with the caching of the images.

As you can see I've tried adjusting the caching options of the bitmap image but it didn't make a difference. I'm also currently building to x64 to give myself extra memory to use.

Is it possible to reduce the memory usage while keeping the playback speed?

You may want to refer to your previous question: http://stackoverflow.com/questions/43321555/playing-an-image-sequence-in-wpf-at-120fps. To prevent duplicate flags. — Mixxiphoid, Apr 11 '17 at 10:49
some good ideas here: http://stackoverflow.com/questions/1684489/how-do-you-make-sure-wpf-releases-large-bitmapsource-from-memory — Colin Smith, Apr 11 '17 at 11:52

Mihai Stoica · Answer 1 · 2018-03-19T23:24:28.730

I managed to find a solution for 1000+ PNGs without converting the image sequence to a movie, so here's my advice for anyone trying to achieve this:

Make sure you compress your images as much as possible. I managed to downsize my PNGs by 70-80% using online tools without noticeable loss in quality.
Load all your images at app start-up. Since BitmapImage takes the number of pixels times 4 bytes of memory (regardless of image compression), you're better off saving the actual file streams as a MemoryStream array. I found that memory usage is quite stable and close to the size of your files.
With each timer tick, convert your current MemoryStream to a BitmapImage and update the source of your Image. Feel free to adjust the framerate as desired.

score 1 · Accepted Answer · edited May 23 '17 at 12:17

Possible Attempts to use Less Bitmap Memory

Try creating the BitmapSource by using the .StreamSource property rather than .UriSource, and use the OnLoad cache option.

var fullFilePath = @file;

//Create a new bitmap to assign our image to

using (FileStream fs = new FileStream(path, FileMode.Open))
{
    BitmapImage bitmap = new BitmapImage();
    bitmap.BeginInit();
    bitmap.StreamSource = fs;
    bitmap.CacheOption = BitmapCacheOption.OnLoad;
    bitmap.EndInit();

    images.Add(bitmap);

    bitmap.Freeze();
}

See:

Alternative Approach - Converting Your Sequence of Images into a Video and Playing that

Effectively what you are trying to do is implement the behaviour of a video player, but with a "selected" set of frames.

Another way of doing what you want in a more efficient manner, would to take those set of frames, and generate a video file (say an MP4).

There are a number of libraries you could use to do that:

ffmpeg https://ffmpeg.org/download.html
ffmpeg Autogen (a wrapper for NET) https://github.com/Ruslan-B/FFmpeg.AutoGen
accord-net-extensions (VideoWriter) https://github.com/dajuric/accord-net-extensions Image sequence to video stream?
dot-imaging library https://www.codeproject.com/Articles/828012/Introducing-Portable-Video-IO-Library-for-Csharp https://github.com/dajuric/dot-imaging/)
Microsoft Media Foundation
Media Foundation .NET wrapper https://sourceforge.net/projects/mfnet/
Some samples on encoding a video using media foundation (note one is UWP...but it should give clues on how to achieve same on FullFramework/WPF).

(H.264 image encoding using Media Foundation .NET) https://codereview.stackexchange.com/questions/136144/h-264-image-encoding-using-media-foundation-net

(How to encode several images to a video in universal Windows apps) https://code.msdn.microsoft.com/windowsapps/How-to-encode-several-to-a-053953d1

You could then just use the standard build in MediaElement to play that file.

To support looping you could use the ideas here:

Replaying a video continuously in a WPF media element

Working Solution

As confirmed below, you ended up using the "Accord" library to generate a video file from the sequence of images, and then played that in a MediaElement.

Image sequence to video stream?

This eliminates the initial loading loop and loads them all in before playing, but it hasn't made a difference to how much memory is being used. — Luke4792, Apr 11 '17 at 12:39
they are all loaded in memory till they aren't used by your imageArray anymore and get garbage collected....do you get the OutOfMemory exception now? You might need to hold fewer images in your imageArray, and keep filling and pruning it when images are no longer onscreen and possibly do some Garbage collection to force out some of the others, Also look at a potential "stream" buffer issue. — Colin Smith, Apr 11 '17 at 13:23
You could also try using https://msdn.microsoft.com/en-us/library/system.runtime.gcsettings.largeobjectheapcompactionmode(v=vs.110).aspx.....to use the "compact" option on the LOH....as you might be hitting fragmentation issues which is causing your Virtual Mem space to enlarge. — Colin Smith, Apr 11 '17 at 13:27
I have tried loading each image when it's needed, but it seems as though I'm stuck, as if I don't load all the images into an array, the program can't read/decompress the images fast enough for a decent framerate, and if i do load them all in I run out of memory. — Luke4792, Apr 11 '17 at 14:55
one lateral thought would be to somehow use those images to generate a movie which you could then play in a looping fashion which appears to be what you are doing. You could change the playback rate to do the variable FPS which also appears what you want. How you generate that movies could be done in a few ways a) with ffmpeg for sure (you just need to find one .NET wrapper library) that lets you call it, or you can shell out to the command line yourself or b) use Media Foundaion library. Then after getting an MP4 file produced, just get it to play in a MediaElement....to be efficiently played — Colin Smith, Apr 11 '17 at 15:56
Thanks for your help! I ended up using the accord .net library to write the sequence to a video file and play it in a MediaElement. I found this question particularly useful: http://stackoverflow.com/questions/9744026/image-sequence-to-video-stream . If you want to edit your question a little to reflect the final solution i'll mark it as an answer — Luke4792, Apr 13 '17 at 14:39

Can I get my WPF application to play an image sequence at 120 FPS without using so much memory?

2 Answers2

Possible Attempts to use Less Bitmap Memory

Alternative Approach - Converting Your Sequence of Images into a Video and Playing that

Working Solution