Applying filters on a video file

Question

I want to apply filters (effects) on a video file while the video is playing.

I'm currently using @BradLarson 's (great) GPUImage framework to do so, the problem here is that the framework doesn't support audio playback while playing the video.

So I have two options:

1) Dive into the GPUImage code and change the GPUImageMovie so it will also process the audio buffers. This requires the knowledge of syncing the audio & video frames, and unfortunately i don't have it. I saw some hacks that try to play the audio with AVAudioPlayer but with a lot of sync problems.

2) Use CoreImage framework instead of GPUImage.

So I want to take a look at the second option of using the native iOS CoreImage and CIFilter to do the job.

The problem is, I couldn't find any example of how to do this with CIFilter, how do I apply filters on a video from a file?

Do I must use an AVAssetReader to read the video and process each frame? if so I'm back to my first problem of syncing the audio & video.
Or is there a way to apply the filters chain directly on the video or on the preview layer?

Appreciate any help :)

I'd certainly appreciate any help in getting audio playback added. It's not an area I'm very familiar with, and even audio recording support was added by someone else. All I've done with it is try to tweak for performance. — Brad Larson, Aug 29 '13 at 21:44
@BradLarson I'm really surprised that nobody implement it until now, especially since so many people use and contribute to GPUImage — Eyal, Aug 31 '13 at 22:13
I've done some work with compositing multiple video and audio tracks using AVComposition and I can relate to the problems with syncing. I don't know how to add audio to GPUImage. I can however offer up a sample project that may help - it's called RosyWriter from Apple and it deals with adding filters during an AVCapture session: apple.https://developer.apple.com/library/ios/samplecode/RosyWriter/Introduction/Intro.html — IanStallings, Sep 06 '13 at 14:40
This recent GitHub comment might be of interest to you: https://github.com/BradLarson/GPUImage/issues/458#issuecomment-23704744 — Brad Larson, Sep 10 '13 at 14:38
thanks @BradLarson actually I had a part in writing this solution :) We couldn't find any other way than writing our own implementation. — Eyal, Sep 17 '13 at 07:13
@Eyal - If you want to clean that up and submit it as a pull request, I'd be glad to bring it in. — Brad Larson, Sep 17 '13 at 17:24
Sure @BradLarson we are planning to do so in the near future. — Eyal, Sep 18 '13 at 15:17

Parvez Belim · Answer 1 · 2013-12-21T07:16:40.373

Use GPUImage framework only that you are using... that is best framework until now for video filters. GO through the documentation of the framework https://github.com/BradLarson/GPUImage scroll down the page you will find the details of the filters available...

this filters are applied on the video and to write the video you have to use the GPUImageMovieWriter class...It automatically handles audio ..

you don't have to maintain it...Use shouldPassThroughAudio property of GPUImageMovieWriter and it will manage the audio on its own.

Use this tutorial for help http://www.sunsetlakesoftware.com/2012/02/12/introducing-gpuimage-framework

Here is the code where I am using GPUImage framework to crop the video and audio is stored not removed after editing.

NSURL *videoUrl = [selectedAsset defaultRepresentation].url;

GPUImageMovie *movieUrl = [[GPUImageMovie alloc] initWithURL:videoUrl];

self.cropFilter = [[GPUImageCropFilter alloc] initWithCropRegion:videoArea];
movieUrl.runBenchmark = YES;
movieUrl.playAtActualSpeed = YES;
[movieUrl addTarget:self.cropFilter];

//Setting path for temporary storing the video in document directory
NSArray *paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
NSString *documentsDirectory = [paths objectAtIndex:0];
NSString *myPathDocs =  [documentsDirectory stringByAppendingPathComponent:
                         [NSString stringWithFormat:@"CroppedVideo-%d.mov",arc4random() % 1000]];
NSURL *movieURL = [NSURL fileURLWithPath:myPathDocs];

AVURLAsset *asset = [AVURLAsset URLAssetWithURL:videoUrl options:nil];
AVAssetTrack *videoAssetTrack = [[asset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0];
CGAffineTransform videoTransform = videoAssetTrack.preferredTransform;

movieWriter = [[GPUImageMovieWriter alloc] initWithMovieURL:movieURL size:CGSizeMake(videoAssetTrack.naturalSize.width, videoAssetTrack.naturalSize.height)];

[_cropFilter addTarget:movieWriter];
movieWriter.shouldPassthroughAudio = YES;

movieUrl.audioEncodingTarget = movieWriter;

[movieUrl enableSynchronizedEncodingUsingMovieWriter:movieWriter];

[self.movieWriter startRecordingInOrientation:videoTransform];
[self.movieWriter startRecording];

[movieUrl startProcessing];
__block BOOL completeRec = NO;
__unsafe_unretained typeof(self) weakSelf = self;
[self.movieWriter setCompletionBlock:^{

    [weakSelf.cropFilter removeTarget:weakSelf.movieWriter];
    [weakSelf.movieWriter finishRecording];
    [movieUrl removeTarget:weakSelf.cropFilter];
    if (!completeRec)
    {
        [weakSelf videoCropDoneUrl:movieURL];
        completeRec = YES;
    }
}];

None of that will help them play audio from a movie using the framework, since that capability doesn't currently exist within the framework. — Brad Larson, Sep 08 '13 at 00:27
@BradLarson Ohh! thats the reason.I have downloaded your class long time ago. — Parvez Belim, Sep 09 '13 at 11:23
@ParvezBelim GPUImage is technically a framework, not a 'class'. — David Schwartz, Dec 20 '13 at 21:14

score 0 · Answer 2 · answered Aug 29 '13 at 09:41

You can use OpenCV framework to play with video Editing..

Here is the link to download OpenCv

http://sourceforge.net/projects/opencvlibrary/files/opencv-ios/2.4.6/opencv2.framework.zip/download

and here is a tutorial to play with OpenCV

http://docs.opencv.org/doc/tutorials/ios/video_processing/video_processing.html#opencviosvideoprocessing

I think this would help you..

score 0 · Answer 3 · edited May 23 '17 at 12:01

Although solution 2 sounds like an "OK" solution if it was my app I'd still prefer using GPUImage Video filters.

Solution:

Get the original video and tear the audio track out of it. (preferably in a background thread) with something like https://github.com/unixpickle/MP4Audio
Let the user choose his video filters with GPUImage, process the video and save it.
Once the video is ready, merge the new edited filtered video with the audio track you saved before GPUImage intervention. (Many ways to do the merge, here's one of them: Merging Audio with Video Objective-C )

Shana Tova

Applying filters on a video file

3 Answers3