How to Simply Remove Duplicate Frames from a Video using ffmpeg

Question

First of all, I'd preface this by saying I'm NO EXPERT with video manipulation, although I've been fiddling with ffmpeg for years (in a fairly limited way). Hence, I'm not too flash with all the language folk often use... and how it affects what I'm trying to do in my manipulations... but I'll have a go with this anyway...

I've checked a few links here, for example: ffmpeg - remove sequentially duplicate frames

...but the content didn't really help me.

I have some hundreds of video clips that have been created under both Windows and Linux using both ffmpeg and other similar applications. However, they have some problems with times in the video where the display is 'motionless'.

As an example, let's say we have some web site that streams a live video into, say, a Flash video player/plugin in a web browser. In this case, we're talking about a traffic camera video stream, for example.

There's an instance of ffmpeg running that is capturing a region of the (Windows) desktop into a video file, viz:-

ffmpeg -hide_banner -y -f dshow ^
      -i video="screen-capture-recorder" ^
      -vf "setpts=1.00*PTS,crop=448:336:620:360" ^
      -an -r 25 -vcodec libx264 -crf 0 -qp 0 ^
      -preset ultrafast SAMPLE.flv

Let's say the actual 'display' that is being captured looks like this:-

123456789 XXXXX 1234567 XXXXXXXXXXX 123456789 XXXXXXX
^---a---^ ^-P-^ ^--b--^ ^----Q----^ ^---c---^ ^--R--^ 

...where each character position represents a (sequence of) frame(s). Owing to a poor internet connection, a "single frame" can be displayed for an extended period (the 'X' characters being an (almost) exact copy of the immediately previous frame). So this means we have segments of the captured video where the image doesn't change at all (to the naked eye, anyway).

How can we deal with the duplicate frames?... and how does our approach change if the 'duplicates' are NOT the same to ffmpeg but LOOK more-or-less the same to the viewer?

If we simply remove the duplicate frames, the 'pacing' of the video is lost, and what used to take, maybe, 5 seconds to display, now takes a fraction of a second, giving a very jerky, unnatural motion, although there are no duplicate images in the video. This seems to be achievable using ffmpeg with an 'mp_decimate' option, viz:-

     ffmpeg -i SAMPLE.flv ^                      ... (i)
        -r 25 ^
        -vf mpdecimate,setpts=N/FRAME_RATE/TB DEC_SAMPLE.mp4

That reference I quoted uses a command that shows which frames 'mp_decimate' will remove when it considers them to be 'the same', viz:-

     ffmpeg -i SAMPLE.flv ^                      ... (ii)
        -vf mpdecimate ^
        -loglevel debug -f null -

...but knowing that (complicated formatted) information, how can we re-organize the video without executing multiple runs of ffmpeg to extract 'slices' of video for re-combining later?

In that case, I'm guessing we'd have to run something like:-

• user specifies a 'threshold duration' for the duplicates (maybe run for 1 sec only)
• determine & save main video information (fps, etc - assuming constant frame rate)
• map the (frame/time where duplicates start)->no. of frames/duration of duplicates
• if the duration of duplicates is less than the user threshold, don't consider this period as a 'series of duplicate frames' and move on
• extract the 'non-duplicate' video segments (a, b & c in the diagram above)
• create 'new video' (empty) with original video's specs
• for each video segment extract the last frame of the segment create a short video clip with repeated frames of the frame just extracted (duration = user spec. = 1 sec) append (current video segment+short clip) to 'new video' and repeat

...but in my case, a lot of the captured videos might be 30 minutes long and have hundreds of 10 sec long pauses, so the 'rebuilding' of the videos will take a long time using this method.

This is why I'm hoping there's some "reliable" and "more intelligent" way to use ffmepg (with/without the 'mp_decimate' filter) to do the 'decimate' function in only a couple of passes or so... Maybe there's a way that the required segments could even be specified (in a text file, for example) and as ffmpeg runs it will stop/restart it's transcoding at specified times/frame numbers?

Short of this, is there another application (for use on Windows or Linux) that could do what I'm looking for, without having to manually set start/stop points, extracting/combining video segments manually...?

I've been trying to do all this with ffmpeg N-79824-gcaee88d under Win7-SP1 and (a different version I don't currently remember) under Puppy Linux Slacko 5.6.4.

Thanks a heap for any clues.

Answer 1

I assume what you want to do is to keep frames with motion and upto 1 second of duplicate frames but discard the rest.

ffmpeg -i in.mp4 -vf
"select='if(gt(scene,0.01),st(1,t),lte(t-ld(1),1))',setpts=N/FRAME_RATE/TB"
trimmed.mp4

What the select filter expression does is make use of an if-then-else operator:

gt(scene,0.01) checks if the current frame has detected motion relative to the previous frame. The value will have to be calibrated based on manual observation by seeing which value accurately captures actual activity as compared to sensor/compression noise or visual noise in the frame. See here on how to get a list of all scene change values.

If the frame is evaluated to have motion, the then clause evaluates st(1,t). The function st(val,expr) stores the value of expr in a variable numbered val and it also returns that expression value as its result. So, the timestamp of the kept frames will keep on being updated in that variable until a static frame is encountered.

The else clause checks the difference between the current frame timestamp and the timestamp of the stored value. If the difference is less than 1 second, the frame is kept, else discarded.

The setpts sanitizes the timestamps of all selected frames.

Edit: I tested my command with a video input I synthesized and it worked.

Answer 2

I've done a bit of work on this question... and have found the following works pretty well...

It seems like the input video has to have a "constant frame rate" for things to work properly, so the first command is:-

ffmpeg -i test.mp4 ^
       -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" ^
       -vsync cfr test01.mp4

I then need to look at the 'scores' for each frame. Such a listing is produced by:-

ffmpeg -i test01.mp4 ^
       -vf select="'gte(scene,0)',metadata=print" -f null - 

I'll look at all those scores... and average them (mean) - a bit dodgy but it seems to work Ok. In this example, that average score is '0.021187'.

I then have to select a 'persistence' value -- how long to let the 'duplicated' frames run. If you force it to only keep one frame, the entire video will tend to run much too quickly... So, I've been using 0.2 seconds as a starting point.

So the next command becomes:-

ffmpeg -i test01.mp4 ^
       -vf "select='if(gt(scene,0.021187),st(1,t),lte(t-ld(1),0.20))', 
                    setpts=N/FRAME_RATE/TB" output.mp4

After that, the resultant 'output.mp4' video seems to work pretty well. It's only a bit of fiddling with the 'persistence' value that might need to be done to compromise between having a smoother-playing video and scenes that change a bit abruptly.

I've put together some Perl code that works Ok, which I'll work out how to post, if folks are interested in it... eventually(!)

Edit: Another advantage of doing this 'decimating', is that files are of shorter duration (obviously) AND they are smaller in size. For example, a sample video that ran for 00:07:14 and was 22MB in size went to 00:05:35 and 11MB.

Answer 3

Variable frame rate encoding is totally possible, but I don't think it does what you think it does. I am assuming that you wish to remove these duplicate frames to save space/bandwidth? If so, it will not work because the codec is already doing it. Codecs use reference frames, and only encode what has changed from the reference. Hence the duplicate frame take almost no space to begin with. Basically frames are just encoded as a packet of data saying, copy the previous frame, and make this change. The X frames have zero changes, so it only takes a few bytes to encode each one.