56

Is there any way to detect duplicate frames within the video using ffmpeg?

I tried -vf flag with select=gt(scene\,0.xxx) for scene change. But, it did not work for my case.

Giacomo1968
  • 23,903
  • 10
  • 59
  • 92
metlira
  • 743
  • 1
  • 6
  • 11

3 Answers3

79

Use the mpdecimate filter, whose purpose is to "Drop frames that do not differ greatly from the previous frame in order to reduce frame rate."

  • This will generate a console readout showing which frames the filter thinks are duplicates.

    ffmpeg -i input.mp4 -vf mpdecimate -loglevel debug -f null -

  • To generate a video with the duplicates removed

    ffmpeg -i input.mp4 -vf mpdecimate,setpts=N/FRAME_RATE/TB out.mp4

The setpts filter expression generates smooth timestamps for a video at FRAME_RATE FPS. See an explanation for timestamps at What is video timescale, timebase, or timestamp in ffmpeg?

Giacomo1968
  • 23,903
  • 10
  • 59
  • 92
Gyan
  • 63,018
  • 7
  • 100
  • 141
36

I also had this problem and Gyan's excellent answer above got me started but the result of it was desynchronized audio so I had to explore more options:

mpdecimate vs decimate filters

  • mpdecimate is the standard recommendation I found all over SO and the internet, but I don't think it should be the first pick
    • it uses heuristics so it may and will skip some duplicate frames
    • you can tweak the detection with frac parameter, but that's extra work you may want to avoid if you can
    • it is not really supposed to work with mp4 container (source), but I was using mkv so this limitation didn't apply on my case, but good to be aware of it
  • decimate removes frames precisely, but it is useful only for periodically occurring duplicates

detected vs actual frame rate

  • so you have multimedia file with duplicate frames, it is good idea to make sure that the detected frame rate matches the actual one

  • ffprobe in.mkv will output the detected FPS; it may look like this

    Stream #0:0: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 1920x1080, SAR 1:1 DAR 16:9, 25 fps, 25 tbr, 1k tbn, 50 tbc (default)

  • the actual frame rate can be found out if you open the media in.mkv in a media player that lets you step one frame at the time; then count the steps needed to advance the playback time for 1 second, in my case it was 30 fps

  • not a big surprise for me, because every 6th frame was duplicate (5 good frames and 1 duplicate), so after 25 good frames there was also 5 duplicates

what is N/FRAME_RATE/TB

  • except the use of FRAME_RATE variable the N/FRAME_RATE/TB is equal to the example below from ffmpeg documentation (source)

    Set fixed rate of 25 frames per second:
    setpts=N/(25*TB)

  • the math behind it perfectly explained in What is video timescale, timebase, or timestamp in ffmpeg?

    • it basically calculates timestamp for each frame and multiplies it with timebase TB to enhance precision

FRAME_RATE variable vs literal FPS value (e.g. 25)

  • this is why it is important to know your detected and actual FPS
  • if the detected FPS matches your actual FPS (e.g. both are 30 fps) you can happily use FRAME_RATE variable in N/FRAME_RATE/TB
  • but if the detected FPS differs than you have to calculate the FRAME_RATE on your own
    • in my case my actual FPS was 30 frames per second and I removed every 6th frame, so the target FPS is 25 which leads to N/25/TB
      • if I used FRAME_RATE (and I actually tried that) it would take the wrong detected fps of 25 frames i.e. FRAME_RATE=25, run it through mpdecimate filter which would remove every 6th frame and it would update to FRAME_RATE=20.833 so N/FRAME_RATE/TB would actually be N/20.833/TB which is completely wrong

to use or not to use setpts

  • so the setpts filter already got pretty complicated especially because of the FPS mess that duplicate frames may create
  • the good news is you actually may not need the setpts filter at all

  • here is what I used with good results

    ffmpeg -i in.mkv -vf mpdecimate out.mkv

    ffmpeg -i in.mkv -vf decimate=cycle=6,setpts=N/25/TB out.mkv

  • but the following gave me desynchronized audio

    ffmpeg -i in.mkv -vf mpdecimate,setpts=N/FRAME_RATE/TB out.mkv

    ffmpeg -i in.mkv -vf mpdecimate,setpts=N/25/TB out.mkv

    ffmpeg -i in.mkv -vf decimate=cycle=6 out.mkv

  • as you see

    • mpdecimate and decimate does not work the same way
    • mpdecimate worked better for me without setpts filter
    • while decimate needed setpts filter and furthermore I need to avoid FRAME_RATE variable and use N/25/TB instead because the actual FPS was not detected properly

note on asetpts

  • it does the same job as setpts does but for audio
  • it didn't really fix desync audio for me but you want to use it something like this -af asetpts=N/SAMPLE_RATE/TB
  • maybe you are supposed to adjust the SAMPLE_RATE according to the ratio of duplicate frames removed, but it seems to me like extra unnecessary work especially when my video had the audio in sync at the beginning, so it is better to use commands that will keep it that way instead of fixing it later

tl;dr

If the usually recommended command ffmpeg -i in.mkv -vf mpdecimate,setpts=N/FRAME_RATE/TB out.mkv does not work for you try this:

ffmpeg -i in.mkv -vf mpdecimate out.mkv

or

ffmpeg -i in.mkv -vf decimate=cycle=6,setpts=N/25/TB out.mkv

(cycle=6 because every 6th frame is duplicate and N/25/TB because after removing the duplicates the video will have 25 fps (avoid the FRAME_RATE variable); adjust for your use case)

Vlastimil Ovčáčík
  • 2,191
  • 24
  • 28
  • 1
    The setpts is needed with mp4 because it is a constant frame rate muxer. Add `-vsync vfr` and remove setpts in my command to keep deduplicated frames with same timestamp when saving to mp4 – Gyan Sep 03 '18 at 08:48
  • `Add -vsync vfr and remove setpts in my command` @Gyan --- Well that's exactly what I did for the usecase with `mpdecimate`, specifically I removed the `setpts` filter and I didn't even needed the `-vsync vfr`. It's all in `tl;dr` section. – Vlastimil Ovčáčík Sep 06 '18 at 18:16
  • `The setpts is needed with mp4 because it is a constant frame rate muxer` @Gyan --- that's interesting, but also notice that while I was able to remove `setpts` filter from the `mpdecimate` filter usecase (as you suggest), I **had to** use `setpts` for the `decimate` usecase. Both are `mkv`. Go figure right?! – Vlastimil Ovčáčík Sep 06 '18 at 18:18
  • When trying `ffmpeg -i in.mkv -vf [mpdecimate][] out.mkv` I get `[AVFilterGraph @ ...] Bad (empty?) label found in the following: "[]".` `Error reinitializing filters!` `Failed to inject frame into filter network: Invalid argument.` `Error while processing the decoded data for stream #0:0` as error. – Lena Apr 27 '20 at 10:08
  • 1
    Hi @Tijmen and thanks for pointing out the problem! I have investigated a bit and it seems there was an [edit to the original answer](https://stackoverflow.com/revisions/52062421/2) that has broken the ffmpeg commands. As I see from your comment, you have used the broken commands. To be specific, the command you used `ffmpeg -i in.mkv -vf [mpdecimate][] out.mkv` is defective and the correct one is `ffmpeg -i in.mkv -vf mpdecimate out.mkv` (notice there are no square brackets). It is all fixed now. – Vlastimil Ovčáčík Apr 29 '20 at 16:34
  • Just tried this solution on a 4 hour video, and the video was still 4 hours afterwards. I expected this solution to totally cut out the repeated frames, but it looks like it only compressed them? If I wanted a shorter video, by removing duplicate frames, how would I go about doing that? – Xavier Horn Aug 10 '20 at 13:09
  • @XavierHorn: You can try the repo [mpdecimate_trim](https://github.com/KenjiTakahashi/mpdecimate_trim). I explained how in a new response. – Human Apr 29 '21 at 17:47
0

I tried the solution here and none of them seem to work when you need to trim the video and keep the audio. There is a mpdecimate_trim repo that does a good job. It basically list all the frames to be dropped (using mpdecimate) and then creates a complex filter to trim all those frames (and the corresponding audio) from the video by splitting the video and only including the portion without duplicate frames.

I did have to tweak a few options in the code though. For instance, in mpdecimate_trim.py, I had to change this line:

dframes2 = get_dframes(ffmpeg(True, "-vf", "mpdecimate=hi=576", "-loglevel", "debug", "-f", "null", "-").stderr)

I had to detect duplicates a bit more aggressively, so I changed the mpdecimate option to mpdecimate=hi=64*32:lo=64*24:frac=0.05

Human
  • 6,849
  • 1
  • 16
  • 14