Anyone else having problems with commercial removal in 0.28?

Have a MythTV related problem? Ask for help from other MythTV users here.

Moderator: Forum Moderators

ricks03
Junior
Posts: 61
Joined: Thu Oct 01, 2015 8:59 pm
United States of America

Re: Anyone else having problems with commercial removal in 0

Post by ricks03 » Thu Nov 24, 2016 4:45 am

Messaged you what I'd changed so you have it.

mr_tea
Newcomer
Posts: 13
Joined: Tue Apr 11, 2017 2:47 am
United States of America

Re: Anyone else having problems with commercial removal in 0

Post by mr_tea » Tue Apr 11, 2017 2:58 am

Is someone planning to post this script to a site like github where we can get access to changes? I've have been looking for a script like this for quite a while, and only when I started writing my own yesterday did I find this discussion.

By the way, my version has a couple of modifications. I've added --ignore_unknown to the ffmpeg call to ignore some of the streams that Comcast puts in that ffmpeg doesn't recognize. I figure if ffmpeg doesn't know what it is, I'm not likely to have a lot of use for it (and ffmpeg can cut it with them there anyway).

cut =subprocess.Popen([ffmpeg,'-ignore_unknown','-i',infile,'-c','copy','-map','0','-f','segment','-segment_list','%scut.ffcat'%(tmpdir),'-segment_frames',cutlist,'%scut%%03d.ts'%(tmpdir)],stdout = subprocess.PIPE, stderr = subprocess.STDOUT,universal_newlines=True)

Also, I am changing how ffmpeg is found so that you set it in the configurations at the top of the script. I'm running on OS X and using macports, so it doesn't exist in the locations the script currently searches. I may also just set it up to use distutils find_executable().

daraden
Senior
Posts: 169
Joined: Tue Feb 23, 2016 7:33 am
United States of America

Re: Anyone else having problems with commercial removal in 0

Post by daraden » Tue Apr 11, 2017 7:37 am

I've been thinking of setting up a git for this and my other scripts. Didn't seem like there was a lot of interest so I never got around to it.

since this is derived from my transcoding script never thought to strip out the other streams since they are removed later. Only benefit might be a few MB saved.

I switched the check for ffmpeg on my other scripts to use distutils find_executable. just replace the old prog_check with

Code: Select all

def prog_check():
    #checks for FFmpeg/FFprobe
    #if not found MythFFmpeg/MythFFprobe is used
    from distutils import spawn

    if spawn.find_executable('ffmpeg') != None:
        ffmpeg = spawn.find_executable('ffmpeg')
    if (spawn.find_executable('ffmpeg') == None and
          spawn.find_executable('mythffmpeg')):
        ffmpeg = spawn.find_executable('mythffmpeg')
    else:
        ffmpeg = None
    if spawn.find_executable('ffprobe') != None:
        ffprobe = spawn.find_executable('ffprobe')
    if (spawn.find_executable('ffprobe') == None and
          spawn.find_executable('mythffprobe')):
        ffprobe = spawn.find_executable('mythffprobe')
    else:
        ffprobe = None
    if ffmpeg != None and ffprobe != None:
        return ffmpeg,ffprobe
    else:
        raise LookupError('Unable to find FFmpeg or MythFFmpeg')
how is the script working on h264? I only have OTA, so i haven't done much testing.

mr_tea
Newcomer
Posts: 13
Joined: Tue Apr 11, 2017 2:47 am
United States of America

Re: Anyone else having problems with commercial removal in 0

Post by mr_tea » Sat Apr 15, 2017 11:53 pm

I need to play with it more. I only had the chance the other evening to see if the files that were created were valid. I need to watch all the way through a couple of test samples, especially the cut points, and make sure everything lines up correctly.

The -ignore_unknown probably does save space, but the real reason I had to add it was that ffmpeg errors when it runs into a stream it doesn't understand while cutting. So without it, the script fails on all the h264 streams coming from Comcast.

mr_tea
Newcomer
Posts: 13
Joined: Tue Apr 11, 2017 2:47 am
United States of America

Re: Anyone else having problems with commercial removal in 0

Post by mr_tea » Sun Apr 16, 2017 3:21 am

I had some more time to look at the file output this evening. While the timings on the mp2 files appear to be correct, the h264 files are being cut significantly earlier than defined. This difference doesn't appear to be keyframe related, because the time can be several seconds.

daraden
Senior
Posts: 169
Joined: Tue Feb 23, 2016 7:33 am
United States of America

Re: Anyone else having problems with commercial removal in 0

Post by daraden » Sun Apr 16, 2017 9:19 am

Figured there would be a timing issue. converting the frame numbers to timestamps may fix it just have to dig through some of my other code to find the bits needed to implement it.
you could also try

Code: Select all

cut =subprocess.Popen([ffmpeg,'-ignore_unknown','-i',infile,'-c','copy','-map','0','-f','segment','-segment_list','%scut.ffcat'%(tmpdir),'-segment_frames',cutlist,'-break_non_keyframes', '1','%scut%%03d.ts'%(tmpdir)],stdout = subprocess.PIPE, stderr = subprocess.STDOUT,universal_newlines=True)
break_non_keyframes 1 could fix the issue if it is key-frame related.

could you post the output of

Code: Select all

ffprobe -shows_streams file.ts 
for one of your h264 recordings.

mr_tea
Newcomer
Posts: 13
Joined: Tue Apr 11, 2017 2:47 am
United States of America

Re: Anyone else having problems with commercial removal in 0

Post by mr_tea » Fri Apr 21, 2017 4:40 pm

ffprobe version 3.2.4 Copyright (c) 2007-2017 the FFmpeg developers
built with Apple LLVM version 7.3.0 (clang-703.0.31)
configuration: --prefix=/opt/local --enable-swscale --enable-avfilter --enable-avresample --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-libtheora --enable-libschroedinger --enable-libopenjpeg --enable-libmodplug --enable-libvpx --enable-libsoxr --enable-libspeex --enable-libass --enable-libbluray --enable-lzma --enable-gnutls --enable-fontconfig --enable-libfreetype --enable-libfribidi --disable-indev=jack --disable-outdev=xv --enable-audiotoolbox --enable-sdl2 --mandir=/opt/local/share/man --enable-shared --enable-pthreads --cc=/usr/bin/clang --enable-vda --enable-videotoolbox --arch=x86_64 --enable-yasm --enable-libx265 --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid
libavutil 55. 34.101 / 55. 34.101
libavcodec 57. 64.101 / 57. 64.101
libavformat 57. 56.101 / 57. 56.101
libavdevice 57. 1.100 / 57. 1.100
libavfilter 6. 65.100 / 6. 65.100
libavresample 3. 1. 0 / 3. 1. 0
libswscale 4. 2.100 / 4. 2.100
libswresample 2. 3.100 / 2. 3.100
libpostproc 54. 1.100 / 54. 1.100
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] SPS unavailable in decode_picture_timing
[h264 @ 0x7fcc5c023000] non-existing PPS 0 referenced
[h264 @ 0x7fcc5c023000] decode_slice_header error
[h264 @ 0x7fcc5c023000] no frame!
[h264 @ 0x7fcc5c023000] Increasing reorder buffer to 2
[mpegts @ 0x7fcc5b00aa00] PES packet size mismatch
Last message repeated 1 times
[mpegts @ 0x7fcc5b00aa00] Could not find codec parameters for stream 3 (Unknown: none (ETV1 / 0x31565445)): unknown codec
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[mpegts @ 0x7fcc5b00aa00] Could not find codec parameters for stream 4 (Unknown: none (ETV1 / 0x31565445)): unknown codec
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, mpegts, from '/Volumes/GarbagePod/mythtv/myth_videos/by_date/Phineas and Ferb/2017-03-02, 12-56 AM - La Candace- Cabra; Happy Birthday, Isabella.mpg':
Duration: 00:38:59.29, start: 81941.554178, bitrate: 3928 kb/s
Program 1
Stream #0:0[0x1109]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1280x720 [SAR 1:1 DAR 16:9], Closed Captions, 59.94 fps, 59.94 tbr, 90k tbn, 119.88 tbc
Stream #0:1[0x110a](eng): Audio: ac3 ([129][0][0][0] / 0x0081), 48000 Hz, 5.1(side), fltp, 384 kb/s
Stream #0:2[0x110b](spa): Audio: ac3 ([129][0][0][0] / 0x0081), 48000 Hz, stereo, fltp, 192 kb/s
Stream #0:3[0x110c]: Unknown: none (ETV1 / 0x31565445)
Stream #0:4[0x110d]: Unknown: none (ETV1 / 0x31565445)
Unsupported codec with id 0 for input stream 3
Unsupported codec with id 0 for input stream 4
[STREAM]
index=0
codec_name=h264
codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
profile=High
codec_type=video
codec_time_base=1001/120000
codec_tag_string=[27][0][0][0]
codec_tag=0x001b
width=1280
height=720
coded_width=1280
coded_height=720
has_b_frames=2
sample_aspect_ratio=1:1
display_aspect_ratio=16:9
pix_fmt=yuv420p
level=32
color_range=N/A
color_space=unknown
color_transfer=unknown
color_primaries=unknown
chroma_location=left
field_order=progressive
timecode=N/A
refs=1
is_avc=false
nal_length_size=0
id=0x1109
r_frame_rate=60000/1001
avg_frame_rate=60000/1001
time_base=1/90000
start_pts=7374827657
start_time=81942.529522
duration_ts=210447988
duration=2338.310978
bit_rate=N/A
max_bit_rate=N/A
bits_per_raw_sample=8
nb_frames=N/A
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=0
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
DISPOSITION:timed_thumbnails=0
[/STREAM]
[STREAM]
index=1
codec_name=ac3
codec_long_name=ATSC A/52A (AC-3)
profile=unknown
codec_type=audio
codec_time_base=1/48000
codec_tag_string=[129][0][0][0]
codec_tag=0x0081
sample_fmt=fltp
sample_rate=48000
channels=6
channel_layout=5.1(side)
bits_per_sample=0
dmix_mode=-1
ltrt_cmixlev=-1.000000
ltrt_surmixlev=-1.000000
loro_cmixlev=-1.000000
loro_surmixlev=-1.000000
id=0x110a
r_frame_rate=0/0
avg_frame_rate=0/0
time_base=1/90000
start_pts=7374739876
start_time=81941.554178
duration_ts=210430848
duration=2338.120533
bit_rate=384000
max_bit_rate=N/A
bits_per_raw_sample=N/A
nb_frames=N/A
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=0
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
DISPOSITION:timed_thumbnails=0
TAG:language=eng
[/STREAM]
[STREAM]
index=2
codec_name=ac3
codec_long_name=ATSC A/52A (AC-3)
profile=unknown
codec_type=audio
codec_time_base=1/48000
codec_tag_string=[129][0][0][0]
codec_tag=0x0081
sample_fmt=fltp
sample_rate=48000
channels=2
channel_layout=stereo
bits_per_sample=0
dmix_mode=-1
ltrt_cmixlev=-1.000000
ltrt_surmixlev=-1.000000
loro_cmixlev=-1.000000
loro_surmixlev=-1.000000
id=0x110b
r_frame_rate=0/0
avg_frame_rate=0/0
time_base=1/90000
start_pts=7374747034
start_time=81941.633711
duration_ts=210430842
duration=2338.120467
bit_rate=192000
max_bit_rate=N/A
bits_per_raw_sample=N/A
nb_frames=N/A
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=0
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
DISPOSITION:timed_thumbnails=0
TAG:language=spa
[/STREAM]
[STREAM]
index=3
codec_name=unknown
codec_long_name=unknown
profile=unknown
codec_type=unknown
codec_tag_string=ETV1
codec_tag=0x31565445
id=0x110c
r_frame_rate=0/0
avg_frame_rate=0/0
time_base=1/90000
start_pts=7374739876
start_time=81941.554178
duration_ts=210535769
duration=2339.286322
bit_rate=N/A
max_bit_rate=N/A
bits_per_raw_sample=N/A
nb_frames=N/A
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=0
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
DISPOSITION:timed_thumbnails=0
[/STREAM]
[STREAM]
index=4
codec_name=unknown
codec_long_name=unknown
profile=unknown
codec_type=unknown
codec_tag_string=ETV1
codec_tag=0x31565445
id=0x110d
r_frame_rate=0/0
avg_frame_rate=0/0
time_base=1/90000
start_pts=7374739876
start_time=81941.554178
duration_ts=210535769
duration=2339.286322
bit_rate=N/A
max_bit_rate=N/A
bits_per_raw_sample=N/A
nb_frames=N/A
nb_read_frames=N/A
nb_read_packets=N/A
DISPOSITION:default=0
DISPOSITION:dub=0
DISPOSITION:original=0
DISPOSITION:comment=0
DISPOSITION:lyrics=0
DISPOSITION:karaoke=0
DISPOSITION:forced=0
DISPOSITION:hearing_impaired=0
DISPOSITION:visual_impaired=0
DISPOSITION:clean_effects=0
DISPOSITION:attached_pic=0
DISPOSITION:timed_thumbnails=0
[/STREAM]

mr_tea
Newcomer
Posts: 13
Joined: Tue Apr 11, 2017 2:47 am
United States of America

Re: Anyone else having problems with commercial removal in 0

Post by mr_tea » Fri Apr 21, 2017 6:51 pm

You can see stream 3 and 4 are the ones that can't be recognized. The decode_slice_header errors all appear to be coming from those streams as they don't appear when processing a file with them stripped out. So, I'm not particularly concerned about them.

daraden
Senior
Posts: 169
Joined: Tue Feb 23, 2016 7:33 am
United States of America

Re: Anyone else having problems with commercial removal in 0

Post by daraden » Sat Apr 22, 2017 3:24 pm

IIRC the error could be the stream not starting with a key-frame.

I've been working on updating the script. Most significant changes are automatic commercial flagging if using the use comm_flag option, automatic removal of any empty audio streams, and new encoding code I back ported from python 3. Most of the new encoding code is for detection of A/V streams and better status output(still need to implement job status updates). only known issue so far is minor rounding issue with encoding status output.

Still working on finalizing new cut-list setup to test if using time-stamps is more effective for h264.

Code: Select all

#!/usr/bin/env python2
# -*- coding: UTF-8 -*-
#MythTV commercial removal -- by Daraden
#usage userjob = /path to script/script.py --jobid %JOBID%
#/path to script/script.py --chanid --starttime can also be used
from __future__ import print_function
from MythTV import (Job, Program, Recorded, System, MythDB, findfile,
                    MythError, MythLog, datetime, VideoGrabber)
import argparse
import shutil
from glob import glob
import time
from datetime import timedelta
import subprocess
import ConfigParser
import logging
import logging.handlers
import sys
import os
import tempfile
# set this to True to use commflag results as cut-list
# this will automaticly flag commercials if no cut-list is available
use_commflag = True
# save copy of original file as file.old if not using output file
save_old = False
# user needs write access to this directory to save logfile
logdir ='/media/windows/test/'
LOG_FILENAME = '{}cut.log'.format(logdir)
# logging setup for file and console
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
lf= logging.Formatter('%(asctime)s:%(levelname)s:%(message)s')
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(lf)
logger.addHandler(ch)
try:
    if os.access(logdir,os.W_OK):
        fh = logging.handlers.TimedRotatingFileHandler(filename=LOG_FILENAME,
                                                       when='W0',interval=1,
                                                       backupCount=10
                                                       )
        fh.setLevel(logging.DEBUG)
        fh.setFormatter(lf)
        logger.addHandler(fh)
    if not os.access(logdir,os.W_OK):
        logging.error('logdir not accessible:%s',logdir)
except exception as e:
    logging.error('logdir not accessible:%s',logdir)
    logging.error(e)
    sys.exit(1)

try:
    db=MythDB()
except exception as e:
    logging.error(e)
    sys.exit(1)

def prog_check():
    """checks for FFmpeg/FFprobe if not found MythFFmpeg/MythFFprobe
    is used"""
    from distutils import spawn

    if spawn.find_executable('ffmpeg') != None:
        ffmpeg = spawn.find_executable('ffmpeg')
    elif (spawn.find_executable('ffmpeg') == None and
          spawn.find_executable('mythffmpeg')):
        ffmpeg = spawn.find_executable('mythffmpeg')
    else:
        ffmpeg = None
    if spawn.find_executable('ffprobe') != None:
        ffprobe = spawn.find_executable('ffprobe')
    elif (spawn.find_executable('ffprobe') == None and
          spawn.find_executable('mythffprobe')):
        ffprobe = spawn.find_executable('mythffprobe')
    else:
        ffprobe = None
    if ffmpeg != None and ffprobe != None:
        return ffmpeg,ffprobe
    else:
        raise LookupError('Unable to find FFmpeg or MythFFmpeg')

#check for ffmpeg or mythffmpeg
ffmpeg = prog_check()[0]
logging.debug('ffmpeg={}'.format(ffmpeg))
ffprobe = prog_check()[1]
logging.debug('ffprobe={}'.format(ffprobe))

def run_cut(jobid=None,chanid=None,starttime=None,outfile=None):
    logging.info('Started')

    # Configure chanid and starttime from userjob input
    if jobid:
        job = Job(jobid, db=db)
        chanid = job.chanid
        starttime = job.starttime
        logging.debug('chanid={} starttime={}'.format(chanid, starttime))
    if not jobid:
        chanid = chanid
        starttime = starttime
        logging.debug('chanid={} starttime={}'.format(chanid, starttime))
    # Get database recording entry
    rec = find_rec(chanid,starttime)
    logging.debug('DB recording entry={}'.format(rec))
    # Find and format full input file path
    sg = findfile('/{}'.format(rec.basename), rec.storagegroup, db=db)
    infile = os.path.join(sg.dirname, rec.basename)
    # Assign and create temporary directory in recording directory
    tmpdir = '{}{}{}/'.format(sg.dirname, 'crtmp.',
                              rec.basename.split('.')[0]
                              )
    logging.debug('tmpdir is: {}'.format(tmpdir))
    tmp_chk(tmpdir)
    # Get info from input file
    avinfo = AVInfo(ffprobe,infile)
    # Get video frame-rate
    rfr = list(int(x) for x in avinfo.video.r_frame_rate.split('/'))
    frame_rate = rfr[0] / float(rfr[1])
    logging.info('Video frame-rate is: {}'.format(frame_rate))

    if save_old == True and outfile == None:
        logging.info('Backing up original file: %s',infile)
        shutil.copyfile(infile,'%s.old' % infile)
        logging.info('Finished backing up original file: {}'.format(infile))
    # Configure output file
    if outfile != None:
        Outfile = outfile
    elif outfile == None:
        Outfile = infile
    if rec.cutlist == 1 or use_commflag == True:
        ##Flag commercials##
        if use_commflag == True and rec.commflagged != 1 and rec.cutlist != 1:
            try:
                logging.info('Flagging commercials')
                startts = datetime.mythformat(rec.starttime)
                rst = subprocess.Popen(['mythcommflag','--chanid',
                                        str(chanid),'--starttime',
                                        str(startts)],
                                       stdout = subprocess.PIPE,
                                       stderr = subprocess.STDOUT)
                output2 = rst.communicate()[0]
                if rst.wait() == 0:
                    logging.info('Commercials flagged for:%s_%s',
                                 chanid,starttime)
                if rst.wait() != 0:
                    logging.error('MYTHcommflag ERROR: Flagging '
                                  'commercials for:{}_{}'
                                  .format(chanid,starttime)
                                  )
                    logging.error('MythCommflag allways exits with '
                                  'decoding error?!'
                                  )
                    # uncoment folowing line to log mythcommflag errors
                    #logging.error('{}'.format(output2))
            except Exception as e:
                logging.error('Mythcommflag ERROR: Flagging commercials %s',e)
            # Reinitalize recording object
            rec = find_rec(chanid,starttime)
        if jobid:
            job.update(
                {'status':job.RUNNING, 'comment':'Removing Cutlist'})
        logging.info('getting cutlist')
        # Retrieve cutlist from database
        print(type(rec.cutlist))
        if rec.cutlist == 1:
            cut_list = (','.join(
                [','.join(str(i) for i in b)
                 for b in rec.markup.getcutlist()]))
            print(cut_list)
        elif rec.cutlist != 1 and rec.commflagged == 1 and use_commflag == True:
            cut_list = (','.join([','.join(str(i) for i in b)
            for b in rec.markup.getskiplist()]))
            if cut_list == '':
                logging.error('Commflag list empty: No commercials flaged!')
                if jobid:
                    job.update(
                        {'status':job.ERRORED, 'comment':'Commflag list empty'})
                sys.exit(1)

        else:
            logging.debug('No cut/skip list found')
            sys.exit(1)

        cutlist = cut_list
        print(cutlist)
        # Format cutlist
        if cutlist.startswith('0,'):
            cutlist = cutlist.replace("0,",'',1)
        if cutlist.endswith(',9999999'):
            cutlist = cutlist.replace(",9999999",'',-1)
        logging.debug('Myth cutlist:{}'.format(cut_list))
        logging.debug('FFmpeg cutlist:{}'.format(cutlist))

        logging.info('cutting started')
        cut(ffmpeg, infile,cutlist, tmpdir)
        logging.info('Merging Cuts')
        avinfo = AVInfo(ffprobe,infile)
        join(ffmpeg, cutlist, tmpdir, Outfile, avinfo)
        rem_tmp(tmpdir)
        if  outfile == None:
            clear_markup(rec,infile,outfile)
        if jobid:
            job.update({'status':job.FINISHED, 'comment':
                        'Commercial removal Completed'
                        }
                       )
class DictToNamespace(object):
    """ convert a dictonary and any nested dictonarys to namespace"""
    def __setitem__(self, key, item):
        self.__dict__[key] = item

    def __getitem__(self, key):
        return self.__dict__[key]

    def has_key(self, k):
        return self.__dict__.has_key(k)

    def update(self, *args, **kwargs):
        return self.__dict__.update(*args, **kwargs)

    def keys(self):
        return self.__dict__.keys()

    def values(self):
        return self.__dict__.values()

    def items(self):
        return self.__dict__.items()

    def __contains__(self, item):
        return item in self.__dict__

    def __iter__(self):
        return iter(self.__dict__)

    def __init__(self, data):
        for item,num in data.items():
            if isinstance(num, str):
                try:
                    num = float(num)
                    if float(num).is_integer():
                        num = int(num)
                    data.update({item: num})
                except ValueError:
                    pass

        self.__dict__.update(data)
        for k, v in data.items():
            if isinstance(v, dict):
                self.__dict__[k] = DictToNamespace(v)

class AVInfo:
    """identify A/V configuration of input file and returns
    self.video dict a list of self.audio.stream dicts,
    and self.duration as float"""

    def __init__(self=None,ffprobe=None,infile=None):
        class Dict(dict):
            def __getattr__(self, name):
                return self[name]
            def __setattr__(self, name, value):
                self[name] = value

        command = [ffprobe, '-v', '-8', '-show_entries',
                   'stream=codec_type,index,codec_name,channels,width,'
                   'height,r_frame_rate:stream_tags=language:'
                   'format=duration', '-of', 'csv=nk=0:p=0', infile
                   ]
        x = subprocess.check_output(command).decode('utf-8').split('\n')

        vcd = {}
        adict = {}
        for vc in x:
            if 'codec_type=video' in vc:
                for item in vc.split(','):
                    k,v = item.split('=')
                    vcd.update({k:v})
        for ac in x:
            if 'codec_type=audio' in ac:
                items =[item.split('=') for item in ac.split(',')]
                streamdict ={k.strip(): v.strip() for (k,v) in items}

                if 'tag:language' in streamdict.keys():
                    streamdict['language'] = (streamdict.pop
                                              ('tag:language')
                                              )
                adict.update({'stream{}'.format(streamdict['index'])
                              : streamdict})
        for d in x:
            if d.startswith('duration='):
                dur = d.split('=')[1]

        self.video = Dict(vcd)
        self.audio = DictToNamespace(adict)
        self.duration = dur

def cut(ffmpeg, infile,cutlist, tmpdir):
    """use ffmpeg segment muxer to cut infile into multipule files
    using a csv formated cutlist placing them into tmpdir"""
    cmd = [ffmpeg, '-ignore_unknown', '-i', infile, '-c', 'copy',
           '-map', '0'
           ]
    av_check = AVInfo(ffprobe,infile)

    for streams,stream in av_check.audio.items():
        if stream.channels == '0':
            cmd.extend(('-map',
                        ('-0:{}'
                         .format(stream.index))))

    cmd.extend(('-f','segment',
                '-segment_frames', cutlist,'-break_non_keyframes', '1',
                '{}cut%03d.ts'.format(tmpdir)
                )
               )
    encode(cmd,av_check,ffmpeg)
def encode(command,AVInfo,ffmpeg):
    """ Run ffmpeg command with status output"""
    # Length of progress bar
    statlen = 10
    # Charicter used for progress bar
    # Use chr() in python 3
    statchar = unichr(9619).encode('UTF-8')
    # Charicter used to pad progress bar
    pad = ' '

    rfr = list(int(x) for x in AVInfo.video.r_frame_rate.split('/'))
    frame_rate = float(rfr[0]) / float(rfr[1])
    duration = float(AVInfo.duration)
    total_frames = duration * frame_rate

    with tempfile.TemporaryFile() as output:
        process = subprocess.Popen(command, stdout=output,
                                   stderr =output,
                                   universal_newlines=True
                                   )

        while True:
            if process.poll() is not None:
                if process.poll() != 0:
                    output.seek(0)
                    print(output.read().decode('UTF-8'))
                    sys.exit(1)
                if process.poll() == 0:
                    print('\rFinished{}'.format(pad * (statlen + 3)))
                    break
            where = output.tell()
            lines = output.read().decode('UTF-8')
            if not lines:
                time.sleep(0.1)
                output.seek(where)
            elif lines.startswith('frame='):
                ln = ' '.join(lines.split()).replace('= ','=').split(' ')
                for item in ln:
                    if item.startswith('frame='):
                        framenum = int(item.replace('frame=',''))
                    if item.startswith('fps='):
                        fps = float(item.replace('fps=',''))
                if int(framenum) == 0:
                    pcomp = 0
                else:
                    # python 2 div
                    pcomp = 100 * (float(framenum) / float(total_frames))
                    #python 3 div
                    #pcomp = 100 * (framenum / total_frames)
                # python 2 div
                stat = int((float(pcomp) / float(100)) * statlen)
                # python 3 div
                #stat = int((int(pcomp) / 100) * statlen)
                padlen = statlen - stat
                status = "|{:6.2f}%|".format(pcomp)
                statusbar = '|{}{}|'.format(statchar * stat, pad * padlen)
                status = '\r{}{}'.format(status,statusbar)
                print(status, end="")
                # Replace with flush=True in print function for python 3
                sys.stdout.flush()

def join(ffmpeg, cutlist, tmpdir, tmpfile,avinfo):
    """ use ffmpeg to join video segments in tempdir into single tempfile.
    using the first value of the cutlist to determine the segments.
    avinfo object from source file used to determine frame-rate for
    status output."""
    j = []
    durlist= []
    # Length of progress bar
    statlen = 10
    # Charicter used for progress bar
    # Use chr() in python 3
    statchar = unichr(9619).encode('UTF-8')
    # Charicter used to pad progress bar
    pad = ' '
    # Get list of segment files
    for root, dirs, files in os.walk(tmpdir):
        for files in files:
            if (files.endswith('.ts') and
                files.startswith('cut')):
                if os.path.isfile(os.path.join(root,files)):
                    j.append(os.path.join(root,files))
    # Create concat string and file list
    if cutlist.startswith('0'):
        filelist = j[0::2]
        concatlist = ','.join(j[0::2]).replace(',','|')
    if not cutlist.startswith('0'):
        filelist = j[1::2]
        concatlist = ','.join(j[1::2]).replace(',','|')
    # Get duration of file segments for output
    for files in filelist:
        durlist.append(float(AVInfo(ffprobe,files).duration.strip('\r')))
    duration = sum(durlist)
    # format refrence framerate to float
    rfr = list(int(x) for x in avinfo.video.r_frame_rate.split('/'))
    frame_rate = rfr[0] / rfr[1]
    # estimate total number of frames
    total_frames = duration * frame_rate

    command = [ffmpeg, '-y', '-i', 'concat:{}'.format(concatlist),
               '-map', '0', '-c', 'copy', '-f', 'mpegts', tmpfile
               ]

    with tempfile.TemporaryFile() as output:
        process = subprocess.Popen(command, stdout=output,
                                   stderr =output,
                                   universal_newlines=True
                                   )

        while True:
            if process.poll() is not None:
                if process.poll() != 0:
                    output.seek(0)
                    print(output.read().decode('UTF-8'))
                    sys.exit(1)
                if process.poll() == 0:
                    if os.path.isfile(tmpfile):
                        for file in j:
                            os.remove(file)
                    print('\rFinished{}'.format(pad * (statlen + 3)))
                    break
            where = output.tell()
            lines = output.read().decode('UTF-8')
            if not lines:
                time.sleep(0.1)
                output.seek(where)
            elif lines.startswith('frame='):
                ln = ' '.join(lines.split()).replace('= ','=').split(' ')
                for item in ln:
                    if item.startswith('frame='):
                        framenum = int(item.replace('frame=',''))
                    if item.startswith('fps='):
                        fps = float(item.replace('fps=',''))
                if int(framenum) == 0:
                    pcomp = 0
                else:
                    # python 2 div
                    pcomp = 100 * (float(framenum) / float(total_frames))
                    # python 3 div
                    #pcomp = 100 * (framenum / total_frames)
                # python 2 div
                stat = int((float(pcomp) / float(100)) * statlen)
                # python 3 div
                #stat = int((int(pcomp) / 100) * statlen)
                padlen = statlen - stat
                status = "|{:6.2f}%|".format(pcomp)
                statusbar = '|{}{}|'.format(statchar * stat, pad * padlen)
                status = '\r{}{}'.format(status,statusbar)

                print(status, end="")
                # Replace with flush=True in print function for python 3
                sys.stdout.flush()

def clear_markup(rec,infile,outfile):
    logging.info('Started Clearing markup')

    logging.debug('rec=%s infile=%s outfile=%s',rec,infile,outfile)
    chanid = rec.chanid
    utcstarttime = rec.starttime
    starttime = str(utcstarttime.utcisoformat().replace(u':', '').replace(u' ', '').replace(u'T', '').replace('-', ''))
    logging.debug('chanid=%s starttime=%s',chanid,starttime)
    logging.info('start clearing markup')
    try:
        rcl = subprocess.Popen(['mythutil','--chanid',str(chanid),'--starttime',str(starttime),'--clearcutlist'], stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
        output1 = rcl.communicate()
        if rcl.wait() == 0:
            logging.info('Cutlist removed for:%s_%s',chanid,starttime)
        if rcl.wait() != 0:
            logging.error('MYTHUTIL ERROR: clearing cutlist for:%s_%s',chanid,starttime)
            logging.error('%s',output1)
    except Exception as e:
        logging.error('Mythutil exception clearing cutlist%s',e)
    try:
        rsl = subprocess.Popen(['mythutil','--chanid',str(chanid),'--starttime',str(starttime),'--clearskiplist'], stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
        output2 = rsl.communicate()[0]
        if rsl.wait() == 0:
            logging.info('Skiplist removed for:%s_%s',chanid,starttime)
        if rsl.wait() != 0:
            logging.error('MYTHUTIL ERROR: clearing skiplist for:%s_%s',chanid,starttime)
            logging.error('%s',output2)
    except Exception as e:
        logging.error('Mythutil exception clearing skiplist:%s',e)

    for index,mark in reversed(list(enumerate(rec.markup))):
        if mark.type in (rec.markup.MARK_COMM_START, rec.markup.MARK_COMM_END):
            del rec.markup[index]
    rec.bookmark = 0
    rec.bookmarkupdate = datetime.now()
    rec.cutlist = 0
    rec.commflagged = 0
    rec.markup.commit()
    rec.basename = os.path.basename(infile)
    rec.filesize = os.path.getsize(infile)
    rec.transcoded = 1
    rec.seek.clean()
    rec.update()

    try:
        logging.info('Removing PNG files')
        for png in glob('%s*.png' % infile):
            os.remove(png)
    except Exception as e:
        logging.error('Error removing png files',e)
    try:
        logging.info('Removing JPG files')
        for jpg in glob('%s*.jpg' % infile):
            os.remove(jpg)
    except Exception as e:
        logging.error('Error removing jpg files',e)
    try:
        logging.info('Rebuilding seektable')
        rst = subprocess.Popen(['mythcommflag','--chanid',str(chanid),'--starttime',str(starttime),'--rebuild'], stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
        output2 = rst.communicate()[0]
        if rst.wait() == 0:
            logging.info('Seektable Rebuilt for:%s_%s',chanid,starttime)
        if rst.wait() != 0:
            logging.error('MYTHcommflag ERROR: Rebuilding Seektable for:%s_%s',chanid,starttime)
            logging.error('%s',output2)
    except Exception as e:
        logging.error('Mythcommflag ERROR clearing skiplist:%s',e)

def tmp_chk(tmpdir):
        try:
            if os.path.isdir(tmpdir):
                logging.info('chk:temp Folder found: %s',tmpdir)
                if os.listdir(tmpdir) != 0:
                    logging.warning('chk:Temp folder not empty!:Removing Files: %s',tmpdir)
                    shutil.rmtree(tmpdir)
                    os.makedirs(tmpdir)
            if not os.path.isdir(tmpdir):
                logging.info('chk:no temp folder found: %s',tmpdir)
                os.makedirs(tmpdir)
                logging.info('chk:Temp folder created: %s',tmpdir)
        except Exception as e:
            logging.error('%s',e)
def rem_tmp(tmpdir):
            try:
                if os.path.isdir(tmpdir):
                    logging.info('rem:temp Folder found %s',tmpdir)
                    shutil.rmtree(tmpdir)
                if not os.path.isdir(tmpdir):
                    logging.info('rem:temp Folder Removed %s',tmpdir)
            except Exception as e:
                logging.error('%s',e)
def find_rec(chanid,starttime):
    def local_time_offset(t=None):
        if t is None:
            t = time.time()

        if time.localtime(t).tm_isdst and time.daylight:
            return -time.altzone
        else:
            return -time.timezone

    def RecordedFromBasename(chanid,starttime):
        bnts = '%s_%s.ts' % (chanid,starttime)
        bnmpg = '%s_%s.mpg' % (chanid,starttime)

        x = list(db.searchRecorded(basename=bnmpg))
        if len(x) == 1:
            for recorded in x:
                return recorded

        if len(x) != 1:
            x = list(db.searchRecorded(basename=bnts))
            if len(x) == 1:
                for recorded in x:
                    return recorded
            if len(x) != 1:
                raise LookupError('unable to find Recorded entry for '
                                  'ChanID {} StartTime {}'
                                  .format(chanid, starttime)
                                  )
    try:
        rec = Recorded((chanid,starttime), db=db)
    except:
        try:
            tzoffset = local_time_offset() / (60*60)
            utcstarttime = datetime.strptime(starttime,"%Y%m%d%H%M%S")
            utcstarttime = utcstarttime + timedelta(hours=tzoffset)
            rec = Recorded((chanid,utcstarttime), db=db)
        except:
            rec = RecordedFromBasename(chanid,starttime)
    return rec

def main():
    parser = argparse.ArgumentParser(description='MythTV Commercial removal and closed caption extraction tool.\nNOTE:Having .srt file in same dir as media files breaks playback in some media players(VLC)')
    parser.add_argument('--chanid',action='store',type=str,dest='chanid',help='Channel-Id of Recording')
    parser.add_argument('--starttime',action='store',type=str,dest='starttime',help='Starttime of recording in utc format')
    parser.add_argument('--jobid',action='store',type=int,dest='jobid',help='JOBID')
    parser.add_argument('-o',action='store',type=str,dest='outfile',help='Output file to be created')
    args = parser.parse_args()
    if args.jobid:
        run_cut(jobid=args.jobid,outfile=args.outfile)
        sys.exit(0)
    if args.chanid and args.starttime:
        run_cut(chanid=args.chanid,starttime=args.starttime,outfile=args.outfile)
        sys.exit(0)
    else:
        print('chanid and starttime or jobid required')
main()

mr_tea
Newcomer
Posts: 13
Joined: Tue Apr 11, 2017 2:47 am
United States of America

Re: Anyone else having problems with commercial removal in 0

Post by mr_tea » Sat Apr 22, 2017 10:42 pm

Glad to hear your still digging in. A quick test with H264 has issues with the cuts still being early and not on keyframe. Let me know when you have a version based on time, and I can run that test for you.

daraden
Senior
Posts: 169
Joined: Tue Feb 23, 2016 7:33 am
United States of America

Re: Anyone else having problems with commercial removal in 0

Post by daraden » Sun Apr 23, 2017 5:27 pm

I spent most of yesterday getting everything almost finished. Rebuilt a lot of the encoding code, completely overhauled the cut-list code, added more job status updates, and changed the console output to info only(log-file still includes debug output). Still need to setup status updates for the encoding process.

Code: Select all

#!/usr/bin/env python2
# -*- coding: UTF-8 -*-
#MythTV commercial removal -- by Daraden
#usage userjob = /path to script/script.py --jobid %JOBID%
#/path to script/script.py --chanid --starttime can also be used
from __future__ import print_function
from MythTV import (Job, Program, Recorded, System, MythDB, findfile,
                    MythError, MythLog, datetime, VideoGrabber)
import argparse
import shutil
from glob import glob
import time
from datetime import timedelta
import subprocess
import ConfigParser
import logging
import logging.handlers
import sys
import os
import tempfile
# set this to True to use commflag results as cut-list
# this will automaticly flag commercials if no cut-list is available
use_commflag = True
# save copy of original file as file.old if not using output file
save_old = False
# user needs write access to this directory to save logfile
logdir ='/media/windows/test/'
LOG_FILENAME = '{}cut.log'.format(logdir)
# logging setup for file and console
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
lf= logging.Formatter('%(asctime)s:%(levelname)s:%(message)s')
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
ch.setFormatter(lf)
logger.addHandler(ch)
try:
    if os.access(logdir,os.W_OK):
        fh = logging.handlers.TimedRotatingFileHandler(filename=LOG_FILENAME,
                                                       when='W0',interval=1,
                                                       backupCount=10
                                                       )
        fh.setLevel(logging.DEBUG)
        fh.setFormatter(lf)
        logger.addHandler(fh)
    if not os.access(logdir,os.W_OK):
        logging.error('logdir not accessible:%s',logdir)
except exception as e:
    logging.error('logdir not accessible:%s',logdir)
    logging.error(e)
    sys.exit(1)

try:
    db=MythDB()
except exception as e:
    logging.error(e)
    sys.exit(1)

def prog_check():
    """checks for FFmpeg/FFprobe if not found MythFFmpeg/MythFFprobe
    is used"""
    from distutils import spawn

    if spawn.find_executable('ffmpeg') != None:
        ffmpeg = spawn.find_executable('ffmpeg')
    elif (spawn.find_executable('ffmpeg') == None and
          spawn.find_executable('mythffmpeg')):
        ffmpeg = spawn.find_executable('mythffmpeg')
    else:
        ffmpeg = None
    if spawn.find_executable('ffprobe') != None:
        ffprobe = spawn.find_executable('ffprobe')
    elif (spawn.find_executable('ffprobe') == None and
          spawn.find_executable('mythffprobe')):
        ffprobe = spawn.find_executable('mythffprobe')
    else:
        ffprobe = None
    if ffmpeg != None and ffprobe != None:
        return ffmpeg,ffprobe
    else:
        raise LookupError('Unable to find FFmpeg or MythFFmpeg')

#check for ffmpeg or mythffmpeg
ffmpeg = prog_check()[0]
logging.debug('ffmpeg={}'.format(ffmpeg))
ffprobe = prog_check()[1]
logging.debug('ffprobe={}'.format(ffprobe))

def run_cut(jobid=None,chanid=None,starttime=None,outfile=None):
    logging.info('Started')

    # Configure chanid and starttime from userjob input
    if jobid:
        job = Job(jobid, db=db)
        chanid = job.chanid
        starttime = job.starttime
        logging.debug('chanid={} starttime={}'.format(chanid, starttime))
    if not jobid:
        chanid = chanid
        starttime = starttime
        logging.debug('chanid={} starttime={}'.format(chanid, starttime))
    # Get database recording entry
    rec = find_rec(chanid,starttime)
    logging.debug('DB recording entry={}'.format(rec))
    # Find and format full input file path
    sg = findfile('/{}'.format(rec.basename), rec.storagegroup, db=db)
    infile = os.path.join(sg.dirname, rec.basename)
    # Assign and create temporary directory in recording directory
    tmpdir = '{}{}{}/'.format(sg.dirname, 'crtmp.',
                              rec.basename.split('.')[0]
                              )
    logging.debug('tmpdir is: {}'.format(tmpdir))
    logging.info('Creating temporary directory')
    tmp_chk(tmpdir)
    # Get info from input file
    avinfo = AVInfo(ffprobe,infile)
    # Get video frame-rate
    rfr = list(int(x) for x in avinfo.video.r_frame_rate.split('/'))
    frame_rate = rfr[0] / float(rfr[1])
    logging.debug('Video frame-rate is: {}'.format(frame_rate))

    if save_old == True and outfile == None:
        if jobid:
            job.update({'status':job.RUNNING,
                        'comment':'Backing up original file'
                        }
                       )

        logging.info('Backing up original file: %s',infile)
        shutil.copyfile(infile,'%s.old' % infile)
        logging.info('Finished backing up original file: {}'.format(infile))
    # Configure output file
    if outfile != None:
        Outfile = outfile
    elif outfile == None:
        Outfile = infile

    if rec.cutlist == 1 or use_commflag == True:
        ##Flag commercials##
        if (use_commflag == True and rec.commflagged != 1
            and rec.cutlist != 1
            ):
            # need check for running commflag job?
            try:
                logging.info('Flagging commercials')
                if jobid:
                    job.update({'status':job.RUNNING,
                                'comment': 'Flagging commercials'
                                }
                               )

                startts = datetime.mythformat(rec.starttime)
                rst = subprocess.Popen(['mythcommflag','--chanid',
                                        str(chanid),'--starttime',
                                        str(startts)],
                                       stdout = subprocess.PIPE,
                                       stderr = subprocess.STDOUT)
                output2 = rst.communicate()[0]
                if rst.wait() == 0:
                    logging.info('Commercials flagged for:%s_%s',
                                 chanid,starttime)
                if rst.wait() != 0:
                    logging.error('MYTHcommflag ERROR: Flagging '
                                  'commercials for:{}_{}'
                                  .format(chanid,starttime)
                                  )
                    logging.error('MythCommflag allways exits with '
                                  'decoding error?!'
                                  )
                    # uncoment folowing line to log mythcommflag errors
                    #logging.error('{}'.format(output2))
            except Exception as e:
                logging.error('Mythcommflag ERROR: Flagging commercials %s',e)
            # Reinitalize recording object
            rec = find_rec(chanid,starttime)
        if jobid:
            job.update({'status':job.RUNNING,
                        'comment':'Getting Cutlist'
                        }
                       )
        logging.info('getting cutlist')
        # Get cut/skip list from database
        if rec.cutlist == 1:
            myth_cut_list = rec.markup.getcutlist()
        if rec.cutlist != 1 and rec.commflagged == 1 and use_commflag:
            myth_cut_list = rec.markup.getskiplist()
        elif rec.cutlist == 0 and rec.commflagged != 1 and not use_commflag:
            logging.debug('No cut/skip list found')
            sys.exit(1)
        logging.debug('Myth cut_list: {}'.format(myth_cut_list))
        # Format cutlist, removing starting 0 and ending 9999999
        cut_list = [mark for cuts in myth_cut_list for mark in cuts]
        if cut_list[0] == 0:
            cut_list.pop(0)
        if cut_list[-1] == 9999999:
            cut_list.pop(-1)
        logging.debug('Cut list: {}'.format(cut_list))
        # Get the name of the video codec
        video_codec = avinfo.video.codec_name
        logging.info('video codec: {}'.format(video_codec))
        # Set cutlist string
        if video_codec == 'h264':
            cut_string = ','.join(str(i / frame_rate) for i in cut_list)
        elif video_codec == 'mpeg2video':
            cut_string = ','.join(str(i) for i in cut_list)
        logging.debug('cut_string: {}'.format(cut_string))
        # Create segment files
        logging.info('Cutting')
        if jobid:
            job.update({'status':job.RUNNING,
                        'comment':'Cutting started'
                        }
                       )
        cut(ffmpeg, infile, cut_string, tmpdir)
        # Create list of files to join
        file_list = []
        # Get list of segment files
        for root, dirs, files in os.walk(tmpdir):
            for File in files:
                if (File.endswith('.ts') and
                    File.startswith('cut')):
                    if os.path.isfile(os.path.join(root,File)):
                        file_list.append(os.path.join(root,File))
        # Set list of files to be joined
        if myth_cut_list[0][0] == 0:
            join_list = file_list[1::2]
        if myth_cut_list[0][0] != 0:
            join_list = file_list[0::2]
        logging.debug('Join file list: {}'.format(join_list))
        # Join segment files
        logging.info('Joining segments')
        if jobid:
            job.update({'status':job.RUNNING,
                        'comment':'Joining segments'
                        }
                       )

        join(ffmpeg, join_list, outfile)
        # Remove temporary directory
        logging.info('removing temporary directory')
        rem_tmp(tmpdir)
        
        if  outfile == None:
            logging.info('clearing database markup')
            if jobid:
                job.update({'status':job.RUNNING,
                            'comment':'Clearing database markup'
                            }
                           )

            clear_markup(rec,infile,outfile)
        if jobid:
            job.update({'status':job.FINISHED, 'comment':
                        'Commercial removal finished'
                        }
                       )
        logging.info('Finished')
class DictToNamespace(object):
    """ convert a dictonary and any nested dictonarys to namespace"""
    def __setitem__(self, key, item):
        self.__dict__[key] = item

    def __getitem__(self, key):
        return self.__dict__[key]

    def has_key(self, k):
        return self.__dict__.has_key(k)

    def update(self, *args, **kwargs):
        return self.__dict__.update(*args, **kwargs)

    def keys(self):
        return self.__dict__.keys()

    def values(self):
        return self.__dict__.values()

    def items(self):
        return self.__dict__.items()

    def __contains__(self, item):
        return item in self.__dict__

    def __iter__(self):
        return iter(self.__dict__)

    def __init__(self, data):
        for item,num in data.items():
            if isinstance(num, str):
                try:
                    num = float(num)
                    if float(num).is_integer():
                        num = int(num)
                    data.update({item: num})
                except ValueError:
                    pass

        self.__dict__.update(data)
        for k, v in data.items():
            if isinstance(v, dict):
                self.__dict__[k] = DictToNamespace(v)

class AVInfo:
    """identify A/V configuration of input file and returns
    self.video dict a list of self.audio.stream dicts,
    and self.duration as float"""

    def __init__(self=None,ffprobe=None,infile=None):
        class Dict(dict):
            def __getattr__(self, name):
                return self[name]
            def __setattr__(self, name, value):
                self[name] = value

        command = [ffprobe, '-v', '-8', '-show_entries',
                   'stream=codec_type,index,codec_name,channels,width,'
                   'height,r_frame_rate:stream_tags=language:'
                   'format=duration', '-of', 'csv=nk=0:p=0', infile
                   ]
        x = subprocess.check_output(command).decode('utf-8').split('\n')

        vcd = {}
        adict = {}
        for vc in x:
            if 'codec_type=video' in vc:
                for item in vc.split(','):
                    k,v = item.split('=')
                    vcd.update({k:v})
        for ac in x:
            if 'codec_type=audio' in ac:
                items =[item.split('=') for item in ac.split(',')]
                streamdict ={k.strip(): v.strip() for (k,v) in items}

                if 'tag:language' in streamdict.keys():
                    streamdict['language'] = (streamdict.pop
                                              ('tag:language')
                                              )
                adict.update({'stream{}'.format(streamdict['index'])
                              : streamdict})
        for d in x:
            if d.startswith('duration='):
                dur = d.split('=')[1]

        self.video = Dict(vcd)
        self.audio = DictToNamespace(adict)
        self.duration = dur

def encode(ffmpeg, command,AVInfo):
    """ Run ffmpeg command with status output"""
    # Length of progress bar
    statlen = 10
    # Charicter used for progress bar
    # Use chr() in python 3
    statchar = unichr(9619).encode('UTF-8')
    # Charicter used to pad progress bar
    pad = ' '

    rfr = list(int(x) for x in AVInfo.video.r_frame_rate.split('/'))
    frame_rate = float(rfr[0]) / float(rfr[1])
    duration = float(AVInfo.duration)
    total_frames = duration * frame_rate

    with tempfile.TemporaryFile() as output:
        process = subprocess.Popen(command, stdout=output,
                                   stderr =output,
                                   universal_newlines=True
                                   )

        while True:
            if process.poll() is not None:
                if process.poll() != 0:
                    output.seek(0)
                    print(output.read().decode('UTF-8'))
                    sys.exit(1)
                if process.poll() == 0:
                    print('\rFinished{}'.format(pad * (statlen + 3)))
                    break
            where = output.tell()
            lines = output.read().decode('UTF-8')
            if not lines:
                time.sleep(0.1)
                output.seek(where)
            elif lines.startswith('frame='):
                ln = ' '.join(lines.split()).replace('= ','=').split(' ')
                for item in ln:
                    if item.startswith('frame='):
                        framenum = int(item.replace('frame=',''))
                    if item.startswith('fps='):
                        fps = float(item.replace('fps=',''))
                if int(framenum) == 0:
                    pcomp = 0
                else:
                    # python 2 div
                    pcomp = 100 * (float(framenum) / float(total_frames))
                    #python 3 div
                    #pcomp = 100 * (framenum / total_frames)
                # python 2 div
                stat = int((float(pcomp) / float(100)) * statlen)
                # python 3 div
                #stat = int((int(pcomp) / 100) * statlen)
                padlen = statlen - stat
                status = "|{:6.2f}%|".format(pcomp)
                statusbar = '|{}{}|'.format(statchar * stat, pad * padlen)
                status = '\r{}{}'.format(status,statusbar)
                print(status, end="")
                # Replace with flush=True in print function for python 3
                sys.stdout.flush()

def cut(ffmpeg, infile, cutlist, tmpdir):
    """use ffmpeg segment muxer to cut infile into multipule files
    using a csv formated cutlist placing them into tmpdir"""
    cmd = [ffmpeg, '-ignore_unknown', '-i', infile, '-c', 'copy',
           '-map', '0'
           ]
    av_check = AVInfo(ffprobe,infile)

    for streams,stream in av_check.audio.items():
        if stream.channels == '0':
            cmd.extend(('-map',
                        ('-0:{}'
                         .format(stream.index))))

    if av_check.video.codec_name == 'mpeg2video':
        cmd.extend(('-f','segment',
                    '-segment_frames', cutlist,
                    '{}cut%03d.ts'.format(tmpdir)
                    )
                   )
    if av_check.video.codec_name == 'h264':
        cmd.extend(('-f','segment',
                    '-segment_times', cutlist,
                    '{}cut%03d.ts'.format(tmpdir)
                    )
                   )

    encode(ffmpeg, cmd, av_check)

def join(ffmpeg, file_list, outfile):
    # Create FFmpeg concat string
    concat_string = ','.join(file_list).replace(',','|')
    duration_list = []
    rfr_list = []
    video_codec_list = []
    for files in file_list:
        file_avinfo = AVInfo(ffprobe,files)
        # get file duration
        duration_list.append(float(file_avinfo.duration.strip('\r')))
        # Get file refrence frame rate
        rfr_list.append(file_avinfo.video.r_frame_rate.strip('\r'))
        # Get file video codec
        video_codec_list.append(file_avinfo.video.codec_name.strip('\r'))
    # Duration of joined files
    duration = sum(duration_list)
    # Check that all files refrence framerate are equal
    rfr_match = all(rfr_list[0] == item for item in rfr_list)
    if len(rfr_list) > 0 and rfr_match:
        # Convert refrence frame rate to float
        rfr = rfr_list[0].split('/')
        frame_rate = int(rfr[0]) * float(rfr[1])
    elif len(rfr_list) == 0 or not rfr_match:
        raise ValueError('Incorrect or missing refrence frame rate')
    # Estimate total number of frames
    total_frames = duration * frame_rate
    #check if all files video codec match
    codec_match = all(video_codec_list[0] == item
                      for item in video_codec_list
                      )
    if codec_match:
        video_codec = video_codec_list[0]
    if not codec_match:
        raise ValueError('Not all video codecs match')
    
    command = [ffmpeg, '-y', '-i', 'concat:{}'.format(concat_string),
               '-map', '0', '-c', 'copy', '-f', 'mpegts', outfile
               ]

    class         """ status update Replacement object for AVInfo  provides
        self.duration and self.video.r_frame_rate for encode()"""
        def __init__(self, duration, rfr):
            class Dict(dict):
                def __getattr__(self, name):
                     return self[name]
                def __setattr__(self, name, value):
                    self[name] = value
            
            self.duration = duration
            self.video = Dict({'r_frame_rate': rfr})

    join_info = AVJoin(duration,rfr_list[0])
    encode(ffmpeg, command, join_info)

def clear_markup(rec,infile,outfile):
    logging.info('Started Clearing markup')

    logging.debug('rec=%s infile=%s outfile=%s',rec,infile,outfile)
    chanid = rec.chanid
    utcstarttime = rec.starttime
    starttime = str(utcstarttime.utcisoformat().replace(u':', '').replace(u' ', '').replace(u'T', '').replace('-', ''))
    logging.debug('chanid=%s starttime=%s',chanid,starttime)
    try:
        rcl = subprocess.Popen(['mythutil','--chanid',str(chanid),'--starttime',str(starttime),'--clearcutlist'], stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
        output1 = rcl.communicate()
        if rcl.wait() == 0:
            logging.info('Cutlist removed for:%s_%s',chanid,starttime)
        if rcl.wait() != 0:
            logging.error('MYTHUTIL ERROR: clearing cutlist for:%s_%s',chanid,starttime)
            logging.error('%s',output1)
    except Exception as e:
        logging.error('Mythutil exception clearing cutlist%s',e)
    try:
        rsl = subprocess.Popen(['mythutil','--chanid',str(chanid),'--starttime',str(starttime),'--clearskiplist'], stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
        output2 = rsl.communicate()[0]
        if rsl.wait() == 0:
            logging.info('Skiplist removed for:%s_%s',chanid,starttime)
        if rsl.wait() != 0:
            logging.error('MYTHUTIL ERROR: clearing skiplist for:%s_%s',chanid,starttime)
            logging.error('%s',output2)
    except Exception as e:
        logging.error('Mythutil exception clearing skiplist:%s',e)

    for index,mark in reversed(list(enumerate(rec.markup))):
        if mark.type in (rec.markup.MARK_COMM_START, rec.markup.MARK_COMM_END):
            del rec.markup[index]
    rec.bookmark = 0
    rec.bookmarkupdate = datetime.now()
    rec.cutlist = 0
    rec.commflagged = 0
    rec.markup.commit()
    rec.basename = os.path.basename(infile)
    rec.filesize = os.path.getsize(infile)
    rec.transcoded = 1
    rec.seek.clean()
    rec.update()

    try:
        logging.info('Removing PNG files')
        for png in glob('%s*.png' % infile):
            os.remove(png)
    except Exception as e:
        logging.error('Error removing png files',e)
    try:
        logging.info('Removing JPG files')
        for jpg in glob('%s*.jpg' % infile):
            os.remove(jpg)
    except Exception as e:
        logging.error('Error removing jpg files',e)
    try:
        logging.info('Rebuilding seektable')
        rst = subprocess.Popen(['mythcommflag','--chanid',str(chanid),'--starttime',str(starttime),'--rebuild'], stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
        output2 = rst.communicate()[0]
        if rst.wait() == 0:
            logging.info('Seektable Rebuilt for:%s_%s',chanid,starttime)
        if rst.wait() != 0:
            logging.error('MYTHcommflag ERROR: Rebuilding Seektable for:%s_%s',chanid,starttime)
            logging.error('%s',output2)
    except Exception as e:
        logging.error('Mythcommflag ERROR clearing skiplist:%s',e)

def tmp_chk(tmpdir):
        try:
            if os.path.isdir(tmpdir):
                logging.debug('chk:temp Folder found: %s',tmpdir)
                if os.listdir(tmpdir) != 0:
                    logging.warning('chk:Temp folder not empty!:Removing Files: %s',tmpdir)
                    shutil.rmtree(tmpdir)
                    os.makedirs(tmpdir)
            if not os.path.isdir(tmpdir):
                logging.debug('chk:no temp folder found: %s',tmpdir)
                os.makedirs(tmpdir)
                logging.debug('chk:Temp folder created: %s',tmpdir)
        except Exception as e:
            logging.error('%s',e)
def rem_tmp(tmpdir):
            try:
                if os.path.isdir(tmpdir):
                    logging.debug('rem:temp Folder found %s',tmpdir)
                    shutil.rmtree(tmpdir)
                if not os.path.isdir(tmpdir):
                    logging.debug('rem:temp Folder Removed %s',tmpdir)
            except Exception as e:
                logging.error('%s',e)
def find_rec(chanid,starttime):
    def local_time_offset(t=None):
        if t is None:
            t = time.time()

        if time.localtime(t).tm_isdst and time.daylight:
            return -time.altzone
        else:
            return -time.timezone

    def RecordedFromBasename(chanid,starttime):
        bnts = '%s_%s.ts' % (chanid,starttime)
        bnmpg = '%s_%s.mpg' % (chanid,starttime)

        x = list(db.searchRecorded(basename=bnmpg))
        if len(x) == 1:
            for recorded in x:
                return recorded

        if len(x) != 1:
            x = list(db.searchRecorded(basename=bnts))
            if len(x) == 1:
                for recorded in x:
                    return recorded
            if len(x) != 1:
                raise LookupError('unable to find Recorded entry for '
                                  'ChanID {} StartTime {}'
                                  .format(chanid, starttime)
                                  )
    try:
        rec = Recorded((chanid,starttime), db=db)
    except:
        try:
            tzoffset = local_time_offset() / (60*60)
            utcstarttime = datetime.strptime(starttime,"%Y%m%d%H%M%S")
            utcstarttime = utcstarttime + timedelta(hours=tzoffset)
            rec = Recorded((chanid,utcstarttime), db=db)
        except:
            rec = RecordedFromBasename(chanid,starttime)
    return rec

def main():
    parser = argparse.ArgumentParser(description='MythTV Commercial removal and closed caption extraction tool.\nNOTE:Having .srt file in same dir as media files breaks playback in some media players(VLC)')
    parser.add_argument('--chanid',action='store',type=str,dest='chanid',help='Channel-Id of Recording')
    parser.add_argument('--starttime',action='store',type=str,dest='starttime',help='Starttime of recording in utc format')
    parser.add_argument('--jobid',action='store',type=int,dest='jobid',help='JOBID')
    parser.add_argument('-o',action='store',type=str,dest='outfile',help='Output file to be created')
    args = parser.parse_args()
    if args.jobid:
        run_cut(jobid=args.jobid,outfile=args.outfile)
        sys.exit(0)
    if args.chanid and args.starttime:
        run_cut(chanid=args.chanid,starttime=args.starttime,outfile=args.outfile)
        sys.exit(0)
    else:
        print('chanid and starttime or jobid required')
main()
I've tested on both mpeg2 and h.264(same file converted from mpeg2), only time i had issues was when using the break_non_keyframes option. The h.264 would be off so i removed it.

mr_tea
Newcomer
Posts: 13
Joined: Tue Apr 11, 2017 2:47 am
United States of America

Re: Anyone else having problems with commercial removal in 0

Post by mr_tea » Sun Apr 23, 2017 5:45 pm

The good news is that the cuts appear to happen on keyframes now. However, the timing issues remain. It appears to get worse the longer into the video the cut is. Perhaps the FPS calculation is off? I haven't had a chance to look into how that is being calculated in the script.

Also, there is a typo in the definitions of AVJoin where it is missing "AVJoin:"

daraden
Senior
Posts: 169
Joined: Tue Feb 23, 2016 7:33 am
United States of America

Re: Anyone else having problems with commercial removal in 0

Post by daraden » Sun Apr 23, 2017 11:32 pm

Fps calculation is fairly basic for 59.94(60fps) 60000/1001 as long as at least one is a float() it should be correct. The scripts calculation is likely more accurate than needed.
you can check your files with

Code: Select all

ffprobe -i file.ts  -select_streams v -show_entries stream=r_frame_rate,avg_frame_rate
after comparing your ffprobe sample with the h264 ts i created I'm thinking it could be a time-stamp issue.
added -copyts -start_at_zero(should force the starting time-stamp to 0) to the cut command hopefully that helps

Code: Select all

#!/usr/bin/env python2
# -*- coding: UTF-8 -*-
#MythTV commercial removal -- by Daraden
#usage userjob = /path to script/script.py --jobid %JOBID%
#/path to script/script.py --chanid --starttime can also be used
from __future__ import print_function
from MythTV import (Job, Program, Recorded, System, MythDB, findfile,
                    MythError, MythLog, datetime, VideoGrabber)
import argparse
import shutil
from glob import glob
import time
from datetime import timedelta
import subprocess
import ConfigParser
import logging
import logging.handlers
import sys
import os
import tempfile
# set this to True to use commflag results as cut-list
# this will automaticly flag commercials if no cut-list is available
use_commflag = True
# save copy of original file as file.old if not using output file
save_old = False
# user needs write access to this directory to save logfile
logdir ='/media/windows/test/'
LOG_FILENAME = '{}cut.log'.format(logdir)
# logging setup for file and console
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
lf= logging.Formatter('%(asctime)s:%(levelname)s:%(message)s')
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
ch.setFormatter(lf)
logger.addHandler(ch)
try:
    if os.access(logdir,os.W_OK):
        fh = logging.handlers.TimedRotatingFileHandler(filename=LOG_FILENAME,
                                                       when='W0',interval=1,
                                                       backupCount=10
                                                       )
        fh.setLevel(logging.DEBUG)
        fh.setFormatter(lf)
        logger.addHandler(fh)
    if not os.access(logdir,os.W_OK):
        logging.error('logdir not accessible:%s',logdir)
except exception as e:
    logging.error('logdir not accessible:%s',logdir)
    logging.error(e)
    sys.exit(1)

try:
    db=MythDB()
except exception as e:
    logging.error(e)
    sys.exit(1)

def prog_check():
    """checks for FFmpeg/FFprobe if not found MythFFmpeg/MythFFprobe
    is used"""
    from distutils import spawn

    if spawn.find_executable('ffmpeg') != None:
        ffmpeg = spawn.find_executable('ffmpeg')
    elif (spawn.find_executable('ffmpeg') == None and
          spawn.find_executable('mythffmpeg')):
        ffmpeg = spawn.find_executable('mythffmpeg')
    else:
        ffmpeg = None
    if spawn.find_executable('ffprobe') != None:
        ffprobe = spawn.find_executable('ffprobe')
    elif (spawn.find_executable('ffprobe') == None and
          spawn.find_executable('mythffprobe')):
        ffprobe = spawn.find_executable('mythffprobe')
    else:
        ffprobe = None
    if ffmpeg != None and ffprobe != None:
        return ffmpeg,ffprobe
    else:
        raise LookupError('Unable to find FFmpeg or MythFFmpeg')

#check for ffmpeg or mythffmpeg
ffmpeg = prog_check()[0]
logging.debug('ffmpeg={}'.format(ffmpeg))
ffprobe = prog_check()[1]
logging.debug('ffprobe={}'.format(ffprobe))

def run_cut(jobid=None,chanid=None,starttime=None,outfile=None):
    logging.info('Started')

    # Configure chanid and starttime from userjob input
    if jobid:
        job = Job(jobid, db=db)
        chanid = job.chanid
        starttime = job.starttime
        logging.debug('chanid={} starttime={}'.format(chanid, starttime))
    if not jobid:
        chanid = chanid
        starttime = starttime
        logging.debug('chanid={} starttime={}'.format(chanid, starttime))
    # Get database recording entry
    rec = find_rec(chanid,starttime)
    logging.debug('DB recording entry={}'.format(rec))
    # Find and format full input file path
    sg = findfile('/{}'.format(rec.basename), rec.storagegroup, db=db)
    infile = os.path.join(sg.dirname, rec.basename)
    # Assign and create temporary directory in recording directory
    tmpdir = '{}{}{}/'.format(sg.dirname, 'crtmp.',
                              rec.basename.split('.')[0]
                              )
    logging.debug('tmpdir is: {}'.format(tmpdir))
    logging.info('Creating temporary directory')
    tmp_chk(tmpdir)
    # Get info from input file
    avinfo = AVInfo(ffprobe,infile)
    # Get video frame-rate
    rfr = list(int(x) for x in avinfo.video.r_frame_rate.split('/'))
    frame_rate = rfr[0] / float(rfr[1])
    logging.debug('Video frame-rate is: {}'.format(frame_rate))

    if save_old == True and outfile == None:
        if jobid:
            job.update({'status':job.RUNNING,
                        'comment':'Backing up original file'
                        }
                       )

        logging.info('Backing up original file: %s',infile)
        shutil.copyfile(infile,'%s.old' % infile)
        logging.info('Finished backing up original file: {}'.format(infile))
    # Configure output file
    if outfile != None:
        Outfile = outfile
    elif outfile == None:
        Outfile = infile

    if rec.cutlist == 1 or use_commflag == True:
        ##Flag commercials##
        if (use_commflag == True and rec.commflagged != 1
            and rec.cutlist != 1
            ):
            # need check for running commflag job?
            try:
                logging.info('Flagging commercials')
                if jobid:
                    job.update({'status':job.RUNNING,
                                'comment': 'Flagging commercials'
                                }
                               )

                startts = datetime.mythformat(rec.starttime)
                rst = subprocess.Popen(['mythcommflag','--chanid',
                                        str(chanid),'--starttime',
                                        str(startts)],
                                       stdout = subprocess.PIPE,
                                       stderr = subprocess.STDOUT)
                output2 = rst.communicate()[0]
                if rst.wait() == 0:
                    logging.info('Commercials flagged for:%s_%s',
                                 chanid,starttime)
                if rst.wait() != 0:
                    logging.error('MYTHcommflag ERROR: Flagging '
                                  'commercials for:{}_{}'
                                  .format(chanid,starttime)
                                  )
                    logging.error('MythCommflag allways exits with '
                                  'decoding error?!'
                                  )
                    # uncoment folowing line to log mythcommflag errors
                    #logging.error('{}'.format(output2))
            except Exception as e:
                logging.error('Mythcommflag ERROR: Flagging commercials %s',e)
            # Reinitalize recording object
            rec = find_rec(chanid,starttime)
        if jobid:
            job.update({'status':job.RUNNING,
                        'comment':'Getting Cutlist'
                        }
                       )
        logging.info('getting cutlist')
        # Get cut/skip list from database
        if rec.cutlist == 1:
            myth_cut_list = rec.markup.getcutlist()
        if rec.cutlist != 1 and rec.commflagged == 1 and use_commflag:
            myth_cut_list = rec.markup.getskiplist()
        elif rec.cutlist == 0 and rec.commflagged != 1 and not use_commflag:
            logging.debug('No cut/skip list found')
            sys.exit(1)
        logging.debug('Myth cut_list: {}'.format(myth_cut_list))
        # Format cutlist, removing starting 0 and ending 9999999
        cut_list = [mark for cuts in myth_cut_list for mark in cuts]
        if cut_list[0] == 0:
            cut_list.pop(0)
        if cut_list[-1] == 9999999:
            cut_list.pop(-1)
        logging.debug('Cut list: {}'.format(cut_list))
        # Get the name of the video codec
        video_codec = avinfo.video.codec_name
        logging.info('video codec: {}'.format(video_codec))
        # Set cutlist string
        if video_codec == 'h264':
            cut_string = ','.join(str(i / frame_rate) for i in cut_list)
        elif video_codec == 'mpeg2video':
            cut_string = ','.join(str(i) for i in cut_list)
        logging.debug('cut_string: {}'.format(cut_string))
        # Create segment files
        logging.info('Cutting')
        if jobid:
            job.update({'status':job.RUNNING,
                        'comment':'Cutting started'
                        }
                       )
        cut(ffmpeg, infile, cut_string, tmpdir)
        # Create list of files to join
        file_list = []
        # Get list of segment files
        for root, dirs, files in os.walk(tmpdir):
            for File in files:
                if (File.endswith('.ts') and
                    File.startswith('cut')):
                    if os.path.isfile(os.path.join(root,File)):
                        file_list.append(os.path.join(root,File))
        # Set list of files to be joined
        if myth_cut_list[0][0] == 0:
            join_list = file_list[1::2]
        if myth_cut_list[0][0] != 0:
            join_list = file_list[0::2]
        logging.debug('Join file list: {}'.format(join_list))
        # Join segment files
        logging.info('Joining segments')
        if jobid:
            job.update({'status':job.RUNNING,
                        'comment':'Joining segments'
                        }
                       )

        join(ffmpeg, join_list, outfile)
        # Remove temporary directory
        logging.info('removing temporary directory')
        rem_tmp(tmpdir)
        
        if  outfile == None:
            logging.info('clearing database markup')
            if jobid:
                job.update({'status':job.RUNNING,
                            'comment':'Clearing database markup'
                            }
                           )

            clear_markup(rec,infile,outfile)
        if jobid:
            job.update({'status':job.FINISHED, 'comment':
                        'Commercial removal finished'
                        }
                       )
        logging.info('Finished')
class DictToNamespace(object):
    """ convert a dictonary and any nested dictonarys to namespace"""
    def __setitem__(self, key, item):
        self.__dict__[key] = item

    def __getitem__(self, key):
        return self.__dict__[key]

    def has_key(self, k):
        return self.__dict__.has_key(k)

    def update(self, *args, **kwargs):
        return self.__dict__.update(*args, **kwargs)

    def keys(self):
        return self.__dict__.keys()

    def values(self):
        return self.__dict__.values()

    def items(self):
        return self.__dict__.items()

    def __contains__(self, item):
        return item in self.__dict__

    def __iter__(self):
        return iter(self.__dict__)

    def __init__(self, data):
        for item,num in data.items():
            if isinstance(num, str):
                try:
                    num = float(num)
                    if float(num).is_integer():
                        num = int(num)
                    data.update({item: num})
                except ValueError:
                    pass

        self.__dict__.update(data)
        for k, v in data.items():
            if isinstance(v, dict):
                self.__dict__[k] = DictToNamespace(v)

class AVInfo:
    """identify A/V configuration of input file and returns
    self.video dict a list of self.audio.stream dicts,
    and self.duration as float"""

    def __init__(self=None,ffprobe=None,infile=None):
        class Dict(dict):
            def __getattr__(self, name):
                return self[name]
            def __setattr__(self, name, value):
                self[name] = value

        command = [ffprobe, '-v', '-8', '-show_entries',
                   'stream=codec_type,index,codec_name,channels,width,'
                   'height,r_frame_rate:stream_tags=language:'
                   'format=duration', '-of', 'csv=nk=0:p=0', infile
                   ]
        x = subprocess.check_output(command).decode('utf-8').split('\n')

        vcd = {}
        adict = {}
        for vc in x:
            if 'codec_type=video' in vc:
                for item in vc.split(','):
                    k,v = item.split('=')
                    vcd.update({k:v})
        for ac in x:
            if 'codec_type=audio' in ac:
                items =[item.split('=') for item in ac.split(',')]
                streamdict ={k.strip(): v.strip() for (k,v) in items}

                if 'tag:language' in streamdict.keys():
                    streamdict['language'] = (streamdict.pop
                                              ('tag:language')
                                              )
                adict.update({'stream{}'.format(streamdict['index'])
                              : streamdict})
        for d in x:
            if d.startswith('duration='):
                dur = d.split('=')[1]

        self.video = Dict(vcd)
        self.audio = DictToNamespace(adict)
        self.duration = dur

def encode(ffmpeg, command,AVInfo):
    """ Run ffmpeg command with status output"""
    # Length of progress bar
    statlen = 10
    # Charicter used for progress bar
    # Use chr() in python 3
    statchar = unichr(9619).encode('UTF-8')
    # Charicter used to pad progress bar
    pad = ' '

    rfr = list(int(x) for x in AVInfo.video.r_frame_rate.split('/'))
    frame_rate = float(rfr[0]) / float(rfr[1])
    duration = float(AVInfo.duration)
    total_frames = duration * frame_rate

    with tempfile.TemporaryFile() as output:
        process = subprocess.Popen(command, stdout=output,
                                   stderr =output,
                                   universal_newlines=True
                                   )

        while True:
            if process.poll() is not None:
                if process.poll() != 0:
                    output.seek(0)
                    print(output.read().decode('UTF-8'))
                    sys.exit(1)
                if process.poll() == 0:
                    print('\rFinished{}'.format(pad * (statlen + 3)))
                    break
            where = output.tell()
            lines = output.read().decode('UTF-8')
            if not lines:
                time.sleep(0.1)
                output.seek(where)
            elif lines.startswith('frame='):
                ln = ' '.join(lines.split()).replace('= ','=').split(' ')
                for item in ln:
                    if item.startswith('frame='):
                        framenum = int(item.replace('frame=',''))
                    if item.startswith('fps='):
                        fps = float(item.replace('fps=',''))
                if int(framenum) == 0:
                    pcomp = 0
                else:
                    # python 2 div
                    pcomp = 100 * (float(framenum) / float(total_frames))
                    #python 3 div
                    #pcomp = 100 * (framenum / total_frames)
                # python 2 div
                stat = int((float(pcomp) / float(100)) * statlen)
                # python 3 div
                #stat = int((int(pcomp) / 100) * statlen)
                padlen = statlen - stat
                status = "|{:6.2f}%|".format(pcomp)
                statusbar = '|{}{}|'.format(statchar * stat, pad * padlen)
                status = '\r{}{}'.format(status,statusbar)
                print(status, end="")
                # Replace with flush=True in print function for python 3
                sys.stdout.flush()

def cut(ffmpeg, infile, cutlist, tmpdir):
    """use ffmpeg segment muxer to cut infile into multipule files
    using a csv formated cutlist placing them into tmpdir"""
    cmd = [ffmpeg, '-ignore_unknown', '-i', infile, '-y', '-copyts',
           '-start_at_zero','-c', 'copy','-map', '0'
           ]
    av_check = AVInfo(ffprobe,infile)

    for streams,stream in av_check.audio.items():
        if stream.channels == '0':
            cmd.extend(('-map',
                        ('-0:{}'
                         .format(stream.index))))

    if av_check.video.codec_name == 'mpeg2video':
        cmd.extend(('-f','segment',
                    '-segment_frames', cutlist,
                    '{}cut%03d.ts'.format(tmpdir)
                    )
                   )
    if av_check.video.codec_name == 'h264':
        cmd.extend(('-f','segment',
                    '-segment_times', cutlist,
                    '{}cut%03d.ts'.format(tmpdir)
                    )
                   )

    encode(ffmpeg, cmd, av_check)

def join(ffmpeg, file_list, outfile):
    # Create FFmpeg concat string
    concat_string = ','.join(file_list).replace(',','|')
    duration_list = []
    rfr_list = []
    video_codec_list = []
    for files in file_list:
        file_avinfo = AVInfo(ffprobe,files)
        # get file duration
        duration_list.append(float(file_avinfo.duration.strip('\r')))
        # Get file refrence frame rate
        rfr_list.append(file_avinfo.video.r_frame_rate.strip('\r'))
        # Get file video codec
        video_codec_list.append(file_avinfo.video.codec_name.strip('\r'))
    # Duration of joined files
    duration = sum(duration_list)
    # Check that all files refrence framerate are equal
    rfr_match = all(rfr_list[0] == item for item in rfr_list)
    if len(rfr_list) > 0 and rfr_match:
        # Convert refrence frame rate to float
        rfr = rfr_list[0].split('/')
        frame_rate = int(rfr[0]) * float(rfr[1])
    elif len(rfr_list) == 0 or not rfr_match:
        raise ValueError('Incorrect or missing refrence frame rate')
    # Estimate total number of frames
    total_frames = duration * frame_rate
    #check if all files video codec match
    codec_match = all(video_codec_list[0] == item
                      for item in video_codec_list
                      )
    if codec_match:
        video_codec = video_codec_list[0]
    if not codec_match:
        raise ValueError('Not all video codecs match')
    
    command = [ffmpeg, '-y', '-i', 'concat:{}'.format(concat_string),
               '-map', '0', '-c', 'copy', '-f', 'mpegts', outfile
               ]

    class AVJoin:
        """ status update Replacement object for AVInfo  provides
        self.duration and self.video.r_frame_rate for encode()"""
        def __init__(self, duration, rfr):
            class Dict(dict):
                def __getattr__(self, name):
                     return self[name]
                def __setattr__(self, name, value):
                    self[name] = value
            
            self.duration = duration
            self.video = Dict({'r_frame_rate': rfr})

    join_info = AVJoin(duration,rfr_list[0])
    encode(ffmpeg, command, join_info)

def clear_markup(rec,infile,outfile):
    logging.info('Started Clearing markup')

    logging.debug('rec=%s infile=%s outfile=%s',rec,infile,outfile)
    chanid = rec.chanid
    utcstarttime = rec.starttime
    starttime = str(utcstarttime.utcisoformat().replace(u':', '').replace(u' ', '').replace(u'T', '').replace('-', ''))
    logging.debug('chanid=%s starttime=%s',chanid,starttime)
    try:
        rcl = subprocess.Popen(['mythutil','--chanid',str(chanid),'--starttime',str(starttime),'--clearcutlist'], stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
        output1 = rcl.communicate()
        if rcl.wait() == 0:
            logging.info('Cutlist removed for:%s_%s',chanid,starttime)
        if rcl.wait() != 0:
            logging.error('MYTHUTIL ERROR: clearing cutlist for:%s_%s',chanid,starttime)
            logging.error('%s',output1)
    except Exception as e:
        logging.error('Mythutil exception clearing cutlist%s',e)
    try:
        rsl = subprocess.Popen(['mythutil','--chanid',str(chanid),'--starttime',str(starttime),'--clearskiplist'], stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
        output2 = rsl.communicate()[0]
        if rsl.wait() == 0:
            logging.info('Skiplist removed for:%s_%s',chanid,starttime)
        if rsl.wait() != 0:
            logging.error('MYTHUTIL ERROR: clearing skiplist for:%s_%s',chanid,starttime)
            logging.error('%s',output2)
    except Exception as e:
        logging.error('Mythutil exception clearing skiplist:%s',e)

    for index,mark in reversed(list(enumerate(rec.markup))):
        if mark.type in (rec.markup.MARK_COMM_START, rec.markup.MARK_COMM_END):
            del rec.markup[index]
    rec.bookmark = 0
    rec.bookmarkupdate = datetime.now()
    rec.cutlist = 0
    rec.commflagged = 0
    rec.markup.commit()
    rec.basename = os.path.basename(infile)
    rec.filesize = os.path.getsize(infile)
    rec.transcoded = 1
    rec.seek.clean()
    rec.update()

    try:
        logging.info('Removing PNG files')
        for png in glob('%s*.png' % infile):
            os.remove(png)
    except Exception as e:
        logging.error('Error removing png files',e)
    try:
        logging.info('Removing JPG files')
        for jpg in glob('%s*.jpg' % infile):
            os.remove(jpg)
    except Exception as e:
        logging.error('Error removing jpg files',e)
    try:
        logging.info('Rebuilding seektable')
        rst = subprocess.Popen(['mythcommflag','--chanid',str(chanid),'--starttime',str(starttime),'--rebuild'], stdout = subprocess.PIPE, stderr = subprocess.STDOUT)
        output2 = rst.communicate()[0]
        if rst.wait() == 0:
            logging.info('Seektable Rebuilt for:%s_%s',chanid,starttime)
        if rst.wait() != 0:
            logging.error('MYTHcommflag ERROR: Rebuilding Seektable for:%s_%s',chanid,starttime)
            logging.error('%s',output2)
    except Exception as e:
        logging.error('Mythcommflag ERROR clearing skiplist:%s',e)

def tmp_chk(tmpdir):
        try:
            if os.path.isdir(tmpdir):
                logging.debug('chk:temp Folder found: %s',tmpdir)
                if os.listdir(tmpdir) != 0:
                    logging.warning('chk:Temp folder not empty!:Removing Files: %s',tmpdir)
                    shutil.rmtree(tmpdir)
                    os.makedirs(tmpdir)
            if not os.path.isdir(tmpdir):
                logging.debug('chk:no temp folder found: %s',tmpdir)
                os.makedirs(tmpdir)
                logging.debug('chk:Temp folder created: %s',tmpdir)
        except Exception as e:
            logging.error('%s',e)
def rem_tmp(tmpdir):
            try:
                if os.path.isdir(tmpdir):
                    logging.debug('rem:temp Folder found %s',tmpdir)
                    shutil.rmtree(tmpdir)
                if not os.path.isdir(tmpdir):
                    logging.debug('rem:temp Folder Removed %s',tmpdir)
            except Exception as e:
                logging.error('%s',e)
def find_rec(chanid,starttime):
    def local_time_offset(t=None):
        if t is None:
            t = time.time()

        if time.localtime(t).tm_isdst and time.daylight:
            return -time.altzone
        else:
            return -time.timezone

    def RecordedFromBasename(chanid,starttime):
        bnts = '%s_%s.ts' % (chanid,starttime)
        bnmpg = '%s_%s.mpg' % (chanid,starttime)

        x = list(db.searchRecorded(basename=bnmpg))
        if len(x) == 1:
            for recorded in x:
                return recorded

        if len(x) != 1:
            x = list(db.searchRecorded(basename=bnts))
            if len(x) == 1:
                for recorded in x:
                    return recorded
            if len(x) != 1:
                raise LookupError('unable to find Recorded entry for '
                                  'ChanID {} StartTime {}'
                                  .format(chanid, starttime)
                                  )
    try:
        rec = Recorded((chanid,starttime), db=db)
    except:
        try:
            tzoffset = local_time_offset() / (60*60)
            utcstarttime = datetime.strptime(starttime,"%Y%m%d%H%M%S")
            utcstarttime = utcstarttime + timedelta(hours=tzoffset)
            rec = Recorded((chanid,utcstarttime), db=db)
        except:
            rec = RecordedFromBasename(chanid,starttime)
    return rec

def main():
    parser = argparse.ArgumentParser(description='MythTV Commercial removal and closed caption extraction tool.\nNOTE:Having .srt file in same dir as media files breaks playback in some media players(VLC)')
    parser.add_argument('--chanid',action='store',type=str,dest='chanid',help='Channel-Id of Recording')
    parser.add_argument('--starttime',action='store',type=str,dest='starttime',help='Starttime of recording in utc format')
    parser.add_argument('--jobid',action='store',type=int,dest='jobid',help='JOBID')
    parser.add_argument('-o',action='store',type=str,dest='outfile',help='Output file to be created')
    args = parser.parse_args()
    if args.jobid:
        run_cut(jobid=args.jobid,outfile=args.outfile)
        sys.exit(0)
    if args.chanid and args.starttime:
        run_cut(chanid=args.chanid,starttime=args.starttime,outfile=args.outfile)
        sys.exit(0)
    else:
        print('chanid and starttime or jobid required')
main()

mr_tea
Newcomer
Posts: 13
Joined: Tue Apr 11, 2017 2:47 am
United States of America

Re: Anyone else having problems with commercial removal in 0

Post by mr_tea » Mon Apr 24, 2017 1:08 am

The cuts appear to be in the same spot. If this is a keyframe issue, can we force cut ends to be on keyframes after the cut?

daraden
Senior
Posts: 169
Joined: Tue Feb 23, 2016 7:33 am
United States of America

Re: Anyone else having problems with commercial removal in 0

Post by daraden » Tue Apr 25, 2017 7:06 pm

The ffmpeg documentation suggests that is what it should be doing by default.

Post Reply