Anyone want to help troubleshoot my transcode script?

For discussion of topics specific to MythTV on linux
Post Reply
KennyB
Junior
Posts: 46
Joined: Fri Sep 25, 2015 3:56 am
United States of America

Anyone want to help troubleshoot my transcode script?

Post by KennyB » Thu Apr 04, 2019 6:53 pm

Well, hardly "my" script. I took https://www.mythtv.org/wiki/Transcode_Mpeg2_to_H264 and modified it a little bit for my own liking.

Code: Select all

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

# 2015 Michael Stucky
# This script is based on Raymond Wagner's transcode wrapper stub.
# Designed to be a USERJOB of the form </path to script/transcode-h264.py %JOBID%>

from MythTV import Job, Recorded, System, MythDB, findfile, MythError, MythLog, datetime

from optparse import OptionParser
from glob import glob
from shutil import copyfile
import sys
import os
import errno
import time
import subprocess

transcoder = '/usr/bin/nice'
flush_commskip = False
build_seektable = False

def runjob(jobid=None, chanid=None, starttime=None):
    db = MythDB()
    if jobid:
        job = Job(jobid, db=db)
        chanid = job.chanid
        starttime = job.starttime
    rec = Recorded((chanid, starttime), db=db)

    sg = findfile('/'+rec.basename, rec.storagegroup, db=db)
    if sg is None:
        print 'Local access to recording not found.'
        sys.exit(1)
    if rec.transcoded == 1:
        if jobid:
            job.update({'status':272, 'comment':'Recording already transcoded'})
        sys.exit(0)

    infile = os.path.join(sg.dirname, rec.basename)
    tmpfile = '%s.tmp' % infile.rsplit('.',1)[0]
    outfile = '%s.mpg' % infile.rsplit('.',1)[0]

    # reformat 'starttime' for use with mythtranscode/ffmpeg/mythcommflag
    starttime = str(starttime.utcisoformat().replace(u':', '').replace(u' ', '').replace(u'T', '').replace('-', ''))

    # Lossless transcode to strip cutlist
    if rec.cutlist == 1:
        if jobid:
            job.update({'status':4, 'comment':'Removing Cutlist'})

        task = System(path='mythtranscode', db=db)
        try:
            output = task('--verbose --chanid "%s"' % chanid,
                          '--starttime "%s"' % starttime,
                          '--mpeg2 -v',
                          '--allaudiotracks',
                          '--honorcutlist',
                          '--profile autodetect',
                          '--allkeys',
                          '-o "%s"' % tmpfile,
                          '> /tmp/mythtranscode_status')
        except MythError, e:
            print 'Command failed with output:\n%s' % e.stderr
            if jobid:
                job.update({'status':304, 'comment':'Error, transcoding without cutlist'})
            #sys.exit(e.retcode)
            #Don't exit, just transcode without the cutlist
            os.remove(tmpfile)
            task = System(path='/usr/bin/mkfifo', db=db)
            try:
                output = task('%s.tmp' % infile.rsplit('.',1)[0])
            except MythError, e:
                print 'Command failed with output:\n%s' % e.stderr
                if jobid:
                    job.update({'status':304, 'comment' : 'Mkfifo failed'})
                sys.exit(e.retcode)
            try:
                subprocess.Popen(['nohup','cp',infile,tmpfile])
            except subprocess.CalledProcessError as e:
                print 'Command failed with returncode:\n%s' % e.returncode
                if jobid:
                    job.update({'status':304, 'comment':'Copying to mkfifo failed'})
                sys.exit(e.returncode)
    else:
        task = System(path='/usr/bin/mkfifo', db=db)
        try:
            output = task('%s.tmp' % infile.rsplit('.',1)[0])
        except MythError, e:
            print 'Command failed with output:\n%s' % e.stderr
            if jobid:
                job.update({'status':304, 'comment' : 'Mkfifo failed'})
            sys.exit(e.retcode)
        try:
            subprocess.Popen(['nohup','cp',infile,tmpfile])
        except subprocess.CalledProcessError as e:
            print 'Command failed with returncode:\n%s' % e.returncode
            if jobid:
                job.update({'status':304, 'comment':'Copying to mkfifo failed'})
            sys.exit(e.returncode)

    # Transcode to x264
    if jobid:
        job.update({'status':4, 'comment':'Transcoding to x264'})

    task = System(path=transcoder, db=db)
    try:

        output = task('-n 19 ffmpeg -y -i "%s"' % tmpfile,
                      '-map 0',
                      '-c:a libfdk_aac -profile:a aac_he -b:a 64k',
                      '-vcodec libx264',
		      '-scodec dvb_teletext',
                      '-preset medium',
                      '-crf 23',
                      '-f mpegts',
                      '"%s"' % outfile,
                      '> /tmp/ffmpeg_status 2>&1 < /dev/null')
    except MythError, e:
        print 'Command failed with output:\n%s' % e.stderr
        if jobid:
            job.update({'status':304, 'comment':'Transcoding to x264 failed'})
        sys.exit(e.retcode)

    rec.basename = os.path.basename(outfile)
    os.remove(infile)
    # Cleanup the old *.png files
    for filename in glob('%s*.png' % infile):
        os.remove(filename)
    os.remove(tmpfile)
    try:
        os.remove('%s.map' % tmpfile)
    except OSError:
        pass
    rec.filesize = os.path.getsize(outfile)
    rec.transcoded = 1
    rec.seek.clean()

    if flush_commskip:
        for index,mark in reversed(list(enumerate(rec.markup))):
            if mark.type in (rec.markup.MARK_COMM_START, rec.markup.MARK_COMM_END):
                del rec.markup[index]
        rec.bookmark = 0
        rec.cutlist = 0
        rec.markup.commit()

    task = System(path='mythutil')
    task.command('--clearcutlist',
        '--chanid %s' % chanid,
        '--starttime %s' % starttime)

    rec.cutlist = 0
    rec.update()

    if jobid:
        job.update({'status':4, 'comment':'Rebuilding seektable'})

    if build_seektable:
	task = System(path='mythtranscode')
        task.command('--mpeg2',
                     '--buildindex',
                     '--allkeys',
                     '--showprogress',
                     '--infile "%s"' % outfile)
    if jobid:
        job.update({'status':272, 'comment':'Transcode Completed'})

def main():
    parser = OptionParser(usage="usage: %prog [options] [jobid]")

    parser.add_option('--chanid', action='store', type='int', dest='chanid',
            help='Use chanid for manual operation')
    parser.add_option('--starttime', action='store', type='int', dest='starttime',
            help='Use starttime for manual operation')
    parser.add_option('-v', '--verbose', action='store', type='string', dest='verbose',
            help='Verbosity level')

    opts, args = parser.parse_args()

    if opts.verbose:
        if opts.verbose == 'help':
            print MythLog.helptext
            sys.exit(0)
        MythLog._setlevel(opts.verbose)

    if len(args) == 1:
        runjob(jobid=args[0])
    elif opts.chanid and opts.starttime:
        runjob(chanid=opts.chanid, starttime=opts.starttime)
    else:
        print 'Script must be provided jobid, or chanid and starttime.'
        sys.exit(1)

if __name__ == '__main__':
    main()
    
So what it SHOULD do is transcode with ffmpeg into h.264 format, copy all audio tracks present, and the DVB teletext, into mpeg-ts. Then it tells the database the new size, and points it toward the new path of the transcode and deletes the old .ts file.

I modified it a bit from the original by making it work with a FIFO instead of copying over the entire .ts into a new .tmp file, instead it copies to the FIFO, forks into the background and runs ffmpeg on the FIFO as 'cp' feeds it from the .ts file.

Well it was working great until recently when I added the '-scodec dvb-teletext' option and now I'm not sure why, but it fails on the first attempt to run it. I tried to 'su' into the mythtv user and run the script manually to troubleshoot what was happening, but that happens to run successfully. It's just the initial launching of the script from the MythTV frontend that doesn't seem to work, and then manually launching works fine, so I can't troubleshoot it much. All I can tell is it's failing at:

Code: Select all

task = System(path=transcoder, db=db)
    try:

        output = task('-n 19 ffmpeg -y -i "%s"' % tmpfile,
                      '-map 0',
                      '-c:a libfdk_aac -profile:a aac_he -b:a 64k',
                      '-vcodec libx264',
		      '-scodec dvb_teletext',
                      '-preset medium',
                      '-crf 23',
                      '-f mpegts',
                      '"%s"' % outfile,
                      '> /tmp/ffmpeg_status 2>&1 < /dev/null')
    except MythError, e:
        print 'Command failed with output:\n%s' % e.stderr
        if jobid:
            job.update({'status':304, 'comment':'Transcoding to x264 failed'})
        sys.exit(e.retcode)
As you can see I'm piping stdin and stderr to /tmp/ffmpeg_status but this doesn't reveal any errors from the intial run either.

It may be evident, but I don't know much about Python. I'm capable of writing basic applications in C so I've just been using that knowledge to fumble through the script.

Anyone want to help me get this working? I could just disable to the teletext, but I'm hearing impaired so I love subtitles and was exciting that I found a way to include them. You'll have to compile ffmpeg with --enable-libzvbi in order to use the dvb_teletext codec. Might also need to compile with --enable-libfdk-aac and --enable-nonfree, or select a different audio codec if your ffmpeg isn't built with libfdk-aac.

daraden
Senior
Posts: 169
Joined: Tue Feb 23, 2016 7:33 am
United States of America

Re: Anyone want to help troubleshoot my transcode script?

Post by daraden » Fri Apr 05, 2019 6:19 pm

Anything in the backend log?

Have you tried running the ffmpeg command directly?

Is dvb_teletext listed as an encoder in your ffmpeg build? documentation suggests libzvbi is only a decoder.

Are you sure you have teletext subtitles and not eia-608/cea-709 closed captions?

Why all the effort with the temp file? Would be a lot simpler just to set "tmpfile = infile" when not using a cut-list. just need to check if the tmpfile exists before trying to remove the file later in the script.

KennyB
Junior
Posts: 46
Joined: Fri Sep 25, 2015 3:56 am
United States of America

Re: Anyone want to help troubleshoot my transcode script?

Post by KennyB » Fri Apr 05, 2019 9:44 pm

daraden wrote:
Fri Apr 05, 2019 6:19 pm
Anything in the backend log?

Have you tried running the ffmpeg command directly?

Is dvb_teletext listed as an encoder in your ffmpeg build? documentation suggests libzvbi is only a decoder.

Are you sure you have teletext subtitles and not eia-608/cea-709 closed captions?

Why all the effort with the temp file? Would be a lot simpler just to set "tmpfile = infile" when not using a cut-list. just need to check if the tmpfile exists before trying to remove the file later in the script.
Nope, only thing that ever shows up in the back-end log is just that the job failed and what the arguments passed to the script were.

All the ffmpeg functionality worked correctly after I manually ran the script after it ran-and-failed initially via Mythfrontend. I've actually managed to get that behavior to stop by adding a single line, but I'm a little confused because it seems like it shouldn't matter that much.

Code: Select all

 except MythError, e:
        print 'Command failed with output:\n%s' % e.stderr
        if jobid:
            job.update({'status':304, 'comment':'Transcoding to x264 failed'})
        sys.exit(e.retcode)
	os.remove('%s.map' % tmpfile)
All I did was add the os.remove() on the tmpfile and suddenly it began working the first time when launched from the interface. Which is puzzling because I don't know why the presence of it would have affected the rest of the script, and definitely don't understand how it changes anything after sys.exit() is called. Maybe I changed something else I'm not realizing and just attributing it to this.

Which seems to bring us back to your question about the tmp file... Honestly since I don't know python that much I was just trying to hack my changes in while changing as little as possible in case I changed something I didn't understand. But seems like you've got a pretty good point.

Still troubleshooting it some more though. Now that I've got it working I tried it on another backend system, and it's having some issues running as the mythtv user. If I run it as my usual user on this PC from the command-line it runs fine, but executed as mythtv or from the user jobs, I get this:

Code: Select all

kenny@AMD:~$ sudo su mythtv
[sudo] password for kenny: 
$ /usr/local/bin/transcode-mpeg4-fifo.py 3
Traceback (most recent call last):
  File "/usr/local/bin/transcode-mpeg4-fifo.py", line 197, in <module>
    main()
  File "/usr/local/bin/transcode-mpeg4-fifo.py", line 189, in main
    runjob(jobid=args[0])
  File "/usr/local/bin/transcode-mpeg4-fifo.py", line 24, in runjob
    db = MythDB()
  File "/usr/lib/python2.7/dist-packages/MythTV/database.py", line 1264, in __init__
    for tmpconfig in dbconfig.test(self.log):
  File "/usr/lib/python2.7/dist-packages/MythTV/database.py", line 938, in test
    for conn in XMLConnection.fromUPNP(5.0):
  File "/usr/lib/python2.7/dist-packages/MythTV/connections.py", line 607, in fromUPNP
    ip, port = reLOC.match(res['location']).group(1,2)
AttributeError: 'NoneType' object has no attribute 'group'
Fairly certain it's some type of permissions issue, but the python errors are always so cryptic to me that I'm not really sure what it's having permission issues with. I thought perhaps the DB wasn't allowing access by user mythtv, but then why would the rest of the front-end/back-end work at all if that were the case.

Anyway it was late last night so I didn't make much headway on the most recent issue.

Edit:
Solved that one. It could not get IP and port number variables from config.xml because the 'mythtv' user had no config.xml in ~/.mythtv/ and for some reason it ignores the global /etc/. I'm leaving this just in case someone Google's that error code, because that's the only reason I figured it out after reading this post: https://lists.gt.net/mythtv/users/620578

daraden
Senior
Posts: 169
Joined: Tue Feb 23, 2016 7:33 am
United States of America

Re: Anyone want to help troubleshoot my transcode script?

Post by daraden » Sat Apr 06, 2019 1:38 pm

I've never used mythtv System() to run subprocesses, it looks like a custom wraper for popen. Not sure if it is intended to be used with non-myth commands.

Code: Select all

os.remove('%s.map' % tmpfile)
should not do anything since sys.exit was called. That assumes System() is correctly throwing a MythError exception. Are you geting 'Transcoding to x264 failed' in the job queue when it fails?

Still wondering if you actually have teletext subtitles. Apparently in the U.S. only a couple of satellite providers have ever used them.

KennyB
Junior
Posts: 46
Joined: Fri Sep 25, 2015 3:56 am
United States of America

Re: Anyone want to help troubleshoot my transcode script?

Post by KennyB » Tue Apr 09, 2019 1:00 am

daraden wrote:
Sat Apr 06, 2019 1:38 pm
I've never used mythtv System() to run subprocesses, it looks like a custom wraper for popen. Not sure if it is intended to be used with non-myth commands.

Code: Select all

os.remove('%s.map' % tmpfile)
should not do anything since sys.exit was called. That assumes System() is correctly throwing a MythError exception. Are you geting 'Transcoding to x264 failed' in the job queue when it fails?

Still wondering if you actually have teletext subtitles. Apparently in the U.S. only a couple of satellite providers have ever used them.
I'm actually recording over-the-air broadcast on ATSC, but it uses the DVB technology for some reason. I don't really understand all the format differences myself personally, but I know that prior to this I had no way to transcode the closed captioning or subtitles that would be present on the .ts files, but since adding '-scodec dvb_teletext' they're included in my transcodes.

Yeah when it errors I get that 'Transcoding to x264' line in the mythtv status window. I think that there's actually a bit of a false-positive problem, because sometimes I would get this error and the script would exit, even though I could view the ffmpeg run log and see that everything had completed without error, and the resulting files would even be playable ( with a little manually tinkering to get the db to point to the tmp file ). I think ffmpeg must be erroneously tripping MythError sometimes.

daraden
Senior
Posts: 169
Joined: Tue Feb 23, 2016 7:33 am
United States of America

Re: Anyone want to help troubleshoot my transcode script?

Post by daraden » Tue Apr 09, 2019 6:05 am

ATSC should be eia-608/cea-709 closed captions. With ffmpeg versions prior to ~3.2 required "-a53cc 1" to preserve closed captions in mpeg2/h264. newer versions have it enabled by default. My initial thought was '-scodec dvb_teletext' may be throwing a warning causing it to have a non 0 return.

give this a try

Code: Select all

mythccextractor -i /path/to/recording.ts -d /tmp
this should output any detected closed captions as srt files into /tmp. eia-608/cea-709 closed captions should end with something like .608-cc1.eng.srt/.708-service-01.eng.srt.

KennyB
Junior
Posts: 46
Joined: Fri Sep 25, 2015 3:56 am
United States of America

Re: Anyone want to help troubleshoot my transcode script?

Post by KennyB » Thu Apr 11, 2019 7:32 am

Yep the .srt files were named as you predicted. That's interseting because without the -scodec option, I don't get the subttitles included by default. Nor with the -a53cc option you mentioned.

Strange thing is '-scodec eia_608' seems to work the same as '-scodec dvb_teletext'. Though interestingly, I don't really see any mention closed captions in the input/output stream listings in ffmpeg's log.

Here is the output from ffmpeg (abbreviated to remove all the status lines)

Code: Select all

ffmpeg version N-91965-gb0cfb2c Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.11) 20160609
  configuration: --prefix=/home/kenny/ffmpeg_build --pkg-config-flags=--static --extra-cflags=-I/home/kenny/ffmpeg_build/include --extra-ldflags=-L/home/kenny/ffmpeg_build/lib --extra-libs='-lpthread -lm' --bindir=/home/kenny/bin --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-nonfree
  libavutil      56. 19.101 / 56. 19.101
  libavcodec     58. 30.100 / 58. 30.100
  libavformat    58. 18.101 / 58. 18.101
  libavdevice    58.  4.103 / 58.  4.103
  libavfilter     7. 32.100 /  7. 32.100
  libswscale      5.  2.100 /  5.  2.100
  libswresample   3.  2.100 /  3.  2.100
  libpostproc    55.  2.100 / 55.  2.100
Input #0, mpegts, from '/mnt/mythtv_recordings/1352_20190411060000.tmp':
  Duration: N/A, start: 73289.566689, bitrate: N/A
  Program 1 
    Stream #0:0[0x41]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p(tv, smpte170m, bottom first), 704x480 [SAR 10:11 DAR 4:3], Closed Captions, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
    Stream #0:1[0x44](eng): Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 192 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (mpeg2video (native) -> h264 (libx264))
  Stream #0:1 -> #0:1 (ac3 (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
[libx264 @ 0x2d94f40] using SAR=10/11
[libx264 @ 0x2d94f40] using cpu capabilities: MMX2 SSE2Fast LZCNT
[libx264 @ 0x2d94f40] profile High, level 3.0
Output #0, mpegts, to '/mnt/mythtv_recordings/1352_20190411060000.mpg':
  Metadata:
    encoder         : Lavf58.18.101
    Stream #0:0: Video: h264 (libx264), yuv420p, 704x480 [SAR 10:11 DAR 4:3], q=-1--1, 29.97 fps, 90k tbn, 29.97 tbc
    Metadata:
      encoder         : Lavc58.30.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
    Stream #0:1(eng): Audio: aac (libfdk_aac) (HE-AAC), 48000 Hz, stereo, s16, 64 kb/s
    Metadata:
      encoder         : Lavc58.30.100 libfdk_aac
 ...
 video:207239kB audio:13971kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 11.118148%
[libx264 @ 0x2d94f40] frame I:424   Avg QP:20.73  size: 25389
[libx264 @ 0x2d94f40] frame P:15561 Avg QP:23.55  size:  8500
[libx264 @ 0x2d94f40] frame B:37731 Avg QP:27.37  size:  1834
[libx264 @ 0x2d94f40] consecutive B-frames:  2.0%  5.3% 22.9% 69.8%
[libx264 @ 0x2d94f40] mb I  I16..4: 16.2% 71.7% 12.1%
[libx264 @ 0x2d94f40] mb P  I16..4:  2.7%  7.9%  1.1%  P16..4: 36.8% 15.0%  8.3%  0.0%  0.0%    skip:28.2%
[libx264 @ 0x2d94f40] mb B  I16..4:  0.2%  0.5%  0.1%  B16..8: 29.5%  3.1%  0.6%  direct: 1.6%  skip:64.5%  L0:36.8% L1:49.7% BI:13.4%
[libx264 @ 0x2d94f40] 8x8 transform intra:67.7% inter:82.0%
[libx264 @ 0x2d94f40] coded y,uvDC,uvAC intra: 59.9% 66.3% 23.1% inter: 12.4% 13.9% 0.4%
[libx264 @ 0x2d94f40] i16 v,h,dc,p: 35% 41%  6% 19%
[libx264 @ 0x2d94f40] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 29% 19% 25%  3%  4%  5%  4%  5%  5%
[libx264 @ 0x2d94f40] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 20% 53% 10%  2%  3%  3%  3%  3%  2%
[libx264 @ 0x2d94f40] i8c dc,h,v,p: 43% 22% 27%  8%
[libx264 @ 0x2d94f40] Weighted P-Frames: Y:2.2% UV:1.6%
[libx264 @ 0x2d94f40] ref P L0: 61.7% 16.9% 17.3%  4.1%  0.0%
[libx264 @ 0x2d94f40] ref B L0: 90.3%  8.1%  1.6%
[libx264 @ 0x2d94f40] ref B L1: 97.7%  2.3%
[libx264 @ 0x2d94f40] kb/s:944.41
Here is the resulting transcode playing in MythTV frontend with the subtitles displayed...
Screenshot_2019-04-11_00-18-25.png
Screenshot_2019-04-11_00-18-25.png (193.6 KiB) Viewed 582 times

daraden
Senior
Posts: 169
Joined: Tue Feb 23, 2016 7:33 am
United States of America

Re: Anyone want to help troubleshoot my transcode script?

Post by daraden » Fri Apr 12, 2019 11:34 am

That is odd, have you tested playback with other media players? You may want to try updating to a more recent version of ffmpeg(without libzvbi).

Are you running with a cut-list?

With eia-608/cea-709 closed captions you wont see subtitle streams since the data is actually encoded into the video. Witch setting a53cc to 1 is supposed to tell the video encoder not to drop the data. This is also why scodec should be entirely irrelevant to the entire process.

KennyB
Junior
Posts: 46
Joined: Fri Sep 25, 2015 3:56 am
United States of America

Re: Anyone want to help troubleshoot my transcode script?

Post by KennyB » Mon Apr 22, 2019 1:31 am

Interesting, so it's supposed to hard-code them into the transcoding then? I don't see them added as another track, but I can toggle them off/on. I haven't had any more failures to transcode so I may just leave it be for now.

Thanks for the help!

daraden
Senior
Posts: 169
Joined: Tue Feb 23, 2016 7:33 am
United States of America

Re: Anyone want to help troubleshoot my transcode script?

Post by daraden » Mon Apr 22, 2019 4:42 am

With Ffmpeg its not so much hard-coding them as it is preserving the top lines of the video frame that already has the closed caption data.

I asked about the cut-list, as I have not used mythtranscode in a long time and I am not sure if it will preserve the closed caption data.

Post Reply