Transcoding/Export userjob script

[daraden]

without some sort of error or log its hard to figure out the issue.
I'm not well versed in writing shell scripts so i have a few questions.
1. why are you trying to get database info? (your userjob command already has all of the data you need and you are not accessing the database)
2. why use -p autodetect when -m is specified for lossless cut?
3. if you are using a temp-file when cutting commercials why rebuild the seek-table?(only makes sense if you are replacing the original file)

my only suggestion is to try the commands individually to find what command is causing the problem.

[Kwisher]

Forum member bonelifer provided the script to me so I cannot answer your questions about the script process. Below is my backend log when trying to run the user job/script.

Code: Select all

Jun 25 10:01:45 garage mythbackend: mythbackend[32134]: I UserJob_7561 jobqueue.cpp:2410 (DoUserJobThread) JobQueue: Started Rec-Export for "Person of Interest":"Return 0" recorded from channel 2291 at 2016-06-22T02:00:00Z
Jun 25 10:03:16 garage mythbackend: mythbackend[32134]: I ProcessRequest mainserver.cpp:1698 (HandleAnnounce) MainServer: MainServer::ANN Monitor
Jun 25 10:03:16 garage mythbackend: mythbackend[32134]: I ProcessRequest mainserver.cpp:1703 (HandleAnnounce) MainServer: adding: garage(233e290) as a client (events: 0)
Jun 25 10:03:16 garage mythbackend: mythbackend[32134]: I ProcessRequest mainserver.cpp:1698 (HandleAnnounce) MainServer: MainServer::ANN Monitor
Jun 25 10:03:16 garage mythbackend: mythbackend[32134]: I ProcessRequest mainserver.cpp:1703 (HandleAnnounce) MainServer: adding: garage(2343ad0) as a client (events: 1)
Jun 25 10:03:17 garage mythbackend: mythbackend[32134]: I MythSocketThread(78) mainserver.cpp:7629 (connectionClosed) Monitor sock(233e290) 'garage' disconnected
Jun 25 10:03:17 garage mythbackend: mythbackend[32134]: I MythSocketThread(85) mainserver.cpp:7629 (connectionClosed) Monitor sock(2343ad0) 'garage' disconnected
Jun 25 10:03:19 garage mythbackend: mythbackend[32134]: I ProcessRequest mainserver.cpp:1698 (HandleAnnounce) MainServer: MainServer::ANN Monitor
Jun 25 10:03:19 garage mythbackend: mythbackend[32134]: I ProcessRequest mainserver.cpp:1703 (HandleAnnounce) MainServer: adding: garage(249bb00) as a client (events: 0)
Jun 25 10:03:19 garage mythbackend: mythbackend[32134]: I ProcessRequest mainserver.cpp:1698 (HandleAnnounce) MainServer: MainServer::ANN Monitor
Jun 25 10:03:19 garage mythbackend: mythbackend[32134]: I ProcessRequest mainserver.cpp:1703 (HandleAnnounce) MainServer: adding: garage(2473a90) as a client (events: 1)
Jun 25 10:03:41 garage mythbackend: mythbackend[32134]: I MythSocketThread(78) mainserver.cpp:7629 (connectionClosed) Monitor sock(249bb00) 'garage' disconnected
Jun 25 10:03:41 garage mythbackend: mythbackend[32134]: I MythSocketThread(94) mainserver.cpp:7629 (connectionClosed) Monitor sock(2473a90) 'garage' disconnected
Jun 25 10:03:42 garage mythbackend: mythbackend[32134]: E UserJob_7561 jobqueue.cpp:2445 (DoUserJobThread) JobQueue: User Job '/usr/bin//mythexport.sh "/media/mythtv/recordings/2291_20160622020000.ts" "/media/vol-01/MythExports/Person of Interest - S5E13 - Return 0.m4v" "2291" "20160622020000"' failed.

[bonelifer]

without some sort of error or log its hard to figure out the issue.
I'm not well versed in writing shell scripts so i have a few questions.
1. why are you trying to get database info? (your userjob command already has all of the data you need and you are not accessing the database)
2. why use -p autodetect when -m is specified for lossless cut?
3. if you are using a temp-file when cutting commercials why rebuild the seek-table?(only makes sense if you are replacing the original file)

my only suggestion is to try the commands individually to find what command is causing the problem.

1. Forgot to remove that as this was a simplified version of a script I use, in my full version, I "move" original files to a "Seen" group one that has never delete set, just in case I need to go back and edit. This is because anything in Deleted gets flushed by another script. I use Kodi exclusively for viewing my collection. MythTV for recording/editing.

2. Can't remember, was having troubles without it. But this script dates back to when I was using .24, so it may of just been needed then due to some mis-configuration on my part.

3. The rebuild command can be removed.

If you've updated to .28, it's probably the commercial cut, as mythutil replaced mythcommflag for this in .28 I think. Not 100% as I haven't updated yet myself.

[Kwisher]

Bonelifer,

Since i can't code my way out of a wet paper bag, can you provide the edited script with the changes you suggested?

TIA

[bonelifer]

Also check that the directory you are writing the file is writable by the mythtv user, ie the account the script is running from. It may of gotten messed up, since I don't know what distro or way you updated. Here's with everything removed that was mentioned earlier.

Userjob command:

Code: Select all

/usr/bin/mythexport.sh "%DIR%/%FILE%" "/media/vol-01/MythExports/%TITLE% - S%SEASON%E%EPISODE% - %SUBTITLE%.m4v" "%CHANID%" "%STARTTIMEUTC%"

Transcode script mythexport.sh:

Code: Select all

#!/bin/bash

# First does a lossless MPEG-2(honorcutlist) transcode removing commericals(cutlist must be manually made before
# hand), then rebuilds the seek table. It then uses handbrakecli to transcode the mpg to a m4a(H264, AAC). 
#
# Loosely based on mythtv-transcode-h264 version 0.8 by Defcronyke Webmaster.
#

# Arguments
# $1 must be the directory/file to be transcoded.
# $2 must be the output directory / file name. The directory must be writeable by the mythtv user
# $3 must be chanid
# $4 must be starttime
# the full userjob command in mythtv-setup should look like this:
#
# Format for TV:
# /home/kevin/Dropbox/scripts/mythexport.sh "%DIR%/%FILE%" "/media/vol-01/MythExports/%TITLE% - S%SEASON%E%EPISODE% - %SUBTITLE%.m4v" "%CHANID%" "%STARTTIMEUTC%"
#
# Format for MOVIES:
# /usr/bin/mythtv-transcode-mkv.sh "%DIR%/%FILE%" "/media/Store/incoming/Movies/%TITLE%.m4v" "%CHANID%" "%STARTTIMEUTC%"
# 
#

# MythTV Install Prefix (make sure this matches with the directory where MythTV is installed)
#
INSTALLPREFIX="/usr/bin"

# don't change these
#
MYPID=$$
DIRNAME=`dirname "$1"`
DIRNAME2=`dirname "$2"`
BASENAME=`echo "$2" | awk -F/ '{print $NF}' | sed 's/://g' | sed 's/?//g' | sed 's/\?//g' | sed 's/*//g' | sed 's/!//g' | sed s/"'"/""/g`
CHAN="$3"
START="$4"


# play nice with other processes
#
renice 19 $MYPID
ionice -c 3 -p $MYPID

# remove commercials
# 
$INSTALLPREFIX/mythtranscode --chanid "$3" --starttime "$4" -m --honorcutlist


# Handbrake
#
$INSTALLPREFIX/HandBrakeCLI -i "$1".tmp -o "$DIRNAME2/$BASENAME" --preset="Android Tablet"
  

# Delete the TEMPORARY commercial edited file
cd $DIRNAME
rm "$1".tmp

[Kwisher]

Sorry to say that the changes did not fix anything. The temp folder is fully read/write accessible to the mythtv user (drwxrwxr-x 2 mythtv mythtv 6 Feb 13 14:51 tmp). I am running Xubuntu 14.04 and I use the Mythbuntu PPA (http://www.ubuntuupdates.org/ppa/mythtv_0.28). For the upgrade from .27 to .28 I just changed the PPA.

[bonelifer]

This seems .28 related to me. I have no experience with it.

[bonelifer]

You say "tmp", but what about /var/lib/mythtv/recordings and "/media/vol-01/MythExports/"? The recordings directory is where the .tmp file get written by mythtranscode(in the same directory as the original TS file). It needs to be readble by the script/handbrakecli. The other one is the folder you said you were writing to in your posts, it needs to be writable by the script/handbrakecli.

[Kwisher]

You say "tmp", but what about /var/lib/mythtv/recordings and "/media/vol-01/MythExports/"? The recordings directory is where the .tmp file get written by mythtranscode(in the same directory as the original TS file). It needs to be readble by the script/handbrakecli. The other one is the folder you said you were writing to in your posts, it needs to be writable by the script/handbrakecli.

I'm going to have to agree that this is a .28 issue as nothing has changed in my setup except for the upgrade to .28. Everything else since the upgrade works as it should in regards to MythTV.

[Kwisher]

I can't let this go. Trying to run the commands manually to see where the issue is.

mythtranscode --infile /media/mythtv/recordings/2291_20160622020000.ts -m --honorcutlist

This does not leave a *.tmp for Handbrake

[bonelifer]

Possibly they've changed the behavior of mythtranscode. Try adding adding the output statement:

Code: Select all

#!/bin/bash

# First does a lossless MPEG-2(honorcutlist) transcode removing commericals(cutlist must be manually made before
# hand), then rebuilds the seek table. It then uses handbrakecli to transcode the mpg to a m4a(H264, AAC).
#
# Loosely based on mythtv-transcode-h264 version 0.8 by Defcronyke Webmaster.
#

# Arguments
# $1 must be the directory/file to be transcoded.
# $2 must be the output directory / file name. The directory must be writeable by the mythtv user
# $3 must be chanid
# $4 must be starttime
# the full userjob command in mythtv-setup should look like this:
#
# Format for TV:
# /home/kevin/Dropbox/scripts/mythexport.sh "%DIR%/%FILE%" "/media/vol-01/MythExports/%TITLE% - S%SEASON%E%EPISODE% - %SUBTITLE%.m4v" "%CHANID%" "%STARTTIMEUTC%"
#
# Format for MOVIES:
# /usr/bin/mythtv-transcode-mkv.sh "%DIR%/%FILE%" "/media/Store/incoming/Movies/%TITLE%.m4v" "%CHANID%" "%STARTTIMEUTC%"
#
#

# MythTV Install Prefix (make sure this matches with the directory where MythTV is installed)
#
INSTALLPREFIX="/usr/bin"

# don't change these
#
MYPID=$$
DIRNAME=`dirname "$1"`
DIRNAME2=`dirname "$2"`
BASENAME=`echo "$2" | awk -F/ '{print $NF}' | sed 's/://g' | sed 's/?//g' | sed 's/\?//g' | sed 's/*//g' | sed 's/!//g' | sed s/"'"/""/g`
CHAN="$3"
START="$4"


# play nice with other processes
#
renice 19 $MYPID
ionice -c 3 -p $MYPID

# remove commercials
#
$INSTALLPREFIX/mythtranscode --chanid "$3" --starttime "$4" -m --honorcutlist -o "$DIRNAME\$1".tmp


# Handbrake
#
$INSTALLPREFIX/HandBrakeCLI -i "$1".tmp -o "$DIRNAME2/$BASENAME" --preset="Android Tablet"
 

# Delete the TEMPORARY commercial edited file
cd $DIRNAME
rm "$1".tmp

[Kwisher]

I read the mythtranscode help section and seen the -o output flag. I tried it with my manual command and it did not produce a file.

Before I did the .28 upgrade I made a Clonezilla image of my / partition and have restored it and I am now back on .27 where the transcode userjob/script work. I lost a few recordings from my DB but, meh.

Thanks for all the input.

[Kwisher]

I may need to open this topic again as my Manjaro/Arch .27 frontend no longer works after some recent updates as per:
viewtopic.php?f=29&t=1655
Going to backup my backend when the wifey goes to town today and give .28 a try again.

[Kwisher]

bonelifer:
The command below does not create a *.tmp file:
mythtranscode --chanid 2061 --starttime 20160314000000 -m --honorcutlist -p autodetect

I modified your script as you mentioned in an earlier post:
$INSTALLPREFIX/mythtranscode --chanid "$3" --starttime "$4" -m --honorcutlist -o "$DIRNAME\$1".tmp

If i make a copy of your script and manually change the variables for CHANID & STARTTIME for a recording and run it, here is the end of the terminal output:

Code: Select all

2016-08-20 19:00:41.906327 N  Transcode Completed
2016-08-20 19:00:41.934358 N  Transcoding /media/myth2/recordings/2061_20151130010000.mpg done
2016-08-20 19:00:41.934747 N  Transcode: delete old file: waiting while program is in use.
2016-08-20 19:00:41.935105 N  Transcode: program is no longer in use.
2016-08-20 19:00:42.222524 E  mythtranscode: Error Renaming '/media/myth2/recordings/2061_20151130010000.mpg.tmp' to '/media/myth2/recordings/2061_20151130010000.mpg'
			eno: No such file or directory (2)
2016-08-20 19:00:42.223742 N  Requesting delete for file 'myth://Default@garage/2061_20151130010000.mpg.old'.
2016-08-20 19:00:42.225950 I  MythCoreContext::ConnectCommandSocket(): Connecting to backend server: 10.1.0.10:6543 (try 1 of 1)
2016-08-20 19:00:42.226963 I  MythCoreContext::CheckProtoVersion(): Using protocol version 88 XmasGift
2016-08-20 19:00:42.229274 N  Requesting delete for file 'myth://Default@garage/2061_20151130010000.mpg.-1.100x56.png'.
2016-08-20 19:00:42.229994 N  Requesting delete for file 'myth://Default@garage/2061_20151130010000.mpg.png'.
2016-08-20 19:01:55.351344 I  Waiting for threads to exit.
MythTV Commercial Flagger, building seek table for:
    
2016-08-20 19:01:58.171412 C  mythcommflag version: fixes/0.28 [v0.28-47-g29d7067] www.mythtv.org
2016-08-20 19:01:58.171430 C  Qt version: compile: 5.2.1, runtime: 5.2.1
2016-08-20 19:01:58.262428 E  Couldn't find file , aborting.
2016-08-20 19:01:58.273104 E  Couldn't find file , aborting.
2016-08-20 19:01:58.273108 E  Unable to find file in defined storage paths.
[19:01:58] hb_init: starting libhb thread
HandBrake 0.10.5 (2016021100) - Linux x86_64 - https://handbrake.fr
8 CPUs detected
Opening %DIR%/%FILE%.tmp...
[19:01:58] CPU: Intel(R) Xeon(R) CPU           X3450  @ 2.67GHz
[19:01:58]  - logical processor count: 8
[19:01:58] OpenCL: library not available
[19:01:58] hb_scan: path=%DIR%/%FILE%.tmp, title_index=1
libbluray/bdnav/index_parse.c:162: indx_parse(): error opening %DIR%/%FILE%.tmp/BDMV/index.bdmv
libbluray/bdnav/index_parse.c:162: indx_parse(): error opening %DIR%/%FILE%.tmp/BDMV/BACKUP/index.bdmv
libbluray/bluray.c:2182: nav_get_title_list(%DIR%/%FILE%.tmp) failed
[19:01:58] bd: not a bd - trying as a stream/file instead
libdvdnav: Using dvdnav version 5.0.1
libdvdread: Can't stat %DIR%/%FILE%.tmp
No such file or directory
libdvdread: Could not open %DIR%/%FILE%.tmp
libdvdnav: vm: failed to open/read the DVD
[19:01:58] dvd: not a dvd - trying as a stream/file instead
[19:01:58] hb_stream_open: open %DIR%/%FILE%.tmp failed
[19:01:58] scan: unrecognized file type
[19:01:58] libhb: scan thread found 0 valid title(s)
No title found.

HandBrake has exited.
/usr/bin//mythexport-test.sh: line 84: cd: %DIR%: No such file or directory
rm: cannot remove ‘%DIR%/%FILE%.tmp’: No such file or directory

Does this provide any help?

[ricks03]

I see two issues with .28. First, as observed, mythtranscode overwrites/deletes the original, and the -o option doesn't work

Second, take a look at mythtv-setup / 1. General / Job Queue (global) about page 9 or so "save original files after transcoding" which will create a .old file when transcoding happens (backing up the original file)