Script to export Mythtv recordings to a folder or NAS

For discussion related to MythTV which doesn't belong in another forum.

Moderator: Forum Moderators

Kwisher
Junior
Posts: 58
Joined: Fri Sep 25, 2015 3:02 pm
United States of America

Re: Script for sending Mythtv recordings to a NAS

Post by Kwisher »

Skinah wrote:I found a few bugs which are now fixed in the very first post of this thread.
Bugs were only in MP4 processing and were causing the cuts to occur in the wrong places and missing ends.
To upgrade make sure you are using MKVMERGE version 9.5 or newer and BC is installed, instructions are in the start of the script. Found I was using a very old version of mkvmerge 6.x until I added the correct source and key. These changes may fix your issues Kwisher.
Yes, these updates/changes fixed my issue with recordings from my HDHRPrime.

Thank you :)
skinah
Junior
Posts: 53
Joined: Sat Oct 17, 2015 11:56 pm

Re: Script to export Mythtv recordings to a folder or NAS

Post by skinah »

No problem.

I'm just updating the script in the first post again...

The script now has a Quality setting you can edit at the start of the script that allows you to choose the handbrake quality or if you set it to 0 zero it will then skip handbrake and output the lossless file. I also re-wrote some code to help if you run more than 1 job at a time and the recordings have similar names like in a TV Series, I probably wont test this myself and doing that will screw up the log file as two jobs or more will be writting to the same log file. If you need a log file to help fault find, you should just run one job at a time as I wont be changing this.
Lastly the backup file to work around the 0.28 mythtranscode bug has been removed and this has sped the script up and lowered HDD wear and tear. Make sure you updated your mythtv to the latest fixes if your on 0.28

I have edited the script heaps to add these changes and the script can branch down multiple paths which will take some time to test all combinations. This script does not touch your myth database or recordings so any bugs will only effect the file the script spits out or it may leave some files behind in your fallback location. Any issues post details on them here and I'll take a look.
User avatar
jfabernathy
Senior
Posts: 577
Joined: Wed Feb 18, 2015 2:37 pm
Location: Raleigh, NC
United States of America

Re: Script to export Mythtv recordings to a folder or NAS

Post by jfabernathy »

FYI, I updated my script to the one you just posted today (11/20/16) and tested it against my MPEG2 files recorded from my ATSC OTA tuner HDHomeRun Connect. It converted it to .m4v as expected with expected quality, and copied the file to my NAS.

I have the latest 0.28 patches so the file deletion problem with mythtranscode -o is fixed and no issues when you removed the cp *.bup commands.

Jim A
skinah
Junior
Posts: 53
Joined: Sat Oct 17, 2015 11:56 pm

Re: Script for sending Mythtv recordings to a NAS

Post by skinah »

Kwisher wrote:Skinah, could you please provide instructions to modify your script so that the finished file is not placed into a folder?
The easiest way would be to give a fake location to the script for the NAS, that way the move fails to work and the files are all left in the fallback location. The Movie script should now work for doing this as I just updated it, the tv show one will need updating when I have time but the Movie script can be run on tv shows as well.
se99paj
Junior
Posts: 26
Joined: Wed Mar 26, 2014 11:43 pm
Great Britain

Re: Script to export Mythtv recordings to a folder or NAS

Post by se99paj »

Thanks for writing the script Skinah - I've tried a few different scripts to attempt this and none of them seem to fit the bill perfectly so keen to give this a try.

Is there anyway to track progress? I've been running my first script for about 40 minutes but not sure if it is still running or stuck. The file in the FALLBACKDIR has reached the file size of the original recording, I've also checked the ScriptsLastRun log but nothing has been added for the last 30 mins. I can't see anything in the final output directory either.

I'm going to try and run a few recordings just to see if I have any different experiences.

[SOLVED]

Just in case anyone has a similar problem, I managed to resolve this by changing the FALLBACKDIR to one that was under my home directory, not sure why it wouldn't work in the original location but I should have probably followed the instructions to begin with.
Kwisher
Junior
Posts: 58
Joined: Fri Sep 25, 2015 3:02 pm
United States of America

Re: Script to export Mythtv recordings to a folder or NAS

Post by Kwisher »

skinah,

It seems your script has stopped working on recordings from my HDHRprime using Comcast. This seems to have started after the new MythTV updates this week. OTA recordings still work fine.

Any insight?

ScriptsLastRun.log:

Code: Select all

The line below records the file details of the last run.
ChanID_StartTimeUTC-Title-SeasonAndEpisodeNumber-Subtitle
2205_20170114135900-Shooter-s1e9-Ballistic Advantage
DIR/Filename
/media/mythtv/recordings/2205_20170114135900.ts
2017-01-14 15:15:58.228109 C  mythutil version: fixes/0.28 [v0.28-105-gd5d1bce] www.mythtv.org
2017-01-14 15:15:58.228132 C  Qt version: compile: 5.5.1, runtime: 5.5.1
2017-01-14 15:15:58.228138 N  Enabled verbose msgs:  general
2017-01-14 15:15:58.228153 N  Setting Log Level to LOG_INFO
2017-01-14 15:15:58.240612 I  Added logging to the console
2017-01-14 15:15:58.240947 I  Setup Interrupt handler
2017-01-14 15:15:58.240954 I  Setup Terminated handler
2017-01-14 15:15:58.240960 I  Setup Segmentation fault handler
2017-01-14 15:15:58.240965 I  Setup Aborted handler
2017-01-14 15:15:58.240969 I  Setup Bus error handler
2017-01-14 15:15:58.240974 I  Setup Floating point exception handler
2017-01-14 15:15:58.240979 I  Setup Illegal instruction handler
2017-01-14 15:15:58.240985 I  Setup Real-time signal 0 handler
2017-01-14 15:15:58.240992 I  Setup Hangup handler
2017-01-14 15:15:58.241056 N  Using runtime prefix = /usr
2017-01-14 15:15:58.241060 N  Using configuration directory = /home/mythtv/.mythtv
2017-01-14 15:15:58.241103 I  Assumed character encoding: en_US.UTF-8
2017-01-14 15:15:58.241301 N  Empty LocalHostName.
2017-01-14 15:15:58.241306 I  Using localhost value of garage
2017-01-14 15:15:58.254602 N  Setting QT default locale to EN_US
2017-01-14 15:15:58.254612 I  Current locale EN_US
2017-01-14 15:15:58.254644 N  Reading locale defaults from /usr/share/mythtv//locales/en_us.xml
Cutlist: 0-3765,21599-36928,67373-72254,107765-121635,147305-159858,179488-193730
Using mythtranscode to remove cutllist after first creating a backup.
2017-01-14 15:15:58.605676 C  mythtranscode version: fixes/0.28 [v0.28-105-gd5d1bce] www.mythtv.org
2017-01-14 15:15:58.605698 C  Qt version: compile: 5.5.1, runtime: 5.5.1
2017-01-14 15:15:58.605704 N  Enabled verbose msgs:  general
2017-01-14 15:15:58.605719 N  Setting Log Level to LOG_INFO
2017-01-14 15:15:58.616304 I  Added logging to the console
2017-01-14 15:15:58.616958 I  Setup Interrupt handler
2017-01-14 15:15:58.616970 I  Setup Terminated handler
2017-01-14 15:15:58.616979 I  Setup Segmentation fault handler
2017-01-14 15:15:58.616988 I  Setup Aborted handler
2017-01-14 15:15:58.616996 I  Setup Bus error handler
2017-01-14 15:15:58.617005 I  Setup Floating point exception handler
2017-01-14 15:15:58.617014 I  Setup Illegal instruction handler
2017-01-14 15:15:58.617024 I  Setup Real-time signal 0 handler
2017-01-14 15:15:58.617035 I  Setup Hangup handler
2017-01-14 15:15:58.617122 N  Using runtime prefix = /usr
2017-01-14 15:15:58.617128 N  Using configuration directory = /home/mythtv/.mythtv
2017-01-14 15:15:58.617196 I  Assumed character encoding: en_US.UTF-8
2017-01-14 15:15:58.617515 N  Empty LocalHostName.
2017-01-14 15:15:58.617522 I  Using localhost value of garage
2017-01-14 15:15:58.633694 N  Setting QT default locale to EN_US
2017-01-14 15:15:58.633709 I  Current locale EN_US
2017-01-14 15:15:58.633760 N  Reading locale defaults from /usr/share/mythtv//locales/en_us.xml
2017-01-14 15:15:58.653136 I  Loading en_us translation for module mythfrontend
2017-01-14 15:15:58.655168 N  Transcoding from /media/mythtv/recordings/2205_20170114135900.ts to /home/mythtv/Shooter - s1e9.mpg
2017-01-14 15:15:58.655180 I  Honoring the cutlist while transcoding
2017-01-14 15:15:58.660308 I  Opening /media/mythtv/recordings/2205_20170114135900.ts
2017-01-14 15:15:58.663156 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.663344 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.663351 I  decode_slice_header error
2017-01-14 15:15:58.663357 I  no frame!
2017-01-14 15:15:58.663413 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.663453 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.663459 I  decode_slice_header error
2017-01-14 15:15:58.663465 I  no frame!
2017-01-14 15:15:58.663485 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.663537 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.663544 I  decode_slice_header error
2017-01-14 15:15:58.663549 I  no frame!
2017-01-14 15:15:58.663592 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.663628 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.663640 I  decode_slice_header error
2017-01-14 15:15:58.663646 I  no frame!
2017-01-14 15:15:58.663661 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.663711 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.663718 I  decode_slice_header error
2017-01-14 15:15:58.663723 I  no frame!
2017-01-14 15:15:58.663755 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.663793 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.663799 I  decode_slice_header error
2017-01-14 15:15:58.663805 I  no frame!
2017-01-14 15:15:58.663833 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.663882 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.663888 I  decode_slice_header error
2017-01-14 15:15:58.663893 I  no frame!
2017-01-14 15:15:58.663911 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.663938 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.663945 I  decode_slice_header error
2017-01-14 15:15:58.663950 I  no frame!
2017-01-14 15:15:58.663999 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.664048 I  non-existing PPS 0 referenced
2017-01-14 15:15:58.664055 I  decode_slice_header error
2017-01-14 15:15:58.664060 I  no frame!
2017-01-14 15:15:58.674295 I  PES packet size mismatch
2017-01-14 15:15:58.674321 I  PES packet size mismatch
2017-01-14 15:15:58.674363 I  Input #0, mpegts, from '/media/mythtv/recordings/2205_20170114135900.ts':
2017-01-14 15:15:58.674378 I    Duration: 01:02:29.80, start: 946.954578, bitrate: 4222 kb/s
2017-01-14 15:15:58.674437 I      Stream #0:0[0x18c9]: Video: h264 (High), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], Closed Captions, 59.94 fps, 59.94 tbr, 90k tbn, 119.88 tbc
2017-01-14 15:15:58.674468 I      Stream #0:1[0x18ca](eng): Audio: ac3, 48000 Hz, stereo, fltp, 192 kb/s
2017-01-14 15:15:58.674493 I      Stream #0:2[0x18cb](spa): Audio: ac3, 48000 Hz, stereo, fltp, 192 kb/s
2017-01-14 15:15:58.674713 W  Warning: partial frame found!
2017-01-14 15:15:58.674781 W  Warning: partial frame found!
2017-01-14 15:15:58.674840 W  Warning: partial frame found!
2017-01-14 15:15:59.643284 E  No more queue slots!
2017-01-14 15:15:59.662856 E  Transcoding /media/mythtv/recordings/2205_20170114135900.ts failed
File must be a mp4 recording as we can not do lossless transcoding with mythtranscode
The cutlist is:
-0 3765-21599 36928-67373 72254-107765 121635-147305 159858-179488 193730-
clipcount=1
Start=3765-End=21599
clipcount=2
Start=36928-End=67373
clipcount=3
Start=72254-End=107765
clipcount=4
Start=121635-End=147305
clipcount=5
Start=159858-End=179488
The keepframes string contains:3765-21599,+36928-67373,+72254-107765,+121635-147305,+159858-179488
Using mkvmerge to create a single file without pieces from the cutlist
Using mkvmerge to create a single file without pieces from the cutlist
mkvmerge v9.7.1 ('Pandemonium') 64bit
Error: Missing start/end specifications for '--split' in '--split parts:'.
Note: Had to delete numerous lines with "Warning: partial frame found!" in order to post.
Skinah2
Newcomer
Posts: 7
Joined: Fri Jan 20, 2017 3:57 am
Switzerland

Re: Script to export Mythtv recordings to a folder or NAS

Post by Skinah2 »

Se99: problem was probably not having write permissions for the folder as the script when run by the backend maybe run as mythtv or a different user to what you use. Thanks for posting it's working now and solution.

Kwisher try using the latest version of the script as I found that mkvmerge was not working very well when using frames for the cuts, by changing to using seconds instead the mkvmerge did a reliable job. I left code in the script for both methods in case a different version of mkvmerge works again. If that fixes your issue my guess is your version of mkvmerge was updated and this caused the issue. The only time I ever have issues is when a cut is less than a second long as commflag can at times create flags very close to the start time so if u have issues with only the odd recording check the cultist for cuts less than a second long.

Once u have a working mythtv setup I would stop doing any kind of updates on it.
axel77
Newcomer
Posts: 8
Joined: Mon Jan 30, 2017 5:35 am
United States of America

Re: Script to export Mythtv recordings to a folder or NAS

Post by axel77 »

Skinah, thank you for this script! I just set it up yesterday and tested it out on a few recordings for the kids. The transcoding/renaming/moving all worked, but I'd like to tweak the output a bit and could use your help.

The only numbers that I changed with the script were fps and quality. However, when hand brake transcoded the tv episodes it drastically changed more that just those two inputs. The original episodes were 1280x720/59.94fps/6-channel audio, but the output was 720x406/30fps/2-channel stereo (I had the fps set to 59.94 NTSC). Is it possible to adjust the script to simply copy the resolution/fps/audio of the source?

Thanks again for the awesome work!
Skinah2
Newcomer
Posts: 7
Joined: Fri Jan 20, 2017 3:57 am
Switzerland

Re: Script to export Mythtv recordings to a folder or NAS

Post by Skinah2 »

See the handbrake website docs or use the man command in Linux to see the switches for handbrake. You will only need to mod the handbrake lines all else can stay as is.

-X 720 will let anything under 720 pass but anything over will be changed so maybe make it 1280 as this makes the file size jump greatly so play with it to find what u prefer. The quality setting is also explained in the handbrake docs.

The same applies to the frame rate. I used the --pfr option so 30fps and under will stay the same as the source and anything over this will change to 30fps. Easy to change if u wish but I would leave as is if file size is what u are after.

Audio is very complex and u will find the settings I have used should create a copy of the ac3 track if it is there PLUS also a second track that will look like stereo but is actually pro logic surround, google what that is if a surround track in two channels is weird.

Hope that helps and please post what u find is a good compromise between the quality and file size with the handbrake line u used. If u have apple devices check out the infuse pro app as the files can be played on an iPhone or iPad.
Kwisher
Junior
Posts: 58
Joined: Fri Sep 25, 2015 3:02 pm
United States of America

Re: Script to export Mythtv recordings to a folder or NAS

Post by Kwisher »

You can also remove all the complex switches & options if you find a HandBrake preset that gives you the desired result. I use the "Android Tablet" preset with good results.
axel77
Newcomer
Posts: 8
Joined: Mon Jan 30, 2017 5:35 am
United States of America

Re: Script to export Mythtv recordings to a folder or NAS

Post by axel77 »

Thanks again Skinah, got everything set up and mostly working perfectly now.

I simply replaced all the mentions of 720 with 1920 to account for full HD width, and then changed the 30fps limitation to 60. I should've checked the audio output in something other than a simple properties view in Windows10, I discovered that there are indeed two audio tracks when opened in vlc.

The size difference when bumping up to 720p/59fps/Q-22 only accounted for roughly a 40% jump in file size. Seems large but storage is cheap enough to account for an extra 100MB or so per half hour show.

Only issue I've discovered is that the file permissions on the transcoded video is still only writable to the mythtv user. I'm still getting used to linux/mythtv but I think that the script is run as root. Do you know of anyway to make the permissions of final output 0777 (or at the very least 0664)?
Skinah2
Newcomer
Posts: 7
Joined: Fri Jan 20, 2017 3:57 am
Switzerland

Re: Script to export Mythtv recordings to a folder or NAS

Post by Skinah2 »

axel77 wrote: The size difference when bumping up to 720p/59fps/Q-22 only accounted for roughly a 40% jump in file size. Seems large but storage is cheap enough to account for an extra 100MB or so per half hour show.

Only issue I've discovered is that the file permissions on the transcoded video is still only writable to the mythtv user. I'm still getting used to linux/mythtv but I think that the script is run as root. Do you know of anyway to make the permissions of final output 0777 (or at the very least 0664)?
That looks about right for the filesize increase for staying at 720 and going to 60fps, it is more the resolution that increases the filesize very quickly as doubling the width will probably give 4 times the filesize depending on the quality setting. I have not played too much in depth with comparing the quality at higher resolutions and using more compression to bring the size back down, as I preferred a more consistant filesize to be desirable as some channels here are in 720i and others 1080i and some are below 720. If filesize is not an issue then you can just leave the files in Myth and add more drives to your backend, myth makes it very easy to do that.

RE write permissions: open a terminal and type 'sudo thunar' and it will open a file browser with a GUI with root control. You can then right hand click (?) and set permissions from a GUI. As I am also not a Linux expert and I find that easiest. Set the permissions for the folder and all files and sub folders should then inherit the same permissions.
axel77
Newcomer
Posts: 8
Joined: Mon Jan 30, 2017 5:35 am
United States of America

Re: Script to export Mythtv recordings to a folder or NAS

Post by axel77 »

Skinah2 wrote: That looks about right for the filesize increase for staying at 720 and going to 60fps, it is more the resolution that increases the filesize very quickly as doubling the width will probably give 4 times the filesize depending on the quality setting.
I actually increased the resolution quite a bit. You have the -X option set which limits the default recording to a horizontal size of 720 max. By changing the horizontal to 1920 or less I was able to get true 720p resolution (1280x720), or almost 4x the resolution of the default 720x406 (406p). If you want to get 720p you'll need to change the -X to -Y. It's actually pretty surprising that the biggest difference isn't in the resolution, but in the compression format.

As to the permission issue, I was debating a cron job that ran every hour to change the folder permissions, but I'd rather a cleaner method that's part of the script (if possible). Do you mind if I post this question/script on a different forum where someone else might be able to answer?

Again, thank you for creating this script for everyone. It really helps those of us just starting to use mythtv!
Kwisher
Junior
Posts: 58
Joined: Fri Sep 25, 2015 3:02 pm
United States of America

Re: Script to export Mythtv recordings to a folder or NAS

Post by Kwisher »

axel77 wrote:Only issue I've discovered is that the file permissions on the transcoded video is still only writable to the mythtv user. I'm still getting used to linux/mythtv but I think that the script is run as root. Do you know of anyway to make the permissions of final output 0777 (or at the very least 0664)?
On my system the folder for the finished file has ownership as mythtv:mythtv. I added my account to the mythtv group and I am able to work with the files. Not sure what you are wanting to do with the final file.
axel77
Newcomer
Posts: 8
Joined: Mon Jan 30, 2017 5:35 am
United States of America

Re: Script to export Mythtv recordings to a folder or NAS

Post by axel77 »

Do you have write permissions? I added my account to the mythtv group as well but can't move/delete the file since it's 644. That's why I was hoping to make it 664 or 666

I've added the below to lines 199 and 233, not sure if it will work since I've never dealt with scripting variables.

chmod 666 "$TVOUTDIR/$TVTITLE/$TVTITLE - $SeasonEpisode - $TVSUBTITLE.m4v"

EDIT: It worked! Rights are now set rw-rw-rw- on the transcoded video
Post Reply