Transcoding full high definition

[njsharp]

Hello. Hope this is the best place for this discussion. In short, I hope to be able to process MPEG-4 full HD in a similar way that I now process MPEG-2 TS. This is because Australian stations are now beginning to offer 1920x1080 using MPEG-4 compression (somewhat to the annoyance of those without adequate TVs or set top boxes!). MPEG-4 is the only way they can fit a Full HD channel into the 7Mbps for all a stations's DTV and radio channels.

What I do after recording a program is:
1 to edit it with E in order to chop off top, tail, and cut out all ads
2 run lossless transcode to remove both the chopped bits and the MPEG-TS stuff
3 run handbrake on the result to re-encode (slooow!) to MPEG-4 for more compact storage

I am expecting to abandon #3 for full HD as the compression is already MPEG-4. Comments on that?

#1 seems to work.
#2 fails, so I ran mythtranscode using CLI and these I think are the interesting bits:

2016-03-09 07:11:30.820289 I Using protocol version 77
2016-03-09 07:11:30.820485 N Found video height of 1088. This is unusual and more than likely the video is actually 1080 so mythtranscode will treat it as such.
2016-03-09 07:11:30.820521 N Transcode: Looking for autodetect profile: Autodetect from 1080i
2016-03-09 07:11:30.820881 E Transcode: Couldn't find profile for : H.264
2016-03-09 07:11:30.820887 E Transcoding aborted, no profile found.
2016-03-09 07:11:30.821938 E Transcoding 2090_20160308194000.mpg failed

I have elsewhere read that somebody set up a different profile for Autodetect from 1080i, but I thought I would here ask the gurus about how best to set about this task. It LOOKS like it ought to finish up easier than with MPEG-2 channels as I wouldn't need to run handbrake, a great program but understandably time and cpu greedy!

Below is some info about channel 90.

Thanks folks.
njsharp
======================

In Australia, Nine's prime channel is now being broadcast on DTV channel 90 in full high definition, Here's VLC's view of the codecs (with an occasional note), which boringly I have to retype as VLC's Codec information sheet text seemingly cannot be copied as text:

Stream 0
Type: Video
Original ID: 513
Codec H264 - MPEG-4 AVC (part 10)(h264)
Resolution 1920x1090 <<<<<<<<<huh?
Display resolution 19020x1080 (1080i????)
Frame rate: 25
Decoded format: Planar 4:2:0 YUV
Stream 1
Type: Audio
Original ID: 650
Codec A52 Audio (aka AC3)(a52)
Language: English
Channels: 3F2R/LFE
Sample rate: 48000 Hz
Bitrate: 448 kb/s
Stream 2
Type: Subtitle
Original ID: 577
Codec: Teletext (telx)
Language: English
Description: Teletext subtitles

and from ffprobe:

Duration: 00:03:32.05, start: 83475.289556, bitrate: 4483 kb/s
Program 1
Stream #0:0[0x201]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 50 tbr, 90k tbn, 50 tbc
Stream #0:1[0x28a](eng): Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, 5.1(side), fltp, 448 kb/s
Stream #0:2[0x241](eng): Subtitle: dvb_teletext ([6][0][0][0] / 0x0006), 492x250

[skinah]

The channel is 1080i interlaced and will be similar to what happened to the BBC HD channel.

From what I read myth transcode does not do lossless cuts to MP4 so even if you do get it to work it will loose quality. What I was reading may have been out of date.

To cut the recording and stay in mp4 this may work...

https://www.mythtv.org/wiki/Lossless_Cut

or maybe

https://www.mythtv.org/wiki/MythDVBcut


If u come up with a good solution then please post how as I will be doing a similar thing when I find time to look into it.

[skinah]

Ok I did the following to get further with mythtranscode to work with the 1088 aka 1080i recordings that channel 9 HD and win HD are now using with mp4 compression. They pad the resolution out so it can be devided by 16 hence why it becomes 1088.

Go to the mythbackend setup.
Go to 'recording profiles'
Transcoders.
Create new profile
Created a new Transcoder profile called "Autodetect from 1080i"
Set the Codec to MPEG-4
For height and width scroll down until they are set to AUTO.
Leave everything else at defaults and then save and exit.

Doing this it will only work if you run mythtranscode without the lossless transcode option. I think it is -m or --mpeg2 options can not be used. If someone with more knowledge can comment on this it would be great as the results were not good when I tried this method ranging from no audio or a static picture for the entire video.

Heres the status of each script I have tried:

Lossless_cut.py
fails with..
This MythTV recording has no recordedseek table first keyframe record.
It is likely that the recordedseek table records for this recording is
invalid, aborting script.
I believe this may have been caused by a script damaging the MySQL database for that recording, not sure what the cause was but the seektable was missing so this script may well work if I re-tried.

hdpvrcutter.pl
fails with..
Can't locate File/Which.pm in @INC (you may need to install the File::Which module) (@INC contains: /etc/perl /usr/local/lib/perl/5.18.2 /usr/local/share/perl/5.18.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.18 /usr/share/perl/5.18 /usr/local/lib/site_perl .) at hdpvrcutter.pl line 41.
BEGIN failed--compilation aborted at hdpvrcutter.pl line 41.
Problem here is that the which modual has been removed from a standard install of perl?!?

Lossless_cut.py uses the same method as this bash script. hdpvrcutter.pl also uses a similar method.
https://www.mythtv.org/wiki/H264_commer ... nd_remuxer
I now have it working after 30+ hours of re-writing the code to work on newer builds. I find a movie length recording takes up 3.2gb of storage when done with lossless cuts and ac3 kept.

And there is this one which is meant to work very quickly with the downside that there is no audio for 2-3 seconds after a cut.
https://gist.github.com/BipedalTV/3174241

[skinah]

I have it 100% working with 3 test recordings and need this tested. It does not replace your recording or play with your mythtv mysql database so feel free to have a play. I will probably add a PAD feature to add extra time to either side of a cut since the cuts jump to the nearest KEYFRAME other methods I believe all do this. Feel free to give some feedback and if you use it please post to say it works or does not. Don't forget to check write permissions for the user that will run the script.

Currently no known bugs as I fixed some sync issues for long recordings... I hope

Code: Select all

#!/bin/sh
# Script written by 'Skinah' using some work created by Ian Thiele from h264cut on myth wiki page.
# This script can have any of the following skipped if you wish. It can remove ads, 
# transcode video to mp4 with AC3, then move the file to a NAS or other location.
# It now supports lossless cutting of MP4 HD footage. 
# All of this is done without changing Mythtv's database or recordings, this way if a 
# mistake is found in the cutting of the recording then before myth expires the recording, we can rerun the 
# job after adjusting the cutlist manually.
#
# Add this next line to the mythbackend setup for a user job
# /home/matt/sendtonas.sh "%CHANID%" "%STARTTIMEUTC%" "%TITLE%" "%SUBTITLE%" "%SEASON%" "%EPISODE%"
#
# to install handbrake use these commands or visit the handbrake website.
#sudo add-apt-repository ppa:stebbins/handbrake-releases
#sudo apt-get update
#sudo apt-get install handbrake-gtk handbrake-cli
#
#To do lossless cutting with MP4 recordings do the following to install the requirements. MP2 does not need these.
#sudo apt-get install mkvtoolnix mkvtoolnix-gui
#sudo apt-get install mediainfo
#sudo apt-get install bc

############# Section for Setting up script and changing options ##################################

# path to MythTV transcoding tools, tested with mythbuntu 14.04
INSTALLPREFIX=/usr/bin

#The folder where the mpeg recordings live, tested with mythbuntu 14.04
MYTHRECORDINGDIR=/var/lib/mythtv/recordings

#If your HD channel is not 25 frames/sec then edit this with the value. Only used for h264 cuts.
fps=25

# Where your finished videos will be placed. This can be a NAS drive that is mounted in etc/fstab
# Mythtv user must have write permissions for this location.
# If you want movies and tv shows to go to the same place, just make both of these the same.
MOVIEOUTDIR="/mnt/nas/Movies"
TVOUTDIR="/mnt/nas/TV Shows"

# If the move command fails due to the NAS not being mounted, the file will remain here so as not to waste the Handbrake time.
# Mythtv user must have write permissions for this location and also enough free disk space.
FALLBACKDIR="/home/matt/Videos"


######## Leave everything below alone unless you know how to mod and debug #############

# Store the title and subtitle that the userjob passes on to this shell script
TVTITLE=$3
TVSUBTITLE=$4

# First clean up any weird tags that the TV channel adds.
# These helped when Shepperd was not setup for XMLTV in Australia.
TVTITLE=`echo "$TVTITLE" | sed 's/Movie Premiere: //g' | sed 's/- Season Premiere//g' | sed 's/Movie: //g' | sed 's/://g' | sed 's/?//g' | sed 's/,//g'`
TVSUBTITLE=`echo "$TVSUBTITLE" | sed 's/\[RPT\] //g' | sed 's/(R) //g' | sed 's/ - //g' | sed 's/(G) //g' | sed 's/(AV) //g' | sed 's/(M) //g' | sed 's/(PG) //g' | sed 's/\[CC\] //g' | sed 's/(MA) //g'`

# clean up the title and subtitle strings by deleting any non-standard characters that can cause issues in file names
TVTITLE=`echo "$TVTITLE" | sed 's/://g' | sed 's/?//g' | sed 's/,//g'`
TVSUBTITLE=`echo "$TVSUBTITLE" | sed 's/://g' | sed 's/?//g' | sed 's/,//g'`

SeasonEpisode="s$5e$6"

# print the modified title_subtitle for logging purposes
echo "The line below records the file details of the last run." > $FALLBACKDIR/ScriptsLastRun.log
echo "ChanID_StartTimeUTC-Title-SeasonAndEpisodeNumber-Subtitle" >> $FALLBACKDIR/ScriptsLastRun.log
echo "$1_$2-$TVTITLE-$SeasonEpisode-$TVSUBTITLE" >> $FALLBACKDIR/ScriptsLastRun.log

# play nice with other processes
MYPID=$$
renice 19 $MYPID
ionice -c 3 -p $MYPID

#if you have installed losslesscut script from the PPA as per the wiki you can use this to jump all cutpoints to nearest keyframe.
#/usr/bin/keyframe_adjust -f "$MYTHRECORDINGDIR/$1_$2.mpg" #-g


#Check to see if we already have a cutlist before we overwrite any manually created skips, otherwise create new cutlist.
if [ -z "`$INSTALLPREFIX/mythutil --chanid "$1" --starttime "$2" --getcutlist | grep Cutlist | sed 's/Cutlist: $//'`" ]; then
    echo "no cutlist found....generating new cutlist"
    $INSTALLPREFIX/mythutil --chanid "$1" --starttime "$2" --gencutlist
fi  

$INSTALLPREFIX/mythutil --chanid "$1" --starttime "$2" --getcutlist

# Remove the commercials without transcoding so as not to lose quality.
$INSTALLPREFIX/mythtranscode --chanid "$1" --starttime "$2" --mpeg2 --honorcutlist -o "$FALLBACKDIR/$TVTITLE - $SeasonEpisode.mpg"

#check exit code
if [ $? = 0 ]
then
	echo "Lossless transcode on MP2 using mythtranscode worked fine" >> $FALLBACKDIR/ScriptsLastRun.log
else
    	echo "File must be a mp4 recording as we can not do lossless transcoding with mythtranscode" >> $FALLBACKDIR/ScriptsLastRun.log

#cutlist provides a list of frames in the format start-end,[start1-end1,....] to cut 
#we swap this list so that it provides the ranges of video we want in the format
#	start-end start1:end1 ....
CUTLIST=`$INSTALLPREFIX/mythutil --chanid "$1" --starttime "$2" --getcutlist | grep Cutlist | sed 's/Cutlist: //' | \
    sed 's/-/ /g' | sed 's/^\|$/-/g' | sed 's/,/-/g'`

#You can view the cutlist if you enable the next line.
#echo $CUTLIST > $FALLBACKDIR/cutlist.log

echo "Removing entire TMP folder to ensure it is empty" >> $FALLBACKDIR/ScriptsLastRun.log
rm -r "$FALLBACKDIR/TMP" >> $FALLBACKDIR/ScriptsLastRun.log

echo "Making a new empty TMP folder" >> $FALLBACKDIR/ScriptsLastRun.log
mkdir -p "$FALLBACKDIR/TMP" >> $FALLBACKDIR/ScriptsLastRun.log
echo "Splitting recording using the cutlist and FFMPEG" >> $FALLBACKDIR/ScriptsLastRun.log


clipcount=0
for i in ${CUTLIST}
do
    start=`echo $i | sed 's/ //g' | sed 's/^\(.*\)-.*$/\1/'`
    end=`echo $i | sed 's/ //g' | sed 's/^.*-\(.*\)$/\1/'`

    	#if $start is empty, deal with it
    	if [ -z $start ]; then
	start=0
	fi

    	#If the end is empty then skip the cut.
	if [ -z $end ]; then
	continue
	fi
	
	#if the start and end are the same then skip the cut
	if [ $start -eq $end ]; then
	continue 
	fi
    
    	#convert start into time in seconds (divide frames by frames per second)    
    	start=$(echo "scale=8; $start / $fps" | bc -l)

	clipcount=$((clipcount+1))

	#Use the program bc to create a floating point value in seconds with 8 decimal places.
	end=$(echo "scale=8; $end / $fps" | bc -l)
	duration=`echo "$end - $start" | bc -l`	

	#Now use ffmpeg to create a separate file for each cut in the list
	ffmpeg -i "$MYTHRECORDINGDIR/$1_$2.mpg" -acodec copy -vcodec copy -f matroska -sn -ss $start -t $duration "$FALLBACKDIR/TMP/264Cut_$clipcount.mkv" >> $FALLBACKDIR/ScriptsLastRun.log
done

mergestr=""
for i in `ls -t -r -1 $FALLBACKDIR/TMP/264Cut_*`
do
    if [ -z "$mergestr" ]; then
	mergestr="$i"
	continue
    fi
    mergestr="$mergestr +$i"
done

echo "The merge string contains:$mergestr" >> $FALLBACKDIR/ScriptsLastRun.log
echo "Merging all the cut recording pieces back into one" >> $FALLBACKDIR/ScriptsLastRun.log
mkvmerge --append-mode file $mergestr -o "$FALLBACKDIR/$TVTITLE.mkv" >> $FALLBACKDIR/ScriptsLastRun.log
#changed from mode track to file to see if sync issues are fixed.

echo "Attempting to send file to the NAS now" >> $FALLBACKDIR/ScriptsLastRun.log

#Script assumes if both season and episode is 0 then it must be a movie. Not the best but it works for me.
if [ "$SeasonEpisode" = "s0e0" ]
then
	mkdir -p "$MOVIEOUTDIR/$TVTITLE"
	mv -u "$FALLBACKDIR/$TVTITLE.mkv" "$MOVIEOUTDIR/$TVTITLE/$TVTITLE.mkv"
else
	mkdir -p "$TVOUTDIR/$TVTITLE"
	mv -u "$FALLBACKDIR/$TVTITLE.mkv" "$TVOUTDIR/$TVTITLE/$TVTITLE - $SeasonEpisode - $TVSUBTITLE.mkv"
fi

#cleanup files no longer needed, to fault find it can be handy to disable these and manually delete them before running script again.
rm "$FALLBACKDIR/$TVTITLE - $SeasonEpisode.mpg"

#echo "Removing entire TMP folder" >> $FALLBACKDIR/ScriptsLastRun.log
#rm -r "$FALLBACKDIR/TMP" >> $FALLBACKDIR/ScriptsLastRun.log

exit 0

fi

#################### Back to processing mp2 files ###################

echo "Running Handbrake" >> $FALLBACKDIR/ScriptsLastRun.log

#Need a better way as the following assumes that it is a Movie if the season and episode is both 0
#Current solution is to run different script for TV Shows but left ability in script.

echo "Attempting to send file to the NAS now" >> $FALLBACKDIR/ScriptsLastRun.log

if [ "$SeasonEpisode" = "s0e0" ]
then
	/usr/bin/HandBrakeCLI -i "$FALLBACKDIR/$TVTITLE - $SeasonEpisode.mpg" -o "$FALLBACKDIR/$TVTITLE - $SeasonEpisode.m4v" -e x264 -q 21 -O -r 30 --pfr -x ref=6:bframes=5:vbv-maxrate=62000:vbv-bufsize=62000 -X 720 --decomb --loose-anamorphic --modulus 2 --x264-tune film --x264-preset medium --h264-profile high --h264-level 4.1 -a 1,1 -E copy:ac3,faac -B auto,160 -R auto,auto -6 auto,dpl2 --audio-copy-mask aac,ac3,dtshd,dts,mp3 --audio-fallback ffac3 -f mp4 >> $FALLBACKDIR/ScriptsLastRun.log
        mkdir -p "$MOVIEOUTDIR/$TVTITLE"
	mv -u "$FALLBACKDIR/$TVTITLE - $SeasonEpisode.m4v" "$MOVIEOUTDIR/$TVTITLE/$TVTITLE.m4v"
else
	/usr/bin/HandBrakeCLI -i "$FALLBACKDIR/$TVTITLE - $SeasonEpisode.mpg" -o "$FALLBACKDIR/$TVTITLE - $SeasonEpisode.m4v" -e x264 -q 25 -O -r 30 --pfr -x ref=6:bframes=5:vbv-maxrate=62000:vbv-bufsize=62000 -X 720 --decomb --loose-anamorphic --modulus 2 --x264-tune film --x264-preset medium --h264-profile high --h264-level 4.1 -a 1,1 -E copy:ac3,faac -B auto,160 -R auto,auto -6 auto,dpl2 --audio-copy-mask aac,ac3,dtshd,dts,mp3 --audio-fallback ffac3 -f mp4 >> $FALLBACKDIR/ScriptsLastRun.log
        mkdir -p "$TVOUTDIR/$TVTITLE"
	mv -u "$FALLBACKDIR/$TVTITLE - $SeasonEpisode.m4v" "$TVOUTDIR/$TVTITLE/$TVTITLE - $SeasonEpisode - $TVSUBTITLE.m4v"
fi

rm "$FALLBACKDIR/$TVTITLE - $SeasonEpisode.mpg"
rm "$FALLBACKDIR/$TVTITLE - $SeasonEpisode.mpg.map"
exit 0

[njsharp]

Thanks Skinah for your posts. I shall have to work through them carefully; I think you are aiming to do more than I am, so I may need to be selective.

Also, my main aim is to get ready for ABC's HD broadcasts due to start about mid 2016. I went just now to channel 90 to set up a recording so I could do tests, and went through several days but did not find anything worth capturing. I'm really a rusted on ABC viewer! I even stopped watching SBS most of the time when they went over to putting ads in the middle of programs.

More later.

[skinah]

Do you know if ABC HD will be using h264 or mpeg2 ?
If you dont need the full script just cut it down to what you need, I'm a newbie to bash scripting so I comment most stuff to explain each step.

The script that is on the wiki called h264cut is what I based some of my code on but dont bother looking at it as it was broken with newer mythtv versions and I had to fix some bugs where it would join the cut files in the wrong order if there were more than 10 cuts. Audio sync issues I think I just fixed and updated my code above.

[njsharp]

I'm guessing ABC will have to use H264 (MPEG-4). Their current sub-full-HD on channel 24 is 1280x720 MPEG-2 with A52 audio (aca AC3). So who wants better definition on a news channel? ABC's answer was that they were regulated to send their primary channel 2 out in SD and did not have enough bandwidth to simulcast it in HD@MPEG-2 on another channel as well as sending 24 out as SD, so they chose to send 2 out only as SD and use sub-full-HD for 24.

Therefore I feel sure ABC must use H264 for HD on their simulcast of the primary channel. For more but not yet complete details see:

https://en.wikipedia.org/wiki/List_of_d ... oadcasters

where you will see that how full the HD will be on channel 20 is "TBA"! 24 will revert to SD.
I suspect all Australian TV will be H264 before long, so resolutions may improve then.

As for SBS, I think their HD channel 30 is MPEG-4 and 1080i but only 1440 wide, to be displayed as 1920.

I DO like the mythtv E edit function for top, tail and ad-ectomies, and hope I will be able to use it on HD as well, hence all this discussion. I am assuming that as the station has compressed the content with MPEG-4 it will not need (indeed will not allow of) any further compression, so I can simply accept the transcode output and file it. It would be nice to skip handbraking, great prog though that is! I have some limited experience of the nice mkvmerge too, but would prefer to do all the editing using mythtv's E! And I'm happy to cut at keyframes.

[halucigenia]

Mythtranscode using the fifo and honorcutlist options and piping the result to ffmpeg would be much simpler than the above code and using "ffmpeg to create a separate file for each cut in the list"
I find that this method cuts more accurately than relying on calculating the cuts and sending them to another program to cut with
I have tried various methods with ffmpeg and mkvmerge but these never seem to cut in the right place for me for UK HD tv.

See:-https://www.mythtv.org/wiki/Mythtransco ... ath_Option
And
https://www.mythtv.org/wiki/Example_Scr ... fodir_mode

[njsharp]

Well! I have just had some excellent results with Fire Down Below which was broadcast on Australian Station 9's channel 90 recently. My work was all manual, but might be of use to those with better scripting skills who would like to automate something.

I first use mythtv E edit function to cut (by key frame) top, tail and ads. Gee I hate commercials! You get to see all the red bits - 8 interruptions of a 1h40min movie. Yuk. I'm only doing this work on commercial channels practicing for ABCs HD channel due about mid 2016! End of rant.

The recording was 2h18m29s at 4.6Gb.
I then used myth E edit to mark up top, ads, tail, but did not transcode as we know it doesn't work. I just like that editor! Then I did this in mysql mythconverg:

select mark*2 from recordedmarkup where chanid="2090" and starttime like "2016-03-18%" order by mark;
and edited the result into this:
14138-49330,+59498-82362,+93530-123162,+130858-171850,+179534-214318,+221958-259270,+266916-296540,+304716-342660,+350332-382794
which I put into mkvmerge with split mode: "split by parts based on frame/field numbers"

Result: perfect new mkv file 2.6Gb, though surprisingly 50fps whereas recording was 25. I suspect that's to do with interlacing, as is the *2 I had to use in the mysql command, seemingly to convert from frames to fields. I don't deeply understand all that, but it works!

When ABC starts its real HD channel, I think I would not bother with mythtv E function, instead simply play the original recording with VLC, note down the actual program start and end times, and use mkvmerge's split by parts based on timecodes. A bit easier and no need to run handbrake as it's aleady mpeg-4.

Hope this helps. Yes, I've skipped some learning, and I hope you'll forgive I don't explain stuff like mysql and the mythconverg database, or mkvmerge.

You know what? Looking at the original recording, when it transitions from movie to ads, I'm wondering whether the movie WAS actually 1920x1080 before it was sent to air? The ads seem a lot crisper! Sigh ...

[njsharp]

I also trialled using VLC to spot the timecodes at each break point, but they are not quite as accurate as using mythtv E function. The latter allows keyframe, which is approx 1/2sec each, whereas VLC reports in seconds. We get TITANIC on 2016-09-22 so I might have a go at that using the above note method, which also has the merit that the computer records the break points rather than me, so is certain to be accurate!

[skinah]

Thanks for the heads up as I did notice that mkvmerge in the man pages said it could do cuts. When h264cut script was written I believe mkvmerge did not have that feature. I'll have a play when I get some time, as my script is working well as it is.

The other option that halucigenia suggested may be a good way to go by using FIFO mode on mythtranscode to create a split audio and video file and then sending that to handbrake directly.
I find a 30 min TV Show is around 600mb with losseless cutting the mp4 recording, if I then handbrake it down with some video loss I can get 160mb and still have surround sound.

Give my script a go as it is fully automated and gives you a friendly named file automatically.