SOLVED (ish): Pi3/MythLight/Stretch 1080i MPEG-2 OTA ATSC Playback Stutters

[jac1d]

Hello everyone,

First a hat tip to Issac and the MythTV team. I realized I have been enjoying MythTV for at least 15 years now.

MY ISSUE: 1080i broadcast sources (stable, very few errors) do not play back smoothly on a Pi3 with MythLight.

PI HARDWARE:
pi@raspberrypi:~ $ awk '/^Revision/ {sub("^1000", "", $3); print $3}' /proc/cpuinfo
a02082

Which Google tells me is:
a02082 Q1 2016 3 Model B 1.2 1 GB (Mfg by Sony)

GPU MEMORY

I've tried 384, 320 and 256. It is now back at 256MB as none of it had any measurable affect.

From /boot/config.txt:
gpu_mem=256

BACKGROUND:

I recently moved and my .29 combined backend/frontend is no longer in the living room so I decided to set up a Raspberry Pi 3 frontend. It went surprisingly well and came together quickly using the MythLight packages referenced on the wiki/MythTV_Light

AUDIO IS PASS-THRU

I have my AC3 passing through to my soundbar over an optical SPDIF cable, and after a bit of tweaking all is well there. The Pi is connected over 100 Mbps Full Duplex ethernet to the closet where the backend is located.

MPEG-2 KEY IS INSTALLED AND CONFIRMED

I have purchased, installed and confirmed the MPEG-2 playback key.

pi@raspberrypi:~ $ !v
vcgencmd codec_enabled MPG2
MPG2=enabled

USING OPENMAX HIGH PROFILE

I have tried both OPENMAX Normal and OPENMAX High. It makes no difference on my system.

GENERAL OPERATION OF MENUS ETC IS OK - PLAYBACK STATS ARE NOT RIGHT

General operation and navigation of the system is fine, but playback is problematic. I only have OTA stations and one of them is 720p. Playback of 720p is generally smooth and occasionally it falls behind 60FPS but usually only to 58 and then immediately recoveres (using the Menu -> Playback -> Playback Stats option when watching a recording). I would say it is totally normal and acceptable.

1080i SOURCES DO NOT PLAY SMOOTHLY

All of my other source channels are in 1080i30 MPEG-2. Looking at the stats during playback, the FPS is NOT reaching 30 very often. Occasionally it meets or exceeds 30 but generally it is running in the 23-25 fps range and results in very jerky video due to the dropped frames.

RUNNING RASPBIAN STRETCH FROM STOCK NOOBS NETWORK INSTALL FROM LAST TWO WEEKS

I am running:
pi@raspberrypi:~ $ cat /etc/issue
Raspbian GNU/Linux 9 \n \l

From a stock Noobs network install.

pi@raspberrypi:~ $ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description: Raspbian GNU/Linux 9.8 (stretch)
Release: 9.8
Codename: stretch

KERNEL IS UPDATED TO 4.19 VIA RPI-UPDATE

I have updated the firmware and kernel with rpi-update per the MythLight/MythTV on Pi wiki:

(I did also try it on a fresh install without using rpi-update and still had the same issue with the original kernel and firmware)

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.19.30-v7+ #1209 SMP Tue Mar 26 13:14:20 GMT 2019 armv7l GNU/Linux

PERFORMANCE GOVERNOR IS SET

The performance governor is set:

pi@raspberrypi:~ $ for GOV in /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_governor; do cat $GOV; done
performance
performance
performance
performance

THE TV & CEC

The TV is a 2013 vintage Sony LCD and the output to the TV is 1080p. I have tried dropping the output resolution of the Pi to 720p60 but it makes no difference in the playback of 1080i content. The TV correctly reports the source as 720p but the dropped frames issue is identical.

I am using CEC and the stock Sony remote to control it (after programming a few of the oddly named Sony button press results in the Myth keybindings config). Works fine.

The soundbar is a Vizio 2.1 system connected over optical SPDIF and works fine with stereo or AC3 pass-through.

I HAVE TRIED VARIOUS ALSA SOUND CONFIGS AND NULL

I went back to stereo output via Also to the soundbar over optical. It made no difference in frame rate (wondering if some kind of audio processing delay was causing the video to fall behind).

I used the NULL sound device to see if removing audio would make a difference, there was no change, same 24-26 FPS performance on 1080i playback.

I have now re-enabled my AC3 pass-through. I do not believe audio is implicated due to the above tests.

KODI & SANDISK CARD

I have not installed Kodi to try it, but I have read of multiple other people reporting 1080i playback issues on Raspbian with Myth that can use a Kodi install and play the same source files without issue. I could do this but I do not believe this is a hardware issue. I am using a Canakit Pi3 with the case and heatsinks installed and a Sumsung Evo SD card and power supply they provided. All looks good. I did try a separate Sandisk high performance card when I did the clean install and it made no difference.

THE STATS DON'T LOOK RIGHT TO ME - POSSIBLY A FEW CLUES

Looking at the screen when the playback stats are up, they don't look right. I will try and take some photos on my phone and include them later today.

When watching 720p60 content the buffer size is 8MB and runs 50% full. 720p content is generally stable and smooth.

When watching 1080i30 content, the buffer size reports as 16MB and reports as being 99% free (if I am reading it correctly). Either that or it is 99% full, but I believe the wording indicates the buffer is free.

The storage to buffer throughput for 1080i30 reports as 45-50Mbps. The buffer to decoder throughput is in the 8-14 Mbps range. So the data appears to be arriving at the Pi just fine, with a few seconds in reserve, but the buffer to deoder throughput looks suspect to me as a video knowledgeable person (but not developer). I am in the Vancouver BC market and out OTA channels do not carry sub channels. I know the bit rates can run up to 17 Mbps for a single program on the video PID which makes me think the data isn't getting out of the buffer and in to the decoder fast enough. Just a hunch.

I am also aware that the 24-25 FPS playback rate range that I am seeing is very close to european broadcast frame rates (1080i25 being the norm IIRC). Being that the Pi comes from Europe I'm curious if someone has accidentally hard coded something in relation to 1080i25 or isn't properly detecting 1080i30 content. Just a shot in the dark, but it made me raise an eyebrow.

I am also aware people on earlier versions of Raspbian did not have this issue, which again makes me suspicious that something has changed (supported by the reports Kodi on OpenELEC and others can play the same recordings [but not yet shown by me, TBD]).

If anyone can provide any suggestions on where to go from here, I would greatly appreciate it.

I'll be prompt and as thorough as I can in answering background questions and trying things.

Thanks,

-Jeff

[pgbennett]

I assume you have checked everything mentioned in https://www.mythtv.org/wiki/Raspberry_P ... erformance

I am watching 1080i content every day on a raspberry pi 2 without a problem.

What is the exact thing you see? Does the sound get interrupted or id the movement visibly jerky or both?

Could the program be a British show or a movie? Perhaps it is being broadcast at 25fps?

The governor does not need to be set for every cpu, just cpu0. Probably others are set at the same time automatically. Please check it by running
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
and see that it responds with "performance".

[jac1d]

Thanks for the response.

I assume you have checked everything mentioned in https://www.mythtv.org/wiki/Raspberry_P ... erformance

Yes, I have done everything in the Wiki.

I am watching 1080i content every day on a raspberry pi 2 without a problem.

I understand, I noted that in your earlier posts. And thanks for preparing the MythLight packages, I appreciate it.

Unfortunately, I am not using a Pi2. I may have one here in my box of older cables and electronics and if I do, I'll pull it out and try it. I believe, based on my research, that it may work fine. I think there were various changes with the Pi3 and I've not found anyone successfully reporting in playing 1080i30 North American programs on the Pi3. The only people who did report issues have wandered off to other methods like Kodi/xELEC builds which apparently don't exhibit the issue. However, I'm a decade plus Myth user (as is the household) and we'd like the regular Mythfrontend experience.

What is the exact thing you see? Does the sound get interrupted or id the movement visibly jerky or both?

My apologies if I wasn't clear enough in my description. I see the effect of frames missing. The video and audio jerk/skip due to the missing data (or late data potentially). At times the frame rate drops down in to the very low 20s and it is totally unwatchable. At 25fps it is uncomfortable and irritating but can be tolerated for some programming. The real issue is it swings wildly from the low 20s to briefly almost 30 every now and then.

It is possible, although I can't see a vector for it, that there are A/V synch issues related to the MPEG clock that are causing the decoder to fail to be able to reassemble the timed video correctly and it has to discard either video or audio frames depending on the nature of the loss. Does default Raspbian have NTP set up? Just wondering about potential clock drift issues in MPEG clock recovery.

Could the program be a British show or a movie? Perhaps it is being broadcast at 25fps?

No, the content is not British. Canada follows the ATSC standard as used in the USA and broadcasts in 1080i30 or 720p60 for all content. It would be frame rate converted from 25 to 30 prior to being broadcast in Canada if the source was in 25 fps for some reason.

The governor does not need to be set for every cpu, just cpu0. Probably others are set at the same time automatically. Please check it by running
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
and see that it responds with "performance".

I confirmed that setting CPU0 only does not cause the other CPUs to report performance, they still reported the default. I am not aware whether setting CPU0 overrides the others due to kernel logic or not, even if the /sys entry still reports the default setting.

Is there any downside in setting all CPUs to performance? Does Myth move itself to CPU0 in all instances? If not, how could we be sure it benefits from the performance change to CPU0?

Per the screenshots I posted, the CPU loads from the Playback stats on the OSD suggest the CPU loads on all covers are mild during the frame drop condition, so I don't believe CPU load is implicated, but I'm not if the refresh cycle there would catch momentary spikes.

-Jeff

[jac1d]

I could grab a 60s sample and figure out how to share it, maybe via Dropbox, if that would be helpful.

-Jeff

[jac1d]

I appear to have isolated the cause to a specific area and I have been able to eliminate the problem. However, there appear to be multiple variables and despite about 45 minutes testing various combinations I haven't completely isolated the issue.

I do believe there may be a situation where some or a combination of variables are not savings and/or applying - resulting in a situation where the settings shown in Setup from the Mythfrontend may not reflect the settings in effect, either due to a bug in Myth or a bug inside the firmware of the Videocore of the Pi.

I will post more detail later, but a combination/sequence of disabling the root prioritization option on the Video -> Playback menu, combined with the audio buffer option directly below that, and combined with cycling from OpenMax High to Normal or vice-versa, seems to ultimately do the trick.

I say ultimately because once I zeroed in on the fact something I had done corrected it, I attempted to back track and re-create the problem. Re-creating it wasn't a simple A/B situation but I was able, by reversing those options in various combinations, to recreate the problem. Then when I tried to determine the exact fix, it got muddy. Ultimately disabling both of those options and switching from High to Normal or Normal to High seems to work but I can't yet offer an exact pathway.

I am suspicious that there may be a bug at play where the settings aren't always saving/applying, as I said above, either in Myth or perhaps the firmware isn't respecting a settings change if applicable. The fact I couldn't get a clear example or combination that was definitive in creating and fixing the issue was highly unusual.

I was able to have smooth 30fps playback under both OpenMax High and Normal settings - but only after I "got it working" each time. If I didn't do the right combination of enable/disable the root prioritization (and it is important that I state that I DO NOT run as root, but as the default user: pi) and the extra audio buffering (my apologies, I don't have the options available to me at the moment to give exact names, but it is immediately below the process prioritization option) then changing profiles from OpenMax Normal/High didn't fix it. However, once I got it stable, I appeared to be able to use either profile successfully.

Separately, and I now believe (to be proven) this to be unnecessary, I disabled the new default time synch controls and added NTP and pointed it at the stratum 2 servers operated by the National Research Council, which reflect the stratum 1 NRC atomic clock reference. So my date and time are now super accurate but I don't think it ended up having any bearing on the issue. I believe the three options above are the implicated items.

I hope this helps the next person who runs across this, or helps the Myth Devs identify the root cause. If I have time I will try and do more detailed and methodical testing, but kids start school again tomorrow and I'm still unpacking at a new place, so I can't say when I will be able to look again in detail. Either way, much happier with the performance now.

-Jeff

[pgbennett]

Regarding pi2 vs pi3: I am watching daily on a pi2 but I have a pi3 as well that I use for development, and it runs fine on there as well.

Regarding the governor, on my system setting cpu0 sets them all, see example below.

Code: Select all

pi@eagle2:~ $ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
ondemand
pi@eagle2:~ $ cat /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
ondemand
pi@eagle2:~ $ cat /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor
ondemand
pi@eagle2:~ $ echo "performance" |sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance
pi@eagle2:~ $ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance
pi@eagle2:~ $ cat /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
performance
pi@eagle2:~ $ cat /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor
performance

There are also a couple of things that are not raspberry pi specific that could cause your jerkiness:

Video -> playback -> advanced playback settings -> Audio read ahead. Try increasing this.

If the problem only exists with LiveTV, Try pausing for a few seconds and then unpausing.

[pgbennett]

The problem you reported, that it sometimes works after a lot of messing around, is the symptom expected if the governor is not set to performance. This had me confused for some time before I discovered the governor setting.

[jac1d]

Thanks for the responses, Peter. I really do apprecaite all your work on making this available to the community.

The problem you reported, that it sometimes works after a lot of messing around, is the symptom expected if the governor is not set to performance. This had me confused for some time before I discovered the governor setting.

Yes, it makes sense it would be the governor, but I created a separate user to login and check the setting without triggering it from the .profile where it is set for user pi before mythfrontend runs, and it is definitely being set correctly during the autostart process at boot.

Regarding pi2 vs pi3: I am watching daily on a pi2 but I have a pi3 as well that I use for development, and it runs fine on there as well.

I did locate my Pi2 so will try and find some time this week to swap over the SD card (and buy another MPEG-2 license) and try it there as well just for the sake of documenting the background on the issue.

Regarding the governor, on my system setting cpu0 sets them all, see example below.

Code: Select all

pi@eagle2:~ $ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
ondemand
pi@eagle2:~ $ cat /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
ondemand
pi@eagle2:~ $ cat /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor
ondemand
pi@eagle2:~ $ echo "performance" |sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance
pi@eagle2:~ $ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance
pi@eagle2:~ $ cat /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
performance
pi@eagle2:~ $ cat /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor
performance

There are also a couple of things that are not raspberry pi specific that could cause your jerkiness:

Video -> playback -> advanced playback settings -> Audio read ahead. Try increasing this.

If the problem only exists with LiveTV, Try pausing for a few seconds and then unpausing.

Good to know about the governor setting, as noted in my post, I had a hunch this might be the case. I thought I recalled seeing different behaviour but I will re-do it on my end by hand and watch what happens. I wouldn't expect it to be different than yours, just trying to check that off the list.

I will definitely look at the audio read ahead setting. That does sound like what is required. I was wondering where that was. Is there a mindmap hierarchy document of the Myth settings available anywhere? I find things aren't always grouped in ways I would expect and I routinely notice something as I am passing by it on the way to something else, but then have trouble finding my way back to the noticed setting if I need it in the future.

I should note that the Vancouver OTA channels do not have sub-channels, whereas most US stations do. As a result, it is quite normal to see video bitrates run up to 16 Mbps or even 17Mbps. Audio, for AC3 5.1 is fixed at 384. But as a result the ratio of video to audio packets in the TS is quite high compared to a US market where there might be 1 HD and 1,2 or even 3 or more SD channels. It sounds like a dynamic buffer read ahead model would address that, but that would be a more core change I suspect.

I'll come back with more updates when time allows. Thanks again.

-Jeff

PS - I forgot to mention this is with both recordings and live TV. I just induced the problem now by switching from OpenMax Normal to OpenMax High Quality with the prioritization and extra audio buffering options unchecked. I'm testing now with moving back to normal to see if I can isolate things further. Going back to Normal on it's own did NOT fix the issue.

** I should note I am on a .29 back end and I do NOT appear to have the "Video -> playback -> advanced playback settings -> Audio read ahead. Try increasing this." option, only the Extra Audio Buffering option. Was that introduced in .30? **

[jac1d]

Still not a clear answer but what does work well is:

Playback Profile -> OpenMAX Normal
Enable realtime priority threads -> Unchecked/Disabled (default is enabled)
Extra audio buffering -> Unchecked/Disabled (default is enabled)

From the descriptions listed here: https://www.mythtv.org/wiki/Configuring ... d#Playback

So on the Pi 3B that setup now smoothly plays high bitrate MPEG-2 OTA ATSC video as received on an antenna.

-Jeff

[pgbennett]

One warning - swapping an SD card from a pi3 to a pi2 may not work. I have had problems with that.

The Video -> playback -> advanced playback settings -> Audio read ahead option is new. I am not sure in what version it was introduced, but it replaced the "extra audio buffer" option. Extra audio buffer simply set a fixed value (I think 100), the new option gives more control over the audio read ahead/audio buffering. You can tell if increasing it (in the next version) would help by adjusting audio sync to a positive value and see if that corrects the stuttering. This is not a solution because then lip sync will be off but it will tell you if the audio offset is causing your problem.

As far as I know, realtime priority threads do not work and the setting has no effect.

It is surprising if disabling extra audio buffering helps. Perhaps your system is very low on memory. Have you set a very high value for gpu memory? I recommend 256 MB. Do you have other things running on the raspberry pi?

[jac1d]

It is surprising if disabling extra audio buffering helps. Perhaps your system is very low on memory. Have you set a very high value for gpu memory? I recommend 256 MB. Do you have other things running on the raspberry pi?

Nothing else running on it that isn't running on a stock Raspbian install with a desktop. I just added the autostart for mythfrontend. I did add NTP during this debugging but other than that, it's sole function is as a frontend so I don't believe load should be an issue. The CPU percentages in the playback screenshots suggest 25-30% per core which seems reasonable.

I played with other memory options for GPU but ended up back at 256MB as 320MB and 384MB didn't seem to make any difference.

-Jeff

[jac1d]

I am almost done my first draft of the detailed install and tweaks for MythLight on the Pi 3 for version .30. Since the options changed from those referenced above, I wanted to mention that I had to turn my audio read ahead buffer level under Setup > Video > Playback > Advanced up to 1000 ms in order to remove the cut outs in the audio. I did most of my testing with stereo and then configured my AC3 pass through so there may be some scope to adjust in here, but wanted to mention that is when everything stabilized as the option has changed from previous versions of the frontend.

-Jeff

[jac1d]

I'm back with another update. Once I upgraded to .30 of the Myth tree the issues returned.

I have -largely- mitigated them by switching the de-interlacer in the playback profile to HW Fast. I did this my creating a new playback profile and selecting it. Still, with 1080i HD OTA content I get a very occasional sound blip but otherwise with this change and using about an 800ms audio buffer, things are smooth on my Pi3.

Overall long term stability is still an open question as I have seen a few unexplained system hangs but I am still investigating those.

-Jeff