Page 1 of 1

[Solved] Two HDHomeruns stop recording at the same time

Posted: Thu Apr 08, 2021 8:23 pm
by squish102
Hi, I have been running mythtv since 2007 and I have a problem recently that I cannot figure out how to correct.

I have 2 hdhomeruns recording and they will stop recording, both at the same time with the error below. The hdhomeruns will still show that they are tuned in but mythtv has lost contact. Later I will receive the error that "ERROR: lock no longer held" in mythbackend and the next few recordings will all fail.

I also set up a job every 2 seconds to get the status from the HDHomeruns and they have about a 30 second no response right at the time of the failure. (log below) I don't believe it is a network issue as I don't see a drop of connectivity to other devices on the same side of the switch (I have a wifi access point also connected)

Anyone have any idea what could be causing this or what else to look for?

Thanks


Version

Code: Select all

MythTV Version : v31.0+fixes.202104012002.563a05b7a8~ubuntu20.04.1
MythTV Branch : fixes/31
Network Protocol : 91
Library API : 31.20200101-1
QT Version : 5.12.8
Options compiled in:
 linux profile use_hidesyms using_alsa using_oss using_pulse using_pulseoutput using_backend using_bindings_perl using_bindings_python using_bindings_php using_dvb using_firewire using_frontend using_hdhomerun using_vbox using_ceton using_hdpvr using_ivtv using_joystick_menu using_libcec using_libcrypto using_gnutls using_libdns_sd using_libfftw3 using_libxml2 using_lirc using_mheg using_opengl using_egl using_qtwebkit using_qtscript using_qtdbus using_taglib using_v4l2 using_x11 using_libbluray_external using_xrandr using_profiletype using_systemd_notify using_systemd_journal using_bindings_perl using_bindings_python using_bindings_php using_freetype2 using_mythtranscode using_opengl using_egl using_vaapi using_nvdec using_vdpau using_ffmpeg_threads using_mheg using_libass using_libxml2 using_libmp3lame
 
HDHOMERUN 1
Hardware Model HDHR5-4US
Firmware Version 20210224

HDHOMERUN 2
Hardware Model HDHR5-2US
Firmware Version 20210224

Error message I see in mythbackend when recording:

Code: Select all

Apr  7 21:41:49 mediaserver mythbackend: mythbackend[3351]: I ProcessRequest mainserver.cpp:1770 (HandleAnnounce) MainServer: adding: mediaserver(55e69d2c1310) as a client (events: 0)
Apr  7 21:41:49 mediaserver mythbackend: mythbackend[3351]: I ProcessRequest mainserver.cpp:1768 (HandleAnnounce) MainServer: MainServer::ANN Monitor
Apr  7 21:41:49 mediaserver mythbackend: mythbackend[3351]: I ProcessRequest mainserver.cpp:1770 (HandleAnnounce) MainServer: adding: mediaserver(55e69d11e8a0) as a client (events: 1)
Apr  7 21:41:51 mediaserver mythbackend: mythbackend[3351]: I MythSocketThread(91) mainserver.cpp:7858 (connectionClosed) Monitor sock(55e69d2c1310) 'mediaserver' disconnected
Apr  7 21:41:51 mediaserver mythbackend: mythbackend[3351]: I MythSocketThread(94) mainserver.cpp:7858 (connectionClosed) Monitor sock(55e69d11e8a0) 'mediaserver' disconnected
Apr  7 21:47:15 mediaserver mythbackend: mythbackend[3351]: E HDHRStreamHandler recorders/hdhrstreamhandler.cpp:470 (TunerSet) HDHRSH[14](1061B172): Set /tuner0/filter to '0x0000-0x0001 0x0030-0x0031 0x0034-0x0035 0x0040 0x0050 0x0060 0x1ffb-0x1ffc' request failed#012#011#011#011eno: Resource temporarily unavailable (11)
Apr  7 21:47:15 mediaserver mythbackend: mythbackend[3351]: E HDHRStreamHandler recorders/hdhrstreamhandler.cpp:470 (TunerSet) HDHRSH[13](10715A68): Set /tuner0/filter to '0x0000-0x0001 0x0030-0x0031 0x0034-0x0035 0x0040 0x0050 0x0060 0x0070 0x1ffb-0x1ffc' request failed#012#011#011#011eno: Resource temporarily unavailable (11)
Log from HDHomerun

Code: Select all

19700101-00:00:00 System: reset reason = power on
19700101-00:00:04 System: network link 100f
19700101-00:00:05 System: ip address obtained: 192.168.2.50 / 255.255.255.0
20210404-17:41:52 System: time changed from Thu Jan  1 00:00:21 1970 to Sun Apr  4 17:41:52 2021
20210405-00:00:01 Tuner: tuner0 streaming rtp to 192.168.2.122:49241
20210405-00:00:02 Tuner: tuner0 rtp stream ended (stop request)
20210405-00:00:02 Tuner: tuner0 streaming rtp to 192.168.2.122:49241
20210405-00:16:11 Tuner: tuner0 rtp stream ended (remote closed)
20210407-01:00:00 Tuner: tuner0 streaming rtp to 192.168.2.122:35947
20210407-01:00:00 Tuner: tuner0 rtp stream ended (stop request)
20210407-01:00:00 Tuner: tuner0 streaming rtp to 192.168.2.122:35947
20210407-02:59:59 Tuner: tuner0 rtp stream ended (stop request)
20210408-01:00:01 Tuner: tuner0 streaming rtp to 192.168.2.122:52415
20210408-01:00:01 Tuner: tuner0 rtp stream ended (stop request)
20210408-01:00:03 Tuner: tuner0 streaming rtp to 192.168.2.122:52415
20210408-01:47:23 Tuner: tuner0 rtp stream ended (remote closed)
My logging from the mythbackend server querying the hdhomerun (You can see the missing values)

Code: Select all

Wed 07 Apr 2021 09:47:07 PM EDT
tun: ch=8vsb:533000000 lock=8vsb:533000000 ss=100 snq=100 seq=100 dbg=-3700/3566
dev: bps=19636224 resync=0 overflow=0
ts:  bps=11410848 te=50 crc=0
net: pps=1084 err=0 stop=0

Wed 07 Apr 2021 09:47:09 PM EDT
tun: ch=8vsb:533000000 lock=8vsb:533000000 ss=100 snq=100 seq=100 dbg=-3725/3566
dev: bps=19315872 resync=0 overflow=0
ts:  bps=10819776 te=50 crc=0
net: pps=1028 err=0 stop=0

Wed 07 Apr 2021 09:47:11 PM EDT
Wed 07 Apr 2021 09:47:13 PM EDT
Wed 07 Apr 2021 09:47:15 PM EDT
Wed 07 Apr 2021 09:47:18 PM EDT
Wed 07 Apr 2021 09:47:20 PM EDT
Wed 07 Apr 2021 09:47:23 PM EDT
Wed 07 Apr 2021 09:47:25 PM EDT
tun: ch=8vsb:533000000 lock=8vsb:533000000 ss=100 snq=100 seq=100 dbg=-3700/3563
dev: bps=19636224 resync=0 overflow=0
ts:  bps=10705472 te=50 crc=0
net: pps=0 err=0 stop=5

Wed 07 Apr 2021 09:47:27 PM EDT
tun: ch=8vsb:533000000 lock=8vsb:533000000 ss=100 snq=100 seq=100 dbg=-3700/3566
dev: bps=19636224 resync=0 overflow=0
ts:  bps=11695104 te=50 crc=0
net: pps=0 err=0 stop=5

Re: Two HDHomeruns stop recording at the same time

Posted: Thu Apr 08, 2021 9:28 pm
by kmdewaal
This error is usually caused by a failing power supply. That two of them stop at the same time is strange however.
Does this also happen when there is only one HDHR in use?

Re: Two HDHomeruns stop recording at the same time

Posted: Thu Apr 08, 2021 9:55 pm
by squish102
I also thought about the failing power supply but as you said, they both had the problem. Logs on both HD's show the same. I also thought I would get that in the HDhomerun log file. This seems to happen every night now (not the same time I don't think), so tonight I have added a ping to my script that gets HD status every 2 seconds and see if the ping continues during that time.

To test with one, I will have to find a night that my wife is not recording 3-4 things at the same time.

Re: Two HDHomeruns stop recording at the same time

Posted: Fri Apr 09, 2021 2:47 pm
by squish102
I had the same problem last night, but I did not have all my logging set up correctly. I did notice that it was about 5 minutes earlier that the problem occurred. Again, on both HDHomeruns at the same second. Mythbackend logging was not configured correctly, so I did not capture the logs. I have now started 12 hours of recording with one tuner and see if the problem happens (might be only a night time issue).

Re: Two HDHomeruns stop recording at the same time

Posted: Fri Apr 09, 2021 4:42 pm
by kmdewaal
That the two HDHRs fail at exactly the same moment rules out that it are the individual HDHR power supplies that give the problems.
However, I still think it is power related. It could be the power supply of the switch for instance.
It could also be a disturbance on your power lines, e.g. by a heavy appliance being switched on or so, and not all power supplies are equally good at handling this.
If possible, try with another switch. Or connect the switch and the HDHR to another wall socket so that it uses other wiring to another fuse in your house. A bit further down the road are netfilters and UPS boxes but I have never used those myself.

Re: Two HDHomeruns stop recording at the same time

Posted: Fri Apr 09, 2021 8:42 pm
by squish102
The HDHRs are in the attic with the switch, as you mentioned it could be the switch. Ordered a new switch, delivery sunday. My script that hits the HDHR's API every 2 seconds, I added a ping command too. I hope to see failed pings when this happens next time. So far 6 hours using one HDHR recording without any problems. I have it set for another 6 hours of recording.

Re: Two HDHomeruns stop recording at the same time

Posted: Wed Apr 14, 2021 2:28 am
by squish102
Replacing the switch did not fix the problem. The ping also times out to both tvtuners when the tuners both become unresponsive for 30 seconds. mythtv shows "(connectionClosed) Monitor sock(563f3719f610) 'mediaserver' disconnected"

I don't know what to check next. I may try a fresh installation of mythtv to try resolve this issue.

Re: Two HDHomeruns stop recording at the same time

Posted: Wed Apr 14, 2021 9:17 am
by mtvf2894
Long shot but check that a software or hardware firewall rule is not turning on mistaking the traffic to the HDHomerun devices as a DOS attack?
The firmware is relatively new did the problem possibly start when you upgraded the firmware on the HDHomeruns?
Could the problem be specific to certain channels perhaps they have changed their transmission parameters?
Has the operating system been updated and it has caused the problem (Linux new kernel, new firewall rule)?

Re: Two HDHomeruns stop recording at the same time

Posted: Fri Apr 16, 2021 6:51 am
by kmdewaal
Another long shot, could it be disk space? Check the free space on your root partition. Is there something happening on the backend when the connection drops? For example, if you do an automatic database backup each night and this fails because of lack of space anything can happen.

Re: [Solved] Two HDHomeruns stop recording at the same time

Posted: Fri Apr 23, 2021 8:19 pm
by squish102
I think I have found the culprit after a lot of digging through logs. Even my pings timed out for about 30 seconds when this went down.

I was losing connectivity to the network from the server (host) This was running in a VM and the host Server had this in the logs at random times during the day but it manifested itself in the streaming.

kernel: e1000e 0000:00:1f.6 eth0: Detected Hardware Unit Hang:

This recovered 30 seconds later but my stream was messed up. HDHomeruns were still tuned but mythtv was throwing errors. It would recover about 2 hours later.