Is there any way to troubleshoot why my MythTV goes offline so often?
Every so often, it just completely fails, resulting in many lost recordings (that is, scheduled recordings that don't record) with the only hint being the words "recorder failed" where the recording is highlighted in red.
I have tried rebooting but it generally won't work until the recordings that are absolutely necessary have been completely missed never to be captured again. Then, after a few more reboots, everything returns to 'normal' and there are no more problems for a few weeks or a month or two.
Also, I have heard mention of logs, but I cannot seem to figure out an easy way to find and view them.
EDIT: I just realized that the last time it happened was exactly 6 weeks ago, almost to the minute.
Thanks
Brian
recorder failed
Moderator: Forum Moderators
Re: recorder failed
Yes, there ways to troubleshoot, but precisely how depends on your mythtv setup.
Need more information about your mythtv setup/configuration:
What is the version of the Operating System you are running mythtv on ?
What version of mythtv are you running ?
What type of tuners are you using ?
If you are running mythtv on mythbuntu there are logs in /var/log/mythtv/, the most important being mythbackend.log
Mike
Need more information about your mythtv setup/configuration:
What is the version of the Operating System you are running mythtv on ?
What version of mythtv are you running ?
What type of tuners are you using ?
If you are running mythtv on mythbuntu there are logs in /var/log/mythtv/, the most important being mythbackend.log
Mike
Re: recorder failed
One Backend/Frontend combo (jack)
and a remote Frontend (dean)
Both running Mythbuntu 16.04 & fixes/0.28
Tuners are HDHomeRun HDHR3 & HDHR4 Dual tuners, a total of 8 units (16 tuners).
I was in the midst of watching a recording (in progress, if that means anything) on the remote frontend when the frontend I was watching suddenly hung. I could not get it restarted no matter what I tried, so I finally rebooted the machine by the power switch.
I also went to the backend and tried to run the frontend on it but it wouldn't go either. I needed to see if the recording had failed and I don't know a way to do that other than through the frontend (if there is a way, maybe somebody could give directions). I didn't want to kill the backend because there was a recording in progress and several more recordings to start before that one ended. But, seeing as I could not get the frontend to run on either machine, I decided that it would be better to sacrifice the earlier recording in order to get the next ones which are a much higher priority. So, I tried rebooting the backend machine. eventually, the only way I could reboot it was by the power supply switch, as even the front panel switch wouldn't kill it. (EDIT: Incidentally, the recordings in progress did not fail until the hard reboot.)
The backend booted back up, but would not record again for a few hours. Then, as before, everything returned to normal and no recordings have suffered since.
I just cut a snippet from the backend log around the time that the system failed, but it is 17,000 lines (from the time I started watching the recording until the time that I was successful in recording again). How would you like me to proceed?
EDIT: I have found a couple of things in the logs that, while I doubt they have anything to do with the problem at hand, greatly worry me. I have never set any recordings to autoexpire as I like to be in control of what gets deleted, but there are hundreds of lines like the following:
And like this as well:
Brian
and a remote Frontend (dean)
Both running Mythbuntu 16.04 & fixes/0.28
Tuners are HDHomeRun HDHR3 & HDHR4 Dual tuners, a total of 8 units (16 tuners).
I was in the midst of watching a recording (in progress, if that means anything) on the remote frontend when the frontend I was watching suddenly hung. I could not get it restarted no matter what I tried, so I finally rebooted the machine by the power switch.
I also went to the backend and tried to run the frontend on it but it wouldn't go either. I needed to see if the recording had failed and I don't know a way to do that other than through the frontend (if there is a way, maybe somebody could give directions). I didn't want to kill the backend because there was a recording in progress and several more recordings to start before that one ended. But, seeing as I could not get the frontend to run on either machine, I decided that it would be better to sacrifice the earlier recording in order to get the next ones which are a much higher priority. So, I tried rebooting the backend machine. eventually, the only way I could reboot it was by the power supply switch, as even the front panel switch wouldn't kill it. (EDIT: Incidentally, the recordings in progress did not fail until the hard reboot.)
The backend booted back up, but would not record again for a few hours. Then, as before, everything returned to normal and no recordings have suffered since.
I just cut a snippet from the backend log around the time that the system failed, but it is 17,000 lines (from the time I started watching the recording until the time that I was successful in recording again). How would you like me to proceed?
EDIT: I have found a couple of things in the logs that, while I doubt they have anything to do with the problem at hand, greatly worry me. I have never set any recordings to autoexpire as I like to be in control of what gets deleted, but there are hundreds of lines like the following:
Code: Select all
Feb 21 20:32:16 JACK mythbackend: mythbackend[5451]: N Expire autoexpire.cpp:637 (SendDeleteMessages) Expiring 3892 MB for 1041 at 2014-01-18T23:59:00Z => "Wheel of Fortune":"Pet Lovers"Code: Select all
Feb 21 21:21:24 JACK mythbackend: mythbackend[1797]: E CoreContext mainserver.cpp:3061 (DoHandleDeleteRecording) MainServer: ERROR when trying to delete file: GetPlaybackURL/UNABLE/TO/FIND/LOCAL/FILE/ON/JACK/1041_20140206192900.mpg. File doesn't exist. Database metadata will not be removed.
Feb 21 21:21:24 JACK mythbackend: mythbackend[1797]: E CoreContext programinfo.cpp:2616 (GetPlaybackURL) ProgramInfo(1041_20140206185900.mpg): GetPlaybackURL: '1041_20140206185900.mpg' should be local, but it can not be found.Re: recorder failed
The messages from the log are worrying, as they indicate the mythdatabase is inconsistent, you might get a few over a long time, but hundreds is not good as it is showing files that mythtv thinks should exist but don't.
Can you get me some information from the backend machine:
All of the following commands need to be run in terminal.
Output of command df -h
Output of command free -h
Output of command mythbackend --testsched
Mike
Can you get me some information from the backend machine:
All of the following commands need to be run in terminal.
Output of command df -h
Output of command free -h
Output of command mythbackend --testsched
Mike
Re: recorder failed
Output of command df -h:
Output of command free -h:
Output of command mythbackend --testsched:
Brian
Code: Select all
Filesystem Size Used Avail Use% Mounted on
udev 3.4G 0 3.4G 0% /dev
tmpfs 694M 79M 615M 12% /run
/dev/sda2 2.7T 2.5T 79G 97% /
tmpfs 3.4G 6.4M 3.4G 1% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup
dean:/var/lib/mythtv/recordings 3.6T 3.4T 80G 98% /mnt/oldrec
tmpfs 694M 12K 694M 1% /run/user/1000Code: Select all
total used free shared buff/cache available
Mem: 6.8G 1.3G 369M 157M 5.1G 5.0G
Swap: 7.0G 221M 6.7GCode: Select all
2017-02-22 17:14:27.977799 C mythbackend version: fixes/0.28 [v0.28.1-6-g7de03a9] www.mythtv.org
2017-02-22 17:14:27.977826 C Qt version: compile: 5.5.1, runtime: 5.5.1
2017-02-22 17:14:27.977830 N Enabled verbose msgs: general
2017-02-22 17:14:27.977844 N Setting Log Level to LOG_INFO
2017-02-22 17:14:27.988532 I Added logging to the console
2017-02-22 17:14:27.988912 I Setup Interrupt handler
2017-02-22 17:14:27.988922 I Setup Terminated handler
2017-02-22 17:14:27.988929 I Setup Segmentation fault handler
2017-02-22 17:14:27.988939 I Setup Aborted handler
2017-02-22 17:14:27.988947 I Setup Bus error handler
2017-02-22 17:14:27.988954 I Setup Floating point exception handler
2017-02-22 17:14:27.988962 I Setup Illegal instruction handler
2017-02-22 17:14:27.988975 I Setup Real-time signal 0 handler
2017-02-22 17:14:27.988986 I Setup Hangup handler
2017-02-22 17:14:27.989075 N Using runtime prefix = /usr
2017-02-22 17:14:27.989079 N Using configuration directory = /home/brian/.mythtv
2017-02-22 17:14:27.989142 I Assumed character encoding: en_CA.UTF-8
2017-02-22 17:14:27.989392 N Empty LocalHostName.
2017-02-22 17:14:27.989397 I Using localhost value of JACK
2017-02-22 17:14:28.000434 N Setting QT default locale to en_CA
2017-02-22 17:14:28.000445 I Current locale en_CA
2017-02-22 17:14:28.000485 N Reading locale defaults from /usr/share/mythtv//locales/en_ca.xml
2017-02-22 17:14:28.009138 I Loading en_ca translation for module mythfrontend
2017-02-22 17:14:28.010318 I New static DB connectionDataDirectCon
2017-02-22 17:14:28.102019 I Found 1 distinct programid authorities
2017-02-22 17:14:28.626295 I Speculative scheduled 97 items in 0.4 = 0.01 match + 0.20 check + 0.18 place
2017-02-22 17:14:28.626340 I --- print list start ---
2017-02-22 17:14:28.626344 I Title - Subtitle Ch Station Day Start End S I T N Pri
2017-02-22 17:14:28.626424 I Evening News at 6 11_1 CHCH-DT 22 17:59-19:00 1 7 A 7 0/-10
2017-02-22 17:14:28.626451 I Wheel of Fortune - Great American 36_1 CITS-HD 22 18:59-19:30 1 4 A 4 40/-2
2017-02-22 17:14:28.626475 I Wheel of Fortune - Great American 4_1 WIVB-HD 22 18:59-19:30 1 2 A 2 40
2017-02-22 17:14:28.626499 I Jeopardy! (Manual Record) - 2017-0 23_1 WNLO-HD 22 18:59-20:01 1 5 D 5 35/-4
2017-02-22 17:14:28.626522 I Jeopardy! - College Championship: 36_1 CITS-HD 22 19:29-20:01 1 3 A 3 50/-2
2017-02-22 17:14:28.626544 I Jeopardy! - College Championship: 4_1 WIVB-HD 22 19:29-20:01 1 1 A 1 50
2017-02-22 17:14:28.626566 I Criminal Minds - Collision Course 4_1 WIVB-HD 22 20:59-22:01 1 1 A 1 0
2017-02-22 17:14:28.626588 I Jeopardy! - College Championship: 23_1 WNLO-HD 23 13:59-14:31 1 1 A 1 50
2017-02-22 17:14:28.626610 I Who Wants to Be a Millionaire 23_1 WNLO-HD 23 14:29-15:01 1 2 A 2 0
2017-02-22 17:14:28.626631 I Evening News at 6 11_1 CHCH-DT 23 17:59-19:00 1 7 A 7 0/-10
2017-02-22 17:14:28.626654 I Wheel of Fortune - Great American 36_1 CITS-HD 23 18:59-19:30 1 4 A 4 40/-2
2017-02-22 17:14:28.626676 I Wheel of Fortune - Great American 4_1 WIVB-HD 23 18:59-19:30 1 2 A 2 40
2017-02-22 17:14:28.626699 I Jeopardy! (Manual Record) - 2017-0 23_1 WNLO-HD 23 18:59-20:01 1 5 D 5 35/-4
2017-02-22 17:14:28.626721 I Jeopardy! - College Championship: 36_1 CITS-HD 23 19:29-20:01 1 3 A 3 50/-2
2017-02-22 17:14:28.626742 I Jeopardy! - College Championship: 4_1 WIVB-HD 23 19:29-20:01 1 1 A 1 50
2017-02-22 17:14:28.626763 I Jeopardy! - College Championship: 23_1 WNLO-HD 24 13:59-14:31 1 1 A 1 50
2017-02-22 17:14:28.626785 I Who Wants to Be a Millionaire 23_1 WNLO-HD 24 14:29-15:01 1 2 A 2 0
2017-02-22 17:14:28.626806 I Evening News at 6 11_1 CHCH-DT 24 17:59-19:00 1 7 A 7 0/-10
2017-02-22 17:14:28.626828 I Wheel of Fortune - Great American 36_1 CITS-HD 24 18:59-19:30 1 4 A 4 40/-2
2017-02-22 17:14:28.626850 I Wheel of Fortune - Great American 4_1 WIVB-HD 24 18:59-19:30 1 2 A 2 40
2017-02-22 17:14:28.626872 I Jeopardy! (Manual Record) - 2017-0 23_1 WNLO-HD 24 18:59-20:01 1 5 D 5 35/-4
2017-02-22 17:14:28.626893 I Jeopardy! - College Championship: 36_1 CITS-HD 24 19:29-20:01 1 3 A 3 50/-2
2017-02-22 17:14:28.626914 I Jeopardy! - College Championship: 4_1 WIVB-HD 24 19:29-20:01 1 1 A 1 50
2017-02-22 17:14:28.626936 I Canada's Worst Driver - The Checke 9_1 CFTO 25 13:59-15:01 1 1 A 1 0
2017-02-22 17:14:28.626956 I Evening News at 6 11_1 CHCH-DT 25 17:59-18:30 1 1 A 1 0
2017-02-22 17:14:28.626977 I Wheel of Fortune - College Road Tr 36_1 CITS-HD 25 18:59-19:30 1 4 A 4 40/-2
2017-02-22 17:14:28.626998 I Wheel of Fortune - College Road Tr 4_1 WIVB-HD 25 18:59-19:30 1 2 A 2 40
2017-02-22 17:14:28.627019 I Jeopardy! 36_1 CITS-HD 25 19:29-20:01 1 3 A 3 50/-2
2017-02-22 17:14:28.627039 I Jeopardy! 4_1 WIVB-HD 25 19:29-20:01 1 1 A 1 50
2017-02-22 17:14:28.627061 I Canada's Worst Driver - Uphill Bat 9_1 CFTO 26 05:59-07:01 1 1 A 1 0
2017-02-22 17:14:28.627081 I Evening News at 6 11_1 CHCH-DT 26 17:59-18:30 1 1 A 1 0
2017-02-22 17:14:28.627101 I Jeopardy! 23_1 WNLO-HD 27 13:59-14:31 1 1 A 1 50
2017-02-22 17:14:28.627122 I Who Wants to Be a Millionaire 23_1 WNLO-HD 27 14:29-15:01 1 2 A 2 0
2017-02-22 17:14:28.627142 I Evening News at 6 11_1 CHCH-DT 27 17:59-19:00 1 7 A 7 0/-10
2017-02-22 17:14:28.627164 I Wheel of Fortune - Teen Best Frien 36_1 CITS-HD 27 18:59-19:30 1 4 A 4 40/-2
2017-02-22 17:14:28.627185 I Wheel of Fortune - Teen Best Frien 4_1 WIVB-HD 27 18:59-19:30 1 2 A 2 40
2017-02-22 17:14:28.627207 I Jeopardy! (Manual Record) - 2017-0 23_1 WNLO-HD 27 18:59-20:01 1 5 D 5 35/-4
2017-02-22 17:14:28.627227 I Jeopardy! 36_1 CITS-HD 27 19:29-20:01 1 3 A 3 50/-2
2017-02-22 17:14:28.627247 I Jeopardy! 4_1 WIVB-HD 27 19:29-20:01 1 1 A 1 50
2017-02-22 17:14:28.627269 I Murdoch Mysteries - Master Lovecra 5_1 CBLT-DT 27 19:59-21:02 1 9 A 9 0/-12
2017-02-22 17:14:28.627291 I The Voice - Blind Auditions Premie 2_1 WGRZ-HD 27 19:59-22:02 1 7 A 7 0/-10
2017-02-22 17:14:28.627311 I Jeopardy! 23_1 WNLO-HD 28 13:59-14:31 1 1 A 1 50
2017-02-22 17:14:28.627331 I Who Wants to Be a Millionaire 23_1 WNLO-HD 28 14:29-15:01 1 2 A 2 0
2017-02-22 17:14:28.627351 I Evening News at 6 11_1 CHCH-DT 28 17:59-19:00 1 7 A 7 0/-10
2017-02-22 17:14:28.627372 I Just for Laughs: Gags 5_1 CBLT-DT 28 18:59-19:31 1 9 A 9 0/-12
2017-02-22 17:14:28.627394 I Wheel of Fortune - Teen Best Frien 36_1 CITS-HD 28 18:59-19:30 1 4 A 4 40/-2
2017-02-22 17:14:28.627415 I Wheel of Fortune - Teen Best Frien 4_1 WIVB-HD 28 18:59-19:30 1 2 A 2 40
2017-02-22 17:14:28.627437 I Jeopardy! (Manual Record) - 2017-0 23_1 WNLO-HD 28 18:59-20:01 1 5 D 5 35/-4
2017-02-22 17:14:28.627457 I Jeopardy! 36_1 CITS-HD 28 19:29-20:01 1 3 A 3 50/-2
2017-02-22 17:14:28.627476 I Jeopardy! 4_1 WIVB-HD 28 19:29-20:01 1 1 A 1 50
2017-02-22 17:14:28.627497 I The Voice 2_1 WGRZ-HD 28 19:59-21:02 1 7 A 7 0/-10
2017-02-22 17:14:28.627517 I Jeopardy! 23_1 WNLO-HD 01 13:59-14:31 1 1 A 1 50
2017-02-22 17:14:28.627537 I Who Wants to Be a Millionaire 23_1 WNLO-HD 01 14:29-15:01 1 2 A 2 0
2017-02-22 17:14:28.627558 I Evening News at 6 11_1 CHCH-DT 01 17:59-19:00 1 7 A 7 0/-10
2017-02-22 17:14:28.627580 I Wheel of Fortune - Teen Best Frien 36_1 CITS-HD 01 18:59-19:30 1 4 A 4 40/-2
2017-02-22 17:14:28.627601 I Wheel of Fortune - Teen Best Frien 4_1 WIVB-HD 01 18:59-19:30 1 2 A 2 40
2017-02-22 17:14:28.627622 I Jeopardy! (Manual Record) - 2017-0 23_1 WNLO-HD 01 18:59-20:01 1 5 D 5 35/-4
2017-02-22 17:14:28.627642 I Jeopardy! 36_1 CITS-HD 01 19:29-20:01 1 3 A 3 50/-2
2017-02-22 17:14:28.627662 I Jeopardy! 4_1 WIVB-HD 01 19:29-20:01 1 1 A 1 50
2017-02-22 17:14:28.627683 I Criminal Minds - Alpha Male 4_1 WIVB-HD 01 20:59-22:01 1 1 A 1 0
2017-02-22 17:14:28.627703 I Jeopardy! 23_1 WNLO-HD 02 13:59-14:31 1 1 A 1 50
2017-02-22 17:14:28.627723 I Who Wants to Be a Millionaire 23_1 WNLO-HD 02 14:29-15:01 1 2 A 2 0
2017-02-22 17:14:28.627743 I Evening News at 6 11_1 CHCH-DT 02 17:59-19:00 1 7 A 7 0/-10
2017-02-22 17:14:28.627765 I Wheel of Fortune - Teen Best Frien 36_1 CITS-HD 02 18:59-19:30 1 4 A 4 40/-2
2017-02-22 17:14:28.627786 I Wheel of Fortune - Teen Best Frien 4_1 WIVB-HD 02 18:59-19:30 1 2 A 2 40
2017-02-22 17:14:28.627808 I Jeopardy! (Manual Record) - 2017-0 23_1 WNLO-HD 02 18:59-20:01 1 5 D 5 35/-4
2017-02-22 17:14:28.627827 I Jeopardy! 36_1 CITS-HD 02 19:29-20:01 1 3 A 3 50/-2
2017-02-22 17:14:28.627846 I Jeopardy! 4_1 WIVB-HD 02 19:29-20:01 1 1 A 1 50
2017-02-22 17:14:28.627868 I The Voice - Blind Auditions Premie 2_1 WGRZ-HD 02 19:59-21:02 1 7 A 7 0/-10
2017-02-22 17:14:28.627887 I Jeopardy! 23_1 WNLO-HD 03 13:59-14:31 1 1 A 1 50
2017-02-22 17:14:28.627907 I Who Wants to Be a Millionaire 23_1 WNLO-HD 03 14:29-15:01 1 2 A 2 0
2017-02-22 17:14:28.627928 I Evening News at 6 11_1 CHCH-DT 03 17:59-19:00 1 7 A 7 0/-10
2017-02-22 17:14:28.627949 I Wheel of Fortune - Teen Best Frien 36_1 CITS-HD 03 18:59-19:30 1 4 A 4 40/-2
2017-02-22 17:14:28.627970 I Wheel of Fortune - Teen Best Frien 4_1 WIVB-HD 03 18:59-19:30 1 2 A 2 40
2017-02-22 17:14:28.627991 I Jeopardy! (Manual Record) - 2017-0 23_1 WNLO-HD 03 18:59-20:01 1 5 D 5 35/-4
2017-02-22 17:14:28.628011 I Jeopardy! 36_1 CITS-HD 03 19:29-20:01 1 3 A 3 50/-2
2017-02-22 17:14:28.628030 I Jeopardy! 4_1 WIVB-HD 03 19:29-20:01 1 1 A 1 50
2017-02-22 17:14:28.628051 I Canada's Worst Driver 9_1 CFTO 04 13:59-15:01 1 1 A 1 0
2017-02-22 17:14:28.628071 I Evening News at 6 11_1 CHCH-DT 04 17:59-18:30 1 1 A 1 0
2017-02-22 17:14:28.628092 I Wheel of Fortune - Sandals Resorts 36_1 CITS-HD 04 18:59-19:30 1 4 A 4 40/-2
2017-02-22 17:14:28.628113 I Wheel of Fortune - Sandals Resorts 4_1 WIVB-HD 04 18:59-19:30 1 2 A 2 40
2017-02-22 17:14:28.628133 I Jeopardy! 36_1 CITS-HD 04 19:29-20:01 1 3 A 3 50/-2
2017-02-22 17:14:28.628153 I Jeopardy! 4_1 WIVB-HD 04 19:29-20:01 1 1 A 1 50
2017-02-22 17:14:28.628174 I Canada's Worst Driver - Judgment 9_1 CFTO 05 05:59-07:01 1 1 A 1 0
2017-02-22 17:14:28.628194 I Evening News at 6 11_1 CHCH-DT 05 17:59-18:30 1 1 A 1 0
2017-02-22 17:14:28.628213 I Jeopardy! 23_1 WNLO-HD 06 13:59-14:31 1 1 A 1 50
2017-02-22 17:14:28.628233 I Who Wants to Be a Millionaire 23_1 WNLO-HD 06 14:29-15:01 1 2 A 2 0
2017-02-22 17:14:28.628253 I Evening News at 6 11_1 CHCH-DT 06 17:59-19:00 1 7 A 7 0/-10
2017-02-22 17:14:28.628275 I Wheel of Fortune - Bella Italia 36_1 CITS-HD 06 18:59-19:30 1 4 A 4 40/-2
2017-02-22 17:14:28.628296 I Wheel of Fortune - Bella Italia 4_1 WIVB-HD 06 18:59-19:30 1 2 A 2 40
2017-02-22 17:14:28.628318 I Jeopardy! (Manual Record) - 2017-0 23_1 WNLO-HD 06 18:59-20:01 1 5 D 5 35/-4
2017-02-22 17:14:28.628338 I Jeopardy! 36_1 CITS-HD 06 19:29-20:01 1 3 A 3 50/-2
2017-02-22 17:14:28.628357 I Jeopardy! 4_1 WIVB-HD 06 19:29-20:01 1 1 A 1 50
2017-02-22 17:14:28.628378 I The Voice - Blind Auditions, Part 2_1 WGRZ-HD 06 19:59-22:02 1 7 A 7 0/-10
2017-02-22 17:14:28.628398 I Jeopardy! 23_1 WNLO-HD 07 13:59-14:31 1 1 A 1 50
2017-02-22 17:14:28.628418 I Who Wants to Be a Millionaire 23_1 WNLO-HD 07 14:29-15:01 1 2 A 2 0
2017-02-22 17:14:28.628438 I Evening News at 6 11_1 CHCH-DT 07 17:59-19:00 1 1 A 1 0
2017-02-22 17:14:28.628439 I --- print list end ---
Calculating Schedule from database.
Inputs, Card IDs, and Conflict info may be invalid if you have multiple tuners.Re: recorder failed
Brian,
Thanks for the outputs.
The output from mythbackend --testsched is good (I was concerned that the scheduler might be having problems, which can cause mythbackend to become unresponsive, but yours is good).
The output from df -h indicates you are low on disk space, but not normally too low to cause major issues. There is one item which I do not recognize (not saying anything wrong, just that I don't understand):
"dean:/var/lib/mythtv/recordings 3.6T 3.4T 80G 98% /mnt/oldrec"
It maybe you have some old recordings and storing them on the remote frontend, no problem with this, as long as mythbackend knows about the storagegroup, otherwise you will get messages in mythbackend that it cannot find files). Can you give me a list of the storagegroups in mythbackend database:
In a terminal on the mythbackend (when finished in mysql type exit to leave):
mysql -uroot -p mythconverg
select * from storagegroup;
I need to see a complete mythbackend.log, I know it likely to be large, so if you can zip the file and put it somewhere I can access, something like Dropbox.
Also there is a utility mythtv-status which allows you to see what the mythbackend is doing at anytime, if you don't have mythweb installed on your backend machine. You can install on your frontend machine (sudo apt-get install mythtv-status) and then do in a terminal:
mythtv-status -h <ip address of mythbackend>
Mike
Thanks for the outputs.
The output from mythbackend --testsched is good (I was concerned that the scheduler might be having problems, which can cause mythbackend to become unresponsive, but yours is good).
The output from df -h indicates you are low on disk space, but not normally too low to cause major issues. There is one item which I do not recognize (not saying anything wrong, just that I don't understand):
"dean:/var/lib/mythtv/recordings 3.6T 3.4T 80G 98% /mnt/oldrec"
It maybe you have some old recordings and storing them on the remote frontend, no problem with this, as long as mythbackend knows about the storagegroup, otherwise you will get messages in mythbackend that it cannot find files). Can you give me a list of the storagegroups in mythbackend database:
In a terminal on the mythbackend (when finished in mysql type exit to leave):
mysql -uroot -p mythconverg
select * from storagegroup;
I need to see a complete mythbackend.log, I know it likely to be large, so if you can zip the file and put it somewhere I can access, something like Dropbox.
Also there is a utility mythtv-status which allows you to see what the mythbackend is doing at anytime, if you don't have mythweb installed on your backend machine. You can install on your frontend machine (sudo apt-get install mythtv-status) and then do in a terminal:
mythtv-status -h <ip address of mythbackend>
Mike
Last edited by MikeB2013 on Thu Feb 23, 2017 1:03 pm, edited 1 time in total.
Re: recorder failed
Just a wag and I could be way off base, but with 8 hdhr dual tuners, are you sure you aren't choking your networks bandwidth? I don't know how much each hdhr tuner requires, but I'd assume 8 is more than a single Gb network could handle. Especially if you're also running running front ends also at the same time watching recorded programs.
BE/FE-Asrock AB350 Pro Ryzen 3 3200G, 6 atsc tuners. FE's-GF8200's Athlon II, Ryzen 3 2200G. Mythtv user since 2005.
Re: recorder failed
Given the relatively low number of simultaneous recordings, network congestion is unlikely assuming a Gigabit network, including router and any switches.
However, it is worth checking that the ethernet ports on both machines are actually running in Gigabit mode ( I recently had an issue with slow file transfers from a server, turned out it was connected at 100Mb/s instead of 1000 Mb/s - replacing ethernet cable fixed it)
Using ethtool in a terminal (if it is not installed just do sudo apt-get install ethtool):
ethtool <name of etherport>
near the end of the output you should see
Speed: 1000Mb/s
Duplex: Full
The <name of ethernetport> can be found using ifconfig, it is likely of the form enp4s0, the last few characters will be different (it could also be something like eth0 or eth1 depending on how the system was installed/configured).
Mike
However, it is worth checking that the ethernet ports on both machines are actually running in Gigabit mode ( I recently had an issue with slow file transfers from a server, turned out it was connected at 100Mb/s instead of 1000 Mb/s - replacing ethernet cable fixed it)
Using ethtool in a terminal (if it is not installed just do sudo apt-get install ethtool):
ethtool <name of etherport>
near the end of the output you should see
Speed: 1000Mb/s
Duplex: Full
The <name of ethernetport> can be found using ifconfig, it is likely of the form enp4s0, the last few characters will be different (it could also be something like eth0 or eth1 depending on how the system was installed/configured).
Mike
Re: recorder failed
To test your network is actually working (and not bad cables) try iperf which sends/receives data.
So, on dean: "sudo iperf -s" (-s ==> server) and on jack "sudo iperf --dualtest -c dean" (-c dean ==> client connect to server dean). This will tell you how much data you can push since sometimes you can have poor h/w. Try running it with the actual cables to your network tuner box too if you can.
If the cables are run through the house then maybe re-run the test with kitchen/laundry equipment running to rule out interference. Again, you shouldn't have problems but I've had cables and h/w that didn't work together (but worked with higher quality h/w).
You can also run "ping -c 60 dean" to see min/max/avg delays in packets and packet loss over a 60-second interval. These numbers should all be roughly equal and less than 1ms and packet loss should be zero.
The autoexpire could just be recordings you have deleted being expired. So, when you delete it puts it in to a deleted state but does not immediately delete in case you want to undelete. Later it will expire and delete the recordings.
I read on one forum about some network turners going offline at 4am for automatic upgrades. I guess your problems occur during the night-time though.
So, on dean: "sudo iperf -s" (-s ==> server) and on jack "sudo iperf --dualtest -c dean" (-c dean ==> client connect to server dean). This will tell you how much data you can push since sometimes you can have poor h/w. Try running it with the actual cables to your network tuner box too if you can.
If the cables are run through the house then maybe re-run the test with kitchen/laundry equipment running to rule out interference. Again, you shouldn't have problems but I've had cables and h/w that didn't work together (but worked with higher quality h/w).
You can also run "ping -c 60 dean" to see min/max/avg delays in packets and packet loss over a 60-second interval. These numbers should all be roughly equal and less than 1ms and packet loss should be zero.
The autoexpire could just be recordings you have deleted being expired. So, when you delete it puts it in to a deleted state but does not immediately delete in case you want to undelete. Later it will expire and delete the recordings.
I read on one forum about some network turners going offline at 4am for automatic upgrades. I guess your problems occur during the night-time though.
Re: recorder failed
The wiki has a find orphans script https://www.mythtv.org/wiki/Find_orphans.py
This can find recordings that are in the database but not on disk and cleanup the database so it no longer references these files.
The file you gave was 1041_20140206192900.mpg. The 1041 is the channel id (in mysql you can "select chanid,name from channel where chanid=1041" to see what channel that is), and the rest is the date/time which suggests 2014-02-06, i.e., 6th Feb 2014 at 7:29pm. So, this suggests that either the db has been screwy for a while or maybe the disk that has the files on was inaccessible.
This can find recordings that are in the database but not on disk and cleanup the database so it no longer references these files.
The file you gave was 1041_20140206192900.mpg. The 1041 is the channel id (in mysql you can "select chanid,name from channel where chanid=1041" to see what channel that is), and the rest is the date/time which suggests 2014-02-06, i.e., 6th Feb 2014 at 7:29pm. So, this suggests that either the db has been screwy for a while or maybe the disk that has the files on was inaccessible.
Re: recorder failed
The more I think about the original problem and the fact that mythtv runs for weeks then goes wrong with mythbackend unresponsive suggests an "out of memory condition", possibly disk space.
I really need to see the mythbackend.log
There is a possible cause of problems with mythbuntu 16.04 which uses a new start up system called systemd, which starts a lot of things in parallel and has been known to cause issues with network tuners. Although this does not explain the initial failure, it might explain the the issue with mythbackend being unresponsive after the reboot. Basically mythbackend can startup before networking is established, so mythbackend tries to connect to network tuners but fails, simply restarting mythbackend (via sudo systemctl restart mythtv-backend.service) fixes it temporarily until the next crash/reboot.
A more permanent solution to this issue I have used in the past is:
In a terminal on the machine running mythtv backend:
sudo systemctl enable NetworkManager-wait-online.service
sudo systemctl edit mythtv-backend.service
Then paste the following 2 lines
[Unit]
After=NetworkManager-wait-online.service
Then exit from the edit screen (it uses nano by default, so Ctrl+x, then y, then Return key)
Now reboot the system (sudo reboot)
Mike
I really need to see the mythbackend.log
There is a possible cause of problems with mythbuntu 16.04 which uses a new start up system called systemd, which starts a lot of things in parallel and has been known to cause issues with network tuners. Although this does not explain the initial failure, it might explain the the issue with mythbackend being unresponsive after the reboot. Basically mythbackend can startup before networking is established, so mythbackend tries to connect to network tuners but fails, simply restarting mythbackend (via sudo systemctl restart mythtv-backend.service) fixes it temporarily until the next crash/reboot.
A more permanent solution to this issue I have used in the past is:
In a terminal on the machine running mythtv backend:
sudo systemctl enable NetworkManager-wait-online.service
sudo systemctl edit mythtv-backend.service
Then paste the following 2 lines
[Unit]
After=NetworkManager-wait-online.service
Then exit from the edit screen (it uses nano by default, so Ctrl+x, then y, then Return key)
Now reboot the system (sudo reboot)
Mike
Re: recorder failed
Also, if you use sudo -i systemctl edit mythtv-backend.service,
then if you have EDITOR and or VISUAL set in .profile or .bashrc, its value will be used (e.g. vim.)
then if you have EDITOR and or VISUAL set in .profile or .bashrc, its value will be used (e.g. vim.)
Re: recorder failed
Another reason mythbackend can be unresponsive after a hard reboot is that mysql can take a while to recover if you pulled the power at the right time, so mythbackend then fails to start. I've had that happen before.
In /etc/rc.local I have the following one line above the "exit 0" to ensure we try and start the backend after two and a further five minutes if it failed to start properly on the initial boot.
I also have a cronjob to restart the mythbackend process every day since it ensures that every day is a clean start.
In /etc/rc.local I have the following one line above the "exit 0" to ensure we try and start the backend after two and a further five minutes if it failed to start properly on the initial boot.
Code: Select all
(sleep 120 ; systemctl start mythtv-backend ; sleep 300 ; systemctl start mythtv-backend &) &
Re: recorder failed
My mythbackend is configured to power down when not in use (basically a power saving measure), it automatically wakes up when its time to record something. I also have remote mythfrontends which are configured to wakeup the mythbackend if it is powered down. The net result being a clean start of mythbackend every day or so, sometimes more than once a day depending on the recording schedule etc.
For an always on system, the use of a daily cron job to restart the backend is a good idea. The only thing you have to make sure of is that no recordings are taking place, there is a utility mythshutdown that can do this.
For an always on system, the use of a daily cron job to restart the backend is a good idea. The only thing you have to make sure of is that no recordings are taking place, there is a utility mythshutdown that can do this.
Re: recorder failed
Sorry. I got a call on Feb 23rd to take a new job which has kept me away from home for weeks at a time and only hours at home in between to look after things like laundry.luc5588 wrote:The wiki has a find orphans script https://www.mythtv.org/wiki/Find_orphans.py
This can find recordings that are in the database but not on disk and cleanup the database so it no longer references these files.
The file you gave was 1041_20140206192900.mpg. The 1041 is the channel id (in mysql you can "select chanid,name from channel where chanid=1041" to see what channel that is), and the rest is the date/time which suggests 2014-02-06, i.e., 6th Feb 2014 at 7:29pm. So, this suggests that either the db has been screwy for a while or maybe the disk that has the files on was inaccessible.
My entire MythTV system has been offline now for quite some time.
In response to this particular issue, I know exactly what show this file contains. My concern is that it was never deleted and was always set to not expire. Should MythTV delete this file, I would be up the proverbial creek. I suppose it is good luck in a way that the system failed at that point.
It seems that the initial problems were caused because the network (after working flawlessly for the past 6 or 7 years) randomly disappears. The only way to bring it back is to issue the command 'dhclient <ethname>', but, of course, if MythTV had started a recording to a remote filesystem when that happened, the recording would show as perfectly successful until the end, when, presto! the file just simply wasn't there.
The output from 'mythtv-status' is (currently):
Code: Select all
Our child has stopped talking to us, kill it off.
Unknown error during retrieval of status from the MythTV backend.Edit again: Logs have been zipped and uploaded to http://www.bpmod.net/mythbackend.zip
Thanks to all for the help so far.
Brian