Mythbackend constantly crashing

mikavir · Post by **mikavir** » Fri Jul 08, 2016 8:53 am

Hello all,

Even though I've been a MythTV user for about ten years, this is my first post to this forum. I usually get my problems solved with people in my Linux flavour's forums. (Sorry for not using pastebinit, it is not found in LinHES repositories)

Code: Select all

[mythtv@taikalaatikko ~]$ uname -a
Linux taikalaatikko 4.4.13-1-ARCH #1 SMP PREEMPT Fri Jun 10 22:12:13 UTC 2016 x86_64 GNU/Linux

Code: Select all

[mythtv@taikalaatikko]$ cat /etc/os-release 
NAME="LinHES"
ID=linhes
PRETTY_NAME="LinHES The Home Entertainment System"
ANSI_COLOR="0;36"
HOME_URL="http://linhes.org/"
SUPPORT_URL="http://forum.linhes.org/"
BUG_REPORT_URL="http://linhes.org/"

Code: Select all

[mythtv@taikalaatikko ~]$ mythbackend --version
Please attach all output as a file in bug reports.
MythTV Version : v0.28-40-g7a4e9ab
MythTV Branch : fixes/0.28
Network Protocol : 88
Library API : 0.28.20160309-1
QT Version : 5.5.1
Options compiled in:
 linux profile use_hidesyms using_alsa using_oss using_backend using_bindings_perl using_bindings_python using_bindings_php using_crystalhd using_dvb using_firewire using_frontend using_hdhomerun using_vbox using_ceton using_hdpvr using_ivtv using_joystick_menu using_libcec using_libcrypto using_libdns_sd using_libfftw3 using_libxml2 using_lirc using_mheg using_opengl using_opengl_video using_qtwebkit using_qtscript using_qtdbus using_sdl using_taglib using_v4l2 using_x11 using_xrandr using_xv using_profiletype using_systemd_notify using_bindings_perl using_bindings_python using_bindings_php using_freetype2 using_mythtranscode using_opengl using_vaapi using_vdpau using_ffmpeg_threads using_mheg using_libass using_libxml2

Now I have a strange problem, which I don't know how to even debug... Mythbackend eats virtual memory until @ 12G (!) it crashes. I believe, it is related to a constant logging:

Code: Select all

[mythtv@taikalaatikko log]$ mythbackend --loglevel debug --syslog local6
2016-07-08 09:15:40.496768 C  mythbackend version: fixes/0.28 [v0.28-40-g7a4e9ab] www.mythtv.org
2016-07-08 09:15:40.496797 C  Qt version: compile: 5.5.1, runtime: 5.5.1
2016-07-08 09:15:40.496804 N  Enabled verbose msgs:  general
2016-07-08 09:15:40.496814 N  Setting Log Level to LOG_DEBUG
2016-07-08 09:15:40.508017 I  Added logging to the console
2016-07-08 09:15:40.508847 I  Setup Interrupt handler
2016-07-08 09:15:40.508861 I  Setup Terminated handler
2016-07-08 09:15:40.508877 I  Setup Segmentation fault handler
2016-07-08 09:15:40.508889 I  Setup Aborted handler
2016-07-08 09:15:40.508905 I  Setup Bus error handler
2016-07-08 09:15:40.508917 I  Setup Floating point exception handler
2016-07-08 09:15:40.508931 I  Setup Illegal instruction handler
2016-07-08 09:15:40.508945 I  Setup Real-time signal 0 handler
2016-07-08 09:15:40.508962 I  Setup Hangup handler
2016-07-08 09:15:40.509138 N  Using runtime prefix = /usr
2016-07-08 09:15:40.509147 N  Using configuration directory = /home/mythtv/.mythtv
2016-07-08 09:15:40.509152 D  appbindir      = /usr/bin/
2016-07-08 09:15:40.509160 D  sharedir       = /usr/share/mythtv/
2016-07-08 09:15:40.509164 D  libdir         = /usr/lib/mythtv/
2016-07-08 09:15:40.509173 D  themedir       = /usr/share/mythtv/themes/
2016-07-08 09:15:40.509178 D  pluginsdir     = /usr/lib/mythtv/plugins/
2016-07-08 09:15:40.509186 D  translationsdir= /usr/share/mythtv/i18n/
2016-07-08 09:15:40.509190 D  filtersdir     = /usr/lib/mythtv/filters/
2016-07-08 09:15:40.509305 I  Assumed character encoding: en_US.UTF-8
2016-07-08 09:15:40.509932 N  Empty LocalHostName.
2016-07-08 09:15:40.509944 I  Using localhost value of taikalaatikko
2016-07-08 09:15:40.526885 D  FindDatabase() - Success!
2016-07-08 09:15:40.527998 N  Setting QT default locale to en_FI
2016-07-08 09:15:40.528009 I  Current locale en_FI
2016-07-08 09:15:40.528046 E  No locale defaults file for en_FI, skipping
2016-07-08 09:15:40.528248 I  Loading en_gb translation for module mythfrontend
2016-07-08 09:15:40.536590 I  Current MythTV Schema Version (DBSchemaVer): 1344
2016-07-08 09:15:40.536749 I  Loading en_gb translation for module mythfrontend
2016-07-08 09:15:40.537416 D  Adding IPv4 loopback to address list.
2016-07-08 09:15:40.537424 D  Adding IPv6 loopback to address list.
2016-07-08 09:15:40.537435 D  Adding BackendServerIP to address list.
2016-07-08 09:15:40.537456 D  Adding link-local 'fe80::a60:6eff:fe71:a5c0%eth0' to address list.
2016-07-08 09:15:40.537782 N  MythBackend: Starting up as the master server.
2016-07-08 09:15:40.608670 I  Added syslogging
2016-07-08 09:15:40.922653 I  DVBChan[19](/dev/dvb/adapter_static_dvb_1/frontend0): Next tuning after less than 1000ms. Delaying by 1000ms
2016-07-08 09:15:42.284932 W  Scheduler: Listings source 'Nettikanavat' is defined, but is not attached to a card input.
2016-07-08 09:15:42.286276 I  Found 0 distinct programid authorities
2016-07-08 09:15:42.286696 I  New static DB connectionSchedCon
2016-07-08 09:15:42.286826 D  LogClean Run window between 79200 - 129600.
2016-07-08 09:15:42.286868 I  Registering HouseKeeperTask 'LogClean'.
2016-07-08 09:15:42.287028 D  DBCleanup Run window between 79200 - 129600.
2016-07-08 09:15:42.287041 I  Registering HouseKeeperTask 'DBCleanup'.
2016-07-08 09:15:42.287162 D  ThemeUpdateNotifications Run window between 79200 - 129600.
2016-07-08 09:15:42.287174 I  Registering HouseKeeperTask 'ThemeUpdateNotifications'.
2016-07-08 09:15:42.287314 D  RecordedArtworkUpdate Run window between 79200 - 129600.
2016-07-08 09:15:42.287327 I  Registering HouseKeeperTask 'RecordedArtworkUpdate'.
2016-07-08 09:15:42.287436 D  MythFillDB Run window between 79200 - 129600.
2016-07-08 09:15:42.288460 D  MythFillDB Run window between 86400 - 99000.
2016-07-08 09:15:42.288470 I  Registering HouseKeeperTask 'MythFillDB'.
2016-07-08 09:15:42.288975 D  UpdateRadioStreams Run window between 79200 - 129600.
2016-07-08 09:15:42.288984 I  Registering HouseKeeperTask 'UpdateRadioStreams'.
2016-07-08 09:15:42.289080 D  JobQueueRecover Run window between 79200 - 129600.
2016-07-08 09:15:42.289089 I  Registering HouseKeeperTask 'JobQueueRecover'.
2016-07-08 09:15:42.289155 I  Registering HouseKeeperTask 'HardwareProfiler'.
2016-07-08 09:15:42.289441 D  MythFillDB Run window between 73782 - 86382.
2016-07-08 09:15:42.289522 D  JobQueueRecover Run window between 71605 - 129600.
2016-07-08 09:15:42.289577 D  LogClean Run window between 70338 - 129600.
2016-07-08 09:15:42.289644 D  DBCleanup Run window between 43200 - 81028.
2016-07-08 09:15:42.289698 D  RecordedArtworkUpdate Run window between 43200 - 80068.
2016-07-08 09:15:42.289754 D  ThemeUpdateNotifications Run window between 72841 - 129600.
2016-07-08 09:15:42.291193 I  Starting HouseKeeper.
2016-07-08 09:15:42.302199 I  Listening on TCP 127.0.0.1:6544
2016-07-08 09:15:42.302228 I  Listening on TCP 192.168.1.200:6544
2016-07-08 09:15:42.302250 I  Listening on TCP [::1]:6544
2016-07-08 09:15:42.302321 I  Listening on TCP [fe80::a60:6eff:fe71:a5c0%eth0]:6544
2016-07-08 09:15:42.302352 I  Listening on TCP 127.0.0.1:6554
2016-07-08 09:15:42.302369 I  Listening on TCP 192.168.1.200:6554
2016-07-08 09:15:42.302385 I  Listening on TCP [::1]:6554
2016-07-08 09:15:42.302412 I  Listening on TCP [fe80::a60:6eff:fe71:a5c0%eth0]:6554
2016-07-08 09:15:42.302455 I  Listening on TCP 127.0.0.1:6549
2016-07-08 09:15:42.302479 I  Listening on TCP 192.168.1.200:6549
2016-07-08 09:15:42.302496 I  Listening on TCP [::1]:6549
2016-07-08 09:15:42.302525 I  Listening on TCP [fe80::a60:6eff:fe71:a5c0%eth0]:6549
2016-07-08 09:15:45.298475 I  Reschedule requested for MATCH 0 0 0 - SchedulerInit
2016-07-08 09:15:45.434430 I  Scheduled 61 items in 0.1 = 0.01 match + 0.03 check + 0.04 place
2016-07-08 09:15:45.461195 I  Scheduler: Seem to be woken up by USER
2016-07-08 09:15:46.504844 I  Main::Registering HttpStatus Extension
2016-07-08 09:15:46.505867 I  Listening on TCP 127.0.0.1:6543
2016-07-08 09:15:46.505903 I  Listening on TCP 192.168.1.200:6543
2016-07-08 09:15:46.505940 I  Listening on TCP [::1]:6543
2016-07-08 09:15:46.506002 I  Listening on TCP [fe80::a60:6eff:fe71:a5c0%eth0]:6543
2016-07-08 09:15:46.507494 N  AutoExpire: CalcParams(): Max required Free Space: 1.0 GB w/freq: 15 min
2016-07-08 09:15:46.508968 E  SG(Videos): Group 'Videos' wants to use directory '/media/video/', but this directory is not writeable.
2016-07-08 09:15:47.438863 I  Bonjour: Service registration complete: name 'Mythbackend on taikalaatikko' type '_mythbackend._tcp.' domain: 'local.'
2016-07-08 09:15:47.525821 I  MainServer: MainServer::ANN Monitor
2016-07-08 09:15:47.525837 I  MainServer: adding: MythExpress.EventListener(1a838b0) as a client (events: 1)
2016-07-08 09:15:47.529237 W  MainServer: Client MythExpress.EventListener is trying to announce a socket multiple times.
2016-07-08 09:15:47.691863 I  MainServer: MainServer::ANN Monitor
2016-07-08 09:15:47.691870 I  MainServer: adding: taikalaatikko(1a7e1a0) as a client (events: 0)
2016-07-08 09:15:47.755403 I  Monitor sock(1a7e1a0) 'taikalaatikko' disconnected
2016-07-08 09:15:47.756280 I  MainServer: MainServer::ANN Monitor
2016-07-08 09:15:47.756290 I  MainServer: adding: taikalaatikko(1a57160) as a client (events: 0)
2016-07-08 09:15:47.756846 I  Monitor sock(1a57160) 'taikalaatikko' disconnected
2016-07-08 09:15:48.576000 I  MainServer: MainServer::ANN Frontend

...and these reoccourring three lines are generated until the crash, which occurs every twenty minutes, or so. Luckily, the system is usable until the crash and the BE starts up after it. I've had this problem for months, but I haven't had time to remedy this until now.

Now, I really don't know, where to look next, since I've never had to struggle with problems like this. Does anybody have ideas?

Just as I'm writing this, I was able to try strace. Loads of incomprehensible information...

Post by **bill6502** » Sun Jul 24, 2016 9:53 pm

Hi,

If "crash" means you're getting a SEGV or similar fault, then to collect the necessary
information, see: https://www.mythtv.org/wiki/Debugging#G ... _Backtrace

If you feel that logging is the issue, then turn it off just as a test. Adjust your startup
.conf/.service/.sh adding --loglevel warning, that will eliminate all the info and notice logs.

Post by **stuarta** » Tue Jul 26, 2016 8:58 am

Given the memory growth you are seeing, it's probably a memory leak of some sort. Can you please install the debug or debuginfo packages for mythtv (or build with --compile-type=debug if you build from source and then run the backend under valgrind?

valgrind --leak-check=full --show-reachable=yes --verbose --log-file=/tmp/valgrind-backend.log <path_to>/mythbackend <options>
> /tmp/mythbackend-valgrind.log

Replace <path_to> with the path to your mythbackend, and <options> with the options you normally run the backend with. I'll need both the log files from /tmp that are created by the above command. Please create a ticket in trac for this https://code.mythtv.org/trac

Regards
Stuart

mikavir · Post by **mikavir** » Fri Jul 29, 2016 8:08 pm

Thank you guys! Exactly the advice I was looking for. As it happened, my life got busier (again). I will look into these suggestions as soon as I get the time. It might take long... But I'll be back!

mikavir · Post by **mikavir** » Mon Sep 26, 2016 4:35 pm

Well, finally I've cleared the problems. I never found the time to start debugging, so I kinda chickened...

I took a new SSD disk and installed the whole system fresh. After all, it wasn't too hard to get all the old recorded data working. Downside on this is, that there's still quite a lot of tweaking, before my system is fully configured. I always have difficulties with remote, getting all the channels and their progam data working and so on...

Anyway, thank you for your help!

MythTV Official Community Forum

Mythbackend constantly crashing

Mythbackend constantly crashing

Re: Mythbackend constantly crashing

Re: Mythbackend constantly crashing

Re: Mythbackend constantly crashing

Re: Mythbackend constantly crashing