Metadata lookup causes Backend to deadlock

Have a MythTV related problem? Ask for help from other MythTV users here.

Moderator: Forum Moderators

Post Reply
jksj
Junior
Posts: 69
Joined: Thu Feb 13, 2014 7:53 pm
Great Britain

Metadata lookup causes Backend to deadlock

Post by jksj » Tue Jan 07, 2020 5:47 pm

I had this problem about a year a go with version 30 and then it went away. Its come back with latest master and seems to occur more often.
v31-Pre-1674-g1c97471de5
It does not happen every time and may occur more often for a recording rule for which no recordings exist yet.
Typically the log shows
Frontend :-

Code: Select all

an  7 17:01:37 tv mythfrontend: mythfrontend[6587]: I MetadataDownload metadatadownload.cpp:183 (run) Returning Metadata Results: Cold Feet 1 1
Jan  7 17:01:40 tv mythfrontend: mythfrontend[6587]: I MetadataImageDownload metadataimagedownload.cpp:236 (run) Metadata Image Download: http://www.thetvdb.com/banners/fanart/original/77152-4.jpg -> myth://Fanart@tv/Cold Feet Season 1_fanart.jpg
Jan  7 17:02:11 tv mythfrontend: mythfrontend[6587]: E MythSocketThread(-1) mythsocket.cpp:815 (ReadStringListReal) MythSocket(7f31d000d650:65): ReadStringList: Error, timed out after 30000 ms.
Jan  7 17:02:11 tv mythfrontend: mythfrontend[6587]: E SendMessage mythsocket.cpp:359 (SendReceiveStringList) MythSocket(7f31d000d650:-1): No response.
Jan  7 17:02:11 tv mythfrontend: mythfrontend[6587]: N SendMessage mythcorecontext.cpp:1381 (SendReceiveStringList) MythCoreContext::SendReceiveStringList(): Connection to backend server lost
Jan  7 17:02:11 tv mythfrontend: mythfrontend[6587]: N MythSocketThread(-1) mythcorecontext.cpp:1631 (connectionClosed) MythCoreContext::connectionClosed(): Event socket closed.  No connection to the backend.
Jan  7 17:02:11 tv mythfrontend: mythfrontend[6587]: I SendMessage mythcorecontext.cpp:457 (ConnectCommandSocket) MythCoreContext::ConnectCommandSocket(): Connecting to backend server: 192.168.0.108:6543 (try 1 of 1)
Jan  7 17:02:18 tv mythfrontend: mythfrontend[6587]: E MythSocketThread(-1) mythsocket.cpp:815 (ReadStringListReal) MythSocket(7f31b0030b80:60): ReadStringList: Error, timed out after 7000 ms.
Jan  7 17:02:18 tv mythfrontend: mythfrontend[6587]: C SendMessage mythcorecontext.cpp:1650 (CheckProtoVersion) Protocol version check failure.#012#011#011#011The response to MYTH_PROTO_VERSION was empty.#012#011#011#011This happens when the backend is too busy to respond,#012#011#011#011or has deadlocked due to bugs or hardware failure.
The backend has

Code: Select all

Jan  7 17:01:41 tv mythbackend: mythbackend[1606]: I ProcessRequest mainserver.cpp:1769 (HandleAnnounce) MainServer: MainServer::ANN Playback
Jan  7 17:01:41 tv mythbackend: mythbackend[1606]: I ProcessRequest mainserver.cpp:1771 (HandleAnnounce) MainServer: adding: tv(564ab87660f0) as a client (events: 0)
Jan  7 17:01:41 tv mythbackend: mythbackend[1606]: W ProcessRequest ringbuffer.cpp:664 (Start) RingBuf(/var/lib/mythtv/fanart//Cold Feet Season 1_fanart.jpg): Not starting read ahead thread, this is a write only RingBuffer
Jan  7 17:01:41 tv mythbackend: mythbackend[1606]: I ProcessRequest mainserver.cpp:2034 (HandleAnnounce) MainServer: adding: tv(564ab87f6520) as a file transfer
Jan  7 17:01:41 tv mythbackend: mythbackend[1606]: I MythSocketThread(109) mainserver.cpp:7905 (connectionClosed) FileTransfer sock(564ab87f6520) disconnected
Jan  7 17:01:42 tv mythbackend: mythbackend[1606]: I MythSocketThread(103) mainserver.cpp:7865 (connectionClosed) Playback sock(564ab87660f0) 'tv' disconnected

User avatar
pgbennett
Developer
Posts: 329
Joined: Mon Apr 27, 2015 5:41 pm
United States of America

Re: Metadata lookup causes Backend to deadlock

Post by pgbennett » Wed Jan 08, 2020 7:31 pm

This may be related to the ticket https://code.mythtv.org/trac/ticket/13479

blm-ubunet
Senior
Posts: 106
Joined: Sun Jun 15, 2014 1:08 am
Cambodia

Re: Metadata lookup causes Backend to deadlock

Post by blm-ubunet » Fri Jan 10, 2020 5:05 am

@jksj

A very similar error happens for me (running version master from couple months back).
Occurs once a week lately .. was 3 times a week a few months back.
Logs imply it is metadata lookup.

For me, the backend can still successfully record & respond to 'mythbackend --testsched' just all network/socket interfaces are dead. I did not try uPnP.
Biggest annoyance is the automatic shutdown fails & a forced shutdown then leads to BE not waking up at right time.

jksj
Junior
Posts: 69
Joined: Thu Feb 13, 2014 7:53 pm
Great Britain

Re: Metadata lookup causes Backend to deadlock

Post by jksj » Fri Jan 10, 2020 11:49 am

Yes sounds the same. Unfortunately there is nothing in the backend log you could look for. The issue has never occurred for me without the frontend running. The frontend string "This happens when the backend is too busy to respond" is unique to this situation, so could be used to trigger a reset of the backend.

The issue seems to happen when the system is busy. I had another fault recently which caused preview generation to fail. That issue aggravated this fault making it noticeable again. Now that the preview issue is fixed the other fault has gone away again. Looking in my logs i had one occurrence of this fault in five weeks then one a day when the other issue was present.

blm-ubunet
Senior
Posts: 106
Joined: Sun Jun 15, 2014 1:08 am
Cambodia

Re: Metadata lookup causes Backend to deadlock

Post by blm-ubunet » Sat Jan 11, 2020 6:49 am

Yes, FE was always running when it happened but that could just be down to chance & the pop-up message notifications in FE make you fully aware.

I try might disable metadata lookup for month & see if this issue stops. TV guide data here has problems anyway (everything is a movie etc).

wmorrison
Junior
Posts: 83
Joined: Sat Dec 01, 2018 12:05 am
United States of America

Re: Metadata lookup causes Backend to deadlock

Post by wmorrison » Thu Jan 16, 2020 6:15 pm

Aha! I'll bet this is the issue I've been occasionally having, when I thought it was mythcommflag.

Last week, I updated all my rules to disable metadata jobs, and limited queue processing to 1 simultaneous job at low cpu use. Haven't had the "Could not connect to backend" problem since. I disabled metadata jobs to narrow down the issue (and because the listings data is enough for me,) but after reading this, it's likely the metadata jobs were the real problem.

BTW for anyone trying this, disabling job processing in setup does not stop jobs from being created. That's in each recording rule. If you just disable processing, your machine may never auto-shutdown due to pending jobs. You can clear them in information center, or just reenable processing in setup.

knappster
Junior
Posts: 18
Joined: Wed Aug 01, 2018 1:42 am
United States of America

Re: Metadata lookup causes Backend to deadlock

Post by knappster » Sat Jan 18, 2020 12:15 pm

@pgbennett, I believe this thread is also related:
viewtopic.php?f=36&t=3020&p=14876#p14876

It still happens from time to time to me, but my script automatically restarts the backend when it does. It typically results in the recording being disrupted around the 1 minute mark and then being restarted when the backend is back up.

Post Reply