[Solved] Cannot pull lineups

Have a MythTV related problem? Ask for help from other MythTV users here.

Moderator: Forum Moderators

Post Reply
rahvin
Junior
Posts: 15
Joined: Sat Jul 02, 2016 12:00 am
United States of America

[Solved] Cannot pull lineups

Post by rahvin »

I've rebuilt my myth backend a dozen times, I've used both Debian and Ubuntu (I prefer Debian). I've built from scratch, used the multimedia archive and used the ubuntu repository copies. No matter what I do I cannot get mythtv-setup to pull the lineups from schedules direct, it shows 50% progress and sits there for a minute or so then just drops back to the screen with no other output. I'm not getting anything logged. Running mythfilldatabase manually with the most verbosity I keep getting this darn QT error:

QIODevice::read (QNetworkReplyHttpImpl): device not open

If anyone can help me with this I would appreciate it.

The below attempt is using Ubuntu Xenial with the repository build.

Code: Select all

root@myth:~/ansible/ansible# mythfilldatabase --manual -v most
###
### Running in manual channel configuration mode.
### This will ask you questions about every channel.
###
2018-04-13 04:58:26.242212 C  mythfilldatabase version: fixes/0.28 [v0.28.2-1-g2d49bc1] www.mythtv.org
2018-04-13 04:58:26.242228 C  Qt version: compile: 5.5.1, runtime: 5.5.1
2018-04-13 04:58:26.242231 N  Enabled verbose msgs: most
2018-04-13 04:58:26.242239 N  Setting Log Level to LOG_INFO
2018-04-13 04:58:26.253240 I  Added logging to the console
2018-04-13 04:58:26.253966 I  Setup Interrupt handler
2018-04-13 04:58:26.253984 I  Setup Terminated handler
2018-04-13 04:58:26.253996 I  Setup Segmentation fault handler
2018-04-13 04:58:26.254009 I  Setup Aborted handler
2018-04-13 04:58:26.254021 I  Setup Bus error handler
2018-04-13 04:58:26.254031 I  Setup Floating point exception handler
2018-04-13 04:58:26.254043 I  Setup Illegal instruction handler
2018-04-13 04:58:26.254057 I  Setup Real-time signal 0 handler
2018-04-13 04:58:26.254071 I  Setup Hangup handler
2018-04-13 04:58:26.254226 N  Using runtime prefix = /usr
2018-04-13 04:58:26.254232 N  Using configuration directory = /root/.mythtv
2018-04-13 04:58:26.254341 I  Assumed character encoding: en_US.UTF-8
2018-04-13 04:58:26.254895 N  Empty LocalHostName.
2018-04-13 04:58:26.254905 I  Using localhost value of mythbuntu
2018-04-13 04:58:26.254947 I  DefaultUPnP() - No default UPnP backend
2018-04-13 04:58:26.264201 I  Testing network connectivity to '192.168.0.35'
2018-04-13 04:58:26.264975 I  Starting process manager
2018-04-13 04:58:26.264986 I  Starting process signal handler
2018-04-13 04:58:26.267377 I  Starting IO manager (read)
2018-04-13 04:58:26.267485 I  Managed child (PID: 60355) has started! * command=ping -t 3 -c 1  192.168.0.35  >/dev/null 2>&1, timeout=0
2018-04-13 04:58:26.267514 I  Starting IO manager (write)
2018-04-13 04:58:26.267519 I  (0x1f33080)::IncrRef() -> 2
2018-04-13 04:58:26.287806 I  Managed child (PID: 60355) has exited! command=ping -t 3 -c 1  192.168.0.35  >/dev/null 2>&1, status=0, result=0
2018-04-13 04:58:26.315198 I  (0x1f33080)::DecrRef() -> 1
2018-04-13 04:58:26.315225 I  (0x1f33080)::DecrRef() -> 0
2018-04-13 04:58:26.318519 N  Setting QT default locale to en_US
2018-04-13 04:58:26.318549 I  Current locale en_US
2018-04-13 04:58:26.318596 N  Reading locale defaults from /usr/share/mythtv//locales/en_us.xml
2018-04-13 04:58:26.325897 E  setHttpProxy() - failed to find a network proxy
2018-04-13 04:58:26.326705 I  Loading en_us translation for module mythfrontend
2018-04-13 04:58:26.328260 I  Current MythTV Schema Version (DBSchemaVer): 1344
2018-04-13 04:58:26.331032 I  MythCoreContext::ConnectCommandSocket(): Connecting to backend server: 192.168.0.35:6543 (try 1 of 1)
2018-04-13 04:58:26.331075 I  MythSocket(1f41560:-1): MythSocket(-1, 0x0) ctor
2018-04-13 04:58:26.331874 I  MythSocket(1f41560:-1): IP is local, using loopback address instead
2018-04-13 04:58:26.331902 I  MythSocket(1f41560:-1): attempting connect() to (127.0.0.1:6543)
2018-04-13 04:58:26.332210 I  MythSocket(1f41560:16): Connected to (127.0.0.1:6543)
2018-04-13 04:58:26.332374 I  MythSocket(1f41560:16): write -> 16 30      MYTH_PROTO_VERSION 88 XmasGift
2018-04-13 04:58:26.332950 I  MythSocket(1f41560:16): read  <- 16 13      ACCEPT[]:[]88
2018-04-13 04:58:26.332991 I  MythCoreContext::CheckProtoVersion(): Using protocol version 88 XmasGift
2018-04-13 04:58:26.333032 I  MythSocket(1f41560:16): write -> 16 24      ANN Playback mythbuntu 0
2018-04-13 04:58:26.333494 I  MythSocket(1f41560:16): read  <- 16 2       OK
2018-04-13 04:58:26.333553 I  Opening blocking connection to master backend
2018-04-13 04:58:26.335015 I  Updating source #1 (Antenna) with grabber schedulesdirect1
2018-04-13 04:58:26.335357 I  No channels are configured to use grabber.
2018-04-13 04:58:26.335424 I  
2018-04-13 04:58:26.335432 I  Checking day @ offset 0, date: Fri Apr 13 2018
2018-04-13 04:58:26.335763 I  Checking program counts for day -1
2018-04-13 04:58:26.336019 I  Checking program counts for day 0
2018-04-13 04:58:26.336027 I  Data refresh needed because no data exists for day @ offset 0 from 8PM - midnight.
2018-04-13 04:58:26.336034 N  Refreshing data for Fri Apr 13 2018
2018-04-13 04:58:26.337846 I  Retrieving datadirect data.
2018-04-13 04:58:26.337866 I  Grabbing data for Fri Apr 13 2018 offset 0
2018-04-13 04:58:26.337890 I  From 2018-04-13T00:00:00Z to 2018-04-14T00:00:00Z (UTC)
2018-04-13 04:58:26.337914 I  DataDirect: Grabbing listing data
2018-04-13 04:58:26.338073 I  Downloading DataDirect feed
QIODevice::read (QNetworkReplyHttpImpl): device not open
2018-04-13 04:59:26.436478 E  DataDirect: Failed to get data: Download error
2018-04-13 04:59:26.436496 E  Encountered error in grabbing data.
2018-04-13 04:59:26.436522 I  
2018-04-13 04:59:26.436530 I  Checking day @ offset 1, date: Sat Apr 14 2018
2018-04-13 04:59:26.436535 I  Data Refresh always needed for tomorrow
2018-04-13 04:59:26.436540 N  Refreshing data for Sat Apr 14 2018
2018-04-13 04:59:26.438827 I  Retrieving datadirect data.
2018-04-13 04:59:26.438843 I  Grabbing data for Fri Apr 13 2018 offset 1
2018-04-13 04:59:26.438865 I  From 2018-04-14T00:00:00Z to 2018-04-15T00:00:00Z (UTC)
2018-04-13 04:59:26.438877 I  DataDirect: Grabbing listing data
2018-04-13 04:59:26.438920 I  Downloading DataDirect feed
Last edited by rahvin on Wed May 30, 2018 4:58 am, edited 1 time in total.
User avatar
bill6502
Developer
Posts: 2307
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: Cannot pull lineups

Post by bill6502 »

Hi,

If you login to http://schedulesdirect.org and press Lineup Support and then
select HERE on the 3rd bullet down from the top, do you see your raw data?

As for no logging, in mythtv-setup, under: "Guide data program", is the entry
exactly: mythfilldatabase (opposed to something like /usr/bin/mythfilldatabase,
which will not log.) [I'll assume you've already checked permissions.]

From the command line: mythfilldatabase --verbose network:debug,file:debug --dd-grab-all --logpath /tmp
is popular and the output in the file, not stdout is better to pastebin.
rahvin
Junior
Posts: 15
Joined: Sat Jul 02, 2016 12:00 am
United States of America

Re: Cannot pull lineups

Post by rahvin »

If you login to http://schedulesdirect.org and press Lineup Support and then
select HERE on the 3rd bullet down from the top, do you see your raw data?
I see a properly formatted XML channel file.
As for no logging, in mythtv-setup, under: "Guide data program", is the entry
exactly: mythfilldatabase (opposed to something like /usr/bin/mythfilldatabase,
which will not log.) [I'll assume you've already checked permissions.]
Yes, I've verified user permissions, database permissions and passwords. I'm not aware of others. I've never used this site so I appoligize if this doesn't work for the log file.

https://hastebin.com/lewixixipi.coffeescript
rahvin
Junior
Posts: 15
Joined: Sat Jul 02, 2016 12:00 am
United States of America

Re: Cannot pull lineups

Post by rahvin »

The only error message I see is the one in the log:
QIODevice::read (QNetworkReplyHttpImpl): device not open
From what little I can find on Google (3 links show up in the search) about this it's an obscure QT5 error. I just don't know enough about how the software works or even QT in general or why mythfilldatabase uses it to give me an indication of what the problem was. At first I thought it was differences between the Debian and Ubuntu on what's included with the packages but I'm getting the error on Ubunutu too.

I'm using a Debian Kernel with the install but I'm not sure how that would have any impact. I've been pulling my hair out for several months trying to get this working again after my hardware upgrade, needless to say I'm frustrated. :(
User avatar
bill6502
Developer
Posts: 2307
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: Cannot pull lineups

Post by bill6502 »

No solution (don't want to get your hopes up.)

My production backend is on: Qt version: compile: 5.5.1, runtime: 5.5.1, just like
yours. Qt is used throughout the project, nothing unusual there. 5.5.1 works fine and
I don't see the error. I'm on v30-Pre, but have been using mythfilldatabase since 0.18.

There is a similar, unsolved thread here: viewtopic.php?f=36&t=593

And I found this discussion that shows where QNetworkReplyHttpImpl is, but it's
on interest only. https://www.peter.hartmann.tk/single-po ... HTTP-stack
The MythTV code has no referrences to QNetworkReplyHttpImpl. But 184 uses of QIODevice.
Uses in the download manager are all logged on failure, which I don't see in your log.

I did try running MFDB as root (which isn't recommended) but didn't get the error.
rahvin
Junior
Posts: 15
Joined: Sat Jul 02, 2016 12:00 am
United States of America

Re: Cannot pull lineups

Post by rahvin »

The thing that frustrates me the most is the absolute lack of logging so I can't see what's wrong. I've tried the MFDB run with multiple users and all of them fail with these QT errors.

If there is a developer that wants to take a closer look at this I'm willing to make it available with root access.
User avatar
dizygotheca
Developer
Posts: 267
Joined: Wed Sep 03, 2014 9:02 am
Great Britain

Re: Cannot pull lineups

Post by dizygotheca »

You need to turn on debug logging, either "-v file --loglevel debug" or "-v file:debug", as Bill said.

It's failing to download the data from SD. OS/user & Myth version will make no difference.
https://code.mythtv.org/doxygen/mythdow ... tml#l00849 has a 60 sec timeout if no data is being received.
So the connection isn't rejected/failing but the download doesn't progress either.

The error comes from Qt internals of the QNetworkRequest. May be https://bugreports.qt.io/browse/QTBUG-48326 or a result of Myth not aborting cleanly. If you upgrade Qt you might get a better error, but it's not the root problem.

With debug logging, you'll see the url, authentication and how far it gets.
rahvin
Junior
Posts: 15
Joined: Sat Jul 02, 2016 12:00 am
United States of America

Re: Cannot pull lineups

Post by rahvin »

I used the debug levels that bill recommended to generate the hastebin linked log file. (it looks like it's timed out at this point as when I click the link it goes to a blank page). Specifically this included debug on network and file.

The command used was: mythfilldatabase --verbose network:debug,file:debug --dd-grab-all --logpath /tmp

In the interest of getting help I'll try to make a longer lasting paste and might even repaste it into the forum when I get home tonight.

Thanks for the input. I'm pretty sure there isn't anything wrong with myth specifically, I suspect it's tied to a library or something or god forbid a kernel patch or enabled feature on the debian kernel that's not on the ubuntu kernel but I really think that's a long shot.
User avatar
bill6502
Developer
Posts: 2307
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: Cannot pull lineups

Post by bill6502 »

Hi,

Thanks for hanging in on this.

If you choose to paste again, please use the file that's created in /tmp.
It has better detail than the command line logs, thanks.
rahvin
Junior
Posts: 15
Joined: Sat Jul 02, 2016 12:00 am
United States of America

Re: Cannot pull lineups

Post by rahvin »

I appreciate your help on this. I'm sorry for the delay getting back to you, I've been working long hours at work the last couple weeks.

The reason I used hastebin last time was because it was impossible to get pastebin to take the paste, it kept give me a too busy error. I've repasted in hastebin and a few other clones. I'll keep trying for another 30 minutes to get it into pastebin, I can use another service it anyone knows of one that lets you set a delete period of more than a few days that works good (I like hastebin's interface and code formating/coloring).

Anyway, because it's been sitting idle for a couple weeks I think the backend is having a problem right now as indicated in the log now it's reporting an empty version. I don't believe this is related at all to the mythfilldatabase problem because I'm getting the same QT error as before but didn't have the empty version error before. I'm not worried about this until I can get it to actually pull lineups from schedules direct so I can actually finish configuring the backend. More importantly I don't have time at the moment to troubleshoot why it's reporting an empty version.

Looking through the log file it's got more information but it's not showing the QT error that appears on STDOUT, I re-pasted the stdout text in hastebin below (labeled STDOUT) in case you want to see the difference. From what I can read in the log file it's successfully authenticating with schedules direct, and then gertting an empty download back, but I need your expert advice on what you think is wrong. :D

This is hastebin of the log file:
https://www.hastebin.com/vubugamufa.hs

This is the hastebin of the STDOUT to the command line of the command: (this is what I posted before)
https://www.hastebin.com/cufomitoci.coffeescript

Here's a codepile paste of the log file:
https://www.codepile.net/pile/kJ20Q47X

Here's a pastefs of the log file:
https://www.pastefs.com/pid/54205

Pastedco of the log file:
http://pasted.co/f88bd49f

Zerobin of the log file:
https://zerobin.net/?013104a40786a1b6#u ... GUvFiDF7w=

Hopefully one of those will last. The forum won't let me put the log in a code bracket because of the length so one of those will have to do.

As always I appreciate all commentary or suggestions on troubleshooting. I'm moderately skilled in Linux, services, servers and such but even though technically inclined I'm not an experienced coder and mythfilldatabase is a blackbox to me.
rahvin
Junior
Posts: 15
Joined: Sat Jul 02, 2016 12:00 am
United States of America

Re: Cannot pull lineups

Post by rahvin »

Ok, strangest thing. I got some time Friday night to play around with the system and tried a few more times with mythfilldatabase with the same result. I decided to see if I could troubleshoot the install and did and apt-get update/upgrade which updated the following packages.
The following packages will be upgraded:
avahi-daemon ifupdown libavahi-client-dev libavahi-client3 libavahi-common-data libavahi-common-dev
libavahi-common3 libavahi-compat-libdnssd-dev libavahi-compat-libdnssd1 libavahi-core7 libpulse0
linux-base linux-libc-dev
After running this upgrade all of a sudden it worked. There is nothing in those packages that were upgraded that should have made this work all of a sudden so I'm even more confused but now it's working, I've been able to pull the lineups and mythfilldatabase was able to pull down the guide data.

I doubt this will help anyone else that has this problem but there it is. Whatever those updates changed must have fixed something.
User avatar
bill6502
Developer
Posts: 2307
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: Cannot pull lineups

Post by bill6502 »

Hmmm, I've been trying to duplicate the failure and haven't had any success.

My next shot was going to be having you use Wireshark to capture the
protocol. No need now.
rahvin
Junior
Posts: 15
Joined: Sat Jul 02, 2016 12:00 am
United States of America

Re: Cannot pull lineups

Post by rahvin »

I really appreciate the help Bill.

So after it started working i went to try again Saturday and it wasn't working again, luckily I was able to figure out what it was and I feel really stupid. The close timing between working and not helped me figure out what happened the night before.

After it quit working again I went back through and tried to redo everything I remember doing/happening Friday night. It occurred to me Snort crashed on my Firewall that night and as a test I went in and disabled it and MFD started working again. The ironic thing is I went through the snort and firewall logs looking for the schedules direct IP (74.117.221.21) before I tried anything else. I know the IP wasn't in the logs because I went over every log and initiated mythfilldatabase and watched the logs for hits/alerts. I did all this weeks/months before I posted to the forum.

In the interest of helping anyone else that may have a similar setup. I'm running a pfsense firewall with snort IDS running with the $25 a year subscription for the daily rule updates. When I get a chance I'm going to setup a VM and re-enable snort and see if I can pin down what's blocking it even if I have to run wireshark.

I do have one question to assist and track down why snort was blocking the communication, does MFD try to talk to any other IP address than the dd.schedulesdirect.com (74.117.221.21) that hosts the schedule data that I should be looking for? In other words does it try to initiate a connection with anything else before connecting to schedulesdirect that would have stalled the process before connecting to SD? I'm 99% sure the SD IP was never blocked so I'm at a loss for how snort was snagging the connection because it logs everything it snags in it's alert database and I scanned that database a dozen times looking for the SD IP.

Again, thanks, your assistance was invaluable and I mean that. Any help you can provide helping me pin down why snort was blocking will hopefully help someone else who has an error like this as I plan to figure out what was blocking SD and post it in this thread.
User avatar
bill6502
Developer
Posts: 2307
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: Cannot pull lineups

Post by bill6502 »

This may help, dd.schedulesdirect.com isn't used with MFDB that
I know of and that text doesn't exit in the code base:

Code: Select all

nslookup webservices.schedulesdirect.tmsdatadirect.com # Existed in *very* old MythTV versions
nslookup dd.schedulesdirect.org # Current
rahvin
Junior
Posts: 15
Joined: Sat Jul 02, 2016 12:00 am
United States of America

Re: Cannot pull lineups

Post by rahvin »

That's interesting, now I've been troubleshooting this for a while but I could have swore I snagged that domain (dd.schedulesdirect.com) from a direct dump from ethernet while running MFD as a redirect when it connected to the schedules direct IP. Anyway I found the problem, and the IP was a generic AWS IP and that's why i missed it.

The Snort rule snagging the connection is from the Emerging Threats Database in the policy section, it's:

ET POLICY Outgoing Basic Auth Base64 HTTP Password detected unencrypted

Apparently MFD is communicating with schedulesdirect with an unencrypted password and that got tagged by a default policy rule. Hopefully if someone else runs into a similar problem and is running snort they find this post and realize the problem considering it took me more than 2 months to figure it out. I've disabled the rule and unblocked the AWS IP and now it works fine with Snort re-enabled.

Thanks for the help Bill, really appreciate it.
Post Reply