mythfilldatabase doesn't do authentication callback

Have a MythTV related problem? Ask for help from other MythTV users here.

Moderator: Forum Moderators

dgriffin
Junior
Posts: 17
Joined: Fri Jan 02, 2015 11:08 pm
United States of America

mythfilldatabase doesn't do authentication callback

Post by dgriffin »

Hello All,

I started with MythTV at version .23 that I built from source (.tar) on Debian Squeeze (6.0). Life was good. I just recently upgraded to version .27-fixes on Debian Wheezy (7.4), and "mythfilldatabase" has stopped working correctly.

I have run the program with "tcpdump" active, and I can see the initial call to dd.schedulesdirect.org which is made by something in the qt4 driven files due to a call to DDpost in the "datadirect.cpp" file. I traced up to a call to something in the files moc_mythdownloadmanager.cpp and/or mythdownloadmanager.cpp
As I don't speak c++ or QT4, I'm stuck.

The tcpdump file shows the initial connect to dd.schedulesdirect.org (which really is ec2-54-85-117-227.compute-1.amazonaws.com) and sends this stuff:
-----
POST /schedulesdirect/tvlistings/xtvdService HTTP/1.1^M
User-Agent: MythTV v0.27.20141016-1 MythDownloadManager^M
Accept-Encoding: gzip^M
Content-Type: application/soap+xml; charset=utf-8^M
Content-Length: 527^M
Connection: Keep-Alive^M
Accept-Language: en-US,*^M
Host: dd.schedulesdirect.org^M
-----

I never see anything that looks like a re-direct and/or authorization-password request and ultimately the response is from the remote is:
----
HTTP/1.1 401 Authorization Required^M
Server: nginx/1.6.2^M
Date: Fri, 02 Jan 2015 22:57:44 GMT^M
Content-Type: text/html; charset=iso-8859-1^M
Content-Length: 472^M
Connection: keep-alive^M
WWW-Authenticate: Basic realm="TMSWebServiceRealm"^M
^M
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>401 Authorization Required</title>
</head><body>
<h1>Authorization Required</h1>
<p>This server could not verify that you
are authorized to access the document
requested. Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.</p>
<hr>
<address>Apache/2.2.29 (Amazon) Server at sd-dd Port 80</address>
</body></html>
----

I would expect somewhere along the line to see a "https://" request to happen for the password authentication similar to when I directly login to my DataDirect account.

Does anyone out there have a suggestion as to curing this very frustrating situation. I've been working on it for the past three weeks.

The particulars:
mythfilldatabase version: fixes/0.27 [b305eb5]
libqt4-dev 4:4.8.2+dfsg-11 (and all "qt4" related packages are of the same vintage)

And everything else is most current in Debian Wheezy. (Nothing from "backports" and nothing that I know of hanging around from older versions.)
"configure" doesn't complain about anything and the output from configure indicates all of the expected items to be correct.

This has the same flavor as people reported a couple of years ago with a bug in qt4 version 4.7.2, but we are much forward advanced from that version.

Regards,
Dan
User avatar
bill6502
Developer
Posts: 2307
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: mythfilldatabase doesn't do authentication callback

Post by bill6502 »

dgriffin wrote: Does anyone out there have a suggestion as to curing this very frustrating situation.
The 1st 2 packet fragments look right. The next should be your backend sending a POST
that includes Authorization: Basic xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==\r\n
Credentials: yourSdLogin:yourSdPassword (in plain text, but that's another issue.)

Any change you're blocked? There's a test in the 3rd paragraph here:
http://forums.schedulesdirect.org/viewt ... f=5&t=2626 .

Running: mythfilldatabase -v file,network --loglevel debug --logpath /tmp --dd-grab-all
is the popular debugging command line.
dgriffin
Junior
Posts: 17
Joined: Fri Jan 02, 2015 11:08 pm
United States of America

Re: mythfilldatabase doesn't do authentication callback

Post by dgriffin »

I believe that blocking is not the issue. If I do "http://dd.schedulesdirect.org" from my browser I get properly redirected to "www.schedulesdirect.org" and then, if I choose, I can log in with the credentials as expected. I've instrumented "mythfilldatabase" with some selected "printf" statements and I know that the correct user name and password are fetched from the database prior to the networking calls. In the tcpdump log for mythfilldatabase, there is never any change in target by the DDPost operation. (And that's buried somewhere in the "callback" functions of the "mythdownloadmanager.cpp" and "moc_mythdownloadmanger.cpp")

Thanks for more things to check.
dgriffin
Junior
Posts: 17
Joined: Fri Jan 02, 2015 11:08 pm
United States of America

Re: mythfilldatabase doesn't do authentication callback

Post by dgriffin »

Oh, and I've done exactly the below until I'm sick of it. It does not log anything in the qt4 procedures, which is where all of the web page accessing happens. By the time of the return back to non QT4 levels happen, data, if present is in a file. In this case it's simply error codes because the web fetching process failed due to lack of authorization.

Running: mythfilldatabase -v file,network --loglevel debug --logpath /tmp --dd-grab-all
is the popular debugging command line.[/quote]
pinacate
Newcomer
Posts: 4
Joined: Fri Dec 05, 2014 3:41 am
United States of America

Re: mythfilldatabase doesn't do authentication callback

Post by pinacate »

I'm setting up MythTV on OS X using MacPorts. All seems to have gone well until I run MythFillDatabase.

I can download lineups from SchedulesDirect. I don't seem to be blocked as the dd.schedulesdirect.org redirects properly.

I ran mythfilldatabase directly with this command:mythfilldatabase -v file,network --loglevel debug --logpath /tmp --dd-grab-all

The problem appears to be related to this output line. I've attached the whole log.

"DownloadManager: downloadError 204 Host requires authentication"

If anyone has suggestions on how to proceed I'd appreciate it!!
Attachments
log.txt
(8.78 KiB) Downloaded 130 times
dgriffin
Junior
Posts: 17
Joined: Fri Jan 02, 2015 11:08 pm
United States of America

Re: mythfilldatabase doesn't do authentication callback

Post by dgriffin »

Hello pinacate,

Your symptoms and log file are exactly the issue I have. Unfortunately no-one has yet responded with a useful fix, or even suggestions as how to go about getting more detailed information for the QT4 stuff in the mythdowloadmanager.cpp and moc_mythdownloadmanager.cpp files.

Given that you are building on OS-X and I'm using Debian Wheezy, I think that it indites the QT4 package or a code bug in the MythTV library subroutines. How we seem to have this issue, and apparently many other don't have a problem is a real mystery. With tcpdump, I see that the redirect and authentication operations do not occur.

I there is a version of tcpdump available for OS-X, I suspect that you'll find a similar situation.

Regards,

Dan
User avatar
bill6502
Developer
Posts: 2307
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: mythfilldatabase doesn't do authentication callback

Post by bill6502 »

Agreed, the: DownloadManager: downloadError 204 Host requires authentication
isn't normal. If you were looking with Wireshark, you'd see a 401 as the 1st response
from SD (it doesn't print in the log.)

I changed my password in mythtv-setup to a bogus value and got the 204 error.
Maybe check yours for some non-printing character, or just retype it.

There was a (very old, and fixed) Qt bug that prevented users with '@' in their
login from working. I haven't seen any reports on the -users mailing list about it
returning.
dgriffin
Junior
Posts: 17
Joined: Fri Jan 02, 2015 11:08 pm
United States of America

Re: mythfilldatabase doesn't do authentication callback

Post by dgriffin »

bill6502 wrote:Agreed, the: DownloadManager: downloadError 204 Host requires authentication
isn't normal. If you were looking with Wireshark, you'd see a 401 as the 1st response
from SD (it doesn't print in the log.)

I changed my password in mythtv-setup to a bogus value and got the 204 error.
Maybe check yours for some non-printing character, or just retype it.

There was a (very old, and fixed) Qt bug that prevented users with '@' in their
login from working. I haven't seen any reports on the -users mailing list about it
returning.
I used tcpdump, and the redirect does not happen. Everything goes to http://dd.schedules.direct.org and the www.schedulesdirect.org redirect doesn't appear. If I go to dd.schedulesdirect.org with a web browser, I get the expected redirect okay. As I stated in the initial post to this thread, I can see that the correct user ID and password are being passed to the lower level routines having been fetched from the SQL request. And I bet that the issues pinacate is having are the same root cause. I know that the bug with QT you are speaking of was fixed back in 2012.

Regards,

Dan
User avatar
bill6502
Developer
Posts: 2307
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: mythfilldatabase doesn't do authentication callback

Post by bill6502 »

dgriffin wrote:I used tcpdump, and the redirect does not happen.
Hi,

Don't expect to see any redirect when mythfilldatabase is run. Only when
doing the manual test from the browser.

I'd use the lineup reporting tool at SD to report your problem. Given the
fairly massive about of posts in -users during the conversion last year,
I think it's reasonable to say that mythfilldatabase is working OK (or
folks would be commenting there.)

I know SD has logging too and they can use it to diagnose problems.

It sounds like you've been over the login/password thing a lot, but here's
the way I look for non-printings anyway:
SELECT userid,HEX(userid),password,HEX(password) FROM videosource;

(I've looked at the download manager code ages ago and was unable to find
additional tools to watch its progress. Out of my league there.)
dgriffin
Junior
Posts: 17
Joined: Fri Jan 02, 2015 11:08 pm
United States of America

Re: mythfilldatabase doesn't do authentication callback

Post by dgriffin »

Okay, since no-one has been able to answer this question, how about a new one.

How do you get MythTv version 27 to build using QT5 on Debian Linux?
wesnewell
Senior
Posts: 731
Joined: Mon Jun 23, 2014 6:54 pm
Location: Wylie TX, USA
United States of America

Re: mythfilldatabase doesn't do authentication callback

Post by wesnewell »

Heck if I know. I think the last one I compiled from source was the first one I ever used 10 years ago, only to later find out it was in the distro repository as an rpm I could have just easily installed.
BE/FE-Asrock AB350 Pro Ryzen 3 3200G, 6 atsc tuners. FE's-GF8200's Athlon II, Ryzen 3 2200G. Mythtv user since 2005.
User avatar
stuartm
Developer
Posts: 129
Joined: Wed Feb 05, 2014 5:17 pm
Great Britain

Re: mythfilldatabase doesn't do authentication callback

Post by stuartm »

QT5 isn't supported in 0.27
jas_il
Newcomer
Posts: 2
Joined: Tue Oct 13, 2015 7:07 pm
United States of America

Re: mythfilldatabase doesn't do authentication callback

Post by jas_il »

Hi,

I seem to be experiencing the same problem reported previously in this thread - mythfilldatabase does not appear to be providing the needed authorization to download the schedules direct data. I'm trying to get this working on OS X (El Capitan) using the March 2015 macports installer. Any help would be greatly appreciated!

tcpdump contains the following response (which appears to be missing the needed authorization):

POST /schedulesdirect/tvlistings/xtvdService HTTP/1.1
User-Agent: MythTV v0.27.20141016-1 MythDownloadManager
Content-Type: application/soap+xml; charset=utf-8
Content-Length: 379
Connection: Keep-Alive
Accept-Encoding: gzip
Accept-Language: en-US,*
Host: dd.schedulesdirect.org
Authorization: Basic Og==

I've attached the full tcpdump as well as the log from the mythfilldatabase command.
Attachments
tcpdump.txt
tcpdump of mythfilldatabase
(13.72 KiB) Downloaded 119 times
mythfilldatabase.txt
mythfilldatabase log
(10.99 KiB) Downloaded 116 times
dgriffin
Junior
Posts: 17
Joined: Fri Jan 02, 2015 11:08 pm
United States of America

Re: mythfilldatabase doesn't do authentication callback

Post by dgriffin »

Exactly the same situation as I have had for nearly a year. I knew if I was patient, somebody would stumble into the same mess.
It really appears that the issue is inside some QT4 function that is supposed to handle the password handshake protocol.
I instrumented a version of the mythfilldatabase subroutines with "printf" statements and all the correct info was being retrieved from my data base and passed into the QT4 functions. I never was able to grok the QT4 code and simply had to give up. What surprises me is why more people don't have this issue. I had this problem first on Debian Squeeze when I updated to MythTV .27 and it persisted when I upgraded to Debian Wheezy. Things were working great on MythTV .23 and Debian Squeeze.
I have just been manually setting recording rules and feeling hoping that the money paid to SchedulesDirect isn't a total waste.
Regards,
Dan Griffin
User avatar
pvr4me
Senior
Posts: 763
Joined: Fri Feb 07, 2014 7:25 pm
Location: near Toronto, Canada
Contact:
Canada

Re: mythfilldatabase doesn't do authentication callback

Post by pvr4me »

jas_il - I think your log indicates quite a different situation. First, it appears your backend is not running.

Next, you have 4 sources configured. Do you really have 4 (groups of) tuners that each receive different sets of channels?

The first source has no channels configured to use the grabber.

Source #2 uses grabber schedulesdirect for 14 channels

The log says:
Source 3 configured with no grabber. Nothing to do.
Source 4 configured with no grabber. Nothing to do.

On the SchedulesDirect web site have you logged in and configured the channels that you need listings data for? You need to enter your Zip code and select the appropriate lineup. And maybe edit it.

Craig
Formerly the MacPorts guy.
Post Reply