XMLTV grabber for Schedules Direct JSON

Have a MythTV related problem? Ask for help from other MythTV users here.

Moderator: Forum Moderators

wmorrison
Junior
Posts: 59
Joined: Sat Dec 01, 2018 12:05 am
United States of America

Re: Important upcoming v31 Changes, you may have work to do!

Post by wmorrison » Mon Dec 02, 2019 6:30 pm

Would you please clear something up?

It's been a few months since I set up my system (v30 I think) and I forget the distinctions between the different grabbers. I got it working and have left it alone since then.

I know I get program data from SchedulesDirect, and that it is downloaded in an XML format, because I snooped the /tmp/mythtv directory while mythfilldatabase was running, before it deleted the file.

I get 13 days of listings, so that's DataDirect, right? So it's not "XMLTV" even though it's in XML format?

XMLTV provides 20 days, right? So I must not be using that.

Here's what really has been confusing me. Above you said JSON grabbers, but the page you linked to said XMLTV grabbers. I always thought those were two separate things, because JSON is not XML. But now, re-reading about XMLTV, I see JSON mentioned on a few pages. What does this mean? Does the "XMLTV Project" now support data in JSON format, even though it's not XML, but they didn't rename the project?

Can I set up the SchedulesDirect JSON grabber (or whatever it's called, getting off DataDirect) with v30, before DataDirect is removed in v31?

User avatar
bill6502
Developer
Posts: 1408
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: Important upcoming v31 Changes, you may have work to do!

Post by bill6502 » Mon Dec 02, 2019 7:58 pm

Good point. The names of the grabbers of interest here are: tv_grab_zz_sdjson_sqlite and tv_grab_zz_sdjson
(users pick one.) They are: "XMLTV grabbers for Schedules Direct JSON service". If you were to look at the file
in /tmp generated by the above, it would still be in XML.

Yes, anyone can switch from the DataDirect (original format/format that's no longer supported by MythTV) to
one of the new grabbers right now.

I'll change the initial post here to clear this up, hopefully.

wmorrison
Junior
Posts: 59
Joined: Sat Dec 01, 2018 12:05 am
United States of America

Re: Important upcoming v31 Changes, you may have work to do!

Post by wmorrison » Mon Dec 02, 2019 8:36 pm

Wait, the file generated in /tmp? Does that mean mythtv downloads JSON but saves it as XML before parsing? As long as it works, lol.

Any advantages of the sqlite one? I read it requires more setup, but is it better in any way?

User avatar
bill6502
Developer
Posts: 1408
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: Important upcoming v31 Changes, you may have work to do!

Post by bill6502 » Mon Dec 02, 2019 9:06 pm

I've only used the sqlite version. Since there are hash sums stored per program, I preferred to have
the info in the DB rather than in files. Here's an example (and answer to your question about how the
data is received.):

Code: Select all

{"titles":[{"title120":"CBS 2 News at 6PM"}],"hasSeriesArtwork":true,"md5":"MZ98dCB4LJB7d23ELYkJXg","resourceID":"605163","hasImageArtwork":true,"genres":["News"],"entityType":"Show","originalAirDate":"2002-09-21","showType":"Series","keyWords":{"General":["Newscast"]},"programID":"SH005371070000"}

wmorrison
Junior
Posts: 59
Joined: Sat Dec 01, 2018 12:05 am
United States of America

Re: Important upcoming v31 Changes, you may have work to do!

Post by wmorrison » Thu Dec 05, 2019 4:53 pm

I am taking the approach of setting up the grabber and testing it before I configure MythTV to use it. (Leaving my existing DataDirect video source working for now.)

As long as MythTV doesn't know about the source yet, it should ignore the DB and config files in .mythtv and .xmltv, right? Won't cause a problem?

So I've got xmltv and sqllite installed, and am going though the setup.

I've read about chanid differences, so I have a question.

My existing SchedulesDirect video source (DataDirect) has an ID of 2. (I think because I started with an EPG one, then added SchedulesDirect, then deleted the EPG one, before I was aware of the IDs in mythconverge.

Consequently, my chanid's start with 2. E.g. channel 3.1 has a chanid of 20301.

If I want to minimize the amount of manual setup of chanids, can I simply edit my existing video source in mythtv-setup (after the new grabber is set up manually and tested) and change the grabber to the new one? I'd do a database backup first to be safe.

If IDs match up, should there be any conflict between "old data" and "new data"? That is, I've got two weeks of data from DataDirect in mythconverg. If I enable the json_sqlite grabber, and it downloads duplicate data for those two weeks (plus the new third week,) will it cause duplicates or other conflicts in mythconverg?

User avatar
bill6502
Developer
Posts: 1408
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: Important upcoming v31 Changes, you may have work to do!

Post by bill6502 » Thu Dec 05, 2019 7:14 pm

Correct the grabber can be set up while still using DataDirect.

Not sure why you mention changing chanids (as opposed to xmltvids.)
I didn't touch my chanids.

I didn't have any problems with the program info. But will admit that once I
switched over, I didn't go back. No duplicate problems.

Check your Private Messages in a few minutes.

wmorrison
Junior
Posts: 59
Joined: Sat Dec 01, 2018 12:05 am
United States of America

Re: Important upcoming v31 Changes, you may have work to do!

Post by wmorrison » Thu Dec 05, 2019 9:24 pm

Got it. My reply still shows it's in my outbox, so I don't know it's been sent. Is that normal for the board? My answer was yes.

User avatar
Steve Goodey
Moderator
Posts: 90
Joined: Fri Feb 07, 2014 6:30 pm
Location: Colchester, England
Great Britain

Re: Important upcoming v31 Changes, you may have work to do!

Post by Steve Goodey » Fri Dec 06, 2019 7:13 am

My reply still shows it's in my outbox, so I don't know it's been sent. Is that normal for the board?
I believe it is. It will leave the outbox when the person reads the PM.

wmorrison
Junior
Posts: 59
Joined: Sat Dec 01, 2018 12:05 am
United States of America

Re: Important upcoming v31 Changes, you may have work to do!

Post by wmorrison » Fri Dec 06, 2019 2:46 pm

Is the xmltvid column used only by mythfilldatabase, to match up channels to myth's internal ids?

If so, updating a channel's xmltvid (and nothing else) while the backend is running (and possibly preparing to record) should not cause any conflict, right?

But if mythfilldatabase runs in the middle of maintenance, it could cause failure to find some or all schedule data, right?

Point is, if you don't have mythfilldatabase scheduled to run at a specific time, MythTV will run it about once a day at a seemingly "random" time (not random, just roughly 24 hours since the last run, when the computer is on,) so watch out for this when updating xmltvid's and grabber, etc.

To be safe, you could run mythtv-setup and disable automatic run of mythfilldatabase until done maintaining. Or run mythfilldatabase manually just before maintenance to give you about a 24 hour window to complete.

These are my guesses. Please correct if I am wrong.

User avatar
bill6502
Developer
Posts: 1408
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: Important upcoming v31 Changes, you may have work to do!

Post by bill6502 » Mon Dec 09, 2019 3:08 am

I've not heard of anyone having problems switching over. It might be reasonable to run MFDB
immediately prior to the change just to give a 24 hour window to complete it. Or, for those that
run it out of cron, to disable it.

I did try running two instances of MFDB (both using the JSON feed) and had no corruption. There
were DB errors when duplicate entries were attempted.

wmorrison
Junior
Posts: 59
Joined: Sat Dec 01, 2018 12:05 am
United States of America

Re: Important upcoming v31 Changes, you may have work to do!

Post by wmorrison » Tue Dec 10, 2019 11:51 pm

Reporting a problem.

When I let mythfilldatabase run automatically on schedule, it fails. mythconverg shows:

mythfilldatabaseLastRunStatus FAILED: XMLTV grabber returned error code 1.

When I run mythfilldatabase from mythwelcome, it works.

I suspect it's a user problem. I log in as my own user, and set up the new grabber that way, so the files are in my home directory and owned by me.

Does the automatic run start the process as user mythtv? That would fail. Maybe from mythwelcome, it runs as me. I haven't checked processes yet.

For now I've made the files (.xmltv and .DB) rw to all, and linked them from mythtv's home directory to mine. I'll check tomorrow if it runs successfully.

Anyway, is this a known problem? In what version was the new grabber support added?

I suppose one way to solve it would be to run via cron job (as me) and set up a daily wake period, but I haven't set a daily wake period yet.

User avatar
bill6502
Developer
Posts: 1408
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: Important upcoming v31 Changes, you may have work to do!

Post by bill6502 » Wed Dec 11, 2019 1:31 pm

Yes, most users run the backend as user mythtv (with its own $HOME.) The configuration file can't
be found. Likely the cause of the error.

It's a configuration issue, not explicitly documented in the Wiki.

I chose to do the same as you (setup as my user) then put files in ~mythtv/.mythtv/<SourceName>.xmltvid
that were hard links to my own. Therefore they pointed to the same ~/.xmltv/SchedulesDirect.DB file. And
as you've made it's writable by user mythtv. I know my own .xmltv is set to group mythtv and is writable
by it. Can't remember if that's required or just me testing.

I do use a systemD service and timer on another host. The timer can be setup for as many runs as desired.
I use one in the morning and one in the evening. The existing method run out of the backend is preferred in
my opinion as the next run time is set by Schedules Direct, even if that's as simple as adding 24 hours to
the current run.

wesnewell
Senior
Posts: 517
Joined: Mon Jun 23, 2014 6:54 pm
Location: Wylie TX, USA
United States of America

Re: Important upcoming v31 Changes, you may have work to do!

Post by wesnewell » Wed Dec 11, 2019 6:57 pm

wmorrison wrote:
Tue Dec 10, 2019 11:51 pm
Reporting a problem.

When I let mythfilldatabase run automatically on schedule, it fails. mythconverg shows:

mythfilldatabaseLastRunStatus FAILED: XMLTV grabber returned error code 1.

When I run mythfilldatabase from mythwelcome, it works.

I suspect it's a user problem. I log in as my own user, and set up the new grabber that way, so the files are in my home directory and owned by me.
Does this help?
wes@mythfe0:~$ locate sd.xmltv
/home/mythtv/.mythtv/sd.xmltv
/home/wes/.mythtv/sd.xmltv
wes@mythfe0:~$ diff /home/mythtv/.mythtv/sd.xmltv /home/wes/.mythtv/sd.xmltv
wes@mythfe0:~$ cat /home/wes/.mythtv/sd.xmltv
cache=/home/mythtv/.xmltv/tv_grab_zz_sdjson.cache
channel-id-format=default
previously-shown-format=date
username=wesnewell
password=xxxxxxxxxx
mode=lineup
lineup=USA-OTA-75098
wes@mythfe0:~$
BE/FE-Asrock AB350 Pro AMD Ryzen 3 2200G, 6 atsc tuners. Frontends-GF8200's,,AMD Athlon II's. Mythtv user since 2005.

User avatar
bill6502
Developer
Posts: 1408
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: Important upcoming v31 Changes, you may have work to do!

Post by bill6502 » Thu Dec 12, 2019 2:13 am

Unfortunately, I run the sqlite version, so can't speak to the above. I'd do: mythbackend --setverbose system:debug
and then look at mythfilldatabase.log for the actual command that gets run both by your user and when the backend
starts it. Then run the grabber from the command line (see if it has a --help option and a --debug switch) and run it
with that to see if the actual error is printed. I have an alias handy to become the mythtv user:

Code: Select all

alias bem='sudo --set-home --login --user=mythtv'
You can just type bem and then run whatever you like as user mythtv.

mythbackend --setverbose system:alert restores logging.

jfabernathy
Senior
Posts: 128
Joined: Wed Feb 18, 2015 2:37 pm
United States of America

Re: Important upcoming v31 Changes, you may have work to do!

Post by jfabernathy » Fri Dec 13, 2019 11:01 am

I've had the opportunity to setup a number of test systems with V30 and tv_grab_zz_sdjson_sqlite and I'm getting good enough at it I've developed a workable process:
- once Mythtv is installed and before I run mythtv-setup, I setup Mythtv XMLTV per the wiki page for sdjson_sqlite https://www.mythtv.org/wiki/XMLTV#Setup ... son_sqlite
BUT you need to "sudo su mythtv" if not already doing the installation of everything mythtv related as user 'mythtv'
I install mythtv and do all my maintenance as my normal user 'jim', but the XMLTV stuff and mythfilldatabase commands need to be run as user 'mythtv'

It says this on the wiki page, but I missed it once or twice.

By changing to user 'mythtv' all the commands on the wiki can be cut and pasted directly.

- Once the XMLTV stuff is done, you run mythtv-setup, in my case as user 'jim', and setup stuff as normal but when you get to the Video Sources, just pick the Multi-national SDJSON grabber and name you source "SD" to match what the scripts used.

- Once mythtv-setup is complete you use systemctl to start mythtv-backend, then you can "sudo su mythtv" again and run mythfilldatabase. It takes several minutes to complete, but then you have about 18 days of EPG listings.

P.S. Since I do a lot of setup of mythtv on "server versions of Ubuntu/Debian Linuxes" via ssh -Y <IP> I avoid issues with that by doing all maintenance as my default user, 'jim'. Limiting "sudo su mythtv" to just the XMLTV setup and mythfilldatabase stuff make it work smoother for me.

Post Reply