[Solved] Configuring XMLTV Prior to V31 upgrade

Have a MythTV related problem? Ask for help from other MythTV users here.

Moderator: Forum Moderators

red192
Newcomer
Posts: 13
Joined: Fri Nov 13, 2015 5:17 am
United States of America

[Solved] Configuring XMLTV Prior to V31 upgrade

Post by red192 » Thu Feb 04, 2021 4:02 am

I really like MythTV and a big thanks to those who make it great. For months I have been stuck trying to migrate from SD DataDirect to the XMLTV format via Schedules Direct JSON API. I have studied the wiki page and every time I read it I wonder how or if I have taken a wrong turn.

For example, the wiki leads off with
If user mythtv is allowed to login and start the window manager, XMLTV can be configured through the Video Sources Setup page in mythtv-setup. Otherwise, follow the steps below and do the configuration from the command line.
Down the page, it reads
Information.png Tip: Setup of XMLTV via command line is less challenging and more consistent with XMLTV instructions/usage.
OK -- so I think I wanna do command line. I know the password of my mythtv user and I can su to be that user. Then I read this:
As long as you have selected your xmltv grabber in MythTV Setup you can perform some or all configuration via the command line.
So ... it seems that one must first login to the desktop as mythtv user, run mythtv backend setup at least as far as setting the xmltv grabber, and then logout, hop back in via command line to finish the configuration?

My experience in running mythtv-setup is that it is full of hidden screens --> and the arrow down and tab sometimes to fill in a variable and at the end of adding and configuring a new video source I am rewarded with "exited with status 0" -- is that a good thing? No clue is given. It's only until you try and run mythfilldatabase you get to see the error that there was an unexpeted end of file, the grabber did not work, etc.

And I start over ... reading more carefully this time.

here is another nugget of wisdom regarding xmltv
You should obtain the latest version, since there have been recent changes. When you download the a more recent version, you must ensure your copies of the programs are found before the version provided for by your Linux distro - do this by placing ensuring you either replace the versions directly in /usr/bin, or adjusting PATH so that your downloaded version is found before the existing version in /usr/bin.

This is really what it says.

Oh yeah -- I need to adjust PATH, of course. Some other nix skill to learn, so I can someday bask in the glory of the improved v31 with my fellow greybeards.

I know I am not alone in my frustration over XMLTV. I will continue to read, google and hunt for a post like the venerable viewtopic.php?f=36&t=3114 which addresses the question HOW ON EARTH DO YOU GET IT WORKING?
Last edited by red192 on Tue Feb 16, 2021 1:59 am, edited 1 time in total.

wesnewell
Senior
Posts: 728
Joined: Mon Jun 23, 2014 6:54 pm
Location: Wylie TX, USA
United States of America

Re: Configuring XMLTV Prior to V31 upgrade

Post by wesnewell » Thu Feb 04, 2021 5:34 pm

I can tell you how I got it working, but first make sure you have all xmltv packages installed. Easiest way for me to do that is to run my package manager and search for xmltv. Synaptic shows 5 packages, including mythtv.
First I ran mythtv-setup form the menu as my user (wes). Configured for the standard sdJson, not sqlite. When exiting it ran fine.
But it did not run from mythtv. This is because when mythtv-setup is run as a regular user it stores the xmltv files under that user, and when run from mythtv it runs as user mythtv. Once I discovered what the problem was, the fix was simple. I just copied the files from wes to the same locations under user mythtv, changing the cache location to user mythtv. This allowed user mythtv to run mfdb without a problem and it also allowed me to manually run it as user wes from cl. Here is what I ended up with. Username and password removed.

wes@mythfe0:~$ cat /home/wes/.mythtv/sd.xmltv
cache=/home/mythtv/.xmltv/tv_grab_zz_sdjson.cache
channel-id-format=default
previously-shown-format=date
username=xxxx
password=xxxxl
mode=lineup
lineup=USA-OTA-75098
wes@mythfe0:~$ cat /home/mythtv/.mythtv/sd.xmltv
cache=/home/mythtv/.xmltv/tv_grab_zz_sdjson.cache
channel-id-format=default
previously-shown-format=date
username=xxxx
password=xxxx
mode=lineup
lineup=USA-OTA-75098
wes@mythfe0:~$

Just make sure both users have write permission to the files.
A way to avoid this would be to run mythtv-setup as user mythtv, but then you couldn't manually run mfdb as a regular user without copying the files to that user. Take your pick. Hope this helps.
BE/FE-Asrock AB350 Pro Ryzen 3 3200G, 6 atsc tuners. FE's-GF8200's Athlon II, Ryzen 3 2200G. Mythtv user since 2005.

red192
Newcomer
Posts: 13
Joined: Fri Nov 13, 2015 5:17 am
United States of America

Re: Configuring XMLTV Prior to V31 upgrade

Post by red192 » Fri Feb 05, 2021 2:32 am

I have the four xmltv packages installed and i think I have run the config correctly this time -- now you say
I just copied the files from wes to the same locations under user mythtv, changing the cache location to user mythtv.
The files ... which files exactly? if the commands you listed already give me that info already I apologize. In my /home/scott/.mythtv i have the following:
scott@mythtv:~/.mythtv$ ls -l
total 240
drwxrwxr-x 2 scott scott 4096 Nov 1 00:27 3rdParty
-rw-rw-r-- 1 scott scott 40 Nov 1 00:47 backuprc
drwxrwxr-x 8 scott scott 4096 Feb 4 19:55 cache
drwxrwxr-x 2 scott scott 4096 Nov 7 23:17 channels
-rw-rw-r-- 1 scott scott 600 Nov 7 01:38 config.xml
-rw-rw-r-- 1 scott scott 209415 Feb 4 20:29 list.xml
drwxrwxr-x 2 scott scott 4096 Nov 1 00:27 themes
drwxrwxr-x 4 scott scott 4096 Nov 7 23:08 tmp
-rw-rw-r-- 1 scott scott 198 Feb 4 20:19 xfinity.xmltv
-rw-rw-r-- 1 scott scott 0 Feb 4 20:25 xfinity.xmltv.TMP

So .. copy the file xfinity.xmltv to /home/mythtv/.mythyv

and change the cache location to user mythtv. No idea how to change the cache location. So I guessed that I needed to edit my xfinity.xmltv config file -- and that's what I did. I suppose the cache file does not exist there until after this pig runs successfully.

Finally, you give sage advice to
make sure both users have permission to the files

One issue for me -- not sure how to do that and what all the files are. So I first did this
sudo chown scott:scott -R /home/scott/.mythtv
sudo chmod -R 777 /home/scott/.mythtv

then

sudo cp /home/scott/.mythtv/xfinity.xmltv /home/mythtv/.mythtv
and
sudo chown mythtv:mythtv -R /home/mythtv/.mythtv

After all that ... I ran mythfilldatabase the way i always have -- start the back end setup and upon leaving says yes, run it.
after all the mucking about it says "no channels are configured to use the grabber" and "XMLTV config file is: /home/scott/.mythtv/xfinity.xmltv and it just hangs there/.

So ... bugger. Back to re-reading and re-doing.

Mythtv has been a sweet pain for me since 2006. For a while, I left and used Microsoft "something awful" -- and then SageTV. I always came back to the myth and it's been great for many years. The only reason I'm obsessed with V31 is that I have this idea that I can run RPi 4 front ends with it -- and v31 might work better with Buster and the pI4. I always tell people If things just worked, I would have never learned anything! So thanks I guess. Exiting with status 0 luck. and remember .. Params::Validate::NO_VALIDATION" used only once: possible typo

wesnewell
Senior
Posts: 728
Joined: Mon Jun 23, 2014 6:54 pm
Location: Wylie TX, USA
United States of America

Re: Configuring XMLTV Prior to V31 upgrade

Post by wesnewell » Fri Feb 05, 2021 7:40 am

Does this help? Both my sd.xmltv files point to the same cache file.

wes@mythfe0:~$ ll /home/wes/.mythtv/sd.*
-rw-rw-r-- 1 wes wes 173 Dec 26 09:21 /home/wes/.mythtv/sd.xmltv
wes@mythfe0:~$ ll /home/mythtv/.mythtv/sd.*
-rw-rw-r-- 1 wes wes 173 Aug 8 2019 /home/mythtv/.mythtv/sd.xmltv
wes@mythfe0:~$ ll /home/mythtv/.xmltv/*
-rw-rw-rw- 1 mysql lxd 57575478 Feb 4 07:25 /home/mythtv/.xmltv/tv_grab_zz_sdjson.cache
wes@mythfe0:~$
BE/FE-Asrock AB350 Pro Ryzen 3 3200G, 6 atsc tuners. FE's-GF8200's Athlon II, Ryzen 3 2200G. Mythtv user since 2005.

red192
Newcomer
Posts: 13
Joined: Fri Nov 13, 2015 5:17 am
United States of America

Re: Configuring XMLTV Prior to V31 upgrade

Post by red192 » Fri Feb 05, 2021 1:40 pm

That cache file is missing for me. Was it supposed to be generated when I ran myth-setup?

wesnewell
Senior
Posts: 728
Joined: Mon Jun 23, 2014 6:54 pm
Location: Wylie TX, USA
United States of America

Re: Configuring XMLTV Prior to V31 upgrade

Post by wesnewell » Fri Feb 05, 2021 2:28 pm

It should have been created under /home/scott/.xmlttv/* if you ran setup from user scott. You would then need to move it under /home/mythtv/.xmltv/ and then also copy the .xmltv file to user mythtv, and edit the cache file to point to the one you copied to under user mythtv.. See previous post example files.
BE/FE-Asrock AB350 Pro Ryzen 3 3200G, 6 atsc tuners. FE's-GF8200's Athlon II, Ryzen 3 2200G. Mythtv user since 2005.

red192
Newcomer
Posts: 13
Joined: Fri Nov 13, 2015 5:17 am
United States of America

Re: Configuring XMLTV Prior to V31 upgrade

Post by red192 » Fri Feb 05, 2021 3:00 pm

Greatly appreciate the guidance! Once I get this dialed in I will make a step by step with pictures for those of use who like this stuff but have weak skills

red192
Newcomer
Posts: 13
Joined: Fri Nov 13, 2015 5:17 am
United States of America

Re: Configuring XMLTV Prior to V31 upgrade

Post by red192 » Sun Feb 07, 2021 2:40 am

Ok I set aside the existing mythtv server i had on xubuntu 18.04 and built a new server using xubuntu 20.04.2
Installed xmltv from the repos
libxmltv-perl/focal,focal,now 0.6.1-1 all [installed,automatic]
xmltv-gui/focal,focal,now 0.6.1-1 all [installed,automatic]
xmltv-util/focal,focal,now 0.6.1-1 all [installed,automatic]
xmltv/focal,focal,now 0.6.1-1 all [installed]
Installed mythtv v31 and ran backend setup. Used the ten-foot interface to choose the Schedules Direct JSON grabber -- exited and did not run mythfilldatabase or start the server.

Checked the /home/scott/.xmltv directory -- its empty, no cache file

Ran the following from the command line for giggles:
scott@mythtv:~$ tv_grab_zz_sdjson --configure --config-file $HOME/.mythtv/SD.xmltv
Cache file for lineups, schedules and programs.
Cache file: [/home/scott/.xmltv/tv_grab_zz_sdjson.cache]
If you are migrating from a different grabber selecting an alternate channel ID format can make the migration easier.
Select channel ID format:
0: Default Format (eg: I12345.json.schedulesdirect.org)
1: tv_grab_na_dd Format (eg: I12345.labs.zap2it.com)
2: MythTV Internal DD Grabber Format (eg: 12345)
Select one: [0,1,2 (default=0)]
As the JSON data only includes the previously shown date normally the XML output should only have the date. However, some programs such as older versions of MythTV also need a time.
Select previously shown format:
0: Date Only
1: Date And Time
Select one: [0,1 (default=0)]
Schedules Direct username.
Username: xxxxxxxx
Schedules Direct password.
Password: xxxxxxxxx
** POST https://json.schedulesdirect.org/20141201/token ==> 200 OK (1s)
** GET https://json.schedulesdirect.org/20141201/status ==> 200 OK
** GET https://json.schedulesdirect.org/20141201/lineups ==> 200 OK
This step configures the lineups enabled for your Schedules Direct account. It impacts all other configurations and programs using the JSON API with your account. A maximum of 4 lineups can by added to your account. In a later step you will choose which lineups or channels to actually use for this configuration.
Current lineups enabled for your Schedules Direct account:
#. Lineup ID | Name | Location | Transport
1. USA-MI21637-X | Comcast South Oakland - Digital | Royal Oak | Cable
2. USA-OTA-48307 | Local Over the Air Broadcast | 48307 | Antenna
Edit account lineups: [continue,add,delete (default=continue)]
Choose whether you want to include complete lineups or individual channels for this configuration.
Select mode: [lineups,channels (default=lineups)]
** GET https://json.schedulesdirect.org/20141201/lineups ==> 200 OK (1s)
Choose lineups to use for this configuration.
USA-MI21637-X [yes,no,all,none (default=no)] yes
USA-OTA-48307 [yes,no,all,none (default=no)]
scott@mythtv:~$ cd .xmltv
scott@mythtv:~/.xmltv$ ls
scott@mythtv:~/.xmltv$
the script just stops after I choose lineups. In the gui I also got the "exited with status 0"
checked the .xmltv directory -- it's empty

No clue what i'm doing wrong.

red192
Newcomer
Posts: 13
Joined: Fri Nov 13, 2015 5:17 am
United States of America

Re: Configuring XMLTV Prior to V31 upgrade

Post by red192 » Sun Feb 07, 2021 2:50 am

In an earlier post Jeff noticed that ubuntu places the xmltv in usr instead of usr/local?
viewtopic.php?t=2120#p10429

Here's what mine shows -- maybe will try and install XMTV from source?
scott@mythtv:~$ tv_find_grabbers
/usr/bin/tv_grab_is|Iceland
/usr/bin/tv_grab_tr|T?rkiye - Digiturk (www.digiturk.com.tr)
/usr/bin/tv_grab_pt_vodafone|Portugal (Vodafone)
/usr/bin/tv_grab_na_tvmedia|North America (XMLTVListings.com by TVMedia)
/usr/bin/tv_grab_eu_dotmedia|Europe tv schedules for free (DEPRECATED)
/usr/bin/tv_grab_zz_sdjson_sqlite|Multinational (Schedules Direct JSON web services with SQLite DB)
/usr/bin/tv_grab_uk_tvguide|UK - TV Guide (tvguide.co.uk)
/usr/bin/tv_grab_dk_dr|TV Oversigten fra Danmarks Radios (2012)
/usr/bin/tv_grab_se_swedb|Sweden (swedb/tvsajten)
/usr/bin/tv_grab_huro|Hungary/Romania
/usr/bin/tv_grab_fi|Osa 9/10. Asiaohjelma, mikä ...
/usr/bin/tv_grab_na_dd|North America (Data Direct)
/usr/bin/tv_grab_eu_epgdata|Parts of Europe (commercial) (www.epgdata.com)
/usr/bin/tv_grab_pt_meo|Portugal (MEO)
/usr/bin/tv_grab_combiner|Combine data from several other grabbers
/usr/bin/tv_grab_se_tvzon|Sweden (DEPRECATED, TVZon)
/usr/bin/tv_grab_zz_sdjson|Schedules Direct JSON API
/usr/bin/tv_grab_dtv_la|Latin America Direct TV listings
/usr/bin/tv_grab_ar|Argentina
/usr/bin/tv_grab_il|Israel (tv-guide.walla.co.il)
/usr/bin/tv_grab_fi_sv|Finland (Swedish)
/usr/bin/tv_grab_uk_bleb|United Kingdom (bleb.org)
/usr/bin/tv_grab_eu_xmltvse|Europe TV schedules for free
/usr/bin/tv_grab_ch_search|Switzerland (tv.search.ch)
/usr/bin/tv_grab_it|Italy
/usr/bin/tv_grab_fr|France
/usr/bin/tv_grab_na_dtv|North America using www.directv.com
/bin/tv_grab_is|Iceland
/bin/tv_grab_tr|T?rkiye - Digiturk (www.digiturk.com.tr)
/bin/tv_grab_pt_vodafone|Portugal (Vodafone)
/bin/tv_grab_na_tvmedia|North America (XMLTVListings.com by TVMedia)
/bin/tv_grab_eu_dotmedia|Europe tv schedules for free (DEPRECATED)
/bin/tv_grab_zz_sdjson_sqlite|Multinational (Schedules Direct JSON web services with SQLite DB)
/bin/tv_grab_uk_tvguide|UK - TV Guide (tvguide.co.uk)
/bin/tv_grab_dk_dr|TV Oversigten fra Danmarks Radios (2012)
/bin/tv_grab_se_swedb|Sweden (swedb/tvsajten)
/bin/tv_grab_huro|Hungary/Romania
/bin/tv_grab_fi|Osa 9/10. Asiaohjelma, mikä ...
/bin/tv_grab_na_dd|North America (Data Direct)
/bin/tv_grab_eu_epgdata|Parts of Europe (commercial) (www.epgdata.com)
/bin/tv_grab_pt_meo|Portugal (MEO)
/bin/tv_grab_combiner|Combine data from several other grabbers
/bin/tv_grab_se_tvzon|Sweden (DEPRECATED, TVZon)
/bin/tv_grab_zz_sdjson|Schedules Direct JSON API
/bin/tv_grab_dtv_la|Latin America Direct TV listings
/bin/tv_grab_ar|Argentina
/bin/tv_grab_il|Israel (tv-guide.walla.co.il)
/bin/tv_grab_fi_sv|Finland (Swedish)
/bin/tv_grab_uk_bleb|United Kingdom (bleb.org)
/bin/tv_grab_eu_xmltvse|Europe TV schedules for free
/bin/tv_grab_ch_search|Switzerland (tv.search.ch)
/bin/tv_grab_it|Italy
/bin/tv_grab_fr|France
/bin/tv_grab_na_dtv|North America using www.directv.com
scott@mythtv:~$

User avatar
bill6502
Developer
Posts: 1895
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: Configuring XMLTV Prior to V31 upgrade

Post by bill6502 » Sun Feb 07, 2021 3:11 am

I assume you're following the Wiki in the forum link above.

Not sure about the giggle comment, but tv_grab_zz_sdjson --configure --config-file $HOME/.mythtv/SD.xmltv
just configures things, I'd expect it to "just stops". [I run the SQLite version, so won't help much, and I'm not
suggesting you need to use it].

red192
Newcomer
Posts: 13
Joined: Fri Nov 13, 2015 5:17 am
United States of America

Re: Configuring XMLTV Prior to V31 upgrade

Post by red192 » Sun Feb 07, 2021 3:30 am

I'm missing the tv_grab_zz_sdjson.cache file --

The wiki reads:
Setup with tv_grab_zz_sdjson
Run MythTV Setup and select Schedules Direct JSON API for your listings source.

You can complete the setup using the Configure button in MythTV Setup. If you prefer to use the command line, run

tv_grab_zz_sdjson --configure --config-file $HOME/.mythtv/SD.xmltv
Replace SD with the name of the source you had set up in MythTV Setup.

Enter your Schedules Direct user and password when prompted.
For channel id format you can select any of the supplied options. Option 0 is the same format as used by tv_grab_zz_sdjson_sqlite. Option 2 is the format used by Schedules Direct Data Direct.
Follow the prompts to complete setup.
When selecting channels the selection process allows you to skip remaining channels in the list and mark them all as yes or no.
If you need to do the channel selection again because of errors or channel changes, your previous values are lost. You have to do the entire selection again.
The reason I say "just stops" is because the wiki makes it sound like more should happen. "Follow the prompts to complete setup." and then goes on to talk about selecting channels. As a developer don't you sometimes give the user some feedback to indicate that things ran as expected? As the developer, you know what is supposed to happen and what the last prompt is. Users don't know. And since the tv_grab_zz_sdjson.cache is not appearing in my .xmltv directory -- I'm wondering if something did not work as expected.

For giggles, because the wiki is unclear (to me) and I have been struggling.

User avatar
bill6502
Developer
Posts: 1895
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: Configuring XMLTV Prior to V31 upgrade

Post by bill6502 » Sun Feb 07, 2021 4:09 am

Understood. Grabbers aren't MythTV products.

Looking at the above, I'm assuming that SD is the name of your MythTV source (which is fine).
MythTV appends .xmltv to the source name and looks in the .mythtv directory of the user running
the command. That's typically user mythtv when the backend runs it. I did all of my setup as user
mythtv and the files appear relative to that user. That way, there's only one set of files.

I'd run mythfilldatabase -v xmltv,system. You may get some clues there and the system logging will
display the grabber command used. I've used the command with the --debug flag (long ago) for
more info. I don't think it's need now.

Since I'm on the SQLite version, I'll go away now. Best of luck.

wesnewell
Senior
Posts: 728
Joined: Mon Jun 23, 2014 6:54 pm
Location: Wylie TX, USA
United States of America

Re: Configuring XMLTV Prior to V31 upgrade

Post by wesnewell » Sun Feb 07, 2021 6:10 am

Just a wag, but if you don't run mfdb, why would you expect to have a cache file? The mfdb run is what writes the cache file. Start the server, then from scott open a terminal and run mythfilldatabase with no args. You should then have a cache file wherever you specified it to be when you configured the grabber in setup. From above, that should be
/home/scott/.xmltv/tv_grab_zz_sdjson.cache
Make sure you run a channel scan in setup before you exit, if you haven't already and that it finds your channels.
BE/FE-Asrock AB350 Pro Ryzen 3 3200G, 6 atsc tuners. FE's-GF8200's Athlon II, Ryzen 3 2200G. Mythtv user since 2005.

red192
Newcomer
Posts: 13
Joined: Fri Nov 13, 2015 5:17 am
United States of America

Re: Configuring XMLTV Prior to V31 upgrade

Post by red192 » Tue Feb 09, 2021 1:52 pm

I have made earnest attempts to follow all of your suggestions, but I could never get everything working together. Something is amiss between my Xubuntu install, HDHR 3 Prime, the channel scanning, the schedules direct grabber, MythTV, and MySQL configuration. I just don't know what it is. I had been randomly trying things and making it all worse.

I made the mistake of changing too many things at once without fully understanding any of them. I added to the folly by thinking -- hey I will just upgrade the whole thing to 20.04. That resulted in a hot mess with a broken maria DB package and a partially upgraded OS that I no longer trusted. Being an idiot, I tinkered for a while -- then went nuclear and repaved, trying to start fresh. With MythTV, there really is no "nirvana setup" where everything "just works" as it used to in the Mythbuntu days. There are all kinds of "gotchas" to overcome -- like not having a MySQL networking configuration set properly.

I tried to follow the approach suggested by Ted here https://htpc.tedsblog.org/2020/10/mytht ... steps.html Ted has you using the very cool Mythbuntu control panel to help with some of the weird configuration workarounds needed ... and I got close but the "Schedules Direct users will need to do something different" part defeated me. I was delighted to find Ted's blog and simple 25-point multi-step guide to the promised land. A non-geek would look at this and say "hard pass." Me? I'm going to make another server and see if I can unlock how to build a v31 that works with Schedules Direct, Xfinity, and HDHR Prime. Thanks to everyone who makes MythTV great. It is wonderful and unique once you get it dialed in. One last thing -- make a backup of your database like I did. I'm back on V30 and using Schedules Direct DD -- works like a champ.

red192
Newcomer
Posts: 13
Joined: Fri Nov 13, 2015 5:17 am
United States of America

Re: Configuring XMLTV Prior to V31 upgrade

Post by red192 » Tue Feb 16, 2021 1:58 am

Update. I built a development version on another computer and finally succeeded in getting xmltv guide data. I'm not ready to write my guide on how to do it, but two things jumped out.

First, the version of XMLTV seems to make a difference. Despite following the wiki and dutifully checking my PATH settings the older version of XMLTV would always run instead of the newly compiled one -- so i installed XMLTV from the 18.04 repo, then removed it leaving all the dependencies in place. THEN made xmltv from source.

The other issue ... file permissions. I should know this by now. It's linux so ... because permissions! Wes said
Just make sure both users have write permission to the files.
I spent some time learning how to set the file ownership and permissions correctly for the directories, cache files and everything really because I'm still not sure what "the files" needing them are. I figure mostly it is the tv_grab_zz_sdjson.cache file that does appear in my user directory after running mythfilldatabase is what needs write permission -- and by the way, the DD method of getting listings was chatty showing lots of feedback while running -- the JSON grabber just appears to hang -- and the first time you run it mythfilldatabase shows a message saying you have no channels listed for SD.xmltv (wrong) while the script runs -- seemingly hung up.

Mythtv -- still the best and easily the most annoying DVR solution in the world -- but think of how much you will learn!! :D

Post Reply