XML mythfilldatabase

[mythfreak]

I just found all of my guide data gone. So it appears that schedules direct is pushing xml data local via some other grabber.

can someone please explain how this works? All of my channels are gone as well as upcoming programs.

So I run something like this: tv_grab_zz_sdjson --configure --config-file \$HOME/.mythtv/CableDigital.xmltv

then somewhere down the line mythfilldatabase has to run. Does the grabber use the config file to write to a local database that I will then run mythfilldatabase against?

[mythfreak]

Setup Floating point exception handler
2024-06-14 19:36:27.135705 I Setup Illegal instruction handler
2024-06-14 19:36:27.135718 I Setup Real-time signal 0 handler
2024-06-14 19:36:27.135728 I Setup Hangup handler
2024-06-14 19:36:27.135844 N Using runtime prefix = /usr
2024-06-14 19:36:27.135848 N Using configuration directory = /home/jamie/.mythtv
2024-06-14 19:36:27.135887 I Assumed character encoding: en_US.UTF-8
2024-06-14 19:36:27.136214 I Empty LocalHostName. This is typical.
2024-06-14 19:36:27.136220 I Using a profile name of: 'mythbackend' (Usually the same as this host's name.)
2024-06-14 19:36:27.136326 I Start up testing connections. DB 192.168.15.6, BE , attempt 0, status start, Delay: 2000
2024-06-14 19:36:28.167842 N Setting QT default locale to en_US
2024-06-14 19:36:28.167856 I Current locale en_US
2024-06-14 19:36:28.167920 N Reading locale defaults from /usr/share/mythtv//locales/en_us.xml
2024-06-14 19:36:28.189698 I PowerDBus: Failed to delay sleep,shutdown: Permission denied
2024-06-14 19:36:28.192439 I PowerDBus: Added UPower.Device '/org/freedesktop/UPower/devices/ups_hiddev0'
2024-06-14 19:36:28.192713 I Power: On AC power
2024-06-14 19:36:28.243731 I Power: Supported actions: None
2024-06-14 19:36:28.244727 I Loading en_us translation for module mythfrontend
2024-06-14 19:36:28.246540 I Current MythTV Schema Version (DBSchemaVer): 1361
2024-06-14 19:36:28.249882 I MythCoreContext::ConnectCommandSocket(): Connecting to backend server: 192.168.15.6:6543 (try 1 of 1)
2024-06-14 19:36:28.250933 I MythCoreContext::CheckProtoVersion(): Using protocol version 91 BuzzOff
2024-06-14 19:36:28.257401 I Opening blocking connection to master backend
2024-06-14 19:36:29.697946 I Updating source #6 (CableDigital) with grabber tv_grab_zz_sdjson
2024-06-14 19:36:29.704755 I Found 1038 channels for source 6 which use grabber
2024-06-14 19:36:29.705011 I Starting process signal handler
2024-06-14 19:36:29.705117 I Starting process manager
2024-06-14 19:36:29.705138 I Starting IO manager (read)
2024-06-14 19:36:29.707021 I Starting IO manager (write)
2024-06-14 19:36:30.356073 I Grabber has capabilities: baseline manualconfig preferredmethod
2024-06-14 19:36:31.006967 I Grabber prefers method: allatonce
2024-06-14 19:36:31.008031 I XMLTV config file is: /home/jamie/.mythtv/CableDigital.xmltv
2024-06-14 19:36:36.866241 E XMLTV grabber returned error code 255
2024-06-14 19:36:37.465691 E Malformed XML file, missing </tv> element, at line 1, Premature end of document.
2024-06-14 19:36:38.903970 I Updating source #17 (test) with grabber tv_grab_zz_sdjson
2024-06-14 19:36:38.904413 I No channels are configured to use grabber.
2024-06-14 19:36:39.517965 I Grabber has capabilities: baseline manualconfig preferredmethod
2024-06-14 19:36:40.168805 I Grabber prefers method: allatonce
2024-06-14 19:36:40.169806 I XMLTV config file is: /home/jamie/.mythtv/test.xmltv
2024-06-14 19:36:40.820935 E XMLTV grabber returned error code 1
2024-06-14 19:36:41.370734 E Malformed XML file, missing </tv> element, at line 1, Premature end of document.
2024-06-14 19:36:41.372193 E Failed to fetch some program info
2024-06-14 19:36:41.372238 I Adjusting program database end times.
2024-06-14 19:36:41.372639 I 0 replacements made
2024-06-14 19:36:41.372644 I Marking generic episodes.
2024-06-14 19:36:43.431556 I Found 0
2024-06-14 19:36:43.431563 I Extending non-unique programids with multiple parts.
2024-06-14 19:36:45.170501 I Found 0
2024-06-14 19:36:45.170509 I Fixing missing original airdates.
2024-06-14 19:36:48.839710 I Found 0 with programids
2024-06-14 19:36:48.840556 I Found 0 without programids
2024-06-14 19:36:48.840562 I Marking repeats.
2024-06-14 19:36:49.128076 I Found 0
2024-06-14 19:36:49.128085 I Unmarking new episode rebroadcast repeats.
2024-06-14 19:36:50.971295 I Found 0
2024-06-14 19:36:54.729151 I Marking episode first showings.
2024-06-14 19:36:59.248366 I Found 54793
2024-06-14 19:36:59.248374 I Marking episode last showings.
2024-06-14 19:37:03.795716 I Found 54187
2024-06-14 19:37:03.826406 I

[white_haired_uncle]

I just found all of my guide data gone. So it appears that schedules direct is pushing xml data local via some other grabber.

SD doesn't push anything, you pull via mythfilldatabase.

can someone please explain how this works? All of my channels are gone as well as upcoming programs.

It is suspicious if you are missing both channels and upcoming programs and not other things. Still, the very first thing I would look for is database corruption. First, check that your filesystem is not full. Then run mysqlcheck.

So I run something like this: tv_grab_zz_sdjson --configure --config-file \$HOME/.mythtv/CableDigital.xmltv

I assume that's a typo and you mean

Code: Select all

$HOME/.mythtv/CableDigital.xmltv

not

Code: Select all

\$HOME/.mythtv/CableDigital.xmltv

then somewhere down the line mythfilldatabase has to run. Does the grabber use the config file to write to a local database that I will then run mythfilldatabase against?

Not with the grabber you are using. It's stored to a file, I forget where. tv_grab_zz_sdjson_sqlite uses a DB.

Note this from your log:

Code: Select all

/home/jamie/.mythtv/test.xmltv

1) Jamie? This is normally done as mythtv, and the backend runs as mythtv. While it's possible to run them separately as different users, it looks like an error to me.
2) Path doesn't match what you're using above.
3) "test" isn't going to work, unless your Video Source is called "test".
4) Also,

Code: Select all

 Malformed XML file, missing </tv> element, at line 1, Premature end of document.

usually indicates there's something wrong with that file (most like caused by running things as the wrong user)

[mythfreak]

I know that I was all over the place in that post. I was fairly stressed and probably didn't make a lot of sense. On June 7th, I noticed that mythfilldatabase wasn't supported by Schedules direct. I had to convert to an XML format. All new to me.

I had one successful grab of 18 days but it doesn't include most of my HD channels. Instead I have a ton of SD channels that I didn't have before, so now I'm seeing most of my records are in SD.

I will go back and start over as mythtv user and delete the test source and see.

I tried to follow the wiki but got lost.

[white_haired_uncle]

mythfilldatabase is still the tool to use to fetch lineups, but the format has changed so you have to use a new method.

The initial configuration is not exactly transparent.

From what you say about HD channels missing, and "new" SD channels, I suspect you need to use the SQLite grabber.

Follow the directions here:

https://www.mythtv.org/wiki/XMLTV

In the Installation section, it talks about building your own grabber on ubuntu. In my experience, this is not necessary and I would skip it at least initially.

Use the command line, not mythtv-setup.

Since (if) you'll be using the sqlite version, skip the "Run the configuration" section.

The name of the xmltv file needs to match your Video Source, for example my Video Source is named "antenna" and my XMLTV config file is /home/mythtv/.mythtv/antenna.xmltv

If you take notes, and find yourself lost, just stop and post what you've done here and we'll get it working.

[mythfreak]

Actually the jamie user is fine. It's been running mythtv for 18 years.

I deleted the test source and stepped through the setup from mythtv-setup for the CableDigital source. After running mythfilldatabase, it has gotten many of the HD channels that I didn't have before but it's still showing all of the SD channels that are not selected in my Schedules Direct lineup.

[white_haired_uncle]

Selecting channels via the SD web site is no longer used, unfortunately.

[mythfreak]

Well that makes perfect sense now. The only lineup that I see at the schedules direct website is the one I can't use. That type is cable digital and NOT SD_JSON. Like you said, I can't edit it.

So command line it is.

After running tv_grab_zz_sdjson_sqlite --days 0 --config-file > channel_lineup.xml I can see all my channels!

mythfilldatabase --do-channel-updates --file --sourceid --xmlfile channel_lineup.xml or something along those lines are what I need to do in the cron for mythfilldatabase.

Thanks to all for the help.

[mythfreak]

Hi,

I'm getting closer to figuring this out at this point. I think it comes down to program guide data for 038 (SD) isn't duplicated to 838 (HD). See the XML:
Ideas?

<channel id="I10035.json.schedulesdirect.org">
<display-name>A&amp;E</display-name>
<display-name>AETV</display-name>
<display-name>38</display-name>
<icon src="https://schedulesdirect-api20141201-log ... w_270h.png" width="360" height="270" />
</channel>


<display-name>A&amp;E HD East</display-name>
<display-name>AETVHD</display-name>
<display-name>838</display-name>
<icon src="https://schedulesdirect-api20141201-log ... w_270h.png" width="360" height="270" />
</channel>

[white_haired_uncle]

You left off the "channel id=" line for AETVHD, which may be important.

Are you missing data for all HD channels which also have an SD channel, or just this one?

What options are you using with mythfilldatabase? The ones you listed above look rather odd to me.

[mythfreak]

Oh .. oops.

Oh ... oops ... yes I did.

<channel id="I60179.json.schedulesdirect.org">
<display-name>Fox News Channel HD</display-name>
<display-name>FNCHD</display-name>
<display-name>837</display-name>
<icon src="https://schedulesdirect-api20141201-log ... w_270h.png" width="360" height="270" />
</channel>

I went back into Mythtv and set the grabber to use tv_grab_zz_sdjson_sqlite. I then simply ran mythfilldatabase. with no options.

And yes ... I'm seeing several channels such as 040 has guide data but 840 doesn't. channel ID is 11150 for 040 but 56905. I'm also seeing some duplicate channel numbers like 2 of channel 807 with one having program guide date.

I think I need to run a channel scan.

[mythfreak]

It's been forever since I've done a channel scan. I believe I had PVR cards then. I seem to recall doing it via command line and importing it.
Sigh ....

[white_haired_uncle]

From the mythfilldatabase --help option, it looks like --file tells it not to use SD at all, but just use what is in the file you gave it (which probably has no guide data at all). I was thinking that your HD channels had no guide data because they didn't exist the last time you ran it "correctly". If that was the case, just running mythfilldatabase with no options should have fixed things.

The channel scan has "moved" in the new web interface. http://mythbackend:6544/setupwizard/input-connections

[mythfreak]

The guide data disappeared back on June 7th. I had a cron doing mythfilldatabase every night for years with no issues. The message from the last failed mythfilldatabase on the 7th was that the lineup had to be converted to the new json/xml format. That's when my troubles began.

[mythfreak]

Actually I think it was because I upgraded mythtv to version 31 that no longer supported lineups not xml/json.