MythFillDatabase not updating the Mythtv programming database when using XMLTV.

[amcamx]

I am at my wits end... I upgraded from Ubuntu 18.04 to 20.04 and with it I got the Mythtv upgrade (.29 to .31). This is ok with me but the changes to the Schedules direct process has rally thrown me for a loop.

The problem I have encountered is related to MythFillDatabase.

I followed the instructions on this page ( The MythTv XMLTV Wiki link ).

I opted to use the Sqlite grabber

When I run mythfilldatabase from terminal, it runs for about 10 minutes and then returns with a Killed message.

When I run it from within the Myth Backend (when exiting the backend I choose to run MythFillDatabase), it runs without any apparent errors.

When I run the MythTv frontend and go to the system information screen, it shows that it has run successfully.

Oddly, there is not any channel information being populated in the Program Guide.

any help would be appreciated.

Thank you,

AmcAmx

[pgbennett]

Try running mythfilldatabase with -v xmltv to see if the grabber is displaying any error messages.

[amcamx]

this is the end of what ran....

Code: Select all

2020-11-02 17:11:01.993864 I  Current MythTV Schema Version (DBSchemaVer): 1361
2020-11-02 17:11:02.000166 I  MythCoreContext::ConnectCommandSocket(): Connecting to backend server: 127.0.0.1:6543 (try 1 of 1)
2020-11-02 17:11:02.001895 I  MythCoreContext::CheckProtoVersion(): Using protocol version 91 BuzzOff
2020-11-02 17:11:02.002852 I  Opening blocking connection to master backend
2020-11-02 17:11:08.678616 I  Updating source #1 (HDHomeRun1) with grabber tv_grab_zz_sdjson_sqlite
2020-11-02 17:11:08.686334 I  Found 1019 channels for source 1 which use grabber
2020-11-02 17:11:10.202348 I  Grabber has capabilities: baseline manualconfig preferredmethod lineups apiconfig 
2020-11-02 17:11:11.253995 I  Grabber prefers method: allatonce
2020-11-02 17:11:11.258884 I  XMLTV config file is: /home/amcamx/.mythtv/HDHomeRun1.xmltv
2020-11-02 17:11:11.267719 I  Grabber Command: nice tv_grab_zz_sdjson_sqlite --config-file '/home/amcamx/.mythtv/HDHomeRun1.xmltv' --output /tmp/mythsoO2rf
2020-11-02 17:11:11.267729 I  ----------------- Start of XMLTV output -----------------
2020-11-02 17:15:23.084693 I  ------------------ End of XMLTV output ------------------
Killed
amcamx@mythman:~$

[amcamx]

Here is the results using the -v all switch:

Code: Select all

2020-11-02 20:14:15.586345 I  (0x7f45bc005b20)::DecrRef() -> 1
2020-11-02 20:14:15.587733 I  Grabber prefers method: allatonce
2020-11-02 20:14:15.587761 I  (0x7f45bc005b20)::DecrRef() -> 0
2020-11-02 20:14:15.592562 I  MSqlQuery::exec(DBManager1) SELECT configpath FROM videosource WHERE sourceid = '1' AND configpath IS NOT NULL <<<< Took 4ms, Returned 0 row(s)
2020-11-02 20:14:15.592582 I  XMLTV config file is: /home/amcamx/.mythtv/HDHomeRun1.xmltv
2020-11-02 20:14:15.594809 I  MSqlQuery::exec(DBManager1) DELETE FROM settings WHERE value = 'MythFillSuggestedRunTime' AND hostname is NULL; <<<< Took 1ms
2020-11-02 20:14:15.595202 I  MSqlQuery::exec(DBManager1) INSERT INTO settings (value,data ) VALUES ( 'MythFillSuggestedRunTime', '2020-11-04T01:14:15Z' ); <<<< Took 0ms
2020-11-02 20:14:15.595689 I  MSqlQuery::exec(DBManager1) DELETE FROM settings WHERE value = 'mythfilldatabaseLastRunStart' AND hostname is NULL; <<<< Took 0ms
2020-11-02 20:14:15.596022 I  MSqlQuery::exec(DBManager1) INSERT INTO settings (value,data ) VALUES ( 'mythfilldatabaseLastRunStart', '2020-11-03T01:14:15Z' ); <<<< Took 0ms
2020-11-02 20:14:15.596441 I  MSqlQuery::exec(DBManager1) DELETE FROM settings WHERE value = 'mythfilldatabaseLastRunStatus' AND hostname is NULL; <<<< Took 0ms
2020-11-02 20:14:15.596773 I  MSqlQuery::exec(DBManager1) INSERT INTO settings (value,data ) VALUES ( 'mythfilldatabaseLastRunStatus', 'currently running.' ); <<<< Took 0ms
2020-11-02 20:14:15.596803 I  Grabber Command: nice tv_grab_zz_sdjson_sqlite --config-file '/home/amcamx/.mythtv/HDHomeRun1.xmltv' --output /tmp/myth3qA1Rr
2020-11-02 20:14:15.596810 I  ----------------- Start of XMLTV output -----------------
2020-11-02 20:14:15.599033 I  Managed child (PID: 47856) has started! * command=nice tv_grab_zz_sdjson_sqlite --config-file '/home/amcamx/.mythtv/HDHomeRun1.xmltv' --output /tmp/myth3qA1Rr, timeout=0
2020-11-02 20:14:15.599088 I  (0x7f45bc005b20)::IncrRef() -> 2
2020-11-02 20:17:48.435118 I  Managed child (PID: 47856) has exited! command=nice tv_grab_zz_sdjson_sqlite --config-file '/home/amcamx/.mythtv/HDHomeRun1.xmltv' --output /tmp/myth3qA1Rr, status=0, result=0
2020-11-02 20:17:48.453030 I  (0x7f45bc005b20)::DecrRef() -> 0
2020-11-02 20:17:48.453045 I  (0x7f45bc005b20)::DecrRef() -> 1
2020-11-02 20:17:48.453112 I  ------------------ End of XMLTV output ------------------
2020-11-02 20:17:48.455698 I  MSqlQuery::exec(DBManager1) DELETE FROM settings WHERE value = 'mythfilldatabaseLastRunEnd' AND hostname is NULL; <<<< Took 1ms
2020-11-02 20:17:48.456333 I  MSqlQuery::exec(DBManager1) INSERT INTO settings (value,data ) VALUES ( 'mythfilldatabaseLastRunEnd', '2020-11-03T01:17:48Z' ); <<<< Took 0ms
2020-11-02 20:17:48.457404 I  MSqlQuery::exec(DBManager1) DELETE FROM settings WHERE value = 'mythfilldatabaseLastRunStatus' AND hostname is NULL; <<<< Took 0ms
2020-11-02 20:17:48.457986 I  MSqlQuery::exec(DBManager1) INSERT INTO settings (value,data ) VALUES ( 'mythfilldatabaseLastRunStatus', 'Successful.' ); <<<< Took 0ms
2020-11-02 20:17:48.467128 I  MSqlQuery::exec(DBManager1) SELECT data FROM settings WHERE value = 'moviegrabber' AND hostname = 'mythman' <<<< Took 0ms, Returned 0 row(s)
2020-11-02 20:17:48.467923 I  MSqlQuery::exec(DBManager1) SELECT data FROM settings WHERE value = 'moviegrabber' AND hostname IS NULL <<<< Took 0ms, Returned 1 row(s)
2020-11-02 20:17:48.470224 I  Managed child (PID: 47893) has started! * command=/usr/share/mythtv/metadata/Movie/tmdb3.py -v, timeout=0
2020-11-02 20:17:48.470295 I  (0x7f45c40036b0)::IncrRef() -> 2
2020-11-02 20:17:48.751613 I  Managed child (PID: 47893) has exited! command=/usr/share/mythtv/metadata/Movie/tmdb3.py -v, status=0, result=0
2020-11-02 20:17:48.755626 I  (0x7f45c40036b0)::DecrRef() -> 1
2020-11-02 20:17:48.756466 I  (0x7f45c40036b0)::DecrRef() -> 0
2020-11-02 20:17:48.758155 I  MSqlQuery::exec(DBManager1) SELECT data FROM settings WHERE value = 'televisiongrabber' AND hostname = 'mythman' <<<< Took 0ms, Returned 0 row(s)
2020-11-02 20:17:48.759006 I  MSqlQuery::exec(DBManager1) SELECT data FROM settings WHERE value = 'televisiongrabber' AND hostname IS NULL <<<< Took 0ms, Returned 1 row(s)
2020-11-02 20:17:48.762961 I  Managed child (PID: 47895) has started! * command=/usr/share/mythtv/metadata/Television/ttvdb.py -v, timeout=0
2020-11-02 20:17:48.763102 I  (0x561dd4570f50)::IncrRef() -> 2
2020-11-02 20:17:49.370364 I  Managed child (PID: 47895) has exited! command=/usr/share/mythtv/metadata/Television/ttvdb.py -v, status=0, result=0
2020-11-02 20:17:49.407662 I  (0x561dd4570f50)::DecrRef() -> 1
2020-11-02 20:17:49.407856 I  (0x561dd4570f50)::DecrRef() -> 0
Killed

[bill6502]

Looks like an old version that doesn't log XMLTV grabber output. Please resist the urge to use -v all,
it's normally too hard to sift through the clutter.

Run the grabber directly: tv_grab_zz_sdjson_sqlite --config-file '/home/amcamx/.mythtv/HDHomeRun1.xmltv' --output /tmp/mytest
You'll need to remove the /tmp/mytest file (which is normally done when mythfilldatabase calls the grabber.)

[amcamx]

Here are the results from:

tv_grab_zz_sdjson_sqlite --config-file '/home/amcamx/.mythtv/HDHomeRun1.xmltv' --output /tmp/mytest

amcamx@mythman:~$ tv_grab_zz_sdjson_sqlite --config-file '/home/amcamx/.mythtv/HDHomeRun1.xmltv' --output /tmp/mytest

Code: Select all

Opening the local database
Obtaining authentication token for Schedules Direct
   Schedules Direct account expires on 2021-04-10T19:54:00
   Schedules Direct data last updated on 2020-11-02T20:04:28
Downloading data from Schedules Direct
   not downloading lineups from Schedules Direct (data current)
   not downloading channel and station maps for lineup USA-FL65018-X (data current)
   not downloading station schedule hashes (data current)
   not downloading daily schedules (data current)
   not downloading programs (data current)
   not downloading programs for supplemental data (data current)
Processing data and creating XMLTV output
   840 channels processed
   235859 program schedules processed
Pruning the local database
amcamx@mythman:~$

[bill6502]

The above just says that all data that's been downloaded previously
is up to date (the hash sums haven't changed) so there's nothing to
do. I'd try again Tuesday maybe after 0900.

I assume your source is named HDHomeRun1 and USA-FL65018-X is
a valid lineup.

Usually, when the results are different when different users run the grabber,
it suggests a setup issue. See if the example here helps:
viewtopic.php?f=36&t=3480&start=15#p19508

[amcamx]

Yea, I was thinking that was what it was doing with no downloading the data......

I tried again this morning and got the same thing.

<I assume your source is named HDHomeRun1 and USA-FL65018-X is
a valid lineup.> yes

This part only shows what is being downloaded. What is the process that actually imports it into Mythtv, that is done by the mythfilldatabase correct ?

ThanX,

AmcAmx

[bill6502]

The grabber gets the data and mythfilldatabase adds it to the MythTV DB (mythconverg).

Lets look at the SchedulesDirect DB. Install sqlite3: sudo apt install sqlite3

cd to the directory where the DB is. Should be: ~mythtv/.xmltv

Run this command: sqlite3 -column -header -readonly SchedulesDirect.DB 'SELECT * FROM channels WHERE selected=1'

[amcamx]

I had done that already and it returned all my channels as it should. After messing around with some manual runs, and looking at the database I am pretty sold that mythfilldatabase is not loading the MythDatabase correctly... maybe something did not work properly in the upgrade process, causing it to not install the latest version of one of the programs. very perplexing...

just an FYI... I am running the backend/frontend on a single system. Have for years and not had any problems. Oh another interesting thing is, I am running a HDHomeRun Prime and all 3 sources have been assigned... but... when running mythfulldatabase it only pulls for the first source... source 2 and 3 are not seen... maybe it could be something to do with the source setups being hosed up during the upgrade ? I considered dumping my sources and rebuilding them to see if it makes a difference... your thoughts on this ?

Here is a sample of the data...

sqlite> select * from channels where selected=1;
lineup station channum selected details
------------- ---------- ---------- ---------- -------------------------------------
USA-FL65018-X 11771 2 1 {"channel":"002","stationID":"11771"}
USA-FL65018-X 11301 3 1 {"channel":"003","stationID":"11301"}
USA-FL65018-X 11349 4 1 {"channel":"004","stationID":"11349"}
USA-FL65018-X 11069 5 1 {"channel":"005","stationID":"11069"}
USA-FL65018-X 11916 6 1 {"channel":"006","stationID":"11916"}
USA-FL65018-X 11857 7 1 {"channel":"007","stationID":"11857"}
USA-FL65018-X 16450 8 1 {"channel":"008","stationID":"16450"}
USA-FL65018-X 11654 9 1 {"channel":"009","stationID":"11654"}
USA-FL65018-X 11780 10 1 {"channel":"010","stationID":"11780"}
USA-FL65018-X 11398 11 1 {"channel":"011","stationID":"11398"}
USA-FL65018-X 23320 12 1 {"channel":"012","stationID":"23320"}
USA-FL65018-X 11836 13 1 {"channel":"013","stationID":"11836"}
USA-FL65018-X 10161 14 1 {"channel":"014","stationID":"10161"}
USA-FL65018-X 66966 15 1 {"channel":"015","stationID":"66966"}
USA-FL65018-X 12896 16 1 {"channel":"016","stationID":"12896"}
USA-FL65018-X 11652 17 1 {"channel":"017","stationID":"11652"}
USA-FL65018-X 32039 19 1 {"channel":"019","stationID":"32039"}
USA-FL65018-X 11991 20 1 {"channel":"020","stationID":"11991"}
USA-FL65018-X 11504 21 1 {"channel":"021","stationID":"11504"}
USA-FL65018-X 36290 22 1 {"channel":"022","stationID":"36290"}
USA-FL65018-X 12013 23 1 {"channel":"023","stationID":"12013"}
USA-FL65018-X 10269 24 1 {"channel":"024","stationID":"10269"}
USA-FL65018-X 91097 25 1 {"channel":"025","stationID":"91097"}
USA-FL65018-X 11207 26 1 {"channel":"026","stationID":"11207"}
USA-FL65018-X 11187 27 1 {"channel":"027","stationID":"11187"}

Bill - Thank you for all your help on this... I have 3 days of schedule info left.. trying to get it done before I have none...

AmcAmx

[pgbennett]

You only need one source. Three capture cards, one video source, three input connections.

I recommend you check your setup of capture cards, video source and input connections. Something is likely inconsistent.

[bill6502]

If that doesn't help, see if there are any hash sums: sqlite3 -column -header -readonly SchedulesDirect.DB 'SELECT * FROM schedules_hash'

It makes sense that there's no new data if matching hashes are returned, but
that shouldn't be the case when programs are updated, I've had to wait
24 hours for new ones to appear.

Note that if you run the grabber from the command line, you can look in the output
file to see what data was returned, assuming there were changes. Also, you can
append the --debug switch to the command for more info.

[bill6502]

See if the XMLTVs have been set to the proper values too: mysql --user=mythtv --password=<from config.xml> --host=localhost --execute="SELECT chanid,callsign,xmltvid FROM channel WHERE chanid=11771" mythconverg
Expect lines like:

Code: Select all

+--------+----------+----------------------------------+
| chanid | callsign | xmltvid                          |
+--------+----------+----------------------------------+
|   1021 | CBS2-HD  | I20454.json.schedulesdirect.org  |

[amcamx]

Bill6502 and pgbennett,

Thank you both for all the help. I decided to drop all my capture cards, the source, and input connections.
Although they all appeared correct... for some reason they were not.

Once I dropped them I was able to see all three capture cards correctly.

BUT.. I am still having the same issue...

I have reverified everything previous to this message.. So either I am very good at duplicating the condition or I still have something weird going on. It appears that the xmltv part is working fine, but the mythfilldatabase process is refusing to load the data from the SchedulesDirect.DB.

This continues to happen ( the proccess ends witha KILLED):

Code: Select all

amcamx@mythman:~/.xmltv$ mythfilldatabase -v xmltv
2020-11-04 21:11:13.777013 C  mythfilldatabase version: fixes/31 [v31.0-v31.0] www.mythtv.org
2020-11-04 21:11:13.777054 C  Qt version: compile: 5.12.5, runtime: 5.12.8
2020-11-04 21:11:13.777367 I  Ubuntu 20.04.1 LTS (x86_64)
2020-11-04 21:11:13.777375 N  Enabled verbose msgs:  general xmltv
2020-11-04 21:11:13.777408 N  Setting Log Level to LOG_INFO
2020-11-04 21:11:13.787978 I  Added logging to the console
2020-11-04 21:11:13.794628 I  Setup Interrupt handler
2020-11-04 21:11:13.794647 I  Setup Terminated handler
2020-11-04 21:11:13.794659 I  Setup Segmentation fault handler
2020-11-04 21:11:13.794677 I  Setup Aborted handler
2020-11-04 21:11:13.794690 I  Setup Bus error handler
2020-11-04 21:11:13.794710 I  Setup Floating point exception handler
2020-11-04 21:11:13.794727 I  Setup Illegal instruction handler
2020-11-04 21:11:13.794743 I  Setup Real-time signal 0 handler
2020-11-04 21:11:13.794759 I  Setup Hangup handler
2020-11-04 21:11:13.794932 N  Using runtime prefix = /usr
2020-11-04 21:11:13.794940 N  Using configuration directory = /home/amcamx/.mythtv
2020-11-04 21:11:13.795006 I  Assumed character encoding: en_US.UTF-8
2020-11-04 21:11:13.798700 I  Empty LocalHostName. This is typical.
2020-11-04 21:11:13.798718 I  Using a profile name of: 'mythman' (Usually the same as this host's name.)
2020-11-04 21:11:13.798803 I  Start up testing connections. DB localhost, BE , attempt 0, status start, Delay: 2000
2020-11-04 21:11:14.801711 I  Start up testing connections. DB localhost, BE , attempt 1, status start, Delay: 2000
2020-11-04 21:11:14.802955 I  Starting process signal handler
2020-11-04 21:11:14.805209 I  Starting process manager
2020-11-04 21:11:14.805328 I  Starting IO manager (read)
2020-11-04 21:11:14.805423 I  Starting IO manager (write)
2020-11-04 21:11:16.073255 N  Setting QT default locale to en_US
2020-11-04 21:11:16.073276 I  Current locale en_US
2020-11-04 21:11:16.076641 N  Reading locale defaults from /usr/share/mythtv//locales/en_us.xml
2020-11-04 21:11:16.237875 I  Power: On AC power
2020-11-04 21:11:16.237907 I  Power: Supported actions: Restart,Shutdown
2020-11-04 21:11:16.241599 I  Loading en_us translation for module mythfrontend
2020-11-04 21:11:16.245979 I  Current MythTV Schema Version (DBSchemaVer): 1361
2020-11-04 21:11:16.251154 I  MythCoreContext::ConnectCommandSocket(): Connecting to backend server: 127.0.0.1:6543 (try 1 of 1)
2020-11-04 21:11:16.265869 I  MythCoreContext::CheckProtoVersion(): Using protocol version 91 BuzzOff
2020-11-04 21:11:16.270083 I  Opening blocking connection to master backend
2020-11-04 21:11:16.289298 I  Updating source #1 (HDHomeRunPrime) with grabber tv_grab_zz_sdjson_sqlite
2020-11-04 21:11:16.289757 I  No channels are configured to use grabber.
2020-11-04 21:11:17.859714 I  Grabber has capabilities: baseline manualconfig preferredmethod lineups apiconfig 
2020-11-04 21:11:18.910524 I  Grabber prefers method: allatonce
2020-11-04 21:11:18.920166 I  XMLTV config file is: /home/amcamx/.mythtv/HDHomeRunPrime.xmltv
2020-11-04 21:11:18.928446 I  Grabber Command: nice tv_grab_zz_sdjson_sqlite --config-file '/home/amcamx/.mythtv/HDHomeRunPrime.xmltv' --output /tmp/mythgOoqry
2020-11-04 21:11:18.928460 I  ----------------- Start of XMLTV output -----------------
2020-11-04 21:14:50.869064 I  ------------------ End of XMLTV output ------------------
Killed
amcamx@mythman:~/.xmltv$

[bill6502]

Break the problem in half. Run the grabber from the command line and put the output
in /tmp/test. Look in your /tmp/test file. The last program in it should look similar to this:

Code: Select all

 <programme channel="I71420.json.schedulesdirect.org" start="20201121233000 +0000" stop="20201122000000 +0000">
  <title>Paid Programming</title>
  <desc lang="en">Paid programming.</desc>
  <credits></credits>
  <category>tvshow</category>
  <category>Shopping</category>
  <category>Paid Programming</category>
  <category>Show</category>
  <episode-num system="dd_progid">SH00000001.0000</episode-num>
  <previously-shown />
  <rating system="CB">
   <value>G</value>
  </rating>
  <rating system="CHVRS">
   <value>G</value>
  </rating>
  <rating system="RTC">
   <value>A</value>
  </rating>
  <rating system="Régie du cinéma">
   <value>G</value>
  </rating>
  <rating system="VCHIP">
   <value>TV-G</value>
  </rating>
 </programme>
</tv>

Then do: mythfilldatabase -v xmltv --file --xmlfile /tmp/test --sourceid 1 assuming
your sourceid is 1. You can verify your sourceid with: <your backend>:6544/Channel/GetVideoSourceList