Upcoming Recordings suddenly blank -- Scheduler error?

Have a MythTV related problem? Ask for help from other MythTV users here.

Moderator: Forum Moderators

MSorice
Junior
Posts: 15
Joined: Wed Oct 27, 2021 1:18 am
United States of America

Upcoming Recordings suddenly blank -- Scheduler error?

Post by MSorice »

Suddenly today, my Upcoming Recordings is blank and several other odd things are happening.

What's normal: I have listings data and these are updating; I can watch live TV and recordings; all recording rules are still present and even list instances that would normally be recorded, though these are all listed as "not recording".

What's not working: upcoming recordings are blank. Setting recordings from e.g. program guide results in nothing. Trying to call up "Previously Recorded" results in infinite delay. (I'm not sure if regularly scheduled programs will record -- the next thing I know I had scheduled is tomorrow at 4:30. I know something recorded today, but this issue arose since 2 days ago - it was normal then.)

I'm running: Ubuntu 20.04 (kernel 5.11.0-342109090909).
MythTV 32 from the Mythbuntu PPA (v32.0~master.202110241541.0374b68ac8~ubuntu20.04.1)
MySQL latest from repo (8.0.27-0ubuntu0.20.04.1).

I noticed in /var/log/mythtvmythbackend.log, I have this error that began showing up about the same time as the problem:
Oct 26 19:57:04 Dangerous mythbackend: mythbackend[1239]: E Scheduler mythdb.cpp
:202 (DBError) DB Error (AddNewRecords):#012Query was:#012SELECT c.chanid,
c.sourceid, p.starttime, p.endtime, p.title,
p.subtitle, p.description, c.channum, c.cal
lsign, c.name, oldrecduplicate, p.category, sched_temp_record.recpriority, sched_temp_record.dupin, recduplicate, findduplicate, sched_temp_record.type, sched_temp_record.recordid, p.starttime - INTERVAL sched_temp_record.startoffset minute AS recstartts, p.endtime + INTERVAL sched_temp_record.endoffset minute AS recendts, p.previouslyshown, sched_temp_record.recgroup, sched_temp_record.dupmethod, c.commmethod, capturecard.cardid, 0, p.seriesid, p.programid, sched_temp_record.inetref, p.category_type, p.airdate, p.stars, p.originalairdate, sched_temp_record.inactive, sched_temp_record.parentid, recordmatch.findid, sched_temp_record.playgroup, oldrecstatus.recstatus, oldrecstatus.reactivate, p.videoprop+0, p.subtitletypes+0, p.audioprop+0, sched_temp_record.storagegroup, capturecard.hostname, recordmatch.oldrecstatus, NULL, oldrecstatus.future, capturecard.schedorder, p.syndicatedepisodenumber, p.partnumber, p.parttotal, c.mplexid, capturecard.displayname, p.season, p.episode, p.totalepisodes, c.recpriority + capturecard.recpriority + (capturecard.cardid = sched_temp_record.prefinput) * 2 + (FIND_IN_SET('SIGNED', p.subtitletypes) > 0) * -1 + (FIND_IN_SET('ONSCREEN', p.subtitletypes) > 0) * -1 AS powerpriority FROM recordmatch INNER JOIN sched_temp_record ON (recordmatch.recordid = sched_temp_record.recordid) INNER JOIN program AS p ON ( recordmatch.chanid = p.chanid AND recordmatch.starttime = p.starttime AND recordmatch.manualid = p.manualid ) INNER JOIN channel AS c ON ( c.chanid = p.chanid ) INNER JOIN capturecard ON ( c.sourceid = capturecard.sourceid AND ( capturecard.schedorder <> 0 OR capturecard.parentid = 0 ) ) LEFT JOIN oldrecorded as oldrecstatus ON ( oldrecstatus.station = c.callsign AND oldrecstatus.starttime = p.starttime AND oldrecstatus.title = p.title ) WHERE p.endtime > (NOW() - INTERVAL 480 MINUTE) ORDER BY sched_temp_record.recordid DESC, p.starttime, p.title, c.callsign, c.channum #012Driver error was [2/1690]:#012QMYSQL3: Unable to store statement results#012Database error was:#012BIGINT UNSIGNED value is out of range in '((`mythconverg`.`c`.`recpriority` + `mythconverg`.`capturecard`.`recpriority`) + ((`mythconverg`.`capturecard`.`cardid` = `mythconverg`.`sched_temp_record`.`prefinput`) * 2))'
This seems pretty likely to be related, but where it came from and how to fix it, I don't know.

I've tried:
Restarting mythbackend
Restarting mysql
Rebooting.
MSorice
Junior
Posts: 15
Joined: Wed Oct 27, 2021 1:18 am
United States of America

Re: Upcoming Recordings suddenly blank -- Scheduler error?

Post by MSorice »

To check off some other issues I've tried based on previous threads:

All relevant disks have at least 10s of GB free.

There's nothing much of interest in /var/log/mysql/error.log:
2021-10-27T01:38:20.127596Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.27-0ubuntu0.20.04.1) starting as process 1149
2021-10-27T01:38:20.144405Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-10-27T01:38:20.468328Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2021-10-27T01:38:20.683286Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
2021-10-27T01:38:20.683305Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
2021-10-27T01:38:20.684493Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2021-10-27T01:38:20.684522Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2021-10-27T01:38:20.715103Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2021-10-27T01:38:20.715143Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.27-0ubuntu0.20.04.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu).
mysqlcheck looks pretty normal to me:
$ sudo mysqlcheck --login-path=mythtv -r mythconverg
mythconverg.archiveitems OK
mythconverg.bdbookmark OK
mythconverg.callsignnetworkmap OK
mythconverg.capturecard OK
mythconverg.cardinput OK
mythconverg.channel OK
mythconverg.channelgroup OK
mythconverg.channelgroupnames OK
mythconverg.channelscan OK
mythconverg.channelscan_channel OK
mythconverg.channelscan_dtv_multiplex OK
mythconverg.codecparams OK
mythconverg.credits OK
mythconverg.customexample OK
mythconverg.diseqc_config OK
mythconverg.diseqc_tree OK
mythconverg.displayprofilegroups OK
mythconverg.displayprofiles OK
mythconverg.dtv_multiplex OK
mythconverg.dtv_privatetypes OK
mythconverg.dvdbookmark OK
mythconverg.dvdinput OK
mythconverg.dvdtranscode OK
mythconverg.eit_cache OK
mythconverg.filemarkup OK
mythconverg.gallery_directories OK
mythconverg.gallery_files OK
mythconverg.gallerymetadata OK
mythconverg.gamemetadata OK
mythconverg.gameplayers OK
mythconverg.housekeeping OK
mythconverg.inputgroup OK
mythconverg.internetcontent OK
mythconverg.internetcontentarticles OK
mythconverg.inuseprograms OK
mythconverg.iptv_channel OK
mythconverg.jobqueue OK
mythconverg.jumppoints OK
mythconverg.keybindings OK
mythconverg.keyword OK
mythconverg.livestream OK
mythconverg.logging OK
mythconverg.movies_movies OK
mythconverg.movies_showtimes OK
mythconverg.movies_theaters OK
mythconverg.music_albumart OK
mythconverg.music_albums OK
mythconverg.music_artists OK
mythconverg.music_directories OK
mythconverg.music_genres OK
mythconverg.music_playlists OK
mythconverg.music_radios
note : The storage engine for the table doesn't support repair
mythconverg.music_smartplaylist_categories OK
mythconverg.music_smartplaylist_items OK
mythconverg.music_smartplaylists OK
mythconverg.music_songs OK
mythconverg.music_stats OK
mythconverg.music_streams
note : The storage engine for the table doesn't support repair
mythconverg.mythlog OK
mythconverg.mythweb_sessions
note : The storage engine for the table doesn't support repair
mythconverg.networkiconmap OK
mythconverg.newssites OK
mythconverg.oldfind OK
mythconverg.oldprogram OK
mythconverg.oldrecorded OK
mythconverg.people OK
mythconverg.phonecallhistory OK
mythconverg.phonedirectory OK
mythconverg.pidcache OK
mythconverg.playgroup OK
mythconverg.powerpriority OK
mythconverg.profilegroups OK
mythconverg.program OK
mythconverg.programgenres OK
mythconverg.programrating OK
mythconverg.recgrouppassword OK
mythconverg.recgroups OK
mythconverg.record OK
mythconverg.record_tmp
note : The storage engine for the table doesn't support repair
mythconverg.recorded OK
mythconverg.recordedartwork OK
mythconverg.recordedcredits OK
mythconverg.recordedfile OK
mythconverg.recordedmarkup OK
mythconverg.recordedprogram OK
mythconverg.recordedrating OK
mythconverg.recordedseek OK
mythconverg.recordfilter OK
mythconverg.recordingprofiles OK
mythconverg.recordmatch OK
mythconverg.roles OK
mythconverg.romdb OK
mythconverg.scannerfile OK
mythconverg.scannerpath OK
mythconverg.schemalock OK
mythconverg.settings OK
mythconverg.storagegroup OK
mythconverg.tvchain OK
mythconverg.tvosdmenu OK
mythconverg.upnpmedia OK
mythconverg.user_permissions OK
mythconverg.user_sessions OK
mythconverg.users OK
mythconverg.videocast OK
mythconverg.videocategory OK
mythconverg.videocollection OK
mythconverg.videocountry OK
mythconverg.videogenre OK
mythconverg.videometadata OK
mythconverg.videometadatacast OK
mythconverg.videometadatacountry OK
mythconverg.videometadatagenre OK
mythconverg.videopart OK
mythconverg.videopathinfo OK
mythconverg.videosource OK
mythconverg.videotypes OK
mythconverg.weatherdatalayout
note : The storage engine for the table doesn't support repair
mythconverg.weatherscreens
note : The storage engine for the table doesn't support repair
mythconverg.weathersourcesettings
note : The storage engine for the table doesn't support repair
mythconverg.websites OK
User avatar
bill6502
Developer
Posts: 2299
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: Upcoming Recordings suddenly blank -- Scheduler error?

Post by bill6502 »

Can you do these from mysql?

Code: Select all

SELECT cardid,recpriority FROM capturecard WHERE recpriority != 0 ORDER BY cardid;
SELECT chanid,recpriority FROM channel WHERE recpriority != 0 ORDER BY chanid;
MSorice
Junior
Posts: 15
Joined: Wed Oct 27, 2021 1:18 am
United States of America

Re: Upcoming Recordings suddenly blank -- Scheduler error?

Post by MSorice »

mysql> SELECT cardid,recpriority FROM capturecard WHERE recpriority != 0 ORDER BY cardid;
+--------+-------------+
| cardid | recpriority |
+--------+-------------+
| 1 | 1 |
| 2 | 1 |
+--------+-------------+
2 rows in set (0.00 sec)

mysql> SELECT chanid,recpriority FROM channel WHERE recpriority != 0 ORDER BY chanid;
+--------+-------------+
| chanid | recpriority |
+--------+-------------+
| 1201 | -1 |
| 1202 | -1 |
| 1203 | -1 |
| 1204 | -1 |
| 1431 | -2 |
| 1551 | -1 |
| 1552 | -1 |
| 1553 | -1 |
+--------+-------------+
8 rows in set (0.00 sec)
MSorice
Junior
Posts: 15
Joined: Wed Oct 27, 2021 1:18 am
United States of America

Re: Upcoming Recordings suddenly blank -- Scheduler error?

Post by MSorice »

Symptom updates:
MSorice wrote:
Wed Oct 27, 2021 1:36 am
...all recording rules are still present and even list instances that would normally be recorded, though these are all listed as "not recording".
Trying to bring up upcoming instances with Recording Rules>Menu>Upcoming still shows the list with "Not recording" next to any instances, but also gives the same kind of endless hang ("Loading..." screen forever) that "Previously Recorded" does.
MSorice wrote:
Wed Oct 27, 2021 1:36 am
...I'm not sure if regularly scheduled programs will record -- the next thing I know I had scheduled is tomorrow at 4:30. I know something recorded today, but this issue arose since 2 days ago - it was normal then.
I've now confirmed that previously scheduled recordings are not happening, as I remembered I should have recorded game 1 of the World Series tonight at 7:00, and that didn't happen.

Whatever caused this seems to have happened at about 7:00, when that recording should have started. Jeopardy! recorded 4:30-5:00 as normal and I watched it at about 6:45 as normal. No updates to software or listings were done that I know of as software is updated manually and listings go at midnight. Not sure if that can have anything to do with this.
JamesF
Newcomer
Posts: 7
Joined: Wed Oct 27, 2021 9:42 pm
Great Britain

Re: Upcoming Recordings suddenly blank -- Scheduler error?

Post by JamesF »

I see the same problem. There was a mysql-8.0 update this morning from 8.0.26 to 8.0.27 at 6:56am and immediately afterwards the DB Error messages start. I'm betting the two are related.

Ubuntu 20.04, mysql 8.0.27-0ubuntu0.20.04.1, mythtv 2:31.0+fixes.20200323.9579662cdc-0ubuntu1

I'm trying the latest 31.0 from the mythbuntu PPA now

James
JamesF
Newcomer
Posts: 7
Joined: Wed Oct 27, 2021 9:42 pm
Great Britain

Re: Upcoming Recordings suddenly blank -- Scheduler error?

Post by JamesF »

Ok, so 31.0+fixes.202108081239.5da2523154~ubuntu20.04.1 does not fix the problem.

I'm thinking that downgrading mysql is the next best option, but I'm not sure I can do that safely yet.

James
MSorice
Junior
Posts: 15
Joined: Wed Oct 27, 2021 1:18 am
United States of America

Re: Upcoming Recordings suddenly blank -- Scheduler error?

Post by MSorice »

I believe you're right! I've found an automatic upgrade to MySQL 8.0.27 in my /var/log/apt/history some hours before the trouble began.

Well, what should we do to fix that? Is there a way to downgrade reliably? Switch to MariaDB perhaps?
User avatar
paulh
Developer
Posts: 909
Joined: Thu Feb 06, 2014 6:09 pm
Great Britain

Re: Upcoming Recordings suddenly blank -- Scheduler error?

Post by paulh »

This would appear to be the problem

Code: Select all

Database error was:#012BIGINT UNSIGNED value is out of range in '((`mythconverg`.`c`.`recpriority` + `mythconverg`.`capturecard`.`recpriority`) + ((`mythconverg`.`capturecard`.`cardid` = `mythconverg`.`sched_temp_record`.`prefinput`) * 2))'
It might take a while to figure out what's changed to cause this.

It could simply be the negative values for recpriority are the problem but why is it now become problem?
JamesF
Newcomer
Posts: 7
Joined: Wed Oct 27, 2021 9:42 pm
Great Britain

Re: Upcoming Recordings suddenly blank -- Scheduler error?

Post by JamesF »

Ok, I believe I have an ugly workaround for Ubuntu 20.04 by downgrading mysql to an earlier version of 8.0. A straight downgrade appears not to be possible because some of the file formats have changed between the most recent earlier version I could find that would install (8.0.19) and 8.0.27.

I've disabled automatic updates (I hadn't realised these were enabled anyhow -- I prefer to manage updates myself). Holding the mysql packages would probably be sufficient if you still want automatic updates.
Stop mythtv-backend
Dump the databases (I just did mysql and mythconverg as I have nothing else on this system)
Stop mysql
Move the entire contents of /var/lib/mysql somewhere safe in case something horrible goes wrong.
Downgrade mysql using:
apt-get install mysql-server=8.0.19-0ubuntu5 libmysqlclient21=8.0.19-0ubuntu5 mysql-client-8.0=8.0.19-0ubuntu5 mysql-client-core-8.0=8.0.19-0ubuntu5 mysql-server-8.0=8.0.19-0ubuntu5 mysql-server-core-8.0=8.0.19-0ubuntu5
Delete all the stuff for tables with names starting "replication_asynchronous..." in the mysql database backup (I don't think these can be reloaded).
Restore the database backups
Restart mysql and check for errors
Restart mythtv-backend and check for errors

I'm not saying this won't break something somewhere, but for the moment it appears to be ok for me.

James
User avatar
paulh
Developer
Posts: 909
Joined: Thu Feb 06, 2014 6:09 pm
Great Britain

Re: Upcoming Recordings suddenly blank -- Scheduler error?

Post by paulh »

@JamesF - If you run those queries Bill6505 suggested do you also see negative recpriorities?
JamesF
Newcomer
Posts: 7
Joined: Wed Oct 27, 2021 9:42 pm
Great Britain

Re: Upcoming Recordings suddenly blank -- Scheduler error?

Post by JamesF »

paulh wrote:
Wed Oct 27, 2021 11:32 pm
@JamesF - If you run those queries Bill6505 suggested do you also see negative recpriorities?
I do indeed.

James
JamesF
Newcomer
Posts: 7
Joined: Wed Oct 27, 2021 9:42 pm
Great Britain

Re: Upcoming Recordings suddenly blank -- Scheduler error?

Post by JamesF »

There are three items I can find in the change logs for mysql 8.0.27 that appear to relate to modifying behaviour regarding the types of expressions. I don't see that they're obviously connected, but it strikes me that this issue could be an unforeseen consequence.

James
User avatar
bill6502
Developer
Posts: 2299
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: Upcoming Recordings suddenly blank -- Scheduler error?

Post by bill6502 »

I can duplicate the failure on v32Pre by setting just a single negative value for
a channel.recpriority.
ashtonp
Newcomer
Posts: 1
Joined: Thu Oct 28, 2021 3:24 am
Jamaica

Re: Upcoming Recordings suddenly blank -- Scheduler error?

Post by ashtonp »

OK, I think you folks have narrowed down the source of the problem!

I was somewhat nervous about applying the workaround that JamesF posted (not enough confidence in my personal abilities, rather than a reflection on his procedure), so I took a different approach and just got rid of any negative priorities on my config. This meant changing the "channel" and "capturecard" priorities that the bill6502's queries identified:
SELECT cardid,recpriority FROM capturecard WHERE recpriority != 0 ORDER BY cardid;
SELECT chanid,recpriority FROM channel WHERE recpriority != 0 ORDER BY chanid;

The channel priorities are easily changed from the Frontend (Channel Recording Priorities), but because I had a recording already in progress (manually started from Live TV at my wife's request -- yes, that method had continued to work), I changed the capturecard priorities directly in mysql:
mysql> UPDATE capturecard SET recpriority = 0 WHERE cardid = xx ;
where xx is the cardid identified by the first query above.

Followed by a mythfilldatabase, and upcoming recordings were successfully restored!
I suspect that the priorities I had applied are now redundant anyway, due to changes/upgrades I have made over time.

After 17 years with mythtv, the WAF had dropped dramatically the last couple of days, and it's really great that you folks figured out the issue so quickly. Thanks! (Yeah, I know, the actual fix will take more work, but having a quick workaround really helps...)
Post Reply