[Solved] questions about upgrading v28 on UBUNTU 16.04LTS to v29 on UBUNTU 18.04LTS

[GregTippitt]

I have some questions about migrating my MythTV system to new hardware and software. I currently have one master MythTV v28 backend with MySQL running on my storage server, with hostname NAS, under UBUNTU 16.04LTS. I have the MythTV v28 frontend running on another UBUNTU 16.04LTS PC, with hostname HTPC. On a new PC, with hostname TV, I want to combine MythTV frontend, backend, and MySQL running UBUNTU 18.04LTS with MythTV v29. (I try really hard to think of original hostnames for my PCs .) I use a SiliconDust HD HomeRun Prime using a CableCard with Comcast cable TV service. I have been waiting to upgrade to MythTV v29 to fix the problem I have with Closed Captions not working on the channels where Comcast uses H264 streams, because I wanted to upgrade to UBUNTU 18.04LTS and new hardware at the same time. I have the new PC ready with the base install of MythTV v29 frontend and backend, with MySQL, but have not configured anything yet on it. I have kept it disconnected from my home LAN to insure no conflicts with the old MythTV v28 setup.

All of the recordings and other MythTV storage groups will remain on the old NAS server and be mounted by the new TV PC using NFS once I shutdown the old MythTV backend. I have migrated old MythTV systems to new hardware a few times in the past, but it has been a couple of years, so I'm rusty. The only major problem I had in the past was when I changed the hostname of the backend and got a mess in the database. This time I again need to change the hostname and IP address of the new TV PC, because the old server named NAS, will still be running lots of non-MythTV stuff that would be really time consuming to change. I want to move the MythTV backend off of the old NAS server, because its MySQL database and Apache2 server run other stuff that I'd prefer to separate from my MythTV stuff.

I would deeply appreciate if others more knowledgeable of this process would review the the following steps I plan to migrate the MythTV database, so that hopefully I might avoid problems caused by changing the hostname of the new system?

On NAS server running MythTV v28 backend
mythconverg_backup.pl --directory /mnt/temp --filename mythdatabase-backup.sql.gz --compress bzip2
sudo service mythtv-backend stop

On new TV PC running MythTV v29 frontend and backend once I have connected it to the network and mounted the folders for storage groups

mythconverg_restore.pl --drop_database --create_database --directory /mnt/temp --filename mythdatabase-backup.sql.gz
mythconverg_restore.pl --change_hostname --old_hostname="NAS" --new_hostname="TV"
sudo service mythtv-backend stop
run mythtv-setup and fix any config stuff that is needed for the backend
sudo service mythtv-backend start
run mythfrontend and fix any config stuff in setup that is needed for the frontend

I am moving my MythTV to new hardware for a couple of reasons rather than upgrading the software on the existing hardware. One is that I want to be able to test the new system and restart the old system quickly and easily if the new one doesn't work right. I also need to replace the hardware for my frontend, which is old, slow, hangs up, and reboots itself at random times. I have been tolerating its dying hardware, while waiting for UBUNTU 18.04LTS so that I could do a full update of both hardware and software and hopefully have a stable setup for a few years.

I had orginally used a low power, quiet system for the frontend, which was why I split MythTV and ran the backend on my storage server, which is a noisy 4U server in another room. The new PC I am going to use for both frontend and backend will have a 1TB hard drive for UBUNTU and applications, while all of my media files and MythTV recordings will stay on the NAS storage server. The NAS server has a bunch of other stuff that runs on it besides the MythTV backend, and once the new TV PC is tested and ready to take over, I will stop the MythTV backend running on the NAS server, and it will stay running UBUNTU 16.04LTS indefinitely. I have about 12TB of recordings, which the new MythTV PC will mount using NFS over a pair of bonded/trunked 1Gigabit direct wired Ethernet cables between the NAS storage server and the new MythTV PC. A third Ethernet cable will connect the new MythTV to my home LAN for the HomeRun tuner and Internet access.

Since I have not needed to restore a backup of the MythTV database in a few years, I have read through the wiki page at the link below. I appreciate any suggestions of anything that I am overlooking. If there are other users that have also been waiting for the new stable LTS release of UBUNTU and are planning similar upgrades, I will post back here with how this works for me, so that any others can benefit from mistakes I make.

https://www.mythtv.org/wiki/Database_Backup_and_Restore

[MikeB2013]

I have recently changed from Xubuntu 16.04 to Xubuntu 18.04 with mythtv 30pre (from the mythbuntu ppa).

I did originally run an in-place upgrade from Xubuntu 16.04 to Xubuntu 18.04, this worked, but in the end I decided to do a clean install of Xubuntu 18.04 and mythtv from ppa with a database restore from the previous system. I stopped mythbackend before doing the mythtv mythconverg backup

Besides the mythconverg backup you might need to copy other files to somewhere safe in case you need to use them on your new system or refer to them.
I copied:
/etc/fstab (contained my mythtv drive mount points)
/etc/modprobe.d/dvb-options (options for my internal tuners)
firmware for my tuners from lib/firmware (my tuner firmware is not in Ubuntu repositories)
any other files/scripts relating to mythtv e.g acpi power down/up

Note I had to reset ownership and permissions for my mythtv storage directories (recordings, db_backups, banners, music, etc). This might have been because I mounted the drives before mythtv was installed (so the mythtv user did not exist).
The required ownership and permissions are:
sudo chown -R mythtv:mythtv <path to root folder>
sudo chmod -R u=rwx,g=rwxs,o=rx <path to root folder>


There currently a few issues to be aware of with Ubuntu 18.04 and myth 0.29.

The version in the Ubuntu repository is, as usual, just a snapshot, you need to add mythbuntu ppa 0.29 to keep mythtv 29 upto date see https://www.mythtv.org/wiki/Installing_MythTV_on_Ubuntu
sudo add-apt-repository ppa:mythbuntu/0.29
Note in Ubuntu 18.04 you don't have to do "sudo apt update afterwards", it does it for you when adding a ppa.

As you are doing a clean install of mythtv, the mythconverg database password will change (the mythtv install creates a random 8 character alpha-numeric password), this is not really a problem, just something to be aware of. The new password is in Password line of /etc/mythtv/config.xml and will be needed for any remote frontends and initial mythtv-setup.

mythtv-setup on first run will check to see if you are in the mythtv group, if not it offers to add you, this fails due to the gksu package no longer being available. Quick fix is to tun in a terminal "sudo adduser $USER mythtv" and then either logout or reboot. See trac ticket https://code.mythtv.org/trac/ticket/13256

In general you should use "sudo systemctl stop mythtv-backend" before running mythtv-setup (mythtv-setup does not stop mythbackend, when it should see trac ticket https://code.mythtv.org/trac/ticket/13160 ) and "sudo systemctl start mythtv-backend" after running mythtv-setup (just say no if mythtv-setup offers to start mythbackend).

If you use mythweb, the install puts a file 20-mythweb.ini in /etc/php/7.0/apache2/conf.d/ should be in /etc/php/7.2/apache2/conf.d
Quick fix "sudo mv /etc/php/7.0/apache2/conf.d/20-mythweb.ini /etc/php/7.2/apache2/conf.d/20-mythweb.ini"
and restart apache2. See trac ticket https://code.mythtv.org/trac/ticket/13255

Also you might need to increase the value of max_input_vars = 10000 depending on how many channels you have
10000 allows for around 600 channels, 30000 allows around 1800 channels,
In UK Freesat and UK Freeview has around 700 channels.

Mike

[GregTippitt]

Thanks Mike for the advice and info.

For others that are considering this same upgrade of both UBUNTU and MythTV together, mine went pretty easy. The great news is that now I am getting the closed captions fine for the Comcast channels using H264 streams, which were not working with V28!!!

On my first attempt, I had something weird going own where the MythTV Frontend would not connect to the backend and/or MySQL database. Rather than wasting time trying to debug the problem, I simply removed the MythTV and MySQL packages, then reinstalled them, which solved whatever the problem was.

If you're used to V28 and haven't tried V29 before, the backend setup facility has had some major rework. If you're accustomed to the setup screens in prior versions, navigation of the new setup screens is a bit confusing until you are familiar with how the new screens work for selecting options from sub-menus. I don't mean to criticize the work of the developers, whose hard work I deeply appreciate, it is simply different from the old one that I had been using for a decade. For a novice, the new user interface is probably a bit easier to learn to navigate. While I love MythTV, it was a pain to setup originally in the old versions until I had used it so much that it was second nature.

The only major problem I had was because of my own stupidity. When I was setting up the capture cards stuff for my Silicon Dust HDHomerun Prime, for which you have to define its 3 tuners as 3 capture cards, I had forgotten they needed to be numbered 0,1,2. I had them numbered 1,2,3 and could not figure out why I kept getting errors whenever it tried to record 2 things while watching a 3rd one on LiveTV. (There are 10 kinds of people in the world, those who start counting from zero and those who start at one.)

The commands I asked about in my first post for restoring the database and changing the backend's name worked fine, except for a minor annoyance I haven't yet solved. I have a folder of videos in addition to MythTV recordings. When I scan the folder for new videos, I get an error that it can't find the Videos storage group (SG) on my old backend named NAS. My database has been through a few hardware and software upgrades, so the database had several old machine names listed in the "settings" table. Using the MySQL utility, I ran the SQL below which deleted all of the old stuff for old hostnames and left the settings for my new hostname 'tv', but I haven't been able to find the table where the Video Storage Group entry for the old hostname 'nas' is saved.

delete from settings where hostname <> 'tv';

Since I am posting this part as info for others that may be reading it later, I will comment on Mike's suggesting about the PPA for getting updates to MythTV. While I make sure that all of my UBUNTU PCs get updated regularly, especially for security fixes, I prefer not to use the MythTV PPA for updates. In the old days when it still had bugs and new features that were desperately needed, I did use the PPA to get the latest versions. Once it got to version 28 on UBUNTU 16.04LTS, it had everything I wanted it to do and was really stable, so I preferred having a fixed platform without any further changes. If not for the problem I was having with closed captions not working on some Comcast cable TV channels using H264, I would probably have left my MythTV system running indefinitely without updating. On my other PCs, I enjoy tinkering with the updates and features added to UBUNTU interim releases, but on my MythTV system I prefer it to change as little as possible. To me it is an appliance, which I love. I don't want the door on my refrigerator to open from the left this week, and next week for the door to change and swing the opposite way, nor for the Hot and Cold water to change sides. When I sit down to watch TV, I want it to work the same way it did yesterday and not have to learn a new way to watch my favorite shows. Once I remembered which theme and menu options I used before on the frontend, it now looks and behaves exactly as it did before!

Depending upon your needs, adding the PPA for updates may or may not be needed. Having the option of a snapshot that will stay the same except for major bug fixes is great, but if you are having problems and need stuff in the latest version, having the PPA available is wonderful. This situation is also different depending upon when and where you live. When the US changed to digital broadcasting of TV in 2008 (or the similar changes in the UK in 2012 with FreeView) it was great to be able to get the latest changes to support new tuner hardware and stuff. On the other hand, if you have other family members in the household, they may be less than enthusiastic than you are when changes are made to the way they watch TV.

A great Thanks and congrats to the developers on doing a great job of making the version update easy for us.

Greg

[bill6502]

... I have a folder of videos in addition to MythTV recordings. When I scan the folder for new videos, I get an error that it can't find the Videos storage group (SG) on my old backend named NAS. ...

Hi,

Do you see entries in the backend log, including at startup, that refer to the
Video Storage Group?

Other than in mythtv-setup, you can browse the Videos Storage Group with:

Code: Select all

http://yourBackendHostNameOrIP:6544/Myth/GetStorageGroupDirs?GroupName=Videos

You can start the BE on the command line after shutting it down with systemctl as above or you can put
additional arguments in: /etc/mythtv/additional.args, like: ADDITIONAL_ARGS="--loglevel debug --logpath /tmp --v file"
and restart the backend with systemctl. Just remember to comment out the lines with # when finished.

[GregTippitt]

Bill, thank you for the response that helped me fix my problem. I've been using MythTV for a decade, so I should have tried that already, but it had run without any problems for so long, I had gotten rusty and forgotten to try the obvious. With the verbose logging, I got some new information that allowed me to find the database problems and fix them. If others come across these posts when searching because they have a similar problem, I am posting what I did to fix my problem. It requires running some SQL commands to update the database, so it's not something a low-tech users should probably try. I don't know anything about the MythTV's code, but I'm a retired Oracle DBA and not afraid of tinkering with SQL to update my database as long as I have a current backup before I start. I'm not explaining how to run these SQL commands for other users, because anyone that cannot do this for themselves, shouldn't try this at home.

When I set up my new system with both frontend and backend on the same machine, I did not use a storage group for Videos and instead just have one folder of videos defined on the frontend. Whenever I scanned this folder for changes, I got a popup error when it finishes that said "Failed to Scan SG Video Hosts: nas". My old backend was called "nas" and the new combined hostname is "tv". When I tried the browser line Bill suggested, I got an error message because I don't have a "Videos" storage group defined on the new 'tv' backend.

When I turned on verbose logging for both the frontend and backend, and then did a scan, I got a message saying it could not find a specific file on SG(nas). It was a file that I remembered having deleted a few days before doing my conversion. I apparently had not done an updated scan of the Video library after deleting the file until after I had done the conversion. This gave me a new scent to get me sniffing through the mythtv database some more. I had been looking for a table with something that was telling MythTV to look for the old hostname for new files. The problem was actually that it had a filename it couldn't find on the new video folder location, so it was looking for the old location.

The storagegroup table only has entries for the ones I have defined on my new machine. I had searched through all of the tables for any columns named 'storagegroup' or 'hostname', but I had not looked for tables with a column named 'host'. I found the the 'videometadata' table has a 'host' column rather than calling it 'hostname' as in other tables. I ran the SQL below to delete the entry for the file it was looking for on the old 'nas' host.

delete from videometadata where host = 'nas';

Since I had deleted this old video file a few days before I did my conversion, but I had not done a new scan to update MythTV's video library, this file in limbo was generating the error message I was getting whenever I scanned to update the library. MythTV automatically updated the library for all of the files that had been in the Video storage group and were later found in the frontend's video folder, but this missing file was giving it problems.

The logging on the backend disclosed some other similar errors I hadn't noticed before that were also related to the old hostname. The backend was trying to expire some old recordings from LiveTV. On my new combined host, the folders in my 'Default' storage group for recordings are NFS mounts from the original backend 'nas'. My old recordings are there, and I use these same folders for new recordings since it has many TBs of empty disk space. To reduce LAN traffic, the new machine uses its local hard drive folder for LiveTV rather than the networked mounts, since they don't take up much room and are not saved long term. The database still thought the old LiveTV recordings need to be deleted, but it couldn't find them, so the expiring of the entries didn't get completed and were tried again later. I ran the two SQLs below to clear these out:

delete from recorded where storagegroup = 'LiveTV' AND hostname = 'nas';
delete from recordedfile where storagegroup = 'LiveTV' AND hostname = 'nas';

Thanks to Mike and Bill for their helpful comments. Thanks to the developers for their many improvements to MythTV. Good luck to others who are planning to upgrade their machine and may read this post.

Greg