cannot restart following system seizure.

Have a MythTV related problem? Ask for help from other MythTV users here.

Moderator: Forum Moderators

Post Reply
pinnerite
Senior
Posts: 107
Joined: Mon Apr 07, 2014 2:34 pm
Great Britain

cannot restart following system seizure.

Post by pinnerite » Wed May 08, 2019 5:21 pm

My dedicated backend and frontend system was working fine until the end of March and then its root partition filled up.
I then fell ill and have just left hospital. Brain left behind I think.
To gain access to the system I deleted the logs only to find the cause was the root .trash folders holding nearly 15GB of garbage.
After deleting that and rebooting I had complete access except now the backend cannot access the database.
Now no log files become created either.
I am frustrated.
The system is Mint 19.1 and mythtv 29.0.

MikeB2013
Senior
Posts: 422
Joined: Mon Jul 25, 2016 4:16 pm
Great Britain

Re: cannot restart following system seizure.

Post by MikeB2013 » Wed May 08, 2019 6:25 pm

You can recreate the /var/log/mythtv/ directory as follows:

Code: Select all

sudo mkdir -p /var/log/mythtv
sudo chown syslog:adm -R /var/log/mythtv/
sudo chmod 2755 /var/log/mythtv
The mythbackend and mythfrontend logs will be automatically recreated on the next start.

Mike

pinnerite
Senior
Posts: 107
Joined: Mon Apr 07, 2014 2:34 pm
Great Britain

Re: cannot restart following system seizure.

Post by pinnerite » Thu May 09, 2019 4:17 pm

Thank you. I was able to generate two logs, in the pastebin page:

https://pastebin.com/s71Ddpx5

Apologies for brevity.

MikeB2013
Senior
Posts: 422
Joined: Mon Jul 25, 2016 4:16 pm
Great Britain

Re: cannot restart following system seizure.

Post by MikeB2013 » Thu May 09, 2019 5:05 pm

From the logs it look like mysql is not running.

In your original post you stated logs were deleted (I presumed this was just /var/log/mythtv/*), did this include /var/log/mysql/ and others ?

The following will recreate /var/log/mysql/ and hopefully allow mysql to start

Code: Select all

sudo mkdir -p /var/log/mysql
sudo chown mysql:adm -R /var/log/mysql/

Suggest you reboot at this point.

Mike

pinnerite
Senior
Posts: 107
Joined: Mon Apr 07, 2014 2:34 pm
Great Britain

Re: cannot restart following system seizure.

Post by pinnerite » Fri May 10, 2019 9:38 am

I did reboot but the logs looked much the same following a restart.

$ sudo service mysqld status

revealed that mysqld was not running. I had not done anything to change any operations on this machine other than to make some space on the root partition by deleting temporary files, the only logs deleted were under /var/log/mythtv and eventually the main culprits under /.trash.

User avatar
bill6502
Developer
Posts: 1358
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: cannot restart following system seizure.

Post by bill6502 » Fri May 10, 2019 1:53 pm

What does: systemctl cat "myth*" | egrep 'After|Requires|Wants' look like in Mint?

Assuming Mint uses systemd, looking to see if the backend expects mysql to be running.
Not sure why that would have changed if only ~root/.trash was cleaned.

pinnerite
Senior
Posts: 107
Joined: Mon Apr 07, 2014 2:34 pm
Great Britain

Re: cannot restart following system seizure.

Post by pinnerite » Fri May 10, 2019 2:18 pm

It is possible that my state of health at the time led me to do more damage than I thought.

thelma@origen ~ $ systemctl cat "myth*" | egrep 'After|Requires|Wants'
After=mysql.service network.target
After=remote-fs.target
After=nss-lookup.target
After=mythtv-backend.service
After=motd.service
RemainAfterExit=yes
thelma@origen ~ $

Thanks for your patience.

User avatar
bill6502
Developer
Posts: 1358
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: cannot restart following system seizure.

Post by bill6502 » Fri May 10, 2019 2:39 pm

After=mysql.service network.target is good.

The After=mythtv-backend.service line above is unusual (to me, at least.)
I've never heard of a service waiting of itself. There's probably another service
with a name that begins with myth.

Make sure there's only one service for the backend: systemctl list-unit-files "*myth*"
I just didn't know what name Mint chose for the backend service. You can see any services
with myth in their name with: systemctl cat "*myth*" along with any overrides.

MikeB2013
Senior
Posts: 422
Joined: Mon Jul 25, 2016 4:16 pm
Great Britain

Re: cannot restart following system seizure.

Post by MikeB2013 » Fri May 10, 2019 5:16 pm

As mysqld is not running there might be some information in /var/log/mysql/error.log

Also the output from systemctl status mysql, the following is good output.

Code: Select all

mike@mike-Aspire-V3-571:~$ systemctl status mysql.service 
● mysql.service - MySQL Community Server
   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-05-10 17:46:50 BST; 26min ago
 Main PID: 1028 (mysqld)
    Tasks: 34 (limit: 4915)
   CGroup: /system.slice/mysql.service
           └─1028 /usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid

May 10 17:46:50 mike-Aspire-V3-571 systemd[1]: Starting MySQL Community Server...
May 10 17:46:50 mike-Aspire-V3-571 systemd[1]: Started MySQL Community Server.

If it helps, here are the default contents of mythtv-backend.service file from Linux Mint 19.1 mythtv version 29.1 using mythbuntu/0.29 ppa.

Code: Select all

mike@mike-Aspire-V3-571:~$ systemctl cat "mythtv-backend.service"
# /lib/systemd/system/mythtv-backend.service
[Unit]
Description=MythTV Backend
Documentation=https://www.mythtv.org/wiki/Mythbackend
After=mysql.service network.target
 
[Service]
User=mythtv
EnvironmentFile=-/etc/mythtv/additional.args
ExecStart=/usr/bin/mythbackend --quiet --syslog local7 $ADDITIONAL_ARGS
StartLimitBurst=10
StartLimitInterval=10m
Restart=on-failure
RestartSec=1
 
[Install]
WantedBy=multi-user.target


Mike

User avatar
bill6502
Developer
Posts: 1358
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: cannot restart following system seizure.

Post by bill6502 » Fri May 10, 2019 5:24 pm

Hey Mike,

Shouldn't Mint have a Wants= line in addition to the After=? I always use it (or
Requires= on occasion.)

https://www.freedesktop.org/software/sy ... .unit.html
"After= ensures that the configured unit is started after the listed unit finished starting up, ..."

pinnerite
Senior
Posts: 107
Joined: Mon Apr 07, 2014 2:34 pm
Great Britain

Re: cannot restart following system seizure.

Post by pinnerite » Fri May 10, 2019 5:43 pm

I am using a tiny keyboard that has a tracker pad on its reverse making moving windows impossible.
So apologies for piecemeal reply.

thelma@origen ~ $ systemctl list-unit-files "*myth*"
UNIT FILE STATE
mythtv-backend.service enabled
mythtv-status.service generated

2 unit files listed.

I will need access to another machine and printer to compare the quoted lines above with this, the target machine.
Have to be tomorrow morning after my batteries are recharged. : )

MikeB2013
Senior
Posts: 422
Joined: Mon Jul 25, 2016 4:16 pm
Great Britain

Re: cannot restart following system seizure.

Post by MikeB2013 » Fri May 10, 2019 5:45 pm

bill6502 wrote:
Fri May 10, 2019 5:24 pm
Hey Mike,

Shouldn't Mint have a Wants= line in addition to the After=? I always use it (or
Requires= on occasion.)

https://www.freedesktop.org/software/sy ... .unit.html
"After= ensures that the configured unit is started after the listed unit finished starting up, ..."
Bill,

Mint, which is basically ubuntu (with a different GUI), uses mythtv-packaging, as does Debian (when built from source, not the deb-multimedia dmo repo).
mythtv-packaging does not have the Wants= line, see https://github.com/MythTV/packaging/blo ... nd.service

EDIT:
just checked history for service file the Wants=mysql.service was removed by commit on 10 Jan 2016

https://github.com/MythTV/packaging/com ... 273e36fbe5

Mike

pinnerite
Senior
Posts: 107
Joined: Mon Apr 07, 2014 2:34 pm
Great Britain

Re: cannot restart following system seizure.

Post by pinnerite » Sat May 11, 2019 12:23 pm

I can now gain access from a remote machine. So
/var/log/mysql/error.log was empty so used the preceding file:

alan@albury:~/Upload$ cat error.log.1
2019-05-10 10:03:48 140052356955264 [Note] CONNECT: Version 1.06.0008 October 06, 2018
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: The InnoDB memory heap is disabled
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Using Linux native AIO
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Using SSE crc32 instructions
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Completed initialization of buffer pool
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Highest supported file format is Barracuda.
2019-05-10 10:03:49 140052356955264 [Note] InnoDB: 128 rollback segment(s) are active.
2019-05-10 10:03:49 140052356955264 [Note] InnoDB: Waiting for purge to start
2019-05-10 10:03:49 140052356955264 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 5839470
2019-05-10 10:03:49 140052356955264 [Note] Plugin 'FEEDBACK' is disabled.
2019-05-10 10:03:49 140051648345856 [Note] InnoDB: Dumping buffer pool(s) not yet started
2019-05-10 10:03:49 140052356955264 [Note] Recovering after a crash using tc.log
2019-05-10 10:03:49 140052356955264 [ERROR] Can't init tc log
2019-05-10 10:03:49 140052356955264 [ERROR] Aborting

===================================================================================================================
thelma@origen ~ $ systemctl status mysql.service
● mariadb.service - MariaDB 10.1.38 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2019-05-10 10:03:52 BST; 1 day 3h ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 1037 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 954 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl se
Process: 934 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 902 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 1037 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"

May 10 10:03:47 origen.asandco systemd[1]: Starting MariaDB 10.1.38 database server...
May 10 10:03:48 origen.asandco mysqld[1037]: 2019-05-10 10:03:48 140052356955264 [Note] /usr/sbin/mysqld (mysqld 10.1.38-MariaDB-0ubuntu0.18.04.1) s
May 10 10:03:52 origen.asandco systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
May 10 10:03:52 origen.asandco systemd[1]: mariadb.service: Failed with result 'exit-code'.
May 10 10:03:52 origen.asandco systemd[1]: Failed to start MariaDB 10.1.38 database server.
lines 1-17/17 (END)

MikeB2013
Senior
Posts: 422
Joined: Mon Jul 25, 2016 4:16 pm
Great Britain

Re: cannot restart following system seizure.

Post by MikeB2013 » Sat May 11, 2019 12:58 pm

if /var/lib/mysql/tc.log exists try deleting it and then reboot.
see https://fransdejonge.com/2018/02/mariad ... log-error/

Mike

pinnerite
Senior
Posts: 107
Joined: Mon Apr 07, 2014 2:34 pm
Great Britain

Re: cannot restart following system seizure.

Post by pinnerite » Sat May 11, 2019 2:30 pm

Unbelievable. It worked!
Thank you all so much.
Alan

Post Reply