cannot restart following system seizure.
Moderator: Forum Moderators
cannot restart following system seizure.
My dedicated backend and frontend system was working fine until the end of March and then its root partition filled up.
I then fell ill and have just left hospital. Brain left behind I think.
To gain access to the system I deleted the logs only to find the cause was the root .trash folders holding nearly 15GB of garbage.
After deleting that and rebooting I had complete access except now the backend cannot access the database.
Now no log files become created either.
I am frustrated.
The system is Mint 19.1 and mythtv 29.0.
I then fell ill and have just left hospital. Brain left behind I think.
To gain access to the system I deleted the logs only to find the cause was the root .trash folders holding nearly 15GB of garbage.
After deleting that and rebooting I had complete access except now the backend cannot access the database.
Now no log files become created either.
I am frustrated.
The system is Mint 19.1 and mythtv 29.0.
Re: cannot restart following system seizure.
You can recreate the /var/log/mythtv/ directory as follows:
The mythbackend and mythfrontend logs will be automatically recreated on the next start.
Mike
Code: Select all
sudo mkdir -p /var/log/mythtv
sudo chown syslog:adm -R /var/log/mythtv/
sudo chmod 2755 /var/log/mythtv
Mike
Re: cannot restart following system seizure.
Thank you. I was able to generate two logs, in the pastebin page:
https://pastebin.com/s71Ddpx5
Apologies for brevity.
https://pastebin.com/s71Ddpx5
Apologies for brevity.
Re: cannot restart following system seizure.
From the logs it look like mysql is not running.
In your original post you stated logs were deleted (I presumed this was just /var/log/mythtv/*), did this include /var/log/mysql/ and others ?
The following will recreate /var/log/mysql/ and hopefully allow mysql to start
Suggest you reboot at this point.
Mike
In your original post you stated logs were deleted (I presumed this was just /var/log/mythtv/*), did this include /var/log/mysql/ and others ?
The following will recreate /var/log/mysql/ and hopefully allow mysql to start
Code: Select all
sudo mkdir -p /var/log/mysql
sudo chown mysql:adm -R /var/log/mysql/
Suggest you reboot at this point.
Mike
Re: cannot restart following system seizure.
I did reboot but the logs looked much the same following a restart.
$ sudo service mysqld status
revealed that mysqld was not running. I had not done anything to change any operations on this machine other than to make some space on the root partition by deleting temporary files, the only logs deleted were under /var/log/mythtv and eventually the main culprits under /.trash.
$ sudo service mysqld status
revealed that mysqld was not running. I had not done anything to change any operations on this machine other than to make some space on the root partition by deleting temporary files, the only logs deleted were under /var/log/mythtv and eventually the main culprits under /.trash.
Re: cannot restart following system seizure.
What does: systemctl cat "myth*" | egrep 'After|Requires|Wants' look like in Mint?
Assuming Mint uses systemd, looking to see if the backend expects mysql to be running.
Not sure why that would have changed if only ~root/.trash was cleaned.
Assuming Mint uses systemd, looking to see if the backend expects mysql to be running.
Not sure why that would have changed if only ~root/.trash was cleaned.
Re: cannot restart following system seizure.
It is possible that my state of health at the time led me to do more damage than I thought.
thelma@origen ~ $ systemctl cat "myth*" | egrep 'After|Requires|Wants'
After=mysql.service network.target
After=remote-fs.target
After=nss-lookup.target
After=mythtv-backend.service
After=motd.service
RemainAfterExit=yes
thelma@origen ~ $
Thanks for your patience.
thelma@origen ~ $ systemctl cat "myth*" | egrep 'After|Requires|Wants'
After=mysql.service network.target
After=remote-fs.target
After=nss-lookup.target
After=mythtv-backend.service
After=motd.service
RemainAfterExit=yes
thelma@origen ~ $
Thanks for your patience.
Re: cannot restart following system seizure.
After=mysql.service network.target is good.
The After=mythtv-backend.service line above is unusual (to me, at least.)
I've never heard of a service waiting of itself. There's probably another service
with a name that begins with myth.
Make sure there's only one service for the backend: systemctl list-unit-files "*myth*"
I just didn't know what name Mint chose for the backend service. You can see any services
with myth in their name with: systemctl cat "*myth*" along with any overrides.
The After=mythtv-backend.service line above is unusual (to me, at least.)
I've never heard of a service waiting of itself. There's probably another service
with a name that begins with myth.
Make sure there's only one service for the backend: systemctl list-unit-files "*myth*"
I just didn't know what name Mint chose for the backend service. You can see any services
with myth in their name with: systemctl cat "*myth*" along with any overrides.
Re: cannot restart following system seizure.
As mysqld is not running there might be some information in /var/log/mysql/error.log
Also the output from systemctl status mysql, the following is good output.
If it helps, here are the default contents of mythtv-backend.service file from Linux Mint 19.1 mythtv version 29.1 using mythbuntu/0.29 ppa.
Mike
Also the output from systemctl status mysql, the following is good output.
Code: Select all
mike@mike-Aspire-V3-571:~$ systemctl status mysql.service
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2019-05-10 17:46:50 BST; 26min ago
Main PID: 1028 (mysqld)
Tasks: 34 (limit: 4915)
CGroup: /system.slice/mysql.service
└─1028 /usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid
May 10 17:46:50 mike-Aspire-V3-571 systemd[1]: Starting MySQL Community Server...
May 10 17:46:50 mike-Aspire-V3-571 systemd[1]: Started MySQL Community Server.
If it helps, here are the default contents of mythtv-backend.service file from Linux Mint 19.1 mythtv version 29.1 using mythbuntu/0.29 ppa.
Code: Select all
mike@mike-Aspire-V3-571:~$ systemctl cat "mythtv-backend.service"
# /lib/systemd/system/mythtv-backend.service
[Unit]
Description=MythTV Backend
Documentation=https://www.mythtv.org/wiki/Mythbackend
After=mysql.service network.target
[Service]
User=mythtv
EnvironmentFile=-/etc/mythtv/additional.args
ExecStart=/usr/bin/mythbackend --quiet --syslog local7 $ADDITIONAL_ARGS
StartLimitBurst=10
StartLimitInterval=10m
Restart=on-failure
RestartSec=1
[Install]
WantedBy=multi-user.target
Mike
Re: cannot restart following system seizure.
Hey Mike,
Shouldn't Mint have a Wants= line in addition to the After=? I always use it (or
Requires= on occasion.)
https://www.freedesktop.org/software/sy ... .unit.html
"After= ensures that the configured unit is started after the listed unit finished starting up, ..."
Shouldn't Mint have a Wants= line in addition to the After=? I always use it (or
Requires= on occasion.)
https://www.freedesktop.org/software/sy ... .unit.html
"After= ensures that the configured unit is started after the listed unit finished starting up, ..."
Re: cannot restart following system seizure.
I am using a tiny keyboard that has a tracker pad on its reverse making moving windows impossible.
So apologies for piecemeal reply.
thelma@origen ~ $ systemctl list-unit-files "*myth*"
UNIT FILE STATE
mythtv-backend.service enabled
mythtv-status.service generated
2 unit files listed.
I will need access to another machine and printer to compare the quoted lines above with this, the target machine.
Have to be tomorrow morning after my batteries are recharged. : )
So apologies for piecemeal reply.
thelma@origen ~ $ systemctl list-unit-files "*myth*"
UNIT FILE STATE
mythtv-backend.service enabled
mythtv-status.service generated
2 unit files listed.
I will need access to another machine and printer to compare the quoted lines above with this, the target machine.
Have to be tomorrow morning after my batteries are recharged. : )
Re: cannot restart following system seizure.
Bill,bill6502 wrote: ↑Fri May 10, 2019 5:24 pmHey Mike,
Shouldn't Mint have a Wants= line in addition to the After=? I always use it (or
Requires= on occasion.)
https://www.freedesktop.org/software/sy ... .unit.html
"After= ensures that the configured unit is started after the listed unit finished starting up, ..."
Mint, which is basically ubuntu (with a different GUI), uses mythtv-packaging, as does Debian (when built from source, not the deb-multimedia dmo repo).
mythtv-packaging does not have the Wants= line, see https://github.com/MythTV/packaging/blo ... nd.service
EDIT:
just checked history for service file the Wants=mysql.service was removed by commit on 10 Jan 2016
https://github.com/MythTV/packaging/com ... 273e36fbe5
Mike
Re: cannot restart following system seizure.
I can now gain access from a remote machine. So
/var/log/mysql/error.log was empty so used the preceding file:
alan@albury:~/Upload$ cat error.log.1
2019-05-10 10:03:48 140052356955264 [Note] CONNECT: Version 1.06.0008 October 06, 2018
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: The InnoDB memory heap is disabled
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Using Linux native AIO
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Using SSE crc32 instructions
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Completed initialization of buffer pool
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Highest supported file format is Barracuda.
2019-05-10 10:03:49 140052356955264 [Note] InnoDB: 128 rollback segment(s) are active.
2019-05-10 10:03:49 140052356955264 [Note] InnoDB: Waiting for purge to start
2019-05-10 10:03:49 140052356955264 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 5839470
2019-05-10 10:03:49 140052356955264 [Note] Plugin 'FEEDBACK' is disabled.
2019-05-10 10:03:49 140051648345856 [Note] InnoDB: Dumping buffer pool(s) not yet started
2019-05-10 10:03:49 140052356955264 [Note] Recovering after a crash using tc.log
2019-05-10 10:03:49 140052356955264 [ERROR] Can't init tc log
2019-05-10 10:03:49 140052356955264 [ERROR] Aborting
===================================================================================================================
thelma@origen ~ $ systemctl status mysql.service
● mariadb.service - MariaDB 10.1.38 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2019-05-10 10:03:52 BST; 1 day 3h ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 1037 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 954 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl se
Process: 934 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 902 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 1037 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"
May 10 10:03:47 origen.asandco systemd[1]: Starting MariaDB 10.1.38 database server...
May 10 10:03:48 origen.asandco mysqld[1037]: 2019-05-10 10:03:48 140052356955264 [Note] /usr/sbin/mysqld (mysqld 10.1.38-MariaDB-0ubuntu0.18.04.1) s
May 10 10:03:52 origen.asandco systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
May 10 10:03:52 origen.asandco systemd[1]: mariadb.service: Failed with result 'exit-code'.
May 10 10:03:52 origen.asandco systemd[1]: Failed to start MariaDB 10.1.38 database server.
lines 1-17/17 (END)
/var/log/mysql/error.log was empty so used the preceding file:
alan@albury:~/Upload$ cat error.log.1
2019-05-10 10:03:48 140052356955264 [Note] CONNECT: Version 1.06.0008 October 06, 2018
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: The InnoDB memory heap is disabled
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Using Linux native AIO
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Using SSE crc32 instructions
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Completed initialization of buffer pool
2019-05-10 10:03:48 140052356955264 [Note] InnoDB: Highest supported file format is Barracuda.
2019-05-10 10:03:49 140052356955264 [Note] InnoDB: 128 rollback segment(s) are active.
2019-05-10 10:03:49 140052356955264 [Note] InnoDB: Waiting for purge to start
2019-05-10 10:03:49 140052356955264 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 5839470
2019-05-10 10:03:49 140052356955264 [Note] Plugin 'FEEDBACK' is disabled.
2019-05-10 10:03:49 140051648345856 [Note] InnoDB: Dumping buffer pool(s) not yet started
2019-05-10 10:03:49 140052356955264 [Note] Recovering after a crash using tc.log
2019-05-10 10:03:49 140052356955264 [ERROR] Can't init tc log
2019-05-10 10:03:49 140052356955264 [ERROR] Aborting
===================================================================================================================
thelma@origen ~ $ systemctl status mysql.service
● mariadb.service - MariaDB 10.1.38 database server
Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2019-05-10 10:03:52 BST; 1 day 3h ago
Docs: man:mysqld(8)
https://mariadb.com/kb/en/library/systemd/
Process: 1037 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
Process: 954 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl se
Process: 934 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
Process: 902 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
Main PID: 1037 (code=exited, status=1/FAILURE)
Status: "MariaDB server is down"
May 10 10:03:47 origen.asandco systemd[1]: Starting MariaDB 10.1.38 database server...
May 10 10:03:48 origen.asandco mysqld[1037]: 2019-05-10 10:03:48 140052356955264 [Note] /usr/sbin/mysqld (mysqld 10.1.38-MariaDB-0ubuntu0.18.04.1) s
May 10 10:03:52 origen.asandco systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
May 10 10:03:52 origen.asandco systemd[1]: mariadb.service: Failed with result 'exit-code'.
May 10 10:03:52 origen.asandco systemd[1]: Failed to start MariaDB 10.1.38 database server.
lines 1-17/17 (END)
Re: cannot restart following system seizure.
if /var/lib/mysql/tc.log exists try deleting it and then reboot.
see https://fransdejonge.com/2018/02/mariad ... log-error/
Mike
see https://fransdejonge.com/2018/02/mariad ... log-error/
Mike
Re: cannot restart following system seizure.
Unbelievable. It worked!
Thank you all so much.
Alan
Thank you all so much.
Alan