Massive disk space wastage.

For discussion related to MythTV which doesn't belong in another forum.

Moderator: Forum Moderators

Post Reply
PhilB
Senior
Posts: 403
Joined: Sun May 11, 2014 6:23 pm
Great Britain

Massive disk space wastage.

Post by PhilB »

Sorry for the rather alarmist title but I think we have a big issue with deleting files. I am posting this as a warning, to tell you of the get-around but also to provoke discussion on how to resolve the situation permanently.

Whilst migrating from a 0.27 system to version 31 I uncovered several hundred mpg files consuming 662GB of disk which were spurious.
Removing them took the free space on my 2TB disk from 18% to 45% - I did not need to buy that 4TB disk for my new system after all!!

The spurious recording files are not listed by frontend and are not discovered by find_orphans.py. The 0.27 GetRecordedList API shows them with a status 'delete pending' but under 31 they are not shown at all. I have run the optimize_database script regularly.

Now the accepted wisdom is that such files are left behind if you have a system crash shortly after requesting a deletion but the accretion rate of one of these every week together with the very low failure rate of my system suggest another mechanism.

I suspect that this behaviour will not be shown if your backend is reliable and is permanently on; it only shows if you closedown regularly such as with mythwelcome. My hypothesis of how they accumulate is as follows. We will typically record (say) 10 weekly episodes of a series and then 'binge watch' them. We will then delete them all then escape to the mythwelcome screen which closes down after a 30 second delay. That backend closedown whilst the recordings are still being deleted causes the problem.

The fix is revealed here:
viewtopic.php?f=36&t=4735 and in viewtopic.php?f=36&t=4071 (thanks to Roland and PGBennett!!).

You can run this script to check whether you have this problem

showpendings.sh

Code: Select all

#!/bin/bash
#  mysql commands to show deletepending recordings
PASS=`grep Password ~/.mythtv/config.xml | sed 's/<\/*Password>//g' | sed 's/ *//g'`
SQLLINE='select deletepending, recgroup, count(*) from recorded group by deletepending, recgroup;' 
echo $SQLLINE | mysql --database=mythconverg --user=mythtv --password=${PASS}

You can expected a response like this:
mysql: [Warning] Using a password on the command line interface can be insecure.
deletepending recgroup count(*)
0 Default 643
1 Deleted 432
That 423 figure is the number of these spurious recordings. The 'Deleted' line will be absent if there are none.
You can remove the spurious recordings (under 0.27 or 31) with this script run at a time when backend is idle:

Code: Select all

#!/bin/bash
#  mysql commands to eliminate deletepending recordings
PASS=`grep Password ~/.mythtv/config.xml | sed 's/<\/*Password>//g' | sed 's/ *//g'`
SQLLINE='select deletepending, recgroup, count(*) from recorded group by deletepending, recgroup;'
REPLY=`echo $SQLLINE | mysql --database=mythconverg --user=mythtv --password=${PASS} | grep Deleted`
[ -z "$REPLY" ] && exit
echo clearing deletepending flags
systemctl stop mythtv-backend.service
SQLLINE="UPDATE recorded SET deletepending = 0 where recgroup = 'Deleted';" 
echo $SQLLINE | mysql --database=mythconverg --user=mythtv --password=${PASS}
systemctl start mythtv-backend.service
(Devs: if backend is idle, is it necessary to stop/start it?).

Then leave backend running to delete the files and their entries in the database - allow about 10 sec/recording. Use showpendings.sh whilst this is going on to check progress.

My solution with my new system which will be 'always on' is a 3am database backup if everything is idle and a monthly optimize. I'll include the fix at that monthly optimize. That does not fix the problem for those systems which are closed down after use.

I'll get round to updating the routine maintenance pages in the wiki, but does the forum think that changes need requesting to major and important mythtv components?

- Including the sql fix during backend initiation should fix this but is that a good solution?
- Should we include it in optimize?
- Should the API change which stopped listing such recording be reversed?
- Is this a bug in find_orphans.py?

Comments appreciated!

Phil
PhilB
Senior
Posts: 403
Joined: Sun May 11, 2014 6:23 pm
Great Britain

Re: Massive disk space wastage.

Post by PhilB »

I have updated the periodic maintenance page https://www.mythtv.org/wiki/User_Manual ... aintenance to describe this problem.
This includes a suggested fix to optimize_mythdb.pl
How do I request inclusion of that code in the distribution?
Phil
User avatar
pvr4me
Senior
Posts: 763
Joined: Fri Feb 07, 2014 7:25 pm
Location: near Toronto, Canada
Contact:
Canada

Re: Massive disk space wastage.

Post by pvr4me »

See the Contributions section in README.md at:

https://github.com/MythTV/mythtv

Craig
Formerly the MacPorts guy.
PhilB
Senior
Posts: 403
Joined: Sun May 11, 2014 6:23 pm
Great Britain

Re: Massive disk space wastage.

Post by PhilB »

Thanks Craig. Would you be kind enough to cast your eye over the github proposal please? I'm a github novice and git looks overwhelming.
It's in mythtv > contribs > maintenance > optimize_mythdb.p
Phil
User avatar
pvr4me
Senior
Posts: 763
Joined: Fri Feb 07, 2014 7:25 pm
Location: near Toronto, Canada
Contact:
Canada

Re: Massive disk space wastage.

Post by pvr4me »

PhilB wrote:
Mon Jan 17, 2022 5:03 pm
Thanks Craig. Would you be kind enough to cast your eye over the github proposal please? I'm a github novice and git looks overwhelming.
It's in mythtv > contribs > maintenance > optimize_mythdb.p
Phil
Sorry, I don't know what you want me to look at. And I'm not a MythTV dev--I've just submitted a few tickets in the past.

Craig
Formerly the MacPorts guy.
PhilB
Senior
Posts: 403
Joined: Sun May 11, 2014 6:23 pm
Great Britain

Re: Massive disk space wastage.

Post by PhilB »

Hi Craig, I was asking whether you thought I'd actually submitted a proposal. Agreed it is up to devs to decide whether it's sensible.
I cannot find the proposal unless I login to Github, but maybe that's ok?
https://github.com/PBrady43/mythtv/pull ... 114c51b5ed
Phil
User avatar
pvr4me
Senior
Posts: 763
Joined: Fri Feb 07, 2014 7:25 pm
Location: near Toronto, Canada
Contact:
Canada

Re: Massive disk space wastage.

Post by pvr4me »

Not really my thing, but I believe you need to raise the pull request on Myth's repository:

https://github.com/MythTV/mythtv/pulls

Then the MythTV devs can review it and decide whether to merge.

Craig
Formerly the MacPorts guy.
gedakc
Junior
Posts: 96
Joined: Fri Jul 18, 2014 1:28 am
Canada

Re: Massive disk space wastage.

Post by gedakc »

Thank you PhilB for posting about this issue and providing a method to recover the lost space.

I support four MythTV PVRs in the family and one had more than 20% lost to these delete pending recordings.

While investigating I also discovered that some space was lost in the 'LiveTV' group as was shown in the showpendings.sh script you provided (thankyou).

Code: Select all

deletepending   recgroup        count(*)
0       Default 104
1       Deleted 4
1       LiveTV  4
Note that the count of 4 in the 'LiveTV' recgroup is just coincidentally the same as the count of 4 in the 'Deleted' recgroup.

To address the issue I changed two lines in the "fixpendings.sh" script.

First from:

Code: Select all

grep Deleted
to:

Code: Select all

egrep 'Deleted|LiveTV'
Second from:

Code: Select all

where recgroup = 'Deleted'
to:

Code: Select all

where recgroup = 'Deleted' or recgroup = 'LiveTV'
This cleaned up the delete pendings in the 'LiveTV' recgroup for me.

Curtis
PhilB
Senior
Posts: 403
Joined: Sun May 11, 2014 6:23 pm
Great Britain

Re: Massive disk space wastage.

Post by PhilB »

Thanks Curtis,

That feedback about the LiveTV is useful thanks.
However, the whole issue of checking for the stuck recordings is more complex than I originally thought.

If you use frontend to delete a recording it hangs around for a while in case you change your mind - the default is 20 mins.
See https://www.mythtv.org/wiki/Setup_Storage_Directories

The showdeletes.sh then gives this:

deletepending recgroup count(*)
0 Default 662
0 Deleted 1

and that is not a 'delete pending' which will stick around.

So my thinking is that the 'fix' routine should not bother with checking - just get on with changing the pending flag.

As for how to proceed in the future, I am now issuing the fix code when starting backend.
A simple 'wrapper' script is doing it.

I have updated
https://www.mythtv.org/wiki/User_Manual ... aintenance

Phil
gedakc
Junior
Posts: 96
Joined: Fri Jul 18, 2014 1:28 am
Canada

Re: Massive disk space wastage.

Post by gedakc »

Thank you PhilB for updating the scripts and the wiki. I'm very happy to be able to recover the lost space due to the delete pending recordings. One of the old recordings that was hidden but hung around was dated 2016!

Similar to you I am running MythTV 0.27. Mine is on Mythbuntu 14.04 and has been very stable and reliable.
User avatar
paulh
Developer
Posts: 909
Joined: Thu Feb 06, 2014 6:09 pm
Great Britain

Re: Massive disk space wastage.

Post by paulh »

I'm undecided whether this is a genuine bug or a setup issue or a misunderstanding on how deleted recordings get removed on your part of maybe on my part :)

My understanding is you can choose if you want recordings deleted right away (with a small delay 5 -20 mins) or you can choose how long they stay around in days, or you can say keep them but if you need the space delete them. This is so you can if you accidentally delete a recording or change your mind later you can un-delete it to recover it.

The behaviour is determined by the Time to retain Deleted Recordings (days) setting under TV Setup -> General -> General (Auto-Expire). What do you have that setting set to?
gedakc
Junior
Posts: 96
Joined: Fri Jul 18, 2014 1:28 am
Canada

Re: Massive disk space wastage.

Post by gedakc »

On my MythTV 0.27, under Setup / Video / General / {Next page - General (Auto-expire)} I have Time to retain deleted recordings (days) set to 0 (zero).

I also have my MythTV PVR set to automatically wake up and shut down using 5 minutes of inactivity as the trigger. For more detail see my article on Configuring MythTV Automatic Wakeup and Shutdown which was written with v0.27 and Mythbuntu 14.04 in mind.

The issue does seem to be some sort of bug or data corruption because a number of the "delete pendings" recordings are a few years old. Also this issue has cropped up on many of the MythTV PVRs I support. Note that these are older PVRs with the oldest being built in the spring of 2011.
PhilB
Senior
Posts: 403
Joined: Sun May 11, 2014 6:23 pm
Great Britain

Re: Massive disk space wastage.

Post by PhilB »

Hi Paul,
thanks for your input. That parameter is set to zero days, so I presume that it defaults to 20 mins. Not sure whether that is clock time
or running time though. My mythwelcome shutdown is set at 30 secs.

I think there are two issues here.

1. The normal deferred delete mechanism. When a recording deletion is requested it shows in Info Centre > autoexpire as deleted. It duly deletes, even if there is a reboot before the timer expires. I agree that is working well and that any changes should not jeopardise that.

2. Two Mythtv users so far have reported finding 'limbo' recordings which gobble up disk space but the exact mechanism which generates them is unclear. We are both running 0.27 though a system migration to 31 shows the same symptoms. We both use mythwelcome. Reproducing the problem is tricky. To recap, the limbo recordings do not appear in the frontend Info Centre > autoexpire listing, are not shown by the API, are not fixed by optimizing the database and re not found by find_orphans. The little sql script counts them, and a program looking for recording files no longer listed in the database by the API can list the files. I presume that python bindings list them since find_orphans shows no mismatch.

In both our cases, clearing the deletepending flag on the recordings resulted in the backend removing the limbo recordings.

I had 432 such recordings. Mythweb shows that since 22 Oct 2009 I have recorded 14402 episodes of 2921 shows. I now have 585 recordings left so have deleted 13,817 recordings. 3% of those went into limbo.That is one every 10 days or so.

I would not have said that my system is particularly unreliable, despite its age. One disk crash resulting in a disk swap and a database restore and the occasional glitch but on the whole good.

I am using mythwelcome and my suspicion is that is the cause. I am surmising that once the delete timeout of 20 minutes has expired then the backend marks the recording as 'delete pending' and starts the deletion. If welcome closes down at that moment then the deletion is incomplete and we are left with a 'limbo' recording. I have deliberately not put in any further preventative measures on my 0.27 system in the hope of catching another incidence but without success. As I now have 55% free on the disk I no longer need to binge delete!

I have considered what might be necessary to Myth to prevent this.
I thought that a change to mythshutdown (or a wrapper shell script) might have done it but the timing is probably too tight.
Clearing the flags in the backend startup code would fix it, as would a modification to the optimize code, provided this were
run at least once in a while.
Likewise, a change to find_orphans.py.

My next system (a low power NUC) will be left on 100% so I'll be unlikely to hit this problem in a big way so I'll content myself with clearing the flags at a monthly periodic maintenance period when the backend is idle.

Incidentally, the 'ProgramFlags' returned by the GetRecordedList seem crucial in this respect - are they documented anywhere?
Phil
User avatar
bill6502
Developer
Posts: 2299
Joined: Fri Feb 07, 2014 5:28 pm
United States of America

Re: Massive disk space wastage.

Post by bill6502 »

I'm not aware of ProgramFlags documentation, but here's the source:
https://github.com/MythTV/mythtv/blob/m ... lags.h#L37
PhilB
Senior
Posts: 403
Joined: Sun May 11, 2014 6:23 pm
Great Britain

Re: Massive disk space wastage.

Post by PhilB »

Thanks Bill. Have added a note to the api wiki.
Post Reply