Removal of deleted recordings still in the database (and filesystem)

[GlenC]

Hi,

I have 2085 total recording count listed in All Programs filter view in Watch Recording of MythTV frontend. This also reports a usage of 4.3TB of space, yet my recordings directories have about up 4170 MPG, TS and NUV files taking up 7.9TB space. That's a 3.6TB discrepancy.

My Recorded table has 2085 out of 4170 records flagged as both recgroup = “Deleted” AND autoexpire = 9999 . Note: exactly half just seems to be a coincidence, sorting all records there seems to be no duplicates, and all basename values are unique.

find_orphans.py doesn’t find any discrepancies between table records and the files in the filesystem.

The 2085 deleted records don’t appear in the Deleted filter view of the recordings list in mythtv’s frontend (and this does work, as I have deleted other recording and they go into this view before being removed by the system).

I’d like to remove the 2085 records and their files (to free up 3.6TB) but am not sure the best way to do this. These recording have been flagged for deletion in the frontend interface more than a month ago (with most years ago), though they are still there in the backend table (and the filesystem).

My “Time to retain deleted recording (days)” variable (in frontend) is set to 0.

I’ve tried running flush_deleted_recgroup.pl but I get the following.

Code: Select all

{about 855 prior of similar format) 
DELETE:  House Hunters Renovation: Massive Master Bath Redo
DELETE:  Fireman Sam: Norman's Big Fossil Adventure
Error reading from MythTV backend: Connection reset by peer
  error:  
DELETE:  Fireman Sam: Pontypandy Heatwave
  error:  
DELETE:  Kazoops, Peppa and Fireman Sam (Manual Record): 2016-11-11 17:25:00
  error:  
{about 1240 repeats of similarly formatted line pairs}

Not sure why this dropped out (have re-tried with restart and no db activity and same issue), but nothing gets removed.

I’m happy to run a query on the table to delete the offending records (in recorded table) then run find_orphans.py to clean up the orphaned files, but understand this is probably a last resort (and without knowing how all the tables work together I don’t want to do without proper advice).

So my question is what is the best way to remove my 2085 deletion flagged records (and their files) from my MythTV?

Thanks, any suggestions appreciated.

[wesnewell]

Wow, in 13 years using mythtv I don;t think I've ever had more than 50 recordings on the system. When I've upgraded versions from scratch, I always either move the recordings to the videos dir or just delete them.
Anyway, could these be live tv recordings? What do you have for days to keep them in the frontend? Default is 1 I think. Not sure what would happen if it's set to 0.

[bill6502]

Could be a case where autoexpire isn't removing anything because there's so
much disk space. I've got lots of watched recordings that won't be removed 'til
space is required. Increasing the 'Extra disk space (GB)' setting might be a
simple solution.

I'd check the backend log too. First, increase the logging like this:

Code: Select all

mythbackend --setverbose file:debug  #change file to general to turn it off

You may need to wait up to an hour for autoexpire to make the pass
that affects the recordings of interest.

Don't do this while recording(s) are in progress. 'file' logging gets very noisy
then!

Here's an example where a recording would autoexpire, but doesn't:

Code: Select all

2018-07-17 12:17:22.467716 I [2239/2420] Expire autoexpire.cpp:1094 (UpdateDontExpireSet) - AutoExpire: Adding Programs to 'Do Not Expire' List
2018-07-17 12:17:22.467836 I [2239/2420] Expire autoexpire.cpp:1112 (UpdateDontExpireSet) -     1261 at 2018-07-17T16:00:00Z in use by flagger on mc0

(this example was for a recording where flagging failed, not one years old)

[paulh]

Bill could be right auto expired or manually deleted recordings will only be deleted from the file system if the space is required.

You can go to the System Status screen and on the 'Auto Expire' tab force the delete of any expired or deleted recordings on there but you can only do one recording at a time.

Since there is a lot of them you may want to try to batch delete them so you can also do it on the Watch Recordings screen. You need to change the 'Group Filter' to 'Deleted' and add all the recordings you want to permanently delete to the playlist, then on the Playlist Options menu select 'Delete'. For me that way doesn't always work and the recordings come back the next time 'Watch Recordings' is reloaded but may be worth a try.

[GlenC]

Hi, thanks for the replies and suggestions.

All the recordings I’m trying to remove have been deleted in the frontend, none are autoexpire records (I have these too), and the issue records don’t seem to be in the Auto Expire tab (in the System Status screen). Nor does the deleted Group filter view show any of these records (the view lists a record if I go and delete one), so I can’t see these records in the frontend anywhere.

I wasn’t sure if adjusting extra disk space would solve my issue and I went down the debug logging path (thanks, though “general” didn’t seem to return back to previous logging level for me—I presumed it did until I saw about 20GB of logs over 2 days—and I needed to set it to file:info to return to normal). For the pertinent part all I got was every 15 minutes a list of just the autoexpire records (column autoexpire = 1 from recorded table). I can’t find any mention of records in the log that align to the autoexpire = 9999 records (ones I need to get rid of).

Something I should have done earlier: I added some test recordings that I then deleted which allowed me to track down the main record differences (in Recorded table) between what did delete and what didn’t. The prior have deletepending = 0 and latter have deletepending = 1

This led me to June 2013 and August 2014 list.mythtv.org posts.

The June 2013 thread says (on an inactive system) to run

Code: Select all

update recorded set deletepending=0 where deletepending=1

yet the August 2014 thread mentions a modified find_orphans.py (it also mentions not to mess with any mythtv tables, but the original poster doesn't say how they resolved the issue). The wiki version appears current from the page update date, but the latest (4 Jan 2012) github entry seems (I don’t really understand the code) more thorough. It interestingly includes a commit (with pertinent comment) from 1 Jan 2012.

Therefore, which approach should I use? The second (updated find_orphans.py) seem better (though I’d suspect the wiki version is more commonly used, so is the github one linked the true updated version and stable for production use? EDIT: it seems the TB and .ts changes (at least) would need to be merged from the wiki version into the 1 Jan 2012 github commit)

Advice about which is the safest way to clean up my mythtv system (and get back my storage space) would be greatly appreciated.

[bill6502]

Is there a one-to-one match between the .mpg or .ts files and
a .nuv file? That is, were the .nuvs created after a recording
was made.

[GlenC]

There is only one file (.mpg or .ts or .nuv) on the file system per record in the recorded table. Whilst find_orphans.py is usually for finding the orphans between the file system and database (and vice versa) I’m asking about the git hub version of the find_orphans.py (as referred to in the August 2014 link) as it seems to also have the specific code to handle recordings that were marked for deletion and failed (i.e. deletepending = 1 in recorded table)(issue I’m having). They aren't an orphan in the normal sense, but suppose if you think about it as though these records (and files) are orphans to the frontend (as they can’t be found anywhere, and hence you can’t normally get rid of them).

[bill6502]

Got it, I was trying to understand the equal number of delete-marked recorded entries' in the initial post.

I did try the github version of find_orphans, but it fires an exception for me. The
original author is no longer active in the project.

I'm not too keen on altering the DB manually either. Wondering if the Services API
will help. If you can get the recordedid of just one recording that you
want to remove, I'd try this (and worry about automating it if it works
later.)

Code: Select all

curl -H Accept:application/json --data RecordedId=1234 --data ForceDelete=true --data AllowRerecord=true localhost:6544/Dvr/DeleteRecording

The response will be bool: true (or false.) Remove the AllowRerecord parameter or
set it to false if you don't want that. If there's an error due to a typo in the line, see the
backend log if the response isn't clear.

You can use ChanId and StartTime in place of RecordedId, but that's just harder.