Storage Directory Setup with Flaky NAS

[TSinge]

I have MythTV up and running well and am saving all of my recordings to a NAS (unraid) that's mounted using nfs.

Code: Select all

192.168.1.201:/mnt/user/Media /mnt/extmedia nfs auto,rw 0 0

Now, things work swimmingly most of the time, but every once in a while, the NAS will crash (still working on resolving this issue) and when it does, nothing records.

So, here's the question: Is there a way to setup the Storage Directory such that if and only if the NAS is unreachable, it will store to a local disk?

Then, once the NAS is back up, I suppose I could create a cron job to try and move all of the recordings back...

Thoughts? Ideas?

[wesnewell]

Sure, just setup a second storage group in the backend setup. Unless things have changed it will use the one with the most free space by default. So if the nas is down it should record to the other. Moving it to nas after that can be done, but the DB will still want to find it in its original recording location. Another option would be to copy it to your videos dir and then delete from within the frontend, if you just have to free up the local space.

[TSinge]

Thanks! I was worried about it recording on the MythTV client when the Nas was available, but if the NAS has less space, then there are other troubles...

Any hints for building the script to move the files to the NAS? Is it as simple as:

Code: Select all

mv /localpath/* /nasmount/recdir/

Inside a script and in cron?

[pgbennett]

See https://www.mythtv.org/wiki/Setup_Storage_Directories
You need to set up a second directory in the same storage group. Also make sure you set the Disk Scheduler to Balanced Free Space.
For moving files you can just move them with mv as you have suggested. The system will search all storage groups to find them. You need to make sure no recordings are taking place while moving them. Moving a file while it is being recorded would not work well.

[TSinge]

How do you recommend I ensure there are no recordings going on?

[pgbennett]

I suggest you start this in your "Pre-shutdown-check command" script. When that is run you can be sure no recordings are running and none are scheduled to be done soon. I suggest you don't hold up that pre shutdown check while it runs but run it as a separate thread that is invoked from there. When starting it you must return non-zero from the pre-shutdown check to ensure the system is not shut down while the move is running. You also need to test in your pre-shutdown-check that no prior instance of the move is still running. If a prior instance is still running you need to prevent shutdown by reruning non-zero and also do not invoke a second move at the same time.

If your plan is never to shut down, you can still use this, but always return false from the pre shutdown check so that it never shuts down.

[mattlach]

Another thing you can do is to have recordings by default go to the local drive, then have an overnight (4am?) script run that moves the oldest recordings to the NAS.

I've been using this script for a few years now with this kind of setup. I stole it years ago from this Lowrenthuis and have been using it with only minor modification since.

You just set a target amount of free disk space (the amount + safety margin you think a days worth of recordings will never exceed) on the local disk, and it will dutifully move the oldest recordings to the mount location defined (your NAS) until it reaches the target free disk space.

Mine is called moveoldfiles.sh

The syntax is "moveoldfiles.sh <mountpoint where files are stored> <target percent diskspace free>", so in my case "moveoldfiles.sh /mnt/scheduled 86"

Then you just edit the one line in the end of the script to tell it what to do with the identified files, and give it a location to move the files to.

Code: Select all

#!/bin/bash
#
############################################################################### 
# Author            :  Louwrentius
# Contact           : louwrentius@gmail.com
# Initial release   : August 2011
# Licence           : Simplified BSD License
############################################################################### 

VERSION=1.01

#
# Mounted volume to be monitored.
#
MOUNT="$1"
#
# Maximum threshold of volume used as an integer that represents a percentage:
# 95 = 95%.
#
MAX_USAGE="$2"
#
# Failsafe mechansim. Delete a maxium of MAX_CYCLES files, raise an error after
# that. Prevents possible runaway script. Disable by choosing a high value.
#
MAX_CYCLES=1000


show_header () {

    echo
    echo MOVE OLD FILES $VERSION
    echo

}

show_header

reset () {
    CYCLES=0
    OLDEST_FILE=""
    OLDEST_DATE=0
    ARCH=`uname`
}

reset

if [ -z "$MOUNT" ] || [ ! -e "$MOUNT" ] || [ ! -d "$MOUNT" ] || [ -z "$MAX_USAGE" ]
then
    echo "Usage: $0 <mountpoint> <threshold>"
    echo "Where threshold is a percentage."
    echo
    echo "Example: $0 /storage 90"
    echo "If disk usage of /storage exceeds 90% the oldest"
    echo "file(s) will be moved until usage is below 90%."
    echo 
    echo "Wrong command line arguments or another error:"
    echo 
    echo "- Directory not provided as argument or"
    echo "- Directory does not exist or"
    echo "- Argument is not a directory or"
    echo "- no/wrong percentage supplied as argument."
    echo
    exit 1
fi

check_capacity () {

    USAGE=`df -h | grep "$MOUNT" | awk '{ print $5 }' | sed s/%//g`
    if [ ! "$?" == "0" ]    
    then
        echo "Error: mountpoint $MOUNT not found in df output."
        exit 1
    fi

    if [ -z "$USAGE" ]
    then
        echo "Didn't get usage information of $MOUNT"
        echo "Mountpoint does not exist or please remove trailing slash."
        exit 1
    fi

    if [ "$USAGE" -gt "$MAX_USAGE" ]
    then
        echo "Usage of $USAGE% exceeded limit of $MAX_USAGE percent."
        return 0
    else
        echo "Usage of $USAGE% is within limit of $MAX_USAGE percent."
        return 1
    fi
}

check_age () {

    FILE="$1"
    if [ "$ARCH" == "Linux" ]
    then
        FILE_DATE=`stat -c %Y "$FILE"`
    elif [ "$ARCH" == "Darwin" ]
    then
        FILE_DATE=`stat -f %Sm -t %s "$FILE"`
    else
        echo "Error: unsupported architecture."
        echo "Send a patch for the correct stat arguments for your architecture."
    fi
        
    NOW=`date +%s`
    AGE=$((NOW-FILE_DATE))
    if [ "$AGE" -gt "$OLDEST_DATE" ]
    then
        export OLDEST_DATE="$AGE"
        export OLDEST_FILE="$FILE"
    fi
}

process_file () {
    
    FILE="$1"

    #
    # Replace the following commands with wathever you want to do with 
    # this file. You can delete files but also move files or do something else.
    #
    echo "Moving oldest file $FILE"
    mv -f "$FILE" /mnt/archive/scheduled/
}

while check_capacity
do
    if [ "$CYCLES" -gt "$MAX_CYCLES" ]
    then
        echo "Error: after $MAX_CYCLES moved files still not enough free space."
        exit 1
    fi
    
    reset

    FILES=`find "$MOUNT" -type f`
    
    IFS=$'\n'
    for x in $FILES
    do
        check_age "$x"
    done

    if [ -e "$OLDEST_FILE" ]
    then
        #
        # Do something with file.
        #
        process_file "$OLDEST_FILE"
    else
        echo "Error: somehow, item $OLDEST_FILE disappeared."
    fi
    ((CYCLES++))
done
echo

The great part about how MythTV is designed is that it doesn't care where the files are located, as long as they are in a folder configured in the back end. So, move the files from one configured folder to another, and the backend just finds them.

Probably wouldn't hurt to regularly run the optimize_mythdb.pl script thought just to make sure everything is cleaned up. I have that one run in cron every night at 4am right after the move operation is done.