so i just setup my new backend with mythbuntu. my main goal is to have it as my main backend for all my tvs and general file server. at this point i have the backend up and running on 1 120gb ssd. i also setup 6 2tb WD red dives in raidz2. i'm a bit new to the zfs stuff
what is the best way to have my recording moved to a zfs file? I don't quite understand how all the mysql stuff works, and I don't want to screw it up. After all I just spend all of last night trying to do a fresh install of 16.04 with no luck (I'm pretty sure mysql was my problem ) so I just went back to 14.04
I'm worried my 120gb ssd will fill up. Also will myth transcode to h.264 so the files aren't so big on the zfs?
Thanks
need help with zfs storage
Re: need help with zfs storage
You can add a second (basically read only) storage group backed by a ZFS file system. Then simply move the recording files over with 'mv' and let MythTV figure out the rest. (MythTV will search the file and update the location when it's not found on the expected storage group).
You may want to give the recordings their own filesystem so you can turn compression off for them.
No, MythTV does not convert to H.264 out of the box. But there are user scripts out there to do that. (Move to videos, apply cutlist, re-encode).
Transcoding to another codec due to size concerns is frowned upon in terms of cost of transcoding vs. cost of more storage. But its a valid thing to do for compatibility reasons.
Depending on the amount of content you cut away a simple (virtually) lossless MPEG-2 transcode will save a huge amount of space, too.
You may want to give the recordings their own filesystem so you can turn compression off for them.
No, MythTV does not convert to H.264 out of the box. But there are user scripts out there to do that. (Move to videos, apply cutlist, re-encode).
Transcoding to another codec due to size concerns is frowned upon in terms of cost of transcoding vs. cost of more storage. But its a valid thing to do for compatibility reasons.
Depending on the amount of content you cut away a simple (virtually) lossless MPEG-2 transcode will save a huge amount of space, too.
-
janicefamily
- Newcomer
- Posts: 8
- Joined: Sun Nov 23, 2014 5:51 pm
Re: need help with zfs storage
Hmm. Ok so when you say read only group, what do you mean? And when you say use mv are you talking about setting up a job?
Sorry I'm not really understandin what your trying to say.
Sorry I'm not really understandin what your trying to say.
Re: need help with zfs storage
The default location for storing recordings is /var/lib/mythtv/recordings. This is controlled via storage groups in mythtv-setup. There are also directories for fanart, etc. but they probably default to saving everything in the recordings directory.
Assuming your new storage is in a zpool called tank then you want to move any existing files on to the new zpool and then alter the mythtv storage groups configuration to use the zpool. You don't need to alter the database directly since mythtv searches the storage pool setting.
I would create a separate filesystem on your tank for mythtv since this will allow you to set a quota in the future and use the pool for other things such as archives. You should probably set up a quota anyway at a maximum of 80% of you capacity to help zfs performance.
All this is done as root (put sudo in front of each command):
Ensure you do NOT (NOT!!) install a zfs snapshot program since it will snapshot your filesystem (save a copy) and will cause your disks to fill which eventually will cause mythtv to delete all your recordings.
Finally, I would say 6*2TB is quite a lot of storage space for mythtv.
You invariably end up with months of programmes that you never watch.
You may find that you need to set up multiple storage groups
(recordings, recordings2, recordings3) since mythtv may not perform
well with 8TB of files in one directory.
You mention h.264 encoding. The amount of storage you have will mean
you don't have to waste cpu transcoding unless you want to. Depending
on your country and channels, your tv channels may already be h.264 or
h.265.
There are tools (I don't use) such as mythexport that can be setup to transcode files for portable use.
If there are very few files that you want to permanently keep outside
of mythtv then you can probably look at mythtvfs which creates
user-friendly names for your recordings (title, subtitle, episode
number) and you could then manually use a tool to transcode them such
as handbrake, ffmpeg, etc. Note that default options in transcoding
often loses features such as subtitles.
Assuming your new storage is in a zpool called tank then you want to move any existing files on to the new zpool and then alter the mythtv storage groups configuration to use the zpool. You don't need to alter the database directly since mythtv searches the storage pool setting.
I would create a separate filesystem on your tank for mythtv since this will allow you to set a quota in the future and use the pool for other things such as archives. You should probably set up a quota anyway at a maximum of 80% of you capacity to help zfs performance.
All this is done as root (put sudo in front of each command):
Code: Select all
zpool create raidz2 tank .....
# Note there is no leading / so it is tank not /tank since
# we are using zfs commands
zfs create tank/mythtv
zfs create tank/mythtv/recordings
chown mythtv /tank/mythtv/recordings
# Don't want any of these ancient features on the filesystem
# since it causes constant file updates.
zfs set atime=off tank/mythtv
# We don't need executables in this hierarchy so disable them.
zfs set exec=off tank/mythtv
# Stop backend so we don't get writes while copying
systemctl stop mythtv-backend
# Now sync the files across.
# Note that here you do need the slashes since
# we are using directories not zfs commands.
rsync -av /var/lib/mythtv/recordings/. /tank/var/mythtv/recordings/.
# If all successful then the two directories are identical.
# Otherwise rsync will report some errors and you should
# check what's wrong.
# If you re-run the rsync command then it should be instant
# since there should be no changes.
# I always use a separate symlink for recordings since it
# allows me to have multiple machines setup easier and
# migrate zpools. For example, on another machine this
# symlink would actually link to /net/mythtvmaster/tank/mythtv/recordings.
#
# But this command allows you to access
# your pool via /mnt/mythtv/recordings instead.
mkdir -p /mnt/mythtv
ln -s /tank/mythtv/recordings /mnt/mythtv/recordings
# We create a temporary file so we can later verify
# files are being stored in the correct location.
touch /tmp/mythtvtest
# Then alter the storage groups in mythtv-setup to
# point to /mnt/mythtv/recordings.
# Storage groups are one of the last menu items
# displayed.
mythtv-setup
# If all works then you can start mythtv-backend.
systemctl start mythtv-backend
# Do some playback in myfrontend and verify it works.
# If it does then do some recording and verify that works.
# Usual problem is incorrect directory permissions on the new
# directory.
# Your log files are in /var/log/mythtv/mythbackend.
# Finally, if all seems ok then we want to verify that
# files are saved to correct location.
#
# This should return no files, otherwise it is filling
# up your ssd.
find /var/lib/mythtv/. -newer /tmp/mythtvtest
# This should return a long filename probably with a .ts
# file extension and containing the date of your recording.
# This shows the file is recording on your pool.
find /mnt/mythtv/recordings/. -newer /tmp/mythtvtest
# Wait a day (or two) without rebooting and repeat the
# find commands for peace of mind that all is ok.
#
# If so then you can carefully delete files from /var/lib/mythtv
# after a backup.
# If you use disk for other things then you can set a quota
# of 6T on your recordings:
zfs set quota=6T tank/mythtv
# If you use the pool for other things you can create new filesystems:
zfs create tank/media
zfs create tank/media/music
zfs create tank/media/movies
zfs create tank/mythtv/dbbackup
# Monthly you should scrub your tank which causes the OS to read all the
# files and verify they are valid against their checksum.
zpool scrub tank
zpool status tank
Ensure you do NOT (NOT!!) install a zfs snapshot program since it will snapshot your filesystem (save a copy) and will cause your disks to fill which eventually will cause mythtv to delete all your recordings.
Finally, I would say 6*2TB is quite a lot of storage space for mythtv.
You invariably end up with months of programmes that you never watch.
You may find that you need to set up multiple storage groups
(recordings, recordings2, recordings3) since mythtv may not perform
well with 8TB of files in one directory.
You mention h.264 encoding. The amount of storage you have will mean
you don't have to waste cpu transcoding unless you want to. Depending
on your country and channels, your tv channels may already be h.264 or
h.265.
There are tools (I don't use) such as mythexport that can be setup to transcode files for portable use.
If there are very few files that you want to permanently keep outside
of mythtv then you can probably look at mythtvfs which creates
user-friendly names for your recordings (title, subtitle, episode
number) and you could then manually use a tool to transcode them such
as handbrake, ffmpeg, etc. Note that default options in transcoding
often loses features such as subtitles.
-
janicefamily
- Newcomer
- Posts: 8
- Joined: Sun Nov 23, 2014 5:51 pm
Re: need help with zfs storage
thank you i will give that a shot. very detailed! thank you. i'm glad you mentioned the quota thing. the main reason i setup the zfs was for all my video editing footage. so really i only want like 1.5 tb for myth anyway.
thank you!
thank you!
Re: need help with zfs storage
You're welcome.
Since you have an ssd, consider setting up a zfs l2arc partition on
it. This acts as a huge read cache for your zpool so should improve
your video editing read performance.
You can then disable l2arc for the mythtv recordings since they
probably won't get read multiple times so this saves the l2arc space
for your video files:
Also ensure you have the faster zfs native (rather than the zfs fuse
version). So this should probably list native (I can't remember on
14.04):
Good luck!
Since you have an ssd, consider setting up a zfs l2arc partition on
it. This acts as a huge read cache for your zpool so should improve
your video editing read performance.
You can then disable l2arc for the mythtv recordings since they
probably won't get read multiple times so this saves the l2arc space
for your video files:
Code: Select all
zpool add tank cache /dev/.....
zfs set secondarycache=metadata tank/mythtv/recordings
Also ensure you have the faster zfs native (rather than the zfs fuse
version). So this should probably list native (I can't remember on
14.04):
Code: Select all
dpkg -l \*zfs\* | grep ^ii
-
janicefamily
- Newcomer
- Posts: 8
- Joined: Sun Nov 23, 2014 5:51 pm
Re: need help with zfs storage
ok i ran into a snag. i for got to change tank to my zfs (NAS1) on this part. for some reason it worked with Tank though?
mythserver@mythserver-GA-880GA-UD3H:~$ sudo mkdir -p /mnt/mythtv
mythserver@mythserver-GA-880GA-UD3H:~$ sudo ln -s /tank/mythtv/recordings /mnt/mythtv/recordings
mythserver@mythserver-GA-880GA-UD3H:~$ sudo ln -s /NAS1/mythtv/recordings /mnt/mythtv/recordings
ln: failed to create symbolic link \u2018/mnt/mythtv/recordings\u2019: File exists
then i tried it again
mythserver@mythserver-GA-880GA-UD3H:~$ sudo mkdir -p /mnt/mythtv
mythserver@mythserver-GA-880GA-UD3H:~$ sudo ln -s /NAS1/mythtv/recordings /mnt/mythtv/recordings
ln: failed to create symbolic link \u2018/mnt/mythtv/recordings\u2019: File exists
mythserver@mythserver-GA-880GA-UD3H:~$ sudo mkdir -p /mnt/mythtv
mythserver@mythserver-GA-880GA-UD3H:~$ sudo ln -s /tank/mythtv/recordings /mnt/mythtv/recordings
mythserver@mythserver-GA-880GA-UD3H:~$ sudo ln -s /NAS1/mythtv/recordings /mnt/mythtv/recordings
ln: failed to create symbolic link \u2018/mnt/mythtv/recordings\u2019: File exists
then i tried it again
mythserver@mythserver-GA-880GA-UD3H:~$ sudo mkdir -p /mnt/mythtv
mythserver@mythserver-GA-880GA-UD3H:~$ sudo ln -s /NAS1/mythtv/recordings /mnt/mythtv/recordings
ln: failed to create symbolic link \u2018/mnt/mythtv/recordings\u2019: File exists
Re: need help with zfs storage
The first time you ran the command it created the symlink so you need to remove it.
So this should show you having the symlink file existing (it will show the filename and then an arrow pointing to tank).
So you can then remove the existing symlink file with this:
And then do the ln command again.
So this should show you having the symlink file existing (it will show the filename and then an arrow pointing to tank).
Code: Select all
ls -l /mnt/mythtv/.
Code: Select all
rm /mnt/mythtv/recordings
-
janicefamily
- Newcomer
- Posts: 8
- Joined: Sun Nov 23, 2014 5:51 pm
Re: need help with zfs storage
Thank you very much. I got all that working!
So I built this machine a few weeks ago. I mainly use it for live tv on kodi front ends. The first week live tv was great, but now live tv does alot of glitching/ buffering. Any advice?
Sent from my SAMSUNG-SM-N910A using Tapatalk
So I built this machine a few weeks ago. I mainly use it for live tv on kodi front ends. The first week live tv was great, but now live tv does alot of glitching/ buffering. Any advice?
Sent from my SAMSUNG-SM-N910A using Tapatalk
Re: need help with zfs storage
I'm glad it works.
For livetv, I actually use tvheadend server instead with kodi since I've always found livetv temperamental on mythtv even on different server configurations. So, since I have multiple tuners, I just reserve one for tvheadend and give the rest of the tuners to mythbackend. Inside kodi you see no difference since it can talk to mythtv and tvheadend at the same time so I use mythtv for recording and tvheadend for live. It may be something you consider.
If it's stuttering, chances are that it's not getting the data fast enough. If you have a keyboard on your kodi device then during playback the letter "o" displays some playback statistics. Even on HD recordings I used to see a "buffer full %" dropping on HD playback going 100% down to 10% and then stutter, buffer, and then back again.
I now connect via gigabit Ethernet rather than powerline adapter/wifi and that seems better. Are both your devices connected via gigabit (1000Mb/s) or 100Mb/s or wifi?
The "o" in kodi will tell you the video bitrate, so might say 10Mb/s or 2Mb/s, etc. Maybe the stuttering only happens on high bitrate channels? Your disks can more than cope (10Mb/s is only roughly 1MB/s and your drives can probably do around 150x more than that).
I can't think why it would work the first week ok and now fail. I have 60% and 40% fragmentation on my two pools (via frag column in zpool list) so I doubt fragmentation is your problem.
The one thing I used to do is switch channel and then pause for a few seconds so that kodi could buffer more data before playback. If you have kodi 16 or earlier then you can search for "livetv minvideocachelevel", I used to have it at 85%, but the setting is removed in kodi 17.
On the server you can also run:
This will show you disk statistics every 1 second. Maybe the stutter corresponds with some specific spike in disk activity?
For livetv, I actually use tvheadend server instead with kodi since I've always found livetv temperamental on mythtv even on different server configurations. So, since I have multiple tuners, I just reserve one for tvheadend and give the rest of the tuners to mythbackend. Inside kodi you see no difference since it can talk to mythtv and tvheadend at the same time so I use mythtv for recording and tvheadend for live. It may be something you consider.
If it's stuttering, chances are that it's not getting the data fast enough. If you have a keyboard on your kodi device then during playback the letter "o" displays some playback statistics. Even on HD recordings I used to see a "buffer full %" dropping on HD playback going 100% down to 10% and then stutter, buffer, and then back again.
I now connect via gigabit Ethernet rather than powerline adapter/wifi and that seems better. Are both your devices connected via gigabit (1000Mb/s) or 100Mb/s or wifi?
The "o" in kodi will tell you the video bitrate, so might say 10Mb/s or 2Mb/s, etc. Maybe the stuttering only happens on high bitrate channels? Your disks can more than cope (10Mb/s is only roughly 1MB/s and your drives can probably do around 150x more than that).
I can't think why it would work the first week ok and now fail. I have 60% and 40% fragmentation on my two pools (via frag column in zpool list) so I doubt fragmentation is your problem.
The one thing I used to do is switch channel and then pause for a few seconds so that kodi could buffer more data before playback. If you have kodi 16 or earlier then you can search for "livetv minvideocachelevel", I used to have it at 85%, but the setting is removed in kodi 17.
On the server you can also run:
Code: Select all
zpool iostat -v 1
-
janicefamily
- Newcomer
- Posts: 8
- Joined: Sun Nov 23, 2014 5:51 pm
Re: need help with zfs storage
Yeah I always thought it was ethernet. But I have 1gb running to each machine. I was thinking maybe it was a buffer thing in kodi
Sent from my SAMSUNG-SM-N910A using Tapatalk
Sent from my SAMSUNG-SM-N910A using Tapatalk
Re: need help with zfs storage
I agree that it's probably some buffer issue or timing issue, but I'm not sure how to work around it.
Is your frontend a beefy box or a pi/android box?
I'd be curious if kodi on your phone has the same issues. If your phone doesn't stutter (despite being wifi and so slower network with higher latency) then it could mean that it's an issue with the frontend box and your backend is fine. If your phone does stutter then it doesn't really prove if it is kodi stuttering due to bugs/network or if it is due to the backend not sending data. But, maybe worth a test?
If it's a linux frontend maybe temporarily try mythfrontend instead of kodi and see if that stutters too. If that doesn't stutter but kodi on that box does stutter then it narrows it down to a kodi issue. But, again, if both stutter it doesn't necessarily mean it's a backend problem since it could mean there is some configuration issue on the frontend that affects both.
Is your frontend a beefy box or a pi/android box?
I'd be curious if kodi on your phone has the same issues. If your phone doesn't stutter (despite being wifi and so slower network with higher latency) then it could mean that it's an issue with the frontend box and your backend is fine. If your phone does stutter then it doesn't really prove if it is kodi stuttering due to bugs/network or if it is due to the backend not sending data. But, maybe worth a test?
If it's a linux frontend maybe temporarily try mythfrontend instead of kodi and see if that stutters too. If that doesn't stutter but kodi on that box does stutter then it narrows it down to a kodi issue. But, again, if both stutter it doesn't necessarily mean it's a backend problem since it could mean there is some configuration issue on the frontend that affects both.