Inconsistent HDD throughput

Do you want advice about what hardware to buy for use with MythTV? Ask here.

Moderator: Forum Moderators

Post Reply
vincepoore
Newcomer
Posts: 5
Joined: Sun Nov 17, 2019 7:59 pm
United States of America

Inconsistent HDD throughput

Post by vincepoore »

I've been seeing many messages like this in my backend log. Granted, the 15 second ones are really abnormal, but 3 seconds seems to happen fairly regular. It's a Seagate Barracuda ST8000DM004-2CX188 8.00 TB.

Code: Select all

Mar 26 20:51:32 mythtv mythbackend: mythbackend[1704]: W TFWWrite threadedfilewriter.cpp:547 (DiskLoop) TFW(/var/lib/mythtv/recordings/1507_20230327023000.ts:89): write(65424) cnt 64 total 4083360 -- took a long time, 3443 ms
Mar 26 20:51:32 mythtv mythbackend: mythbackend[1704]: W TFWWrite threadedfilewriter.cpp:547 (DiskLoop) TFW(/var/lib/mythtv/recordings/1539_20230327020300.ts:90): write(65424) cnt 75 total 4094076 -- took a long time, 3443 ms
Mar 26 20:51:35 mythtv mythbackend: mythbackend[1704]: W TFWWrite threadedfilewriter.cpp:547 (DiskLoop) TFW(/var/lib/mythtv/recordings/1580_20230327020000.ts:106): write(65424) cnt 123 total 8258840 -- took a long time, 6724 ms
Mar 26 20:51:40 mythtv mythbackend: mythbackend[1704]: W TFWWrite threadedfilewriter.cpp:547 (DiskLoop) TFW(/var/lib/mythtv/recordings/1539_20230327020300.ts:90): write(49820) cnt 65 total 3641748 -- took a long time, 3478 ms
Mar 26 20:51:40 mythtv mythbackend: mythbackend[1704]: W TFWWrite threadedfilewriter.cpp:547 (DiskLoop) TFW(/var/lib/mythtv/recordings/1507_20230327023000.ts:89): write(13912) cnt 58 total 3629904 -- took a long time, 3477 ms
Mar 26 20:51:40 mythtv mythbackend: mythbackend[1704]: W TFWWrite threadedfilewriter.cpp:547 (DiskLoop) TFW(/var/lib/mythtv/recordings/1580_20230327020000.ts:106): write(168260) cnt 62 total 4023200 -- took a long time, 3529 ms
Mar 26 20:51:59 mythtv mythbackend: mythbackend[1704]: W TFWWrite threadedfilewriter.cpp:547 (DiskLoop) TFW(/var/lib/mythtv/recordings/1507_20230327023000.ts:89): write(1692) cnt 267 total 17111572 -- took a long time, 15193 ms
Mar 26 20:51:59 mythtv mythbackend: mythbackend[1704]: W TFWWrite threadedfilewriter.cpp:547 (DiskLoop) TFW(/var/lib/mythtv/recordings/1539_20230327020300.ts:90): write(33464) cnt 326 total 17184516 -- took a long time, 15194 ms
Mar 26 20:51:59 mythtv mythbackend: mythbackend[1704]: W TFWWrite threadedfilewriter.cpp:547 (DiskLoop) TFW(/var/lib/mythtv/recordings/1580_20230327020000.ts:106): write(65424) cnt 278 total 17750584 -- took a long time, 15244 ms
I have 4 tuners and sometimes there may be playback from 2 different front ends so possibly 4 write and 2 read streams going at once. When these issues happen it can mess up both the recording and also make playback frustrating.

Here is some sar data for above problem period. As you can see, things were pretty smooth in the minutes before and after the issue. It just seems to come out of nowhere.

Code: Select all

LC_ALL=C sar -dp --dev=sda -f /var/log/sysstat/sa26 -s 20:48:00 -e 20:55:00 
Linux 5.15.0-67-generic (mythtv) 	03/26/23 	_x86_64_	(12 CPU)

20:46:01          DEV       tps     rkB/s     wkB/s     dkB/s   areq-sz    aqu-sz     await     %util
20:46:01          sda     15.15   1085.37   3383.71      0.00    295.00      2.44    132.30     30.65
20:47:01          sda     15.90   1553.75   3377.14      0.00    310.14      4.10    223.41     40.99
20:48:01          sda     33.95   1359.74   3433.39      0.00    141.18      6.99    191.91     53.37
20:49:01          sda     14.92   1123.12   3200.81      0.00    289.82      2.80    156.75     33.04
20:50:01          sda     14.87   1086.40   3562.46      0.00    312.66      3.83    224.59     35.68
20:51:01          sda     13.35   1063.62   3359.49      0.00    331.26      4.43    287.75     41.54
20:52:01          sda      8.51   1058.40   1921.01      0.00    349.95     10.61   1212.14     67.87
20:53:01          sda     12.21    499.03   4749.89      0.00    429.77      9.18    703.79     78.47
20:54:01          sda     14.64    929.61   3438.52      0.00    298.40      3.29    192.31     40.98
20:55:01          sda     15.46   1389.49   3200.59      0.00    296.88      3.31    180.16     47.71
20:56:01          sda     15.20   1072.22   3429.53      0.00    296.12      3.51    196.66     38.59
20:57:01          sda     14.55   1127.51   3228.57      0.00    299.38      3.10    172.59     38.74
I have an available M.2 slot and with a 1T NVMe SSD down to 55 USD I'm thinking I will create a new file system on that and add to my default storage group. New recordings will go there initially since it will be empty, but my plan is to move recordings overnight via a cron job to the Seagate so that the SSD always has more free space and will therefore always be picked for new recordings.

Does this plan seem reasonable? Should I be looking harder at why the Seagate can't handle the load?
white_haired_uncle
Senior
Posts: 265
Joined: Thu Feb 23, 2023 8:55 pm
Location: Safe outside my gilded cage
United States of America

Re: Inconsistent HDD throughput

Post by white_haired_uncle »

That's an SMR drive, and the wrong workload for it. Then again, at least in your example, it's barely doing anything. I've certainly pushed them a LOT (like 20x) harder without issue. I'd at least have a glance at smartctl -a, and maybe run a long test.

It's also the wrong workload for an SSD, at least if you want it to last to a grand old age. Then again, at least in your example, it's barely doing anything. If I went that route, I'd plan for the SSD to fail at an inconvenient time, which means at a minimum having a spare and being willing to lose some current recordings, keeping the OS off of it unless you go RAID1 (which is a good idea anyway, I don't leave home without it).

If you have a decent UPS, a motherboard that can handle a decent amount of RAM, and don't mind possibly losing a recording or two if something bad happens at just the wrong time, I'd look at using tmpfs for recordings with a user job that moved them immediately after completion. Probably a bit more money up front, but also probably a better/cheaper approach over the long term (at least cheaper if you're not the type that needs the latest HW the day it comes out, but plan to keep that RAM for several years).

If you go with a cron job, you might consider only copying files with a time since last modification >= X, where X is long enough that things you will watch shortly after recording and delete do not have to be copied, and you're not trying to copy a file that's still being written. Of course, the longer it sits before going to HDD the greater the chance of data loss.
User avatar
kmdewaal
Developer
Posts: 644
Joined: Wed Dec 07, 2016 8:01 pm
Netherlands

Re: Inconsistent HDD throughput

Post by kmdewaal »

A sharp observation of the white_haired_uncle that your Seagate is a SMR disk. I think that all your problems disappear when you change to a CMR (i.e. not shingled) disk.
The suggestion to use part of the RAM as a disk is interesting, but "a decent amount of RAM" can become quite a lot. My biggest recordings can be 30GB and if I then have four of them in parallel then that is quite a lot of RAM.
On my living room backend I do have a small SSD (256GB) as system disk and a few CMR conventional disks for the recording. The SSD does already run for 10 years 24/7 and the CMR disks also tend to last 10 years.
A reason for using a separate system disk, which also has the MySQL database, is that the daily database backup and repair/optimization is rather disk-intensive and this is likely to interfere with recordings if you make recordings at the same time.
vincepoore
Newcomer
Posts: 5
Joined: Sun Nov 17, 2019 7:59 pm
United States of America

Re: Inconsistent HDD throughput

Post by vincepoore »

Ok, so I take it I shouldn't expect more from the Seagate because it's an SMR drive. I already have the OS on a 256GB m.2 NVMe, so the 1TB NVMe will be exclusively recordings. No worries at all when it fails since it would be less than 24 hours of recordings. The drive has a 600 TBW rating and I checked my average recording size per day and found it to be 34 GB/day.

Check my calculations here, but 600TBW ~ 600,000GBW so 600,000 GB / 34 GB per day means I could expect the SSD to handle 17,647 days or 48 years of recordings before wearing out? If true, then I think the SSD as a short term stop for recordings solves my issue.
vincepoore
Newcomer
Posts: 5
Joined: Sun Nov 17, 2019 7:59 pm
United States of America

Re: Inconsistent HDD throughput

Post by vincepoore »

I'm thinking I will create a new file system on that and add to my default storage group. New recordings will go there initially since it will be empty, but my plan is to move recordings overnight via a cron job to the Seagate so that the SSD always has more free space and will therefore always be picked for new recordings.
I've taken some time to study https://www.mythtv.org/wiki/Setup_Storage_Directories and I don't think the above plan will work like I want.

The problem is in how autoexpire processing is done. If nothing records directly to the Seagate, then nothing will autoexpire on it. Eventually my cron job will fail to find space on the Seagate.

I think what I need to do is change the Default Storage Group (the only one my recording rules reference) to be the SSD only. Then create a new Storage Group for the Seagate. To solve the autoexpire issue, I will setup one recording rule to go to the Seagate. Probably the local 10pm news. I'm also going to set the "Extra Disk Space" global setting to 300GB. So each night, when the news records it should trim the Seagate back to 300GB free. I'll then run my cron job an hour later and 300GB free should be plenty of free margin to handle de-staging the day's recordings off the SSD and onto the slower Seagate.

Does this sound workable?

P.S. I feel completely burned by this SMR technology. Never heard of it until this week. It really should be disclosed prominently to the buyer. I would not have bought this Seagate had I known. I found a quote on one site: "Even the people using Seagate SMR drives are reporting 10 second pauses in writes at times..."
blm-ubunet
Senior
Posts: 265
Joined: Sun Jun 15, 2014 1:08 am
Cambodia

Re: Inconsistent HDD throughput

Post by blm-ubunet »

I use an 8TB SMR in storage group "Archive".
You can still delete from MythTV GUI but does NOT use to record..
And use 2TB normal HDD for recordings & manually move files to SMR after 6 months of aging..
The SMR HDD seems to work very well with large file transfers.
System & home on SSD.

CMR HDD
Seagate Ironwolf (5400rpm) should be lowest noise & best price/performance.
Seagate Ironwolf Pro (7200rpm)
Seagate Exos* SATA (7200rpm)

* I have hard time believing a cheap enclosure can hold on to its helium for 5 years.
blm-ubunet
Senior
Posts: 265
Joined: Sun Jun 15, 2014 1:08 am
Cambodia

Re: Inconsistent HDD throughput

Post by blm-ubunet »

The old WD Green (with default 10sec? sleeping disabled) had started to become unpredictable at light loads.

Seagate Exos 4G 7e8:
- runs hotter than old WD-Green (still air, lying on horizontal surface)
- appears to be much faster IO.
- more rotational noise (as expected)
- more head seeking noise
- need to use better mounting isolation in HTPC case..
Post Reply