Editor
17
edits
Jlcampbell (talk | contribs) m |
Jlcampbell (talk | contribs) m |
||
Line 137: | Line 137: | ||
The approach taken works like this: | The approach taken works like this: | ||
Open up batch taskq, aggregation limits, write threads, and ZIO throttle | * Open up batch taskq, aggregation limits, write threads, and ZIO throttle: | ||
<pre> | |||
$ /etc/modprobe.d/zfs.conf: | |||
$ # zfs_delay_scale = | |||
$ options zfs zfs_delay_scale = 1000 * blocksize / {expected writes per sec in MB/s) | |||
$ # so 1000 * 128k / 384 = 333000 | |||
$ # make the taskq the limiting part of the flow | |||
$ options zfs zfs_sync_taskq_batch_pct=1 | |||
$ # disabling the throttle greatly aids merge | |||
$ options zfs zio_dva_throttle_enabled=0 | |||
$ # txg commit every 30 seconds | |||
$ options zfs zfs_txg_timeout=30 | |||
$ # start txg commit just before writers ramp up | |||
$ options zfs zfs_dirty_data_sync = {zfs_dirty_data_max * zfs_async_dirty_min * 0.9} | |||
$ # save last 100 txg's information | |||
$ options zfs zfs_txg_history=100 | |||
$ # IO aggregation | |||
$ # limit total agg to 3.5m and read gap to 0.75m | |||
$ options zfs zfs_vdev_aggregation_limit=blocksize * K * 3 | |||
$ options zfs zfs_vdev_write_gap_limit=ashift * 4 (16k for ashift=12) | |||
$ options zfs zfs_vdev_read_gap_limit=blocksize + 64k | |||
$ # initial thread settings | |||
$ options zfs zfs_vdev_sync_read_min_active=4 | |||
$ options zfs zfs_vdev_sync_read_max_active=30 | |||
$ options zfs zfs_vdev_sync_write_min_active=10 | |||
$ options zfs zfs_vdev_sync_write_max_active=20 | |||
$ options zfs zfs_vdev_async_read_min_active=2 | |||
$ options zfs zfs_vdev_async_read_max_active=30 | |||
$ options zfs zfs_vdev_async_write_min_active=2 | |||
$ options zfs zfs_vdev_async_write_max_active=30 | |||
</pre> | |||
TxG commit should now drive writes without throttling for latency. | |||
* Turn zfs_sync_taskq_batch_pct down until speed reduces 10%. This sets the pace of the initial flow within the TxG commit. | |||
* Verify dirty data is stable and roughly at the midpoint of the dirty data throttle, when under high throughput workloads. | |||
* Decrease agg limits to K * blocksize | |||
Decrease | * Decrease write threads until speed starts to reduce | ||
* Verify IO merge | |||
* Decrease async read threads until speed reduces 20% | |||
* Decrease sync read threads until speed starts to reduce | |||
* Raise agg limit to K * blocksize * 3 | |||
Check agg size | * Check agg size | ||
Test and verify dirty data | * optionally: set and enable throttle match | ||
* Check agg size and throughput | |||
* Test and verify dirty data | |||
------ | |||
IO prioritization: assume SRmax is the highest max (it usually will be). If not, find a compromise value for it so that | IO prioritization: assume SRmax is the highest max (it usually will be). If not, find a compromise value for it so that |