Difference between revisions of "ZFS on high latency devices"

Jump to navigation Jump to search
m
no edit summary
m
m
Line 137: Line 137:
The approach taken works like this:
The approach taken works like this:


Open up batch taskq, aggregation limits, write threads, and ZIO throttle.  TxG commit should now drive writes without throttling for latency.
* Open up batch taskq, aggregation limits, write threads, and ZIO throttle:


Turn zfs_sync_taskq_batch_pct down until speed reduces 10%. This sets the pace of the initial flow within the TxG commit.
<pre>
$ /etc/modprobe.d/zfs.conf:
$ # zfs_delay_scale =
$ options zfs zfs_delay_scale = 1000 * blocksize / {expected writes per sec in MB/s)
$ # so 1000 * 128k / 384 = 333000
$ # make the taskq the limiting part of the flow
$ options zfs zfs_sync_taskq_batch_pct=1
$ # disabling the throttle greatly aids merge
$ options zfs zio_dva_throttle_enabled=0
$ # txg commit every 30 seconds
$ options zfs zfs_txg_timeout=30
$ # start txg commit just before writers ramp up
$ options zfs zfs_dirty_data_sync = {zfs_dirty_data_max * zfs_async_dirty_min * 0.9}
$ # save last 100 txg's information
$ options zfs zfs_txg_history=100
$ # IO aggregation
$ # limit total agg to 3.5m and read gap to 0.75m
$ options zfs zfs_vdev_aggregation_limit=blocksize * K * 3
$ options zfs zfs_vdev_write_gap_limit=ashift * 4 (16k for ashift=12)
$ options zfs zfs_vdev_read_gap_limit=blocksize + 64k
$ # initial thread settings
$ options zfs zfs_vdev_sync_read_min_active=4
$ options zfs zfs_vdev_sync_read_max_active=30
$ options zfs zfs_vdev_sync_write_min_active=10
$ options zfs zfs_vdev_sync_write_max_active=20
$ options zfs zfs_vdev_async_read_min_active=2
$ options zfs zfs_vdev_async_read_max_active=30
$ options zfs zfs_vdev_async_write_min_active=2
$ options zfs zfs_vdev_async_write_max_active=30
</pre>


Verify dirty data is stable and roughly at the midpoint of the dirty data throttle, when under high throughput workloads.
TxG commit should now drive writes without throttling for latency.


Decrease agg limits to K * blocksize
* Turn zfs_sync_taskq_batch_pct down until speed reduces 10%.  This sets the pace of the initial flow within the TxG commit.


Decrease write threads until speed starts to reduce
* Verify dirty data is stable and roughly at the midpoint of the dirty data throttle, when under high throughput workloads.


Verify IO merge
* Decrease agg limits to K * blocksize


Decrease async read threads until speed reduces 20%
* Decrease write threads until speed starts to reduce


Decrease sync read threads until speed starts to reduce
* Verify IO merge


Raise agg limit to K * blocksize * 3
* Decrease async read threads until speed reduces 20%


Check agg size
* Decrease sync read threads until speed starts to reduce


optionally: set and enable throttle match
* Raise agg limit to K * blocksize * 3


Check agg size and throughput
* Check agg size


Test and verify dirty data
* optionally: set and enable throttle match
 
* Check agg size and throughput
 
* Test and verify dirty data
 
------


IO prioritization: assume SRmax is the highest max (it usually will be).  If not, find a compromise value for it so that
IO prioritization: assume SRmax is the highest max (it usually will be).  If not, find a compromise value for it so that
Editor
17

edits

Navigation menu