2019 Storage Spaces Write Performance Guide

This guide was posted on the Microsoft forms.

RS5 (Build 17763, Windows 10 1809) update brings improved parity write performance to storage spaces. The improvement comes from being able to bypass the parity space write cache for full stripe writes. Previously created storage spaces will also benefit from these improvements (once the storage pool is upgraded with Update-StoragePool). For best results, you will need to create a new storage space with specific interleave size.

Step 1

Upgrade your storage pool to the latest version.

Get-StoragePool <NameOfPool> | Update-StoragePool

Confirm

Are you sure you want to perform this action?

This will upgrade the StoragePool “TestPool” to the latest version. This is an irreversible action.

[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is “Y”):  

Verify that your pool is at least at “Server 2019” version or later

Get-StoragePool | ? IsPrimordial -eq $false | ft FriendlyName,Version

FriendlyName Version

———— ——-

NameOfPool   Windows Server 2019

Step 2

 Create a new parity virtual disk, with an interleave size of 32KB, 3 columns. This maximizes your flexibility in adding capacity to your space, and ensures that the data stripe size is 64KB, which will match the NTFS allocation unit (cluster) size of 64KB that you will use in the next step. (If you use the Storage Spaces Control Panel UI to create the space, it will typically have an interleave size of 256KB and an NTFS cluster size of 4KB, which doesn’t guarantee that all writes will be aligned to data stripe boundaries)

New-VirtualDisk -StoragePoolFriendlyName <NameOfPool> -ProvisioningType Thin -Interleave 32KB -FriendlyName FastParity -Size 1TB -ResiliencySettingName Parity -NumberOfColumns 3

Step 3

Go to disk management, initialize the disk corresponding to the newly created virtual disk, and format it with NTFS (or REFS) filesystem with an allocation unit (cluster) size of 64KB.

Step 4

Verify that copying large files to this volume is fast. Provided you are copying from a source that is *different* from any of the virtual disks in the storage pool, you should be able to achieve a write performance that is close to 2x the write performance of the slowest physical disk in your storage pool. With typical consumer SATA hard disks, if your source is sufficiently fast (e.g. internal SSD), you should be able to hit 200MB/sec for copying large files.

You can use the performance monitor (perfmon.exe) to verify that your new virtual disk has a high “Write Bypass %”. When correctly configured, you should expect this value to be >99%. The  Counter set name is “Storage Spaces Write Cache”

Parity (Single disk failure resilient) recommended configurations for Archival workloads

Storage Spaces Interleave Size

Number of Columns

Data Stripe Size

FileSystem

Allocation Unit Size
(Cluster Size)

Expected Write Performance
(Multiples of single disk performance)

32KB

3

64KB

NTFS

64KB

2x

32KB

3

64KB

REFS

64KB

2x

16KB

5

64KB

NTFS

64KB

4x

16KB

5

64KB

REFS

64KB

4x

Dual Parity (Two disk failure resilient) recommended configurations for Archival workloads

Storage Spaces Interleave Size

Number of Columns

Data Stripe Size

FileSystem

Allocation Unit Size
(Cluster Size)

Expected Write Performance
(Multiples of single disk performance)

16KB

7

64KB

NTFS

64KB

4x

16KB

7

64KB

REFS

64KB

4x

DrivePool – Alternative to Storage Spaces

As you all probably know, I’ve had a lot of problems with storage spaces. Many of these issues are from not knowing how to correctly configure storage spaces. Specific settings need to be enabled in order to get the most out of your array. The wrong setting can heavily impact your read/write speed, which is what many of the comments have been about. It struck me that there had to be a better application for doing this. This is where we come to DrivePool (This isn’t sponsored – I just love this application).

I have a simple media server at home. I wanted a way to combine disks without using a full-on approach like storage spaces. I did quite a bit of research where I came across and tested a great tool called DrivePool.

DrivePool

DrivePool allows you to configure spare disks on your system to create a redundant storage pool. It’s very flexible.

All my disks were formatted with NTFS. Some of the disk even had existing data on them. That’s actually fine (not recommended, but fine non the less). There are a number of different options to use. For example, I can say that my d:\downloads folder is not duplicated for redundancy, but other folders are. You can set how to balance the disks etc.

The application is very flexible.

What you get is a pool created out of your existing disks that don’t have to be formatted, which can have mixed file systems (NTFS with ReFS).

I am very impressed by this program and highly recommend it. I have purchased a license – it is very affordable. Find out more at https://stablebit.com/

 

Storage Spaces 2019 – Slow with parity

I’ve written a lot about Storage Spaces and slow performance. You can find some of my articles here:

We’ve done a lot of work on Storage Spaces recently to try and find out why our new parity array on server 2019 was slow.

The hardware is the following:

  • Lenovo SR650
  • 32GB Memory
  • Xeon CPU
  • 12x 8TB 7200RPM 512e drives

When creating the Storage Space, the logical sector size is set from the disk. These disks are 512e drives with a 512 sector size. The default sector size created is 4k.

You can either buy native 4k drives at the outset, or set your Storage Space to 4k.

Here is our storage spaces before the change:



From my techs

By default when creating a storage pool, the logical sector size should match the highest physical sector in the disk pool, in this case it should be 4K, but maybe due to this drive is 512e drive, so the logical sector stays in 512. This will cause 8 times delay and performance delay due to the OS and disk controller is doing so called “RMW” operation. The picture below described how the performance get affected by this way.

Once we change our Storage Space to 4k, we started to get 120-160mb/s across the array, all the time. Before this, we were getting 30-60mb/s with many stutters.

I hope this helps.

TRIM over iSCSI/NFS

I’ve taken this post from Reddit, which you can find here. The author is BloodyIron.

So I just went down a rabbit hole for about 3hrs, as I NEEDED to know if iSCSI and NFS can pass TRIM over their protocols to SSD-backed storage. And here are my findings.

  1. iSCSI is capable of passing TRIM correctly
  2. NFS requires v4.2 on server and client to pass TRIM correctly, otherwise earlier versions DO NOT

I cobbled this together from an eye-spinning number of sources on the internet. So if you feel you can conclusively prove me wrong, by all means.

I’m primarily posting this for myself (as my blog/website is not yet production ready), and maybe it can help some other people who are looking.

Furthermore:

  1. RHEL 7.4+ has official support for NFS v4.2
  2. FreeBSD, unsure when it will get NFS v4.2, I’m trying to find out, so far I haven’t found info
  3. Proxmox, to get it to use NFS v4.2 (not sure if it can or not) you have to change NFS options for mounts at the CLI, I’ve opened a feature request to add options/settings like this to the webGUI
  4. FreeNAS seems to conclusively not be NFS v4.2 capable, as of this writing (since it also relies on FreeBSD)

Hope this helps someone! 😀

IBM v3700 + Fusion MT HBA + Lenovo x3650 M5 – Multipath issue on VMWare 6

I’ve been working on an issue for the past week with the following hardware/software:

3x Lenovo x3650 M5 Type 5462
6x Fusion-MPT 12GSAS SAS3008 (two each host)
1x IBM v3700 SAN
VMWare 6.0 U2 (Lenovo image)

The HBA’s and SAN were configured in the following manner:

FC-attach+(1)

What I didn’t realise early on was that multipathing from the SAN to VMWare was not working. As I was in a rush, I saw the SAS connections were live. The SAN said everything was ok, so I didn’t think twice.

However, on closer inspection on the SAN, I found that only one SAS HBA on each host was active. Hmm, what was going on?

Capture (1)

VMWare was also reporting the same issue:

cap2

Initially, I thought this was a SAN issue. I contacted support who checked out the SAN and couldn’t find any issue.

I then contacted VMWare who initially said the configuration was not supported (driver wise). Actually, what I found is VMWare were referring to the wrong driver.

After about a week of going back and forward, I noticed the drivers that were shipped with the Lenovo VMWare image were not the latest. I proceed to update the drivers which in turn, enabled multi-pathing in VMWare.

VMWare:

chrome_2016-08-09_21-50-40

SAN:

chrome_2016-08-09_21-57-30

This was quite a simple issue but made a bit more complicated as all the hardware seemed supported and at the right driver level.

The correct driver was the lsi-msgpt3 driver found here. lsi-msgpt3 version lsi-msgpt3 version 13.00.00.00-1OEM. The installed version was lsi-msgpt3 version 12.00.00.00-1OEM.

Sometimes it pays to check the basics.