tl;dr: VM virtual disks stored on physical hosts disks are extreamly slow when tested from inside the VM. Test from the host they appear fine. 180 MiB/s vs 0.11 MiB/s
I have two identical physical Dell PE R640 servers, With internal storage. Internal Storage is PERC H740P
Both Servers have 3 Virtual disks:
Disk 1, SSD-SATA Raid 10 , 8 x 1787GB SSD disks (mounted to the host as E:\)
Disk 2, HDD-SAS Raid-5, 4 x 1862GB SAS HDD (mounted to the host as G:\)
Disk 3, OS raid-1 SSD 111 GB (mounted to the host as c:\)
These Servers have been installed with windows server 2016 Standard and Hyper-V. 2 Virtual Machines on each physical server, The virtual machines are all also 2016 standard.
The virtual machines have 4 virtual disk drives, all VHDX's are fixed in size
C:\ stored on the HDD volume G:\ of the host
D:\ stored on the SSD volume E:\ of the host
E:\ stored on the SSD volume E:\ on the host
G:\ stored on the HDD volume G:\ of the host
All servers have been updated to the latest microsoft patches, the firmware, drivers, Bios, Storage controlers and Lifecycle controlers have been updated to the latest versions as reccomended by Dell product support. Im currently working with Dell product support
on this issue
When we first Setup these servers everything worked as expected disk access from the VM's was normal. At some point before Christmas, both VM's on one of the hosts dropped their disk perfromance from super fast to super slow, the other two vm's on the other physical server remained super fast.
We have been troubleshooting since then when all of a sudden the remaining two VM's dropped disk speed yesterday from supper fast to super slow.
Whats the difference, here are the numbers from diskspd.exe, I used the same optoins in all tests
diskspd.exe -d15 -o4 -t4 -b8k -r -L -w50 -c1G testdiskspd.dat
https://pastebin.com/gKQvKTTG
Above is the SSD Storage of one of the two hosts benchmarked with diskspd.exe right now: some exerpts from that file:
Total IO = ~371 MiB/s
Read IO = ~180 MiB/s
Write IO = ~180 MiB/s
Total IO = ~47565.65 I/O per s
Read IO = ~23757.19 I/O per s
Write IO = ~23808.46 I/O per s
Now im going to perform the same test on one of the VM Guests, Neither of these VM's are currently performing any operations on the disk other than having been booted up. This test is performed from the VM, on its d:\ disk which is stored on the above disk
The same details as above
Total IO = ~0.21 MiB/s
Read IO = ~0.11 MiB/s
Write IO = ~0.10 MiB/s
Total IO = ~27.06 I/O per s
Read IO = ~13.93 I/O per s
Write IO = ~13.13 I/O per s
These results are consistent across both disks in the VM and on both Virtual machines hosted. Whats crazy is I did this test on one of the virtual Guests last week and its performace was almost identical to that of the host, and when this set up was first created, the performace of all the VMs was almost exactly that of the host. and now its almost at a standstil.
Im at a loss of where to go from here, Its not something ive seen before so any help woudl be appriciated. thanks