Wednesday, May 1, 2013

ODX Trap! DPM 2012 & Hyper-V 2012 a CSV volume goes offline when you try to create a backup

I recently assisted a customer that was experiencing outages of their Hyper-V VM's that corresponded with a DPM 2012 CSV snapshot. After further investigation the following Failover Cluster event logs were found (around the same time of the VM outages):

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: Date and time
Event ID: 5120
Task Category: Cluster Shared Volume
Level: Error
Keywords:
User: SYSTEM
Computer: Computer name
Description: Cluster Shared Volume 'Volume1' ('name’) is no longer available on this node because of 'STATUS_IO_TIMEOUT(c00000b5)'. All I/O will temporarily be queued until a path to the volume is reestablished.

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: Date and time
Event ID: 5142
Task Category: Cluster Shared Volume
Level: Error
Keywords:
User: SYSTEM
Computer: Computer name
Description: Cluster Shared Volume 'Volume3' ('Cluster Disk 4') is no longer accessible from this cluster node because of error 'ERROR_TIMEOUT(1460)'. Please troubleshoot this node's connectivity to the storage device and network connectivity.

It was evident that DPM 2012 snapshots were causing the IO errors. I came across this hotfix: http://support.microsoft.com/kb/2813630

The hotfix was applied across all the Hyper-V hosts.

A great feature of Windows 2012 is that it is Offloaded Data Transfer (ODX) capable straight out of the box! However when your Storage Array is not ODX capable this is where the CSV I/O errors will start occurring on snapshot operations.

As a result I had to run the following PowerShell command to turn off ODX on all the Hyper-V hosts (see: http://technet.microsoft.com/en-us/library/jj200627):

Check to see if ODX is Enabled/Disabled
Get-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "FilterSupportedFeaturesMode"

Disable ODX
Set-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "FilterSupportedFeaturesMode" -Value 1

1 comment:

  1. do I need to restart host server after registy has been modified?
    Thx
    Mike

    ReplyDelete