Friday, March 25, 2016

NetApp CDOT SnapMirror Throttle Script

I ran into an issue with SnapProtect resetting my SnapMirror throttles to unlimited every time it would initiate a Aux Copy; saturating the 1Gbps link between two datacenters.  I tried a script provided by NetApp Professional services that did not work.  I have created the following script that is working for me.

UPDATE - 23 JAN 2017: I ran into an issue with the script managing uninitialized SnapMirror relationships; so I have updated the script to include logic for addressing this issue.

################################################################################
# Author: MikeTheTechGeek #
# Version: 1.2 #
# #
# Scenario: Auto throttling on CDOT SnapMirrors, where throttling is reset to #
# unlimited by third-party application (like SnapProtect). #
# Created scheduled task on a server with NetApp Powershell Toolkit installed #
# to run every 15-60 seconds #
################################################################################
#Module import
Import-Module DataONTAP
#CDOT cluster connection info
$cluster = "cluster.domain.local"
$user = "********"
$pass = "********" | ConvertTo-SecureString -AsPlainText -Force
$creds = New-Object System.Management.Automation.PsCredential($user, $pass)
$connect = Connect-NcController $cluster -Credential $creds
#Max allowed throttle in kilobytes / second (100 megabits/sec = 12500 kilobytes/sec)
[Int]$throttle = 62500 * 1024
#Collects all SnapMirrors on cluster
$SnapMirrors = Get-NcSnapmirror
#Determines SnapMirrors that are running
$RunningSnapMirrors = $SnapMirrors | Where-Object {($_.Status -eq "transferring")}
#Determines SnapMirrors that are running without a throttle set 0 = unlimited
$RunningWOThrottle = $RunningSnapMirrors | Where-Object {$_.MaxTransferRate -eq 0}
#Determines total number of running SnapMirrors
$count = $RunningSnapMirrors.count
#If statement to check for running SnapMirrors
If ($count -gt 0) {
#Calculate currently running SnapMirrors total MaxTransferRate
$CurrentRate = $RunningSnapMirrors | Measure-Object -sum MaxTransferRate
#Calculate theortical additonal transfer rate added for any SnapMirrors running currently without throttling
$AddRate = $RunningWOThrottle.count * 10240000
#Calculate total theortical transfer rate both current and additional
$SumRate = $CurrentRate.Sum + $AddRate
#Switch Statement to determine throttle conditions based of SumRate
switch ($SumRate)
{
#If SumRate is less than allowed throttle and has SnapMirrors running without throttling
{(($_ -lt ($throttle + 1)) -and ($RunningWOThrottle.count -gt 0))}
{
#Set throttling on SnapMirrors running without throttle; default max transfer rate
[Void]($RunningWOThrottle | Set-NcSnapmirror -MaxTransferRate 10240000)
#Abort SnapMirrors running without throttle
[Void]($RunningWOThrottle | Invoke-NcSnapmirrorAbort)
#Pause for 5 seconds
Start-Sleep -Seconds 5
#Start previously aborted SnapMirrors now with throttling
foreach ($mirror in $RunningWOThrottle)
{
if ($mirror.MirrorState -eq "Snapmirrored")
{ [Void]($mirror | Invoke-NcSnapmirrorUpdate -MaxTransferRate 10240000) }
ElseIf ($mirror.MirrorState -eq "Uninitialized")
{ [Void]($mirror | Invoke-NcSnapmirrorInitialize -MaxTransferRate 10240000) }
}
}
#If SumRate is greater than allowed throttle and has SnapMirrors running without throttling, or if SumRate is greater than allowed throttle and all SnapMirrors are throttled
{(($_ -gt $throttle) -and ($RunningWOThrottle.count -gt 0)) -or (($_ -gt $throttle) -and ($RunningWOThrottle.count -eq 0))}
{
#Calculate max transfer rate for each running SnapMirror based of max allowed transfer and number of running SnapMirrors
$avgThrottle = [Int64]($throttle / $count)
#Set throttling on all running SnapMirrors based on above calculation
[Void]($RunningSnapMirrors | Set-NcSnapmirror -MaxTransferRate $avgThrottle)
#Abort all running SnapMirrors
[Void]($RunningSnapMirrors | Invoke-NcSnapmirrorAbort)
#Pause for 5 seconds
Start-Sleep -Seconds 5
#Start previously aborted SnapMirrors now with throttling
foreach ($mirror in $RunningSnapMirrors)
{
if ($mirror.MirrorState -eq "Snapmirrored")
{ [Void]($mirror | Invoke-NcSnapmirrorUpdate -MaxTransferRate $avgThrottle) }
ElseIf ($mirror.MirrorState -eq "Uninitialized")
{ [Void]($mirror | Invoke-NcSnapmirrorInitialize -MaxTransferRate $avgThrottle) }
}
}
#If SumRate is less than half allowed throttle and has no SnapMirrors running without throttling
{(($_ -lt (($throttle*.75)+1)) -and ($RunningWOThrottle.count -eq 0))}
{
#Calculate max transfer rate for each running SnapMirror based of max allowed transfer and number of running SnapMirrors
$avgThrottle = [Int64]($throttle / $count)
#Set throttling on all running SnapMirrors based on above calculation
[Void]($RunningSnapMirrors | Set-NcSnapmirror -MaxTransferRate $avgThrottle)
#Abort all running SnapMirrors
[Void]($RunningSnapMirrors | Invoke-NcSnapmirrorAbort)
#Pause for 5 seconds
Start-Sleep -Seconds 5
#Start previously aborted SnapMirrors now with throttling
foreach ($mirror in $RunningSnapMirrors)
{
if ($mirror.MirrorState -eq "Snapmirrored")
{ [Void]($mirror | Invoke-NcSnapmirrorUpdate -MaxTransferRate $avgThrottle) }
ElseIf ($mirror.MirrorState -eq "Uninitialized")
{ [Void]($mirror | Invoke-NcSnapmirrorInitialize -MaxTransferRate $avgThrottle) }
}
}
#Default switch
default {break}
}
}
view raw gistfile1.txt hosted with ❤ by GitHub

2 comments:

  1. This looks good for what I need, but I am not a programmer. Is there a simple way to modify this so that I set the throttle during my production time and backup window and then have it open it wide during the night and weekends. Am trying to seed my DR box before shipping to another data center and have multi tb to seed over.

    ReplyDelete
    Replies
    1. My apologies for not replying sooner; to answer your question.

      I currently have a scheduled task on a server that runs this script every minute. The easiest way to tackle your challenge is to create two copies of this script, one for production time and one for your backup windows. Create a scheduled task to run the prod script during the hours of production and the second script for the backup window during that time. (https://technet.microsoft.com/en-us/library/cc748993(v=ws.11).aspx).

      The only thing you would need to change in each of the scripts is the first number in line 22 ([Int]$throttle = 56250 * 1024), which is what defines the max throughput for the SnapMirrors. 56250 = 500Mbps; so if you want production hours to only use 100Mbps then change 56250 to 12500, and if you want your backup window throttle to use only 400Mbps then change 56250 to 50000. Every 100Mbps equals 12500.

      I also never set my throttle to be more than 80% the total bandwidth of the line the SnapMirrors are going over (1Gpbs, I would never allow the max throttle setting to go over 800Mbps).

      I hope this helps, let me know if you have any other questions.

      Delete