FlashArray: Veeam - BfSS/Snapshot-Integration QoS Throttling PowerShell Script (over OAuth 2.0)
[ NOTE: machine translation with the help of DeepL translator without additional proofreading and spell checking ]
Are you using FlashArray(s), a fast SAN, Veeam Backup & Replication and Storage Integration for backups? Then I might have something for you here, but first... what does backup storage integration actually do?
Backup from Storage Snapshots (BfSS) lets you accelerate backup and replication for VMware vSphere VMs whose disks are hosted on storage systems. When you backup from storage snapshots, Veeam Backup & Replication uses previously created storage snapshots to process VM data. With storage snapshot backups, you can reduce the impact of backup and replication activities on the production environment and improve RPOs. - Translated from the official Veeam User Guide for VMware vSphere (VBR10).
Below I briefly explain the main difference in the use of integration.
Regular processing during backup / streaming backup
If you do not use storage integration, then a VMware vSphere snapshot is used as the basis for a backup. The consistent VM state with its data is frozen at the snapshot time for the duration of the backup.
VBR triggers a snapshot of the virtual machine. The VM's vDisks are placed in a read-only state.
Veeam then copies the VM data from the VM's protected disks. Any changes while the backup is in progress are written to delta files.
After the VM copy is complete, the vSphere snapshot is "committed." The delta files are merged with the vDisks and the vSphere snapshots are finally cleaned up.
The problem here: regular backups (also often referred to as streaming backups) can take a long time to process VM data - depending on the machine/VM size. Now, when a backup is performed for a high-traffic transactional application VM, the delta file can get large. A snapshot commit process will then take a long time and the VM may hang or be impacted during this process. To eliminate this problem, you can use Backup from Storage Snapshots.
Backup from Storage Snapshot
VBR triggers a snapshot of the virtual machine. The VM's vDisks are placed in a read-only state.
Veeam controls the creation of a FlashArray snapshot of the volumes containing the VM and the VMware vSphere snapshot via the integration.
The VMware vSphere snapshot on the original storage volume is deleted immediately after the FlashArray snapshot is created. Veeam Backup & Replication accesses the "cloned" VMware vSphere snapshot within the storage snapshot and copies the VM data from it. The vSphere snapshot therefore only exists for an extremely short time, usually a few seconds. As a result, the delta files do not grow large, and the time to "commit" the VMware vSphere snapshot is minimal.
When the backup is complete, the temporarily created storage snapshot is removed.
For more information about integration: "Veeam Backup & Replication 10: User Guide for VMware vSphere".
With Pure Storage FlashArrays you can use the snapshot integration without additional licenses/costs (all features incl.), the functionality/licensing with Veeam must be given (to the Veeam license overview).
Configuration Pure Storage Plugins for Veeam B&R
If you are not yet using snapshot integration, but plan to use BfSS in the future, then I refer you to the installation guide from "JD". JD Wallace [twitter | LinkedIn] is a Pure Storage Principal Systems Engineer and has a history with Veeam. He has already published a good post here and allowed me to reference his posts.
Go to his blog post here: "Use Pure Storage® FlashArray™ Snapshots with Veeam® for Faster Backups and Improved RPOs".
He has also created a video about the configuration: "Setting up the Pure Storage® plug-in for Veeam® Backup and Replication™".
Thanks JD for that and Greetings to US.
On the actual topic
It is not possible with the latest version of Veeam Backup & Replication and the Pure Storage integration to define bandwidth limits/QoS limits when backing up storage snapshots at the volume level (snapshots).
According to my research, this is not natively possible with any competitor via the respective Veeam integrations.
However, such requirements are found more and more often in enterprise/midmarket/MSP environments, but also in performant SAN infrastructures (from 16G FC or 25G iSCSI). A performant infrastructure (backup infrastructure included) can cause you to "suck" large amounts of data (read) out of the storage system during backup. This in turn can lead to the available performance of the system being used uncontrollably for backup traffic - impractical. A backup should always be completed as quickly as possible, but in no way should it limit the performance of your actual production systems.
Here the mixture of Pure Storage's Storage Integration (BfSS) and Veeam Backup & Replication with the "Pure Storage Veeam BfSS Volume QoS Throttling" PowerShell script is a conceivable solution to prevent such behavior.
So what does the script actually do?:
In short, the script does nothing else than:
periodically check if Veeam BfSS jobs are running.
if yes, it initiates a connection via OAuth 2.0 (connection via key on API client) with the FlashArray and sets the previously defined bandwidth limit on the Veeam BfSS volumes.
I have created a flowchart to the script, this is worth a thousand words:
It looks more complicated than it actually is! The work for the activation of the QoS then takes over the script anyway ツ .
The script use
Two PowerShell scripts are supplied, one of which is only required initially to create the API client.
The scripts are executed on the VBR server, as the Veeam PowerShell add-on is required to use them. If Pure Storage PowerShell SDK2 is not yet installed on the VBR, do this first. The installation can be done via the PowerShell Gallery or the attached MSI package.
HINT: Please read the release notes included in the ZIP!
1. Create API client on the FlashArray (OAuth 2.0)
Run the "Pure_Storage_create_FlashArray_API_client_Creator.ps1" as administrator. Enter the connection details (IP or DNS) to the FlashArray VIP and the user/password to use with "array_admin" privileges.
HINT: The user used will be used later for the interactions in the user context of auditing with the FlashArray.
I created a unique user (veeamqos) as "array_admin" on the FlashArray in advance to make the tracking in the audit trail as clear as possible.
An API client with the name "veeamqos" and SSH connection keys/certificates is now automatically created in the background on the FlashArray.
A log is created to store the future information for the connection. Save this information for future use. Make sure that the login contents are always overwritten after re-executing the scripts (on additional FlashArrays).
If you plan to use multiple FlashArrays/ActiveClusters with the script, repeat these steps for the remaining systems.
The generated connection keys are stored in the local user directory in the \.ssh\... folder. folder. If these are ever lost, then the existing API client can be deleted and regenerated using the steps above.
(The used script is not needed for using the "Pure_Storage_Veeam_BfSS_Volume_QoS_Throttling.ps1" script).
OAuth 2.0 Overview
2. Customization script variables to your system
Using the previously created API Client Creator log file, now customize the main script "Pure_Storage_Veeam_BfSS_Volume_QoS_Throttling.ps1".
To change (copy-paste) are the variables for the connection setup in the lines:
Line 13 ($FlashArrayEndPoint)
line 14 ($ArrayClientName)
line 17 ($ArrayUsername)
Line 20 ($clientId)
Line 21 ($keyId)
Line 23 ($privateKeyFile)
To change (copy-paste) is the variable for the desired QoS value of the volume bandwidths in the lines:
Line 27 ($BandwidthLimit) - specified in bytes (default value: 1Gb/s)
Finally to be changed (copy-paste) are the variables for the later configured scheduled task (this is needed for terminating the script after periodic execution) and the log file name:
line 116 - name of the associated Scheduled Task.
line 120 - name of the log file name
Finally, save the script to a suitable location (e.g. C:\Scripts) on the VBR server.
If you plan to use multiple FlashArrays/ActiveClusters with the script, repeat these steps for the remaining systems.
3. Import/Create a scheduled task
Finally, as mentioned above, the Scheduled Task must be configured for periodic activation of the script. Also included is a template (.XML) for easy import of the task. With a few adjustments, the scheduled task can be created.
By default, the task is scheduled to run every 2 minutes (1 minute is also conceivable), but this value can be adjusted as desired.
Imported tasks must always be activated initially!
In case you do not want to import the task template, please refer to the RELEASENOTES for a description of how to set up the task with the necessary options to ensure a smooth script flow.
If you plan to use multiple FlashArrays/ActiveClusters with the script, repeat these steps for the remaining systems.
If you use multiple FlashArrays/ActiveClusters with the integration and script, the location of the scripts could be as follows.
Via the FlashArray Audit Trail (or Pure1) corresponding interactions of the script are logged.
The script can be downloaded free of charge in the PUREFLASH.blog member area. The registration is done quickly, it is non-binding and revocable at any time. Beside the script there are already several other contents available for download. So it is worth to register: to the protected area.
I have packed the script together with the release notes into a ZIP file, this is about 3MB in size. If not already done, you still need to install the PowerShell SDK 2.0.
More info - Links
All officially published setting options in the GUI but also CLI can be read via the "on-board" user guides of the Pure Storage systems.
Click on "Help" in the Purity main menu.
The User Guide is structured like the main menu and can be opened downwards. A search function is also integrated - within here you can also search for keywords.
WEB: Pure Storage (Pure1) support portal - Ticket system and support *(requires registered FlashSystems)
PHONE: Pure Storage phone support: GER - (+49) (0)800 7239467; INTERNATIONAL - (+1) 650 7294088
WEB: Pure Storage OFFICIAL blog