FlashArray: Thales enable full end-to-end/host-to-storage encryption without any dedupe compromises
[ NOTE: machine translation with the help of DeepL translator without additional proofreading and spell checking ]
[Authors: Kai Wolff and Marcel Düssil].
Ensuring enterprise data privacy using end-to-end non-disruptive encryption requires a trade-off in storage efficiency and data reduction mechanisms - using encryption nullifies (or drastically reduces to a minimum - possibly global compression/dedupe still scores a few "points") data reduction technologies like compression and deduplication.
Data reduction in general does nothing else than identify and remove patterns. Both compression and deduplication are fundamentally based on this concept.
Encryption, on the other hand, is based on just the opposite: removing identifiable patterns.
As a good example to illustrate the problem:
Host data encryption and array data reduction are like oil and water - they just don't mix. Oil, gasoline, grease - operating fluids in general are non-polar, but water is polar. Polar and non-polar simply do not mix.
The combination of "host-based encryption" and "dedupe storage/data reduction developed storage" leads in extreme cases to storage capacity requirements increasing by up to 500% compared to the storage of unencrypted data.
It is by far not new news anymore, but probably the first article on this topic in German. Thales announced in March 2019 at the RSA Conference in San Francisco/CA-USA the First "end-to-end" data encryption for storage systems, a data transport without compromise or the renunciation of data reduction.
A transparent end-to-end/E2 encryption, so to speak.
With Vormetric (= a Thales brand) Transparent Encryption for Efficient Storage (VTE-ES) and Pure Storage, users no longer have to choose between maximum data security and storage efficiency.
ATTENTION: Of course, Pure Storage offers always-on encryption on the storage side, but the aim here is to ensure "end-to-end" encryption across the entire data transport path. Host-to-storage vice versa. Purity continuously encrypts all data in the FlashArray using AES256 encryption validated to the FIPS 140-2 standard, meeting the U.S. government's highest security standard for encryption of data in storage. Encryption is integrated, always active and inline - natively embedded in the operating system.
Using a secured key exchange between Vormetric Transparent Encryption (VTE) and storage arrays, encrypted data from hosts running VTE can now be analyzed, compressed, deduplicated, and then securely stored on the array in an encrypted format by enterprise storage solutions. With VTE-ES, there is no impact on data security during data reduction.
Pure Storage FlashArray is the first storage solution Thales has integrated with VTE-ES.
As a result, this industry-first solution provides end-to-end encryption while retaining the benefits of Pure Storage's industry-leading data reduction.
Why E2?
The amount of data that is classified as sensitive and has increased security requirements has increased due to new and stricter data protection mandates. Studies (such as the Thales Data Threat Report-Global Edition 2019) state that 60% of organizations (30% of which have experienced a data breach within the last year). Common reasons for this shortcoming: cost and complexity.
Pure Storage is a leader in simplicity and data reduction mechanisms, so Thales selected Pure Storage as its first Alliance Partner to deliver Vormetric Transparent Encryption.
Encrypting data to meet compliance requirements increases storage costs. Encrypting data at the storage layer (data-at-rest encryption - D@RE) is achieved through software or self-encrypting drives (SEDs). D@RE only protects unencrypted data from being read in the event of physical access (e.g., theft or improper disposal) by storing the data in an encrypted format on the media.
Organizations or departments with more stringent security policies often have a requirement that application and end-user data be encrypted when written to protect it from unauthorized access if the host or server is tampered with. Securing host data is called Transparent Encryption (TE) because the encryption is not visible to the application or user. TE has the problem that the use of data reduction mechanisms (deduplication and compression) are almost completely nullified *. As a result, the need for storage capacity increases enormously with "classic" host encryption.
* Pure Storage guarantees an OVERALL DataReduction of 1.3:1 on the overall system, independently of this, on already encrypted data. However, this data reduction factor is neglected here and is not included in the reduction factors.
Take it practical
VTE takes encryption even further by not only providing efficient (data reduction) and transparent encryption, but also enabling granular access control, privileged user access policies and auditing. The integration of Pure Storage and Vormetric Transparent Encryption is ingenious; you don't have to choose between data reduction and encryption - you can use both without restriction.
In the example, a publicly accessible data set of 5.3 GB was stored on three different volumes of a FlashArray. These are not POD volumes.
The same test data set was written to all three volumes, so the test results can be reproduced at any time. We write to one:
HOST ENCRYPTED VOLUME
Volume name: Encrypted
Volume size: 5.19 G
Data reduction achieved: 1.0:1
VOLUME ENCRYPTED WITH VTE
Volume name: Encrypted_with_VTE
Volume size: 1.08 G
achieved data reduction: 4.8:1
UN-ENCRYPTED VOLUME
Volume name: Non-encrypted
Volume size: 1.08 G
achieved data reduction: 4.8:1
Technical Overview
It is clear: to be able to offer this type of encryption, it is mandatory to have a third instance that takes care of the key management:
There is a Key Management Server (KMS) from Thales, called Vormetric Data Security Manager (DSM), which communicates between FlashArray and host via the Key Management Interoperability Protocol (KMIP). The DSM enables FlashArray and host to exchange the key for the encrypted volumes. The VTE Agent provides the volume with an identifier that allows the FlashArray to identify the volume. The VTE Agent also ensures that all data written to the volume is encrypted. The key management/exchange itself takes place over the network.
In the following example, a RHEL host with highly sensitive data is to be backed up:
The write process/write-IO
Vormetric File System Agent has been installed on the Linux host. The host checks out an encryption key at the DSM. The FlashArray registers as a KMIP client with the DSM and checks the host encryption key. The host writes its encrypted data to the FlashArray.
The FlashArray decrypts the data with the Host Key *, compresses, deduplicates, and encrypts it with the FlashArray Key before writing it to memory.
* decryption of the data with the host key is part of the integration.
The read operation/read IO
When the host wants to read the data, the FlashArray decrypts the data with the FlashArray key and re-encrypts it with the host key before transferring it. This step is also part of the integration.
System requirements
Vormetric DSM/Data Security Manager
DSM VM or appliance (redundant design)
DSM 6.3
VTE 6.2 or newer
Thales KMIP client license
Pure Storage FlashArray
Purity 5.3 or newer
iSCSI or FibreChannel
Host OS
RHEL 7 & 8, SLES 12 & 15, Ubuntu 18
Bare Metal Server
VMware VMs mit RDMs (raw device mappings) oder vVols keine VMDKs *
* Volume-level virtual machine encryption is achieved with the Vormetric Transparent Encryption File System Agent. This is available for AIX, HPUX (EOS 2019), Solaris (EOS 2019), OEL, Red Hat, SUSE, Ubuntu, Windows, Docker and Hadoop.
Configuration Overview
Three devices need to be provisioned and configured. The basic configuration is straightforward, the longest process is likely to be the establishment of the trust relationship (TCP/IP) between the array, DSM and host. The trust position is based on certificates, either self-signed or from a certificate authority via TLS/SSL. This can be a challenge for a beginner who has not had much to do with certificates in the past.
In general, the installation proceeds in this order:
Installation and configuration of Thales DSM (those who already own a DSM only need to install a KMIP license)
Configuration of the DSM by defining the FlashArray as KMIP client (-> make the FlashArray known to the DSM)
Configuration of the FlashArray for KMIP (exchange of device certificates between FlashArray and DSM; make the KMIP/DSM known to the FlashArray)
Installation and configuration of the VTE agent on the host
I cannot make a complete installation guide publicly available for everyone at this point. The setup is only available for authorized partners. However, it can be said: the setup is done quickly with the help of the documentation (workload 30 minutes).
Award
Vormetric's Transparent Encryption (VTE) was awarded "Winner Gold" in Encryption by CyberSecurity Excellence Awards.
Conclusion
As you can see, Thales and Pure Storage have each created real innovation with their technologies, and have been able to deliver integration that was previously thought to be unachievable. It is now possible to provide "end-to-end" data security, while minimizing storage requirements and ultimately enabling customers to meet the highest compliance requirements.
Vormetric Transparent Encryption takes transparent encryption further by not only transparently encrypting and preserving data reduction, but also adding other options (such as access control, policy-driven user access and auditing).
Acknowledgement
This article was written in collaboration with Kai Wolff Channel Sales Manager at Thales [XING | LinkedIn].
Kai and I had the pleasure of getting to know each other in the course of the EncryptReduce story from Thales-Pure Storage. Since then, we meet again and again on occasion and exchange news about the integration and its ecosystems. Together, we have set a goal: to secure every customer environment with simple E2 encryption - mission: E2-everywhere. We agree: the only way to really secure your company data completely.
Thanks again for your support and the great cooperation - Kai.
More info - Links
All officially published setting options in the GUI but also CLI can be read via the "on-board" user guides of the Pure Storage systems.
Click on "Help" in the Purity main menu.
The User Guide is structured like the main menu and can be opened downwards. A search function is also integrated - within here you can also search for keywords.
WEB: Pure Storage (Pure1) support portal - Ticket system and support *(requires registered FlashSystems)
PHONE: Pure Storage phone support: GER - (+49) (0)800 7239467; INTERNATIONAL - (+1) 650 7294088
WEB: Pure Storage OFFICIAL blog