ONTAP snapshots for Proxmox VE

In modern IT infrastructure, virtualization is the key to efficient resource management. With virtualization, memory, cpu, network and storage resources can easily be assigned to and shared between virtual machines (VM). Virtualization also comes with the advantage of being able to easily change the resources assigned to virtual machines or to clone virtual machines as needed. Since the virtual machines hard disk is just a file, it can easily be resized, copied and backuped.

Motivation

One of the many advantages of virtualization is the ability to easily create snapshots of the virtual machine disk images.

Proxmox VE offers this ability for the qcow2 disk image format, but not for the raw format on NFS based storage. When creating a VM disk snapshot with Proxmox VE on the local filesystem, Proxmox VE uses the features of the underlying filesystem to create snapshots. In cases, in which the VM disk is placed on a NFS storage, it is not possible for Proxmox VE to use any filesystem features to create a snapshot and therefore has to fallback to file based snapshots.

Since NetApp ONTAP offers snapshot features on file level and on volume level using its own filesystem features, it could be an advantage to use this features over the Proxmox VE file based snapshots.

The common way to connect Proxmox VE with NetApp ONTAP would be NFS. NetApp ONTAP also supports iSCSI, but that is out of scope, as an iSCSI connected storage offers full access to the filesystem and therefore Proxmox VE could use the filesystem features.

ONTAP snapshots

A FileClone is a copy of a file, that points to the same blocks as the original file, only changes are written to new blocks. Therefore creating a FileClone happens instantly and writing to a FileClone does not create any overhead as a file based snapshot in Proxmox VE does. Also deleting or better merging the snapshot back into the virtual disk image is not necessary, as a FileClone is a full copy of the original virtual disk. Just delete the original virtual disk and continue using the FileClone or switch back to the original virtual disk image and delete the clone if a roll-back is necessary.

A VolumeClone works the same way as the FileClone, but on volume level. A VolumeClone creates an instant copy of a complete volume, a Snapshot. The Snapshot references the same blocks as the original volume and changes on the original volume are written to new blocks. It is also possible to access the Snapshot by creating a new volume from it, this is called a FlexClone volume.

This can be used to create a snapshot of a Proxmox VE storage with all its virtual machine disk images instead of creating snapshots of single virtual machine disk images. With the FlexClone volume functionality it is easy to access the data in the clone.

Since Proxmox VE does not support ONTAP features directly, there is a small Python script that uses the ONTAP Rest API and the Proxmox VE API to make this features easily accessible. The script is able to create a FileClone of a virtual machine disk image, it can create the clone from a running VM, but is also able to suspend or stop a VM before creating the clone and then start the VM again.

The script also gives easy access to the VolumeClone and FlexClone features, it is able to create, manage, mount, unmount and delete clones of a volume. When mounting a clone of a volume, it will also automatically be attached as an additional storage to Proxmox VE.

NFS features

In Linux kernel 5.3 the NFS mount option `nconnect` was introduced. By default a NFS client will use one TCP connection to the server, this can be a bottleneck in the case of high NFS work loads. With the nconnect option the number of TCP connections per server can be increased up to 16 connections.

nconnect is supported by Proxmox VE since version 6.2 and by ONTAP 9 and can be easily used by adding the nconnect=<value> to the mount options for the NFS share on the client side.

The other interesting NFS mount option is max_connect. max_connect sets the maximum number of connections to different server IPs belonging to the same NFSv4.1+ server. This is called trunking and is a multipath functionality. The difference to nconnect is, that nconnect sets the number of TCP connections to one server IP, while max_connect sets the number of TCP connections to the same server over multiple IPs. This is supported by ONTAP since version 9.14.1.

Combining the options is possible, but might not lead to the desired result.

More

To further optimize the resource usage, data deduplication on storage systems is an important feature, since especially virtual machine disk images share the same (operating system) data between each other. Deduplication is reducing the used space in this use case very effective. ONTAP supports deduplication.