Configuring the OpenShift API for Data Protection with {VirtProductName}
You can install the OpenShift API for Data Protection (OADP) with OpenShift Virtualization by installing the OADP Operator and configuring a backup location. Then, you can install the Data Protection Application.
Back up and restore virtual machines by using the OpenShift API for Data Protection.
OpenShift API for Data Protection with OpenShift Virtualization supports the following backup and restore storage options:
-
Container Storage Interface (CSI) backups
-
Container Storage Interface (CSI) backups with DataMover
The following storage options are excluded:
-
File system backup and restore
-
Volume snapshot backups and restores
For more information, see Backing up applications with File System Backup: Kopia or Restic.
To install the OADP Operator in a restricted network environment, you must first disable the default software catalog sources and mirror the Operator catalog. See Using Operator Lifecycle Manager in disconnected environments for details.
Important
Red Hat only supports the combination of OADP versions 1.3.0 and later, and OpenShift Virtualization versions 4.14 and later.
OADP versions before 1.3.0 are not supported for back up and restore of OpenShift Virtualization.
Installing and configuring OADP with OpenShift Virtualization
As a cluster administrator, you install OADP by installing the OADP Operator.
The latest version of the OADP Operator installs Velero 1.16.
-
Access to the cluster as a user with the
cluster-adminrole.
-
Install the OADP Operator according to the instructions for your storage provider.
-
Install the Data Protection Application (DPA) with the
kubevirtandopenshiftOADP plugins. -
Back up virtual machines by creating a
Backupcustom resource (CR).Warning
Red Hat support is limited to only the following options:
-
CSI backups
-
CSI backups with DataMover.
You restore the
BackupCR by creating aRestoreCR. -
Installing the Data Protection Application
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication API.
-
You must install the OADP Operator.
-
You must configure object storage as a backup location.
-
If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
-
If the backup and snapshot locations use the same credentials, you must create a
Secretwith the default name,cloud-credentials.Note
If you do not want to specify backup or snapshot locations during the installation, you can create a default
Secretwith an emptycredentials-velerofile. If there is no defaultSecret, the installation will fail.
-
Click Ecosystem → Installed Operators and select the OADP Operator.
-
Under Provided APIs, click Create instance in the DataProtectionApplication box.
-
Click YAML View and update the parameters of the
DataProtectionApplicationmanifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: configuration: velero: defaultPlugins: - kubevirt - gcp - csi - openshift resourceTimeout: 10m nodeAgent: enable: true uploaderType: kopia podConfig: nodeSelector: <node_selector> backupLocations: - velero: provider: gcp default: true credential: key: cloud name: <default_secret> objectStorage: bucket: <bucket_name> prefix: <prefix>where:
namespace-
Specifies the default namespace for OADP which is
openshift-adp. The namespace is a variable and is configurable. kubevirt-
Specifies that the
kubevirtplugin is mandatory for OpenShift Virtualization. gcp-
Specifies the plugin for the backup provider, for example,
gcp, if it exists. csi-
Specifies that the
csiplugin is mandatory for backing up PVs with CSI snapshots. Thecsiplugin uses the Velero CSI beta snapshot APIs. You do not need to configure a snapshot location. openshift-
Specifies that the
openshiftplugin is mandatory. resourceTimeout-
Specifies how many minutes to wait for several Velero resources such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability, before timeout occurs. The default is 10m.
nodeAgent-
Specifies the administrative agent that routes the administrative requests to servers.
enable-
Set this value to
trueif you want to enablenodeAgentand perform File System Backup. uploaderType-
Specifies the uploader type. Enter
kopiaas your uploader to use the Built-in DataMover. ThenodeAgentdeploys a daemon set, which means that thenodeAgentpods run on each working node. You can configure File System Backup by addingspec.defaultVolumesToFsBackup: trueto theBackupCR. nodeSelector-
Specifies the nodes on which Kopia are available. By default, Kopia runs on all nodes.
provider-
Specifies the backup provider.
name-
Specifies the correct default name for the
Secret, for example,cloud-credentials-gcp, if you use a default plugin for the backup provider. If specifying a custom name, then the custom name is used for the backup location. If you do not specify aSecretname, the default name is used. bucket-
Specifies a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
prefix-
Specifies a prefix for Velero backups, for example,
velero, if the bucket is used for multiple purposes.
-
Click Create.
-
Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:
$ oc get all -n openshift-adpNAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/node-agent-9cq4q 1/1 Running 0 94s pod/node-agent-m4lts 1/1 Running 0 94s pod/node-agent-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s service/openshift-adp-velero-metrics-svc ClusterIP 172.30.10.0 <none> 8085/TCP 8h NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/node-agent 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/velero-588db7f655 1 1 1 96s
-
Verify that the
DataProtectionApplication(DPA) is reconciled by running the following command:$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]} -
Verify the
typeis set toReconciled. -
Verify the backup storage location and confirm that the
PHASEisAvailableby running the following command:$ oc get backupstoragelocations.velero.io -n openshift-adpNAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h true
Warning
If you run a backup of a Microsoft Windows virtual machine (VM) immediately after the VM reboots, the backup might fail with a PartiallyFailed error. This is because, immediately after a VM boots, the Microsoft Windows Volume Shadow Copy Service (VSS) and Guest Agent (GA) service are not ready. The VSS and GA service being unready causes the backup to fail. In such a case, retry the backup a few minutes after the VM boots.
Backing up a single VM
If you have a namespace with multiple virtual machines (VMs), and want to back up only one of them, you can use the label selector to filter the VM that needs to be included in the backup. You can filter the VM by using the app: vmname label.
-
You have installed the OADP Operator.
-
You have multiple VMs running in a namespace.
-
You have added the
kubevirtplugin in theDataProtectionApplication(DPA) custom resource (CR). -
You have configured the
BackupStorageLocationCR in theDataProtectionApplicationCR andBackupStorageLocationis available.
-
Configure the
BackupCR as shown in the following example:ExampleBackupCRapiVersion: velero.io/v1 kind: Backup metadata: name: vmbackupsingle namespace: openshift-adp spec: snapshotMoveData: true includedNamespaces: - <vm_namespace> labelSelector: matchLabels: app: <vm_app_name> storageLocation: <backup_storage_location_name>where:
vm_namespace-
Specifies the name of the namespace where you have created the VMs.
vm_app_name-
Specifies the VM name that needs to be backed up.
backup_storage_location_name-
Specifies the name of the
BackupStorageLocationCR.
-
To create a
BackupCR, run the following command:$ oc apply -f <backup_cr_file_name>where:
backup_cr_file_name-
Specifies the name of the
BackupCR file.
Restoring a single VM
After you have backed up a single virtual machine (VM) by using the label selector in the Backup custom resource (CR), you can create a Restore CR and point it to the backup. This restore operation restores a single VM.
-
You have installed the OADP Operator.
-
You have backed up a single VM by using the label selector.
-
Configure the
RestoreCR as shown in the following example:ExampleRestoreCRapiVersion: velero.io/v1 kind: Restore metadata: name: vmrestoresingle namespace: openshift-adp spec: backupName: vmbackupsingle restorePVs: truewhere:
vmbackupsingle-
Specifies the name of the backup of a single VM.
-
To restore the single VM, run the following command:
$ oc apply -f <restore_cr_file_name>where:
restore_cr_file_name-
Specifies the name of the
RestoreCR file.Note
When you restore a backup of VMs, you might notice that the Ceph storage capacity allocated for the restore is higher than expected. This behavior is observed only during the
kubevirtrestore and if the volume type of the VM isblock.Use the
rbd sparsifytool to reclaim space on target volumes. For more details, see Reclaiming space on target volumes.
Restoring a single VM from a backup of multiple VMs
If you have a backup containing multiple virtual machines (VMs), and you want to restore only one VM, you can use the LabelSelectors section in the Restore CR to select the VM to restore. To ensure that the persistent volume claim (PVC) attached to the VM is correctly restored, and the restored VM is not stuck in a Provisioning status, use both the app: <vm_name> and the kubevirt.io/created-by labels. To match the kubevirt.io/created-by label, use the UID of DataVolume of the VM.
-
You have installed the OADP Operator.
-
You have labeled the VMs that need to be backed up.
-
You have a backup of multiple VMs.
-
Before you take a backup of many VMs, ensure that the VMs are labeled by running the following command:
$ oc label vm <vm_name> app=<vm_name> -n openshift-adp -
Configure the label selectors in the
RestoreCR as shown in the following example:ExampleRestoreCRapiVersion: velero.io/v1 kind: Restore metadata: name: singlevmrestore namespace: openshift-adp spec: backupName: multiplevmbackup restorePVs: true LabelSelectors: - matchLabels: kubevirt.io/created-by: <datavolume_uid> - matchLabels: app: <vm_name>where:
datavolume_uid-
Specifies the UID of
DataVolumeof the VM that you want to restore. For example,b6…53a-ddd7-4d9d-9407-a0c…e5. vm_name-
Specifies the name of the VM that you want to restore. For example,
test-vm.
-
To restore a VM, run the following command:
$ oc apply -f <restore_cr_file_name>where:
restore_cr_file_name-
Specifies the name of the
RestoreCR file.
Configuring the DPA with client burst and QPS settings
The burst setting determines how many requests can be sent to the velero server before the limit is applied. After the burst limit is reached, the queries per second (QPS) setting determines how many additional requests can be sent per second.
You can set the burst and QPS values of the velero server by configuring the Data Protection Application (DPA) with the burst and QPS values. You can use the dpa.configuration.velero.client-burst and dpa.configuration.velero.client-qps fields of the DPA to set the burst and QPS values.
-
You have installed the OADP Operator.
-
Configure the
client-burstand theclient-qpsfields in the DPA as shown in the following example:Example Data Protection ApplicationapiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: test-dpa namespace: openshift-adp spec: backupLocations: - name: default velero: config: insecureSkipTLSVerify: "true" profile: "default" region: <bucket_region> s3ForcePathStyle: "true" s3Url: <bucket_url> credential: key: cloud name: cloud-credentials default: true objectStorage: bucket: <bucket_name> prefix: velero provider: aws configuration: nodeAgent: enable: true uploaderType: restic velero: client-burst: 500 client-qps: 300 defaultPlugins: - openshift - aws - kubevirtwhere:
client-burst-
Specifies the
client-burstvalue. In this example, theclient-burstfield is set to 500. client-qps-
Specifies the
client-qpsvalue. In this example, theclient-qpsfield is set to 300.
Configuring the node agent as a non-root and non-privileged user
To enhance the node agent security, you can configure the OADP Operator node agent daemonset to run as a non-root and non-privileged user by using the spec.configuration.velero.disableFsBackup setting in the DataProtectionApplication (DPA) custom resource (CR).
By setting the spec.configuration.velero.disableFsBackup setting to true, the node agent security context sets the root file system to read-only and sets the privileged flag to false.
Note
Setting spec.configuration.velero.disableFsBackup to true enhances the node agent security by removing the need for privileged containers and enforcing a read-only root file system.
However, it also disables File System Backup (FSB) with Kopia. If your workloads rely on FSB for backing up volumes that do not support native snapshots, then you should evaluate whether the disableFsBackup configuration fits your use case.
-
You have installed the OADP Operator.
-
Configure the
disableFsBackupfield in the DPA as shown in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: ts-dpa namespace: openshift-adp spec: backupLocations: - velero: credential: key: cloud name: cloud-credentials default: true objectStorage: bucket: <bucket_name> prefix: velero provider: gcp configuration: nodeAgent: enable: true uploaderType: kopia velero: defaultPlugins: - csi - gcp - openshift disableFsBackup: truewhere:
nodeAgent-
Specifies to enable the node agent in the DPA.
disableFsBackup-
Specifies to set the
disableFsBackupfield totrue.
-
Verify that the node agent security context is set to run as non-root and the root file system is
readOnlyby running the following command:$ oc get daemonset node-agent -o yamlThe example output is as following:
apiVersion: apps/v1 kind: DaemonSet metadata: ... name: node-agent namespace: openshift-adp ... spec: ... template: metadata: ... spec: containers: ... securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL privileged: false readOnlyRootFilesystem: true ... nodeSelector: kubernetes.io/os: linux os: name: linux restartPolicy: Always schedulerName: default-scheduler securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault serviceAccount: velero serviceAccountName: velero ....where:
allowPrivilegeEscalation-
Specifies that the
allowPrivilegeEscalationfield is false. privileged-
Specifies that the
privilegedfield is false. readOnlyRootFilesystem-
Specifies that the root file system is read-only.
runAsNonRoot-
Specifies that the node agent is run as a non-root user.
Configuring node agents and node labels
The Data Protection Application (DPA) uses the nodeSelector field to select which nodes can run the node agent. The nodeSelector field is the recommended form of node selection constraint.
-
Run the node agent on any node that you choose by adding a custom label:
$ oc label node/<node_name> node-role.kubernetes.io/nodeAgent=""Note
Any label specified must match the labels on each node.
-
Use the same custom label in the
DPA.spec.configuration.nodeAgent.podConfig.nodeSelectorfield, which you used for labeling nodes:configuration: nodeAgent: enable: true podConfig: nodeSelector: node-role.kubernetes.io/nodeAgent: ""The following example is an anti-pattern of
nodeSelectorand does not work unless both labels,node-role.kubernetes.io/infra: ""andnode-role.kubernetes.io/worker: "", are on the node:configuration: nodeAgent: enable: true podConfig: nodeSelector: node-role.kubernetes.io/infra: "" node-role.kubernetes.io/worker: ""
Configuring node agent load affinity
You can schedule the node agent pods on specific nodes by using the spec.podConfig.nodeSelector object of the DataProtectionApplication (DPA) custom resource (CR).
See the following example in which you can schedule the node agent pods on nodes with the label label.io/role: cpu-1 and other-label.io/other-role: cpu-2.
...
spec:
configuration:
nodeAgent:
enable: true
uploaderType: kopia
podConfig:
nodeSelector:
label.io/role: cpu-1
other-label.io/other-role: cpu-2
...
You can add more restrictions on the node agent pods scheduling by using the nodeagent.loadAffinity object in the DPA spec.
-
You must be logged in as a user with
cluster-adminprivileges. -
You have installed the OADP Operator.
-
You have configured the DPA CR.
-
Configure the DPA spec
nodegent.loadAffinityobject as shown in the following example.In the example, you ensure that the node agent pods are scheduled only on nodes with the label
label.io/role: cpu-1and the labellabel.io/hostnamematching with eithernode1ornode2.... spec: configuration: nodeAgent: enable: true loadAffinity: - nodeSelector: matchLabels: label.io/role: cpu-1 matchExpressions: - key: label.io/hostname operator: In values: - node1 - node2 ...where:
loadAffinity-
Specifies the
loadAffinityobject by adding thematchLabelsandmatchExpressionsobjects. matchExpressions-
Specifies the
matchExpressionsobject to add restrictions on the node agent pods scheduling.
Node agent load affinity guidelines
Use the following guidelines to configure the node agent loadAffinity object in the DataProtectionApplication (DPA) custom resource (CR).
-
Use the
spec.nodeagent.podConfig.nodeSelectorobject for simple node matching. -
Use the
loadAffinity.nodeSelectorobject without thepodConfig.nodeSelectorobject for more complex scenarios. -
You can use both
podConfig.nodeSelectorandloadAffinity.nodeSelectorobjects, but theloadAffinityobject must be equal or more restrictive as compared to thepodConfigobject. In this scenario, thepodConfig.nodeSelectorlabels must be a subset of the labels used in theloadAffinity.nodeSelectorobject. -
You cannot use the
matchExpressionsandmatchLabelsfields if you have configured bothpodConfig.nodeSelectorandloadAffinity.nodeSelectorobjects in the DPA. -
See the following example to configure both
podConfig.nodeSelectorandloadAffinity.nodeSelectorobjects in the DPA.... spec: configuration: nodeAgent: enable: true uploaderType: kopia loadAffinity: - nodeSelector: matchLabels: label.io/location: 'US' label.io/gpu: 'no' podConfig: nodeSelector: label.io/gpu: 'no'
Configuring node agent load concurrency
You can control the maximum number of node agent operations that can run simultaneously on each node within your cluster.
You can configure it using one of the following fields of the Data Protection Application (DPA):
-
globalConfig: Defines a default concurrency limit for the node agent across all nodes. -
perNodeConfig: Specifies different concurrency limits for specific nodes based onnodeSelectorlabels. This provides flexibility for environments where certain nodes might have different resource capacities or roles.
-
You must be logged in as a user with
cluster-adminprivileges.
-
If you want to use load concurrency for specific nodes, add labels to those nodes:
$ oc label node/<node_name> label.io/instance-type='large' -
Configure the load concurrency fields for your DPA instance:
configuration: nodeAgent: enable: true uploaderType: kopia loadConcurrency: globalConfig: 1 perNodeConfig: - nodeSelector: matchLabels: label.io/instance-type: large number: 3where:
globalConfig-
Specifies the global concurrent number. The default value is 1, which means there is no concurrency and only one load is allowed. The
globalConfigvalue does not have a limit. label.io/instance-type-
Specifies the label for per-node concurrency.
number-
Specifies the per-node concurrent number. You can specify many per-node concurrent numbers, for example, based on the instance type and size. The range of per-node concurrent number is the same as the global concurrent number. If the configuration file contains a per-node concurrent number and a global concurrent number, the per-node concurrent number takes precedence.
Configuring repository maintenance
OADP repository maintenance is a background job, you can configure it independently of the node agent pods. This means that you can schedule the repository maintenance pod on a node where the node agent is or is not running.
You can use the repository maintenance job affinity configurations in the DataProtectionApplication (DPA) custom resource (CR) only if you use Kopia as the backup repository.
You have the option to configure the load affinity at the global level affecting all repositories. Or you can configure the load affinity for each repository. You can also use a combination of global and per-repository configuration.
-
You must be logged in as a user with
cluster-adminprivileges. -
You have installed the OADP Operator.
-
You have configured the DPA CR.
-
Configure the
loadAffinityobject in the DPA spec by using either one or both of the following methods:-
Global configuration: Configure load affinity for all repositories as shown in the following example:
... spec: configuration: repositoryMaintenance: global: podResources: cpuRequest: "100m" cpuLimit: "200m" memoryRequest: "100Mi" memoryLimit: "200Mi" loadAffinity: - nodeSelector: matchLabels: label.io/gpu: 'no' matchExpressions: - key: label.io/location operator: In values: - US - EUwhere:
repositoryMaintenance-
Specifies the
repositoryMaintenanceobject as shown in the example. global-
Specifies the
globalobject to configure load affinity for all repositories.
-
Per-repository configuration: Configure load affinity per repository as shown in the following example:
... spec: configuration: repositoryMaintenance: myrepositoryname: loadAffinity: - nodeSelector: matchLabels: label.io/cpu: 'yes'where:
myrepositoryname-
Specifies the
repositoryMaintenanceobject for each repository.
-
Configuring Velero load affinity
With each OADP deployment, there is one Velero pod and its main purpose is to schedule Velero workloads. To schedule the Velero pod, you can use the velero.podConfig.nodeSelector and the velero.loadAffinity objects in the DataProtectionApplication (DPA) custom resource (CR) spec.
Use the podConfig.nodeSelector object to assign the Velero pod to specific nodes. You can also configure the velero.loadAffinity object for pod-level affinity and anti-affinity.
The OpenShift scheduler applies the rules and performs the scheduling of the Velero pod deployment.
-
You must be logged in as a user with
cluster-adminprivileges. -
You have installed the OADP Operator.
-
You have configured the DPA CR.
-
Configure the
velero.podConfig.nodeSelectorand thevelero.loadAffinityobjects in the DPA spec as shown in the following examples:-
velero.podConfig.nodeSelectorobject configuration:... spec: configuration: velero: podConfig: nodeSelector: some-label.io/custom-node-role: backup-core -
velero.loadAffinityobject configuration:... spec: configuration: velero: loadAffinity: - nodeSelector: matchLabels: label.io/gpu: 'no' matchExpressions: - key: label.io/location operator: In values: - US - EU
-
Overriding the imagePullPolicy setting in the DPA
In OADP 1.4.0 or earlier, the Operator sets the imagePullPolicy field of the Velero and node agent pods to Always for all images.
In OADP 1.4.1 or later, the Operator first checks if each image has the sha256 or sha512 digest and sets the imagePullPolicy field accordingly:
-
If the image has the digest, the Operator sets
imagePullPolicytoIfNotPresent. -
If the image does not have the digest, the Operator sets
imagePullPolicytoAlways.
You can also override the imagePullPolicy field by using the spec.imagePullPolicy field in the Data Protection Application (DPA).
-
You have installed the OADP Operator.
-
Configure the
spec.imagePullPolicyfield in the DPA as shown in the following example:Example Data Protection ApplicationapiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: test-dpa namespace: openshift-adp spec: backupLocations: - name: default velero: config: insecureSkipTLSVerify: "true" profile: "default" region: <bucket_region> s3ForcePathStyle: "true" s3Url: <bucket_url> credential: key: cloud name: cloud-credentials default: true objectStorage: bucket: <bucket_name> prefix: velero provider: aws configuration: nodeAgent: enable: true uploaderType: kopia velero: defaultPlugins: - openshift - aws - kubevirt - csi imagePullPolicy: Neverwhere:
imagePullPolicy-
Specifies the value for
imagePullPolicy. In this example, theimagePullPolicyfield is set toNever.
About incremental back up support
OADP supports incremental backups of block and Filesystem persistent volumes for both containerized, and OpenShift Virtualization workloads. The following table summarizes the support for File System Backup (FSB), Container Storage Interface (CSI), and CSI Data Mover:
| Volume mode | FSB - Restic | FSB - Kopia | CSI | CSI Data Mover |
|---|---|---|---|---|
Filesystem |
S [1], I [2] |
S [1], I [2] |
S [1] |
S [1], I [2] |
Block |
N [3] |
N [3] |
S [1] |
S [1], I [2] |
| Volume mode | FSB - Restic | FSB - Kopia | CSI | CSI Data Mover |
|---|---|---|---|---|
Filesystem |
N [3] |
N [3] |
S [1] |
S [1], I [2] |
Block |
N [3] |
N [3] |
S [1] |
S [1], I [2] |
-
Backup supported
-
Incremental backup supported
-
Not supported
Note
The CSI Data Mover backups use Kopia regardless of uploaderType.