Configuring the OpenShift API for Data Protection with {ibm-cloud-title}
You install the OpenShift API for Data Protection (OADP) Operator on an IBM Cloud cluster to back up and restore applications on the cluster. You configure IBM Cloud Object Storage (COS) to store the backups.
Configuring the COS instance
You create an IBM Cloud Object Storage (COS) instance to store the OADP backup data. After you create the COS instance, configure the HMAC service credentials.
-
You have an IBM Cloud Platform account.
-
You installed the IBM Cloud CLI.
-
You are logged in to IBM Cloud.
-
Install the IBM Cloud Object Storage (COS) plugin by running the following command:
$ ibmcloud plugin install cos -f -
Set a bucket name by running the following command:
$ BUCKET=<bucket_name> -
Set a bucket region by running the following command:
$ REGION=<bucket_region>where:
<bucket_region>-
Specifies the bucket region. For example,
eu-gb.
-
Create a resource group by running the following command:
$ ibmcloud resource group-create <resource_group_name> -
Set the target resource group by running the following command:
$ ibmcloud target -g <resource_group_name> -
Verify that the target resource group is correctly set by running the following command:
$ ibmcloud targetExample outputAPI endpoint: https://cloud.ibm.com Region: User: test-user Account: Test Account (fb6......e95) <-> 2...122 Resource group: DefaultIn the example output, the resource group is set to
Default. -
Set a resource group name by running the following command:
$ RESOURCE_GROUP=<resource_group>where:
<resource_group>-
Specifies the resource group name. For example,
"default".
-
Create an IBM Cloud
service-instanceresource by running the following command:$ ibmcloud resource service-instance-create \ <service_instance_name> \ <service_name> \ <service_plan> \ <region_name>where:
<service_instance_name>-
Specifies a name for the
service-instanceresource. <service_name>-
Specifies the service name. Alternatively, you can specify a service ID.
<service_plan>-
Specifies the service plan for your IBM Cloud account.
<region_name>-
Specifies the region name.
Refer to the following example command:
$ ibmcloud resource service-instance-create test-service-instance cloud-object-storage \ standard \ global \ -d premium-global-deploymentwhere:
cloud-object-storage-
Specifies the service name.
-d premium-global-deployment-
Specifies the deployment name.
-
Extract the service instance ID by running the following command:
$ SERVICE_INSTANCE_ID=$(ibmcloud resource service-instance test-service-instance --output json | jq -r '.[0].id') -
Create a COS bucket by running the following command:
$ ibmcloud cos bucket-create \ --bucket $BUCKET \ --ibm-service-instance-id $SERVICE_INSTANCE_ID \ --region $REGIONVariables such as
$BUCKET,$SERVICE_INSTANCE_ID, and$REGIONare replaced by the values you set previously. -
Create
HMACcredentials by running the following command.$ ibmcloud resource service-key-create test-key Writer --instance-name test-service-instance --parameters {\"HMAC\":true} -
Extract the access key ID and the secret access key from the
HMACcredentials and save them in thecredentials-velerofile. You can use thecredentials-velerofile to create asecretfor the backup storage location. Run the following command:$ cat > credentials-velero << __EOF__ [default] aws_access_key_id=$(ibmcloud resource service-key test-key -o json | jq -r '.[0].credentials.cos_hmac_keys.access_key_id') aws_secret_access_key=$(ibmcloud resource service-key test-key -o json | jq -r '.[0].credentials.cos_hmac_keys.secret_access_key') __EOF__
Creating a default Secret
You create a default Secret if your backup and snapshot locations use the same credentials or if you do not require a snapshot location.
Note
The DataProtectionApplication custom resource (CR) requires a default Secret. Otherwise, the installation will fail. If the name of the backup location Secret is not specified, the default name is used.
If you do not want to use the backup location credentials during the installation, you can create a Secret with the default name by using an empty credentials-velero file.
-
Your object storage and cloud storage, if any, must use the same credentials.
-
You must configure object storage for Velero.
-
Create a
credentials-velerofile for the backup storage location in the appropriate format for your cloud provider. -
Create a
Secretcustom resource (CR) with the default name:$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-veleroThe
Secretis referenced in thespec.backupLocations.credentialblock of theDataProtectionApplicationCR when you install the Data Protection Application.
Creating secrets for different credentials
Create separate Secret objects when your backup and snapshot locations require different credentials. This allows you to configure distinct authentication for each storage location while maintaining secure credential management.
-
Create a
credentials-velerofile for the snapshot location in the appropriate format for your cloud provider. -
Create a
Secretfor the snapshot location with the default name:$ oc create secret generic cloud-credentials -n openshift-adp --from-file cloud=credentials-velero -
Create a
credentials-velerofile for the backup location in the appropriate format for your object storage. -
Create a
Secretfor the backup location with a custom name:$ oc create secret generic <custom_secret> -n openshift-adp --from-file cloud=credentials-velero -
Add the
Secretwith the custom name to theDataProtectionApplicationCR, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> namespace: openshift-adp spec: ... backupLocations: - velero: provider: <provider> default: true credential: key: cloud name: <custom_secret> objectStorage: bucket: <bucket_name> prefix: <prefix>where:
custom_secret-
Specifies the backup location
Secretwith custom name.
Installing the Data Protection Application
You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication API.
-
You must install the OADP Operator.
-
You must configure object storage as a backup location.
-
If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
-
If the backup and snapshot locations use the same credentials, you must create a
Secretwith the default name,cloud-credentials.Note
If you do not want to specify backup or snapshot locations during the installation, you can create a default
Secretwith an emptycredentials-velerofile. If there is no defaultSecret, the installation will fail.
-
Click Ecosystem → Installed Operators and select the OADP Operator.
-
Under Provided APIs, click Create instance in the DataProtectionApplication box.
-
Click YAML View and update the parameters of the
DataProtectionApplicationmanifest:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: namespace: openshift-adp name: <dpa_name> spec: configuration: velero: defaultPlugins: - openshift - aws - csi backupLocations: - velero: provider: aws default: true objectStorage: bucket: <bucket_name> prefix: velero config: insecureSkipTLSVerify: 'true' profile: default region: <region_name> s3ForcePathStyle: 'true' s3Url: <s3_url> credential: key: cloud name: cloud-credentialswhere:
provider-
Specifies that the provider is
awswhen you use IBM Cloud as a backup storage location. bucket-
Specifies the IBM Cloud Object Storage (COS) bucket name.
region-
Specifies the COS region name, for example,
eu-gb. s3Url-
Specifies the S3 URL of the COS bucket. For example,
http://s3.eu-gb.cloud-object-storage.appdomain.cloud. Here,eu-gbis the region name. Replace the region name according to your bucket region. name-
Specifies the name of the secret you created by using the access key and the secret access key from the
HMACcredentials.
-
Click Create.
-
Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:
$ oc get all -n openshift-adpNAME READY STATUS RESTARTS AGE pod/oadp-operator-controller-manager-67d9494d47-6l8z8 2/2 Running 0 2m8s pod/node-agent-9cq4q 1/1 Running 0 94s pod/node-agent-m4lts 1/1 Running 0 94s pod/node-agent-pv4kr 1/1 Running 0 95s pod/velero-588db7f655-n842v 1/1 Running 0 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/oadp-operator-controller-manager-metrics-service ClusterIP 172.30.70.140 <none> 8443/TCP 2m8s service/openshift-adp-velero-metrics-svc ClusterIP 172.30.10.0 <none> 8085/TCP 8h NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/node-agent 3 3 3 3 3 <none> 96s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/oadp-operator-controller-manager 1/1 1 1 2m9s deployment.apps/velero 1/1 1 1 96s NAME DESIRED CURRENT READY AGE replicaset.apps/oadp-operator-controller-manager-67d9494d47 1 1 1 2m9s replicaset.apps/velero-588db7f655 1 1 1 96s
-
Verify that the
DataProtectionApplication(DPA) is reconciled by running the following command:$ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'{"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]} -
Verify the
typeis set toReconciled. -
Verify the backup storage location and confirm that the
PHASEisAvailableby running the following command:$ oc get backupstoragelocations.velero.io -n openshift-adpNAME PHASE LAST VALIDATED AGE DEFAULT dpa-sample-1 Available 1s 3d16h true
Setting Velero CPU and memory resource allocations
You set the CPU and memory resource allocations for the Velero pod by editing the DataProtectionApplication custom resource (CR) manifest.
-
You must have the OpenShift API for Data Protection (OADP) Operator installed.
-
Edit the values in the
spec.configuration.velero.podConfig.ResourceAllocationsblock of theDataProtectionApplicationCR manifest, as in the following example:apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: <dpa_sample> spec: # ... configuration: velero: podConfig: nodeSelector: <node_selector> resourceAllocations: limits: cpu: "1" memory: 1024Mi requests: cpu: 200m memory: 256Miwhere:
nodeSelector-
Specifies the node selector to be supplied to Velero podSpec.
resourceAllocations-
Specifies the resource allocations listed for average usage.
Note
Kopia is an option in OADP 1.3 and later releases. You can use Kopia for file system backups, and Kopia is your only option for Data Mover cases with the built-in Data Mover.
Kopia is more resource intensive than Restic, and you might need to adjust the CPU and memory requirements accordingly.
Configuring node agents and node labels
The Data Protection Application (DPA) uses the nodeSelector field to select which nodes can run the node agent. The nodeSelector field is the recommended form of node selection constraint.
-
Run the node agent on any node that you choose by adding a custom label:
$ oc label node/<node_name> node-role.kubernetes.io/nodeAgent=""Note
Any label specified must match the labels on each node.
-
Use the same custom label in the
DPA.spec.configuration.nodeAgent.podConfig.nodeSelectorfield, which you used for labeling nodes:configuration: nodeAgent: enable: true podConfig: nodeSelector: node-role.kubernetes.io/nodeAgent: ""The following example is an anti-pattern of
nodeSelectorand does not work unless both labels,node-role.kubernetes.io/infra: ""andnode-role.kubernetes.io/worker: "", are on the node:configuration: nodeAgent: enable: true podConfig: nodeSelector: node-role.kubernetes.io/infra: "" node-role.kubernetes.io/worker: ""
Configuring the DPA with client burst and QPS settings
The burst setting determines how many requests can be sent to the velero server before the limit is applied. After the burst limit is reached, the queries per second (QPS) setting determines how many additional requests can be sent per second.
You can set the burst and QPS values of the velero server by configuring the Data Protection Application (DPA) with the burst and QPS values. You can use the dpa.configuration.velero.client-burst and dpa.configuration.velero.client-qps fields of the DPA to set the burst and QPS values.
-
You have installed the OADP Operator.
-
Configure the
client-burstand theclient-qpsfields in the DPA as shown in the following example:Example Data Protection ApplicationapiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: test-dpa namespace: openshift-adp spec: backupLocations: - name: default velero: config: insecureSkipTLSVerify: "true" profile: "default" region: <bucket_region> s3ForcePathStyle: "true" s3Url: <bucket_url> credential: key: cloud name: cloud-credentials default: true objectStorage: bucket: <bucket_name> prefix: velero provider: aws configuration: nodeAgent: enable: true uploaderType: restic velero: client-burst: 500 client-qps: 300 defaultPlugins: - openshift - aws - kubevirtwhere:
client-burst-
Specifies the
client-burstvalue. In this example, theclient-burstfield is set to 500. client-qps-
Specifies the
client-qpsvalue. In this example, theclient-qpsfield is set to 300.
Configuring node agent load affinity
You can schedule the node agent pods on specific nodes by using the spec.podConfig.nodeSelector object of the DataProtectionApplication (DPA) custom resource (CR).
See the following example in which you can schedule the node agent pods on nodes with the label label.io/role: cpu-1 and other-label.io/other-role: cpu-2.
...
spec:
configuration:
nodeAgent:
enable: true
uploaderType: kopia
podConfig:
nodeSelector:
label.io/role: cpu-1
other-label.io/other-role: cpu-2
...
You can add more restrictions on the node agent pods scheduling by using the nodeagent.loadAffinity object in the DPA spec.
-
You must be logged in as a user with
cluster-adminprivileges. -
You have installed the OADP Operator.
-
You have configured the DPA CR.
-
Configure the DPA spec
nodegent.loadAffinityobject as shown in the following example.In the example, you ensure that the node agent pods are scheduled only on nodes with the label
label.io/role: cpu-1and the labellabel.io/hostnamematching with eithernode1ornode2.... spec: configuration: nodeAgent: enable: true loadAffinity: - nodeSelector: matchLabels: label.io/role: cpu-1 matchExpressions: - key: label.io/hostname operator: In values: - node1 - node2 ...where:
loadAffinity-
Specifies the
loadAffinityobject by adding thematchLabelsandmatchExpressionsobjects. matchExpressions-
Specifies the
matchExpressionsobject to add restrictions on the node agent pods scheduling.
Node agent load affinity guidelines
Use the following guidelines to configure the node agent loadAffinity object in the DataProtectionApplication (DPA) custom resource (CR).
-
Use the
spec.nodeagent.podConfig.nodeSelectorobject for simple node matching. -
Use the
loadAffinity.nodeSelectorobject without thepodConfig.nodeSelectorobject for more complex scenarios. -
You can use both
podConfig.nodeSelectorandloadAffinity.nodeSelectorobjects, but theloadAffinityobject must be equal or more restrictive as compared to thepodConfigobject. In this scenario, thepodConfig.nodeSelectorlabels must be a subset of the labels used in theloadAffinity.nodeSelectorobject. -
You cannot use the
matchExpressionsandmatchLabelsfields if you have configured bothpodConfig.nodeSelectorandloadAffinity.nodeSelectorobjects in the DPA. -
See the following example to configure both
podConfig.nodeSelectorandloadAffinity.nodeSelectorobjects in the DPA.... spec: configuration: nodeAgent: enable: true uploaderType: kopia loadAffinity: - nodeSelector: matchLabels: label.io/location: 'US' label.io/gpu: 'no' podConfig: nodeSelector: label.io/gpu: 'no'
Configuring node agent load concurrency
You can control the maximum number of node agent operations that can run simultaneously on each node within your cluster.
You can configure it using one of the following fields of the Data Protection Application (DPA):
-
globalConfig: Defines a default concurrency limit for the node agent across all nodes. -
perNodeConfig: Specifies different concurrency limits for specific nodes based onnodeSelectorlabels. This provides flexibility for environments where certain nodes might have different resource capacities or roles.
-
You must be logged in as a user with
cluster-adminprivileges.
-
If you want to use load concurrency for specific nodes, add labels to those nodes:
$ oc label node/<node_name> label.io/instance-type='large' -
Configure the load concurrency fields for your DPA instance:
configuration: nodeAgent: enable: true uploaderType: kopia loadConcurrency: globalConfig: 1 perNodeConfig: - nodeSelector: matchLabels: label.io/instance-type: large number: 3where:
globalConfig-
Specifies the global concurrent number. The default value is 1, which means there is no concurrency and only one load is allowed. The
globalConfigvalue does not have a limit. label.io/instance-type-
Specifies the label for per-node concurrency.
number-
Specifies the per-node concurrent number. You can specify many per-node concurrent numbers, for example, based on the instance type and size. The range of per-node concurrent number is the same as the global concurrent number. If the configuration file contains a per-node concurrent number and a global concurrent number, the per-node concurrent number takes precedence.
Configuring repository maintenance
OADP repository maintenance is a background job, you can configure it independently of the node agent pods. This means that you can schedule the repository maintenance pod on a node where the node agent is or is not running.
You can use the repository maintenance job affinity configurations in the DataProtectionApplication (DPA) custom resource (CR) only if you use Kopia as the backup repository.
You have the option to configure the load affinity at the global level affecting all repositories. Or you can configure the load affinity for each repository. You can also use a combination of global and per-repository configuration.
-
You must be logged in as a user with
cluster-adminprivileges. -
You have installed the OADP Operator.
-
You have configured the DPA CR.
-
Configure the
loadAffinityobject in the DPA spec by using either one or both of the following methods:-
Global configuration: Configure load affinity for all repositories as shown in the following example:
... spec: configuration: repositoryMaintenance: global: podResources: cpuRequest: "100m" cpuLimit: "200m" memoryRequest: "100Mi" memoryLimit: "200Mi" loadAffinity: - nodeSelector: matchLabels: label.io/gpu: 'no' matchExpressions: - key: label.io/location operator: In values: - US - EUwhere:
repositoryMaintenance-
Specifies the
repositoryMaintenanceobject as shown in the example. global-
Specifies the
globalobject to configure load affinity for all repositories.
-
Per-repository configuration: Configure load affinity per repository as shown in the following example:
... spec: configuration: repositoryMaintenance: myrepositoryname: loadAffinity: - nodeSelector: matchLabels: label.io/cpu: 'yes'where:
myrepositoryname-
Specifies the
repositoryMaintenanceobject for each repository.
-
Configuring Velero load affinity
With each OADP deployment, there is one Velero pod and its main purpose is to schedule Velero workloads. To schedule the Velero pod, you can use the velero.podConfig.nodeSelector and the velero.loadAffinity objects in the DataProtectionApplication (DPA) custom resource (CR) spec.
Use the podConfig.nodeSelector object to assign the Velero pod to specific nodes. You can also configure the velero.loadAffinity object for pod-level affinity and anti-affinity.
The OpenShift scheduler applies the rules and performs the scheduling of the Velero pod deployment.
-
You must be logged in as a user with
cluster-adminprivileges. -
You have installed the OADP Operator.
-
You have configured the DPA CR.
-
Configure the
velero.podConfig.nodeSelectorand thevelero.loadAffinityobjects in the DPA spec as shown in the following examples:-
velero.podConfig.nodeSelectorobject configuration:... spec: configuration: velero: podConfig: nodeSelector: some-label.io/custom-node-role: backup-core -
velero.loadAffinityobject configuration:... spec: configuration: velero: loadAffinity: - nodeSelector: matchLabels: label.io/gpu: 'no' matchExpressions: - key: label.io/location operator: In values: - US - EU
-
Overriding the imagePullPolicy setting in the DPA
In OADP 1.4.0 or earlier, the Operator sets the imagePullPolicy field of the Velero and node agent pods to Always for all images.
In OADP 1.4.1 or later, the Operator first checks if each image has the sha256 or sha512 digest and sets the imagePullPolicy field accordingly:
-
If the image has the digest, the Operator sets
imagePullPolicytoIfNotPresent. -
If the image does not have the digest, the Operator sets
imagePullPolicytoAlways.
You can also override the imagePullPolicy field by using the spec.imagePullPolicy field in the Data Protection Application (DPA).
-
You have installed the OADP Operator.
-
Configure the
spec.imagePullPolicyfield in the DPA as shown in the following example:Example Data Protection ApplicationapiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: test-dpa namespace: openshift-adp spec: backupLocations: - name: default velero: config: insecureSkipTLSVerify: "true" profile: "default" region: <bucket_region> s3ForcePathStyle: "true" s3Url: <bucket_url> credential: key: cloud name: cloud-credentials default: true objectStorage: bucket: <bucket_name> prefix: velero provider: aws configuration: nodeAgent: enable: true uploaderType: kopia velero: defaultPlugins: - openshift - aws - kubevirt - csi imagePullPolicy: Neverwhere:
imagePullPolicy-
Specifies the value for
imagePullPolicy. In this example, theimagePullPolicyfield is set toNever.
Configuring the DPA with more than one BSL
Configure the DataProtectionApplication (DPA) custom resource (CR) with multiple BackupStorageLocation (BSL) resources to store backups across different locations using provider-specific credentials. This provides backup distribution and location-specific restore capabilities.
For example, you have configured the following two BSLs:
-
Configured one BSL in the DPA and set it as the default BSL.
-
Created another BSL independently by using the
BackupStorageLocationCR.
As you have already set the BSL created through the DPA as the default, you cannot set the independently created BSL again as the default. This means, at any given time, you can set only one BSL as the default BSL.
-
You must install the OADP Operator.
-
You must create the secrets by using the credentials provided by the cloud provider.
-
Configure the
DataProtectionApplicationCR with more than oneBackupStorageLocationCR. See the following example:Example DPAapiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication #... backupLocations: - name: aws velero: provider: aws default: true objectStorage: bucket: <bucket_name> prefix: <prefix> config: region: <region_name> profile: "default" credential: key: cloud name: cloud-credentials - name: odf velero: provider: aws default: false objectStorage: bucket: <bucket_name> prefix: <prefix> config: profile: "default" region: <region_name> s3Url: <url> insecureSkipTLSVerify: "true" s3ForcePathStyle: "true" credential: key: cloud name: <custom_secret_name_odf> #...where:
name: aws-
Specifies a name for the first BSL.
default: true-
Indicates that this BSL is the default BSL. If a BSL is not set in the
Backup CR, the default BSL is used. You can set only one BSL as the default. <bucket_name>-
Specifies the bucket name.
<prefix>-
Specifies a prefix for Velero backups. For example,
velero. <region_name>-
Specifies the AWS region for the bucket.
cloud-credentials-
Specifies the name of the default
Secretobject that you created. name: odf-
Specifies a name for the second BSL.
<url>-
Specifies the URL of the S3 endpoint.
<custom_secret_name_odf>-
Specifies the correct name for the
Secret. For example,custom_secret_name_odf. If you do not specify aSecretname, the default name is used.
-
Specify the BSL to be used in the backup CR. See the following example.
Example backup CRapiVersion: velero.io/v1 kind: Backup # ... spec: includedNamespaces: - <namespace> storageLocation: <backup_storage_location> defaultVolumesToFsBackup: truewhere:
<namespace>-
Specifies the namespace to back up.
<backup_storage_location>-
Specifies the storage location.
Disabling the node agent in DataProtectionApplication
If you are not using Restic, Kopia, or DataMover for your backups, you can disable the nodeAgent field in the DataProtectionApplication custom resource (CR). Before you disable nodeAgent, ensure the OADP Operator is idle and not running any backups.
-
To disable the
nodeAgent, set theenableflag tofalse. See the following example:ExampleDataProtectionApplicationCR# ... configuration: nodeAgent: enable: false uploaderType: kopia # ...where:
enable-
Enables the node agent.
-
To enable the
nodeAgent, set theenableflag totrue. See the following example:ExampleDataProtectionApplicationCR# ... configuration: nodeAgent: enable: true uploaderType: kopia # ...where:
enable-
Enables the node agent.
You can set up a job to enable and disable the
nodeAgentfield in theDataProtectionApplicationCR. For more information, see "Running tasks in pods using jobs".