Configuring higher VM workload density

You can increase the number of virtual machines (VMs) on nodes by overcommitting memory (RAM). Increasing VM workload density can be useful in the following situations:

You have many similar workloads.
You have underused workloads.

Note

Memory overcommitment can lower workload performance on a highly utilized system.

Using wasp-agent to increase VM workload density

The wasp-agent component facilitates memory overcommitment by assigning swap resources to worker nodes.

The wasp-agent component is deployed automatically if memoryOvercommitPercentage is set to more than 100 when you first create the HyperConverged custom resource (CR).

Important

Swap resources can be only assigned to virtual machine workloads (VM pods) of the Burstable Quality of Service (QoS) class. VM pods of the Guaranteed QoS class and pods of any QoS class that do not belong to VMs cannot swap resources.

For descriptions of QoS classes, see Configure Quality of Service for Pods (Kubernetes documentation).

Using spec.domain.resources.requests.memory in the VM manifest disables the memory overcommit configuration. Use spec.domain.memory.guest instead.

Prerequisites

You have installed the OpenShift CLI (oc).
You are logged into the cluster with the cluster-admin role.
A memory overcommit ratio is defined.
The node belongs to a worker pool.

Note

The wasp-agent component deploys an Open Container Initiative (OCI) hook to enable swap usage for containers on the node level. The low-level nature requires the DaemonSet object to be privileged.

Procedure

Provision swap by creating a MachineConfig object:

Create a MachineConfig file with the parameters shown in the following example:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 90-worker-swap
spec:
  config:
    ignition:
      version: 3.5.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,YXBpVmVyc2lvbjoga3ViZWxldC5jb25maWcuazhzLmlvL3YxYmV0YTEKa2luZDogS3ViZWxldENvbmZpZ3VyYXRpb24KZmFpbFN3YXBPbjogZmFsc2UK
        mode: 420
        overwrite: true
        path: /etc/openshift/kubelet.conf.d/90-swap.conf
    systemd:
      units:
        - contents: |
            [Unit]
            Description=Enable swap
            ConditionFirstBoot=no
            ConditionPathExists=/var/tmp/swapfile

            [Service]
            Type=oneshot
            ExecStart=/bin/sh -c "sudo swapon /var/tmp/swapfile"

            [Install]
            RequiredBy=kubelet-dependencies.target
          enabled: true
          name: swap-enable.service
        - contents: |
            [Unit]
            Description=Provision and enable swap
            ConditionFirstBoot=no
            ConditionPathExists=!/var/tmp/swapfile

            [Service]
            Type=oneshot
            Environment=SWAP_SIZE_MB=5000
            ExecStart=/bin/sh -c "sudo fallocate -l ${SWAP_SIZE_MB}M /var/tmp/swapfile && \
            sudo chmod 600 /var/tmp/swapfile && \
            sudo mkswap /var/tmp/swapfile && \
            sudo swapon /var/tmp/swapfile && \
            free -h"

            [Install]
            RequiredBy=kubelet-dependencies.target
          enabled: true
          name: swap-provision.service
        - contents: |
            [Unit]
            Description=Restrict swap for system slice
            ConditionFirstBoot=no

            [Service]
            Type=oneshot
            ExecStart=/bin/sh -c "sudo systemctl set-property --runtime system.slice MemorySwapMax=0 IODeviceLatencyTargetSec=\"/ 50ms\""

            [Install]
            RequiredBy=kubelet-dependencies.target
          enabled: true
          name: cgroup-system-slice-config.service

To have enough swap space for the worst-case scenario, make sure to have at least as much swap space provisioned as overcommitted RAM. Calculate the amount of swap space to be provisioned on a node by using the following formula:

NODE_SWAP_SPACE = NODE_RAM * (MEMORY_OVER_COMMIT_PERCENT / 100% - 1)

Example:

NODE_SWAP_SPACE = 16 GB * (150% / 100% - 1)
               = 16 GB * (1.5 - 1)
               = 16 GB * (0.5)
               =  8 GB

Wait for the worker nodes to sync with the new configuration by running the following command:
```
$ oc wait mcp worker --for condition=Updated=True --timeout=-1s
```
1. Enable memory overcommitment in OpenShift Virtualization by using the web console or the CLI.
  Web console
  
  In the OpenShift Container Platform web console, go to Virtualization → Overview → Settings → General settings → Memory density.
  
  Set Configure memory density to on.
  
  Expand the Current memory density line.
  
  Set the density value by moving the Requested memory density slider. You can increase the density from 100% up to 400% in increments of 25%.
  
  The Memory density field shows the actual and requested values.
  
  Click Save.
  
  CLI
  
  Configure your OpenShift Virtualization to enable higher memory density and set the overcommit rate:
  
  $ oc -n openshift-cnv patch HyperConverged/kubevirt-hyperconverged --type='json' -p='[ \ { \ "op": "replace", \ "path": "/spec/higherWorkloadDensity/memoryOvercommitPercentage", \ "value": 150 \ } \ ]'
  
  Successful output:
  
  hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched

Verification

To verify the deployment of wasp-agent, run the following command:
```
$ oc rollout status ds wasp-agent -n openshift-cnv
```
If the deployment is successful, the following message is displayed:

Example output:
```
daemon set "wasp-agent" successfully rolled out
```
To verify that swap is correctly provisioned, complete the following steps:
1. View a list of worker nodes by running the following command:
  $ oc get nodes -l node-role.kubernetes.io/worker
2. Select a node from the list and display its memory usage by running the following command:
  $ oc debug node/<selected_node> -- free -m
  Replace <selected_node> with the node name.
  
  If swap is provisioned, an amount greater than zero is displayed in the Swap: row.
  
  Table 1. Example output
  
  total
  
  used
  
  free
  
  shared
  
  buff/cache
  
  available
  
  Mem:
  
  31846
  
  23155
  
  1044
  
  6014
  
  14483
  
  8690
  
  Swap:
  
  8191
  
  2337
  
  5854
Verify the OpenShift Virtualization memory overcommitment configuration by running the following command:
```
$ oc -n openshift-cnv get HyperConverged/kubevirt-hyperconverged -o jsonpath='{.spec.higherWorkloadDensity}{"\n"}'
```
Example output:
```
{"memoryOvercommitPercentage":150}
```
The returned value must match the value you had previously configured.

Removing the wasp-agent component

If you no longer need memory overcommitment, you can remove the wasp-agent component and associated resources from your cluster.

Prerequisites

You have logged in to the cluster with the cluster-admin role.
You have installed the OpenShift CLI (oc).

Procedure

Revert the memory overcommitment configuration by running the following command:

$ oc -n openshift-cnv patch HyperConverged/kubevirt-hyperconverged \
  --type='json' \
  -p='[{"op": "remove", "path": "/spec/higherWorkloadDensity"}]'

Delete the MachineConfig that provisions swap memory by running the following command:
```
$ oc delete machineconfig 90-worker-swap
```

Verification

Confirm that swap is no longer enabled on a node, by running the following command and observing the output:
```
$ oc debug node/<selected_node> -- free -m
```
Ensure that the Swap: row shows 0 or that no swap space shows as provisioned.

	total	used	free	shared	buff/cache	available
Mem:	31846	23155	1044	6014	14483	8690
Swap:	8191	2337	5854