Skip to content

Deploying hosted control planes on bare metal in a disconnected environment

When you provision hosted control planes on bare metal, you use the Agent platform. The Agent platform and multicluster engine for Kubernetes Operator work together to enable disconnected deployments. The Agent platform uses the central infrastructure management service to add worker nodes to a hosted cluster. For an introduction to the central infrastructure management service, see Enabling the central infrastructure management service.

Disconnected environment architecture for bare metal

The following diagram illustrates an example architecture of a disconnected environment:

Disconnected architecture diagram

  1. Configure infrastructure services, including the registry certificate deployment with TLS support, web server, and DNS, to ensure that the disconnected deployment works.

  2. Create a config map in the openshift-config namespace. In this example, the config map is named registry-config. The content of the config map is the Registry CA certificate. The data field of the config map must contain the following key/value:

    • Key: <registry_dns_domain_name>..<port>, for example, registry.hypershiftdomain.lab..5000:. Ensure that you place .. after the registry DNS domain name when you specify a port.

    • Value: The certificate content

      For more information about creating a config map, see Configuring TLS certificates for a disconnected installation of hosted control planes.

  3. Modify the images.config.openshift.io custom resource (CR) specification and adds a new field named additionalTrustedCA with a value of name: registry-config.

  4. Create a config map that contains two data fields. One field contains the registries.conf file in RAW format, and the other field contains the Registry CA and is named ca-bundle.crt. The config map belongs to the multicluster-engine namespace, and the config map name is referenced in other objects. For an example of a config map, see the following sample configuration:

    apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-registries
  namespace: multicluster-engine
  labels:
    app: assisted-service
data:
  ca-bundle.crt: |
    -----BEGIN CERTIFICATE-----
    # ...
    -----END CERTIFICATE-----
  registries.conf: |
    unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]

    [[registry]]
    prefix = ""
    location = "registry.redhat.io/openshift4"
    mirror-by-digest-only = true

    [[registry.mirror]]
      location = "registry.ocp-edge-cluster-0.qe.lab.redhat.com:5000/openshift4"

    [[registry]]
    prefix = ""
    location = "registry.redhat.io/rhacm2"
    mirror-by-digest-only = true
# ...
# ...

  5. In the multicluster engine Operator namespace, you create the multiclusterengine CR, which enables both the Agent and hypershift-addon add-ons. The multicluster engine Operator namespace must contain the config maps to modify behavior in a disconnected deployment. The namespace also contains the multicluster-engine, assisted-service, and hypershift-addon-manager pods.

  6. Create the following objects that are necessary to deploy the hosted cluster:

    • Secrets: Secrets contain the pull secret, SSH key, and etcd encryption key.

    • Config map: The config map contains the CA certificate of the private registry.

    • HostedCluster: The HostedCluster resource defines the configuration of the cluster that the user intends to create.

    • NodePool: The NodePool resource identifies the node pool that references the machines to use for the data plane.

  7. After you create the hosted cluster objects, the HyperShift Operator establishes the HostedControlPlane namespace to accommodate control plane pods. The namespace also hosts components such as Agents, bare metal hosts (BMHs), and the InfraEnv resource. Later, you create the InfraEnv resource, and after ISO creation, you create the BMHs and their secrets that contain baseboard management controller (BMC) credentials.

  8. The Metal3 Operator in the openshift-machine-api namespace inspects the new BMHs. Then, the Metal3 Operator tries to connect to the BMCs to start them by using the configured LiveISO and RootFS values that are specified through the AgentServiceConfig CR in the multicluster engine Operator namespace.

  9. After the worker nodes of the HostedCluster resource are started, an Agent container is started. This agent establishes contact with the Assisted Service, which orchestrates the actions to complete the deployment. Initially, you need to scale the NodePool resource to the number of worker nodes for the HostedCluster resource. The Assisted Service manages the remaining tasks.

  10. At this point, you wait for the deployment process to be completed.

Requirements to deploy hosted control planes on bare metal in a disconnected environment

To configure hosted control planes in a disconnected environment, you must meet the following prerequisites:

  • CPU: The number of CPUs provided determines how many hosted clusters can run concurrently. In general, use 16 CPUs for each node for 3 nodes. For minimal development, you can use 12 CPUs for each node for 3 nodes.

  • Memory: The amount of RAM affects how many hosted clusters can be hosted. Use 48 GB of RAM for each node. For minimal development, 18 GB of RAM might be sufficient.

  • Storage: Use SSD storage for multicluster engine Operator.

    • Management cluster: 250 GB.

    • Registry: The storage needed depends on the number of releases, operators, and images that are hosted. An acceptable number might be 500 GB, preferably separated from the disk that hosts the hosted cluster.

    • Web server: The storage needed depends on the number of ISOs and images that are hosted. An acceptable number might be 500 GB.

  • Production: For a production environment, separate the management cluster, the registry, and the web server on different disks. This example illustrates a possible configuration for production:

    • Registry: 2 TB

    • Management cluster: 500 GB

    • Web server: 2 TB

Extracting the release image digest

You can extract the OpenShift Container Platform release image digest by using the tagged image.

Procedure

  • Obtain the image digest by running the following command:

    $ oc adm release info <tagged_openshift_release_image> | grep "Pull From"

    Replace <tagged_openshift_release_image> with the tagged image for the supported OpenShift Container Platform version, for example, quay.io/openshift-release-dev/ocp-release:4.14.0-x8_64.

    Example output
    Pull From: quay.io/openshift-release-dev/ocp-release@sha256:69d1292f64a2b67227c5592c1a7d499c7d00376e498634ff8e1946bc9ccdddfe

DNS configurations on bare metal

The API Server for the hosted cluster is exposed as a NodePort service. A DNS entry must exist for api.<hosted_cluster_name>.<base_domain> that points to destination where the API Server can be reached.

The DNS entry can be as simple as a record that points to one of the nodes in the management cluster that is running the hosted control plane. The entry can also point to a load balancer that is deployed to redirect incoming traffic to the ingress pods.

Example DNS configuration
api.example.krnl.es.    IN A 192.168.122.20
api.example.krnl.es.    IN A 192.168.122.21
api.example.krnl.es.    IN A 192.168.122.22
api-int.example.krnl.es.    IN A 192.168.122.20
api-int.example.krnl.es.    IN A 192.168.122.21
api-int.example.krnl.es.    IN A 192.168.122.22
`*`.apps.example.krnl.es. IN A 192.168.122.23

Note

In the previous example, *.apps.example.krnl.es. IN A 192.168.122.23 is either a node in the hosted cluster or a load balancer, if one has been configured.

If you are configuring DNS for a disconnected environment on an IPv6 network, the configuration looks like the following example.

Example DNS configuration for an IPv6 network
api.example.krnl.es.    IN A 2620:52:0:1306::5
api.example.krnl.es.    IN A 2620:52:0:1306::6
api.example.krnl.es.    IN A 2620:52:0:1306::7
api-int.example.krnl.es.    IN A 2620:52:0:1306::5
api-int.example.krnl.es.    IN A 2620:52:0:1306::6
api-int.example.krnl.es.    IN A 2620:52:0:1306::7
`*`.apps.example.krnl.es. IN A 2620:52:0:1306::10

If you are configuring DNS for a disconnected environment on a dual stack network, be sure to include DNS entries for both IPv4 and IPv6.

Example DNS configuration for a dual stack network
host-record=api-int.hub-dual.dns.base.domain.name,192.168.126.10
host-record=api.hub-dual.dns.base.domain.name,192.168.126.10
address=/apps.hub-dual.dns.base.domain.name/192.168.126.11
dhcp-host=aa:aa:aa:aa:10:01,ocp-master-0,192.168.126.20
dhcp-host=aa:aa:aa:aa:10:02,ocp-master-1,192.168.126.21
dhcp-host=aa:aa:aa:aa:10:03,ocp-master-2,192.168.126.22
dhcp-host=aa:aa:aa:aa:10:06,ocp-installer,192.168.126.25
dhcp-host=aa:aa:aa:aa:10:07,ocp-bootstrap,192.168.126.26

host-record=api-int.hub-dual.dns.base.domain.name,2620:52:0:1306::2
host-record=api.hub-dual.dns.base.domain.name,2620:52:0:1306::2
address=/apps.hub-dual.dns.base.domain.name/2620:52:0:1306::3
dhcp-host=aa:aa:aa:aa:10:01,ocp-master-0,[2620:52:0:1306::5]
dhcp-host=aa:aa:aa:aa:10:02,ocp-master-1,[2620:52:0:1306::6]
dhcp-host=aa:aa:aa:aa:10:03,ocp-master-2,[2620:52:0:1306::7]
dhcp-host=aa:aa:aa:aa:10:06,ocp-installer,[2620:52:0:1306::8]
dhcp-host=aa:aa:aa:aa:10:07,ocp-bootstrap,[2620:52:0:1306::9]

Deploying a registry for hosted control planes in a disconnected environment

For development environments, deploy a small, self-hosted registry by using a Podman container. For production environments, deploy an enterprise-hosted registry, such as Red Hat Quay, Nexus, or Artifactory.

Procedure

To deploy a small registry by using Podman, complete the following steps:

  1. As a privileged user, access the ${HOME} directory and create the following script:

    #!/usr/bin/env bash

set -euo pipefail

PRIMARY_NIC=$(ls -1 /sys/class/net | grep -v podman | head -1)
export PATH=/root/bin:$PATH
export PULL_SECRET="/root/baremetal/hub/openshift_pull.json" 

if [[ ! -f $PULL_SECRET ]];then
  echo "Pull Secret not found, exiting..."
  exit 1
fi

dnf -y install podman httpd httpd-tools jq skopeo libseccomp-devel
export IP=$(ip -o addr show $PRIMARY_NIC | head -1 | awk '{print $4}' | cut -d'/' -f1)
REGISTRY_NAME=registry.$(hostname --long)
REGISTRY_USER=dummy
REGISTRY_PASSWORD=dummy
KEY=$(echo -n $REGISTRY_USER:$REGISTRY_PASSWORD | base64)
echo "{\"auths\": {\"$REGISTRY_NAME:5000\": {\"auth\": \"$KEY\", \"email\": \"sample-email@domain.ltd\"}}}" > /root/disconnected_pull.json
mv ${PULL_SECRET} /root/openshift_pull.json.old
jq ".auths += {\"$REGISTRY_NAME:5000\": {\"auth\": \"$KEY\",\"email\": \"sample-email@domain.ltd\"}}" < /root/openshift_pull.json.old > $PULL_SECRET
mkdir -p /opt/registry/{auth,certs,data,conf}
cat <<EOF > /opt/registry/conf/config.yml
version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
  delete:
    enabled: true
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
compatibility:
  schema1:
    enabled: true
EOF
openssl req -newkey rsa:4096 -nodes -sha256 -keyout /opt/registry/certs/domain.key -x509 -days 3650 -out /opt/registry/certs/domain.crt -subj "/C=US/ST=Madrid/L=San Bernardo/O=Karmalabs/OU=Guitar/CN=$REGISTRY_NAME" -addext "subjectAltName=DNS:$REGISTRY_NAME"
cp /opt/registry/certs/domain.crt /etc/pki/ca-trust/source/anchors/
update-ca-trust extract
htpasswd -bBc /opt/registry/auth/htpasswd $REGISTRY_USER $REGISTRY_PASSWORD
podman create --name registry --net host --security-opt label=disable --replace -v /opt/registry/data:/var/lib/registry:z -v /opt/registry/auth:/auth:z -v /opt/registry/conf/config.yml:/etc/docker/registry/config.yml -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry" -e "REGISTRY_HTTP_SECRET=ALongRandomSecretForRegistry" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd -v /opt/registry/certs:/certs:z -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key docker.io/library/registry:latest
[ "$?" == "0" ] || !!
systemctl enable --now registry
    1. Replace the location of the PULL_SECRET with the appropriate location for your setup.

  2. Name the script file registry.sh and save it. When you run the script, it pulls in the following information:

    • The registry name, based on the hypervisor hostname

    • The necessary credentials and user access details

  3. Adjust permissions by adding the execution flag as follows:

    $ chmod u+x ${HOME}/registry.sh

  4. To run the script without any parameters, enter the following command:

    $ ${HOME}/registry.sh

    The script starts the server. The script uses a systemd service for management purposes.

  5. If you need to manage the script, you can use the following commands:

    $ systemctl status
    $ systemctl start
    $ systemctl stop

The root folder for the registry is in the /opt/registry directory and contains the following subdirectories:

  • certs contains the TLS certificates.

  • auth contains the credentials.

  • data contains the registry images.

  • conf contains the registry configuration.

Setting up a management cluster for hosted control planes in a disconnected environment

To set up an OpenShift Container Platform management cluster, you need to ensure that the multicluster engine for Kubernetes Operator is installed. The multicluster engine Operator plays a crucial role in deploying clusters across providers.

Prerequisites

  • There must be bidirectional connectivity between the management cluster and the Baseboard Management Controller (BMC) of the target Bare Metal Host (BMH). As an alternative, you follow a Boot It Yourself approach through the Agent provider.

  • The hosted cluster must be able to resolve and reach the API hostname of the management cluster hostname and *.apps hostname. Here is an example of the API hostname of the management cluster and *.apps hostname:

    • api.management-cluster.internal.domain.com

    • console-openshift-console.apps.management-cluster.internal.domain.com

  • The management cluster must be able to resolve and reach the API and *.apps hostname of the hosted cluster. Here is an example of the API hostname of the hosted cluster and *.apps hostname:

    • api.sno-hosted-cluster-1.internal.domain.com

    • console-openshift-console.apps.sno-hosted-cluster-1.internal.domain.com

Procedure

  1. Install multicluster engine Operator 2.4 or later on an OpenShift Container Platform cluster. You can install multicluster engine Operator as an Operator from the OpenShift Container Platform software catalog. The HyperShift Operator is included with multicluster engine Operator. For more information about installing multicluster engine Operator, see "Installing and upgrading multicluster engine operator" in the Red Hat Advanced Cluster Management documentation.

  2. Ensure that the HyperShift Operator is installed. The HyperShift Operator is automatically included with multicluster engine Operator, but if you need to manually install it, follow the steps in "Manually enabling the hypershift-addon managed cluster add-on for local-cluster".

Next steps

Next, configure the web server.

Additional resources

Configuring the web server for hosted control planes in a disconnected environment

You need to configure an additional web server to host the Red Hat Enterprise Linux CoreOS (RHCOS) images that are associated with the OpenShift Container Platform release that you are deploying as a hosted cluster.

Procedure

To configure the web server, complete the following steps:

  1. Extract the openshift-install binary from the OpenShift Container Platform release that you want to use by entering the following command:

    $ oc adm -a ${LOCAL_SECRET_JSON} release extract --command=openshift-install \
  "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}"

  2. Run the following script. The script creates a folder in the /opt/srv directory. The folder contains the RHCOS images to provision the worker nodes.

    #!/bin/bash

WEBSRV_FOLDER=/opt/srv
ROOTFS_IMG_URL="$(./openshift-install coreos print-stream-json | jq -r '.architectures.x86_64.artifacts.metal.formats.pxe.rootfs.location')" 
LIVE_ISO_URL="$(./openshift-install coreos print-stream-json | jq -r '.architectures.x86_64.artifacts.metal.formats.iso.disk.location')" 

mkdir -p ${WEBSRV_FOLDER}/images
curl -Lk ${ROOTFS_IMG_URL} -o ${WEBSRV_FOLDER}/images/${ROOTFS_IMG_URL##*/}
curl -Lk ${LIVE_ISO_URL} -o ${WEBSRV_FOLDER}/images/${LIVE_ISO_URL##*/}
chmod -R 755 ${WEBSRV_FOLDER}/*

## Run Webserver
podman ps --noheading | grep -q websrv-ai
if [[ $? == 0 ]];then
    echo "Launching Registry pod..."
    /usr/bin/podman run --name websrv-ai --net host -v /opt/srv:/usr/local/apache2/htdocs:z quay.io/alosadag/httpd:p8080
fi
    1. You can find the ROOTFS_IMG_URL value on the OpenShift CI Release page.
    2. You can find the LIVE_ISO_URL value on the OpenShift CI Release page.

After the download is completed, a container runs to host the images on a web server. The container uses a variation of the official HTTPd image, which also enables it to work with IPv6 networks.

Configuring image mirroring for hosted control planes in a disconnected environment

Image mirroring is the process of fetching images from external registries, such as registry.redhat.com or quay.io, and storing them in your private registry.

In the following procedures, the oc-mirror tool is used, which is a binary that uses the ImageSetConfiguration object. In the file, you can specify the following information:

  • The OpenShift Container Platform versions to mirror. The versions are in quay.io.

  • The additional Operators to mirror. Select packages individually.

  • The extra images that you want to add to the repository.

Prerequisites

  • Ensure that the registry server is running before you start the mirroring process.

Procedure

To configure image mirroring, complete the following steps:

  1. Ensure that your ${HOME}/.docker/config.json file is updated with the registries that you are going to mirror from and with the private registry that you plan to push the images to.

  2. By using the following example, create an ImageSetConfiguration object to use for mirroring. Replace values as needed to match your environment:

    apiVersion: mirror.openshift.io/v2alpha1
kind: ImageSetConfiguration
mirror:
  platform:
    channels:
    - name: candidate-4.19
      minVersion: <4.x.y-build>  
      maxVersion: <4.x.y-build> 
      type: ocp
    kubeVirtContainer: true 
    graph: true
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.19
    packages:
    - name: lvms-operator
    - name: local-storage-operator
    - name: odf-csi-addons-operator
    - name: odf-operator
    - name: mcg-operator
    - name: ocs-operator
    - name: metallb-operator
    - name: kubevirt-hyperconverged
    1. Replace <4.x.y-build> with the supported OpenShift Container Platform version you want to use.
    2. Set this optional flag to true if you want to also mirror the container disk image for the Red Hat Enterprise Linux CoreOS (RHCOS) boot image for the KubeVirt provider. This flag is available with oc-mirror v2 only.
    3. For deployments that use the KubeVirt provider, include this line.

  3. Start the mirroring process by entering the following command:

    $ oc-mirror --v2 --config imagesetconfig.yaml \
  --workspace file://mirror-file docker://<registry>

    After the mirroring process is finished, you have a new folder named mirror-file, which contains the ImageDigestMirrorSet (IDMS), ImageTagMirrorSet (ITMS), and the catalog sources to apply on the hosted cluster.

  4. Mirror the nightly or CI versions of OpenShift Container Platform by configuring the imagesetconfig.yaml file as follows:

    apiVersion: mirror.openshift.io/v2alpha1
kind: ImageSetConfiguration
mirror:
  platform:
    graph: true
    release: registry.ci.openshift.org/ocp/release:<4.x.y-build> 
    kubeVirtContainer: true 
# ...
    1. Replace <4.x.y-build> with the supported OpenShift Container Platform version you want to use.
    2. Set this optional flag to true if you want to also mirror the container disk image for the Red Hat Enterprise Linux CoreOS (RHCOS) boot image for the KubeVirt provider. This flag is available with oc-mirror v2 only.

  5. If you have a partially disconnected environment, mirror the images from the image set configuration to a registry by entering the following command:

    $ oc mirror -c imagesetconfig.yaml \
  --workspace file://<file_path> docker://<mirror_registry_url> --v2

    For more information, see "Mirroring an image set in a partially disconnected environment".

  6. If you have a fully disconnected environment, perform the following steps:

    1. Mirror the images from the specified image set configuration to the disk by entering the following command:

      $ oc mirror -c imagesetconfig.yaml file://<file_path> --v2

      For more information, see "Mirroring an image set in a fully disconnected environment".

    2. Process the image set file on the disk and mirror the contents to a target mirror registry by entering the following command:

      $ oc mirror -c imagesetconfig.yaml \
  --from file://<file_path> docker://<mirror_registry_url> --v2

  7. Mirror the latest multicluster engine Operator images by following the steps in Install on disconnected networks.

Additional resources

Applying objects in the management cluster

After the mirroring process is complete, you need to apply two objects in the management cluster:

  • ImageContentSourcePolicy (ICSP) or ImageDigestMirrorSet (IDMS)

  • Catalog sources

When you use the oc-mirror tool, the output artifacts are in a folder named oc-mirror-workspace/results-XXXXXX/.

The ICSP or IDMS initiates a MachineConfig change that does not restart your nodes but restarts the kubelet on each of them. After the nodes are marked as READY, you need to apply the newly generated catalog sources.

The catalog sources initiate actions in the openshift-marketplace Operator, such as downloading the catalog image and processing it to retrieve all the PackageManifests that are included in that image.

Procedure

  1. To check the new sources, run the following command by using the new CatalogSource as a source:

    $ oc get packagemanifest

  2. To apply the artifacts, complete the following steps:

    1. Create the ICSP or IDMS artifacts by entering the following command:

      $ oc apply -f oc-mirror-workspace/results-XXXXXX/imageContentSourcePolicy.yaml

    2. Wait for the nodes to become ready, and then enter the following command:

      $ oc apply -f catalogSource-XXXXXXXX-index.yaml

  3. Mirror the OLM catalogs and configure the hosted cluster to point to the mirror.

    When you use the management (default) OLMCatalogPlacement mode, the image stream that is used for OLM catalogs is not automatically amended with override information from the ICSP on the management cluster.

    1. If the OLM catalogs are properly mirrored to an internal registry by using the original name and tag, add the hypershift.openshift.io/olm-catalogs-is-registry-overrides annotation to the HostedCluster resource. The format is "sr1=dr1,sr2=dr2", where the source registry string is a key and the destination registry is a value.

    2. To bypass the OLM catalog image stream mechanism, use the following four annotations on the HostedCluster resource to directly specify the addresses of the four images to use for OLM Operator catalogs:

      • hypershift.openshift.io/certified-operators-catalog-image

      • hypershift.openshift.io/community-operators-catalog-image

      • hypershift.openshift.io/redhat-marketplace-catalog-image

      • hypershift.openshift.io/redhat-operators-catalog-image

In this case, the image stream is not created, and you must update the value of the annotations when the internal mirror is refreshed to pull in Operator updates.

Next steps

Deploy the multicluster engine Operator by completing the steps in Deploying multicluster engine Operator for a disconnected installation of hosted control planes.

Additional resources

Deploying AgentServiceConfig resources

The AgentServiceConfig custom resource is an essential component of the Assisted Service add-on that is part of multicluster engine Operator. It is responsible for bare metal cluster deployment. When the add-on is enabled, you deploy the AgentServiceConfig resource to configure the add-on.

In addition to configuring the AgentServiceConfig resource, you need to include additional config maps to ensure that multicluster engine Operator functions properly in a disconnected environment.

Procedure

  1. Configure the custom registries by adding the following config map, which contains the disconnected details to customize the deployment:

    apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-registries
  namespace: multicluster-engine
  labels:
    app: assisted-service
data:
  ca-bundle.crt: |
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----
  registries.conf: |
    unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]

    [[registry]]
    prefix = ""
    location = "registry.redhat.io/openshift4"
    mirror-by-digest-only = true

    [[registry.mirror]]
      location = "registry.dns.base.domain.name:5000/openshift4" 

    [[registry]]
    prefix = ""
    location = "registry.redhat.io/rhacm2"
    mirror-by-digest-only = true
    # ...
    # ...
    1. Replace dns.base.domain.name with the DNS base domain name.

      The object contains two fields:

      • Custom CAs: This field contains the Certificate Authorities (CAs) that are loaded into the various processes of the deployment.

      • Registries: The Registries.conf field contains information about images and namespaces that need to be consumed from a mirror registry rather than the original source registry.

  2. Configure the Assisted Service by adding the AssistedServiceConfig object, as shown in the following example:

    apiVersion: agent-install.openshift.io/v1beta1
kind: AgentServiceConfig
metadata:
  annotations:
    unsupported.agent-install.openshift.io/assisted-service-configmap: assisted-service-config 
  name: agent
  namespace: multicluster-engine
spec:
  mirrorRegistryRef:
    name: custom-registries 
  databaseStorage:
    storageClassName: lvms-vg1
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi
  filesystemStorage:
    storageClassName: lvms-vg1
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 20Gi
  osImages: 
  - cpuArchitecture: x86_64 
    openshiftVersion: "4.14"
    rootFSUrl: http://registry.dns.base.domain.name:8080/images/rhcos-414.92.202308281054-0-live-rootfs.x86_64.img 
    url: http://registry.dns.base.domain.name:8080/images/rhcos-414.92.202308281054-0-live.x86_64.iso
    version: 414.92.202308281054-0
  - cpuArchitecture: x86_64
   openshiftVersion: "4.15"
   rootFSUrl: http://registry.dns.base.domain.name:8080/images/rhcos-415.92.202403270524-0-live-rootfs.x86_64.img
   url: http://registry.dns.base.domain.name:8080/images/rhcos-415.92.202403270524-0-live.x86_64.iso
   version: 415.92.202403270524-0
    1. The metadata.annotations["unsupported.agent-install.openshift.io/assisted-service-configmap"] annotation references the config map name that the Operator consumes to customize behavior.
    2. The spec.mirrorRegistryRef.name annotation points to the config map that contains disconnected registry information that the Assisted Service Operator consumes. This config map adds those resources during the deployment process.
    3. The spec.osImages field contains different versions available for deployment by this Operator. This field is mandatory. This example assumes that you already downloaded the RootFS and LiveISO files.
    4. Add a cpuArchitecture subsection for every OpenShift Container Platform release that you want to deploy. In this example, cpuArchitecture subsections are included for 4.14 and 4.15.
    5. In the rootFSUrl and url fields, replace dns.base.domain.name with the DNS base domain name.

  3. Deploy all of the objects by concatenating them into a single file and applying them to the management cluster. To do so, enter the following command:

    $ oc apply -f agentServiceConfig.yaml

    The command triggers two pods.

    Example output
    assisted-image-service-0                               1/1     Running   2             11d 
assisted-service-668b49548-9m7xw                       2/2     Running   5             11d
    1. The assisted-image-service pod is responsible for creating the Red Hat Enterprise Linux CoreOS (RHCOS) boot image template, which is customized for each cluster that you deploy.
    2. The assisted-service refers to the Operator.
Next steps

Configure TLS certificates.

Configuring TLS certificates for a disconnected installation of hosted control planes

To ensure proper function in a disconnected deployment, you need to configure the registry CA certificates in the management cluster and the worker nodes for the hosted cluster.

Adding the registry CA to the management cluster

To add the registry CA to the management cluster, complete the following steps.

Procedure

  1. Create a config map that resembles the following example:

    apiVersion: v1
kind: ConfigMap
metadata:
  name: <config_map_name> 
  namespace: <config_map_namespace> 
data: 
  <registry_name>..<port>: | 
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----
  <registry_name>..<port>: |
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----
  <registry_name>..<port>: |
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----
    1. Specify the name of the config map.
    2. Specify the namespace for the config map.
    3. In the data field, specify the registry names and the registry certificate content. Replace <port> with the port where the registry server is running; for example, 5000.
    4. Ensure that the data in the config map is defined by using | only instead of other methods, such as | -. If you use other methods, issues can occur when the pod reads the certificates.

  2. Patch the cluster-wide object, image.config.openshift.io to include the following specification:

    spec:
  additionalTrustedCA:
    - name: registry-config

    As a result of this patch, the control plane nodes can retrieve images from the private registry and the HyperShift Operator can extract the OpenShift Container Platform payload for hosted cluster deployments.

    The process to patch the object might take several minutes to be completed.

Adding the registry CA to the worker nodes for the hosted cluster

In order for the data plane workers in the hosted cluster to be able to retrieve images from the private registry, you need to add the registry CA to the worker nodes.

Procedure

  1. In the hc.spec.additionalTrustBundle file, add the following specification:

    spec:
  additionalTrustBundle:
    - name: user-ca-bundle
    1. The user-ca-bundle entry is a config map that you create in the next step.

  2. In the same namespace where the HostedCluster object is created, create the user-ca-bundle config map. The config map resembles the following example:

    apiVersion: v1
data:
  ca-bundle.crt: |
    // Registry1 CA
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----

    // Registry2 CA
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----

    // Registry3 CA
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----

kind: ConfigMap
metadata:
  name: user-ca-bundle
  namespace: <hosted_cluster_namespace>
    1. Specify the namespace where the HostedCluster object is created.

Creating a hosted cluster on bare metal

A hosted cluster is an OpenShift Container Platform cluster with its control plane and API endpoint hosted on a management cluster. The hosted cluster includes the control plane and its corresponding data plane.

Deploying hosted cluster objects

Typically, the HyperShift Operator creates the HostedControlPlane namespace. However, in this case, you want to include all the objects before the HyperShift Operator begins to reconcile the HostedCluster object. Then, when the Operator starts the reconciliation process, it can find all of the objects in place.

Procedure

  1. Create a YAML file with the following information about the namespaces:

    ---
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: null
  name: <hosted_cluster_namespace>-<hosted_cluster_name> 
spec: {}
status: {}
---
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: null
  name: <hosted_cluster_namespace> 
spec: {}
status: {}
    1. Replace <hosted_cluster_name> with your hosted cluster.
    2. Replace <hosted_cluster_namespace> with the name of your hosted cluster namespace.

  2. Create a YAML file with the following information about the config maps and secrets to include in the HostedCluster deployment:

    ---
apiVersion: v1
data:
  ca-bundle.crt: |
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----
kind: ConfigMap
metadata:
  name: user-ca-bundle
  namespace: <hosted_cluster_namespace> 
---
apiVersion: v1
data:
  .dockerconfigjson: xxxxxxxxx
kind: Secret
metadata:
  creationTimestamp: null
  name: <hosted_cluster_name>-pull-secret 
  namespace: <hosted_cluster_namespace> 
---
apiVersion: v1
kind: Secret
metadata:
  name: sshkey-cluster-<hosted_cluster_name> 
  namespace: <hosted_cluster_namespace> 
stringData:
  id_rsa.pub: ssh-rsa xxxxxxxxx
---
apiVersion: v1
data:
  key: nTPtVBEt03owkrKhIdmSW8jrWRxU57KO/fnZa8oaG0Y=
kind: Secret
metadata:
  creationTimestamp: null
  name: <hosted_cluster_name>-etcd-encryption-key 
  namespace: <hosted_cluster_namespace> 
type: Opaque
    1. Replace <hosted_cluster_namespace> with the name of your hosted cluster namespace.
    2. Replace <hosted_cluster_name> with your hosted cluster.

  3. Create a YAML file that contains the RBAC roles so that Assisted Service agents can be in the same HostedControlPlane namespace as the hosted control plane and still be managed by the cluster API:

    apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  creationTimestamp: null
  name: capi-provider-role
  namespace: <hosted_cluster_namespace>-<hosted_cluster_name>  
rules:
- apiGroups:
  - agent-install.openshift.io
  resources:
  - agents
  verbs:
  - '*'
    1. Replace <hosted_cluster_namespace> with the name of your hosted cluster namespace.
    2. Replace <hosted_cluster_name> with your hosted cluster.

  4. Create a YAML file with information about the HostedCluster object, replacing values as necessary:

    apiVersion: hypershift.openshift.io/v1beta1
kind: HostedCluster
metadata:
  name: <hosted_cluster_name> 
  namespace: <hosted_cluster_namespace> 
spec:
  additionalTrustBundle:
    name: "user-ca-bundle"
  olmCatalogPlacement: guest
  configuration:
    operatorhub:
      disableAllDefaultSources: true 
  imageContentSources: 
  - source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
    mirrors:
    - registry.<dns.base.domain.name>:5000/openshift/release 
  - source: quay.io/openshift-release-dev/ocp-release
    mirrors:
    - registry.<dns.base.domain.name>:5000/openshift/release-images 
  - mirrors:
    - registry.<dns.base.domain.name>:5000/openshift/release-images 
  - source: registry.redhat.io/multicluster-engine
    mirrors:
    - registry.<dns.base.domain.name>:5000/openshift/multicluster-engine 
  # ...
  autoscaling: {}
  controllerAvailabilityPolicy: SingleReplica
  dns:
    baseDomain: <dns.base.domain.name> 
  etcd:
    managed:
      storage:
        persistentVolume:
          size: 8Gi
        restoreSnapshotURL: null
        type: PersistentVolume
    managementType: Managed
  fips: false
  networking:
    clusterNetwork:
    - cidr: 10.132.0.0/14
    - cidr: fd01::/48
    networkType: OVNKubernetes
    serviceNetwork:
    - cidr: 172.31.0.0/16
    - cidr: fd02::/112
  platform:
    agent:
      agentNamespace: <hosted_cluster_namespace>-<hosted_cluster_name>  
    type: Agent
  pullSecret:
    name: <hosted_cluster_name>-pull-secret 
  release:
    image: registry.<dns.base.domain.name>:5000/openshift/release-images:<4.x.y>-x86_64  
  secretEncryption:
    aescbc:
      activeKey:
        name: <hosted_cluster_name>-etcd-encryption-key 
    type: aescbc
  services:
  - service: APIServer
    servicePublishingStrategy:
      type: LoadBalancer
  - service: OAuthServer
    servicePublishingStrategy:
      type: Route
  - service: OIDC
    servicePublishingStrategy:
      type: Route
  - service: Konnectivity
    servicePublishingStrategy:
      type: Route
  - service: Ignition
    servicePublishingStrategy:
      type: Route
  sshKey:
    name: sshkey-cluster-<hosted_cluster_name> 
status:
  controlPlaneEndpoint:
    host: ""
    port: 0
    1. Replace <hosted_cluster_name> with your hosted cluster.
    2. Replace <hosted_cluster_namespace> with the name of your hosted cluster namespace.
    3. For the disableAllDefaultSources parameter, specify true if you want to disable all default OLM catalog resources. The default value is false, which enables all default OLM catalog resources.
    4. The imageContentSources section contains mirror references for user workloads within the hosted cluster.
    5. Replace <dns.base.domain.name> with the DNS base domain name.
    6. Replace <4.x.y> with the supported OpenShift Container Platform version you want to use.

  5. Create all of the objects that you defined in the YAML files by concatenating them into a file and applying them against the management cluster. To do so, enter the following command:

    $ oc apply -f 01-4.14-hosted_cluster-nodeport.yaml
    Example output
    NAME                                                  READY   STATUS    RESTARTS   AGE
capi-provider-5b57dbd6d5-pxlqc                        1/1     Running   0          3m57s
catalog-operator-9694884dd-m7zzv                      2/2     Running   0          93s
cluster-api-f98b9467c-9hfrq                           1/1     Running   0          3m57s
cluster-autoscaler-d7f95dd5-d8m5d                     1/1     Running   0          93s
cluster-image-registry-operator-5ff5944b4b-648ht      1/2     Running   0          93s
cluster-network-operator-77b896ddc-wpkq8              1/1     Running   0          94s
cluster-node-tuning-operator-84956cd484-4hfgf         1/1     Running   0          94s
cluster-policy-controller-5fd8595d97-rhbwf            1/1     Running   0          95s
cluster-storage-operator-54dcf584b5-xrnts             1/1     Running   0          93s
cluster-version-operator-9c554b999-l22s7              1/1     Running   0          95s
control-plane-operator-6fdc9c569-t7hr4                1/1     Running   0          3m57s
csi-snapshot-controller-785c6dc77c-8ljmr              1/1     Running   0          77s
csi-snapshot-controller-operator-7c6674bc5b-d9dtp     1/1     Running   0          93s
csi-snapshot-webhook-5b8584875f-2492j                 1/1     Running   0          77s
dns-operator-6874b577f-9tc6b                          1/1     Running   0          94s
etcd-0                                                3/3     Running   0          3m39s
hosted-cluster-config-operator-f5cf5c464-4nmbh        1/1     Running   0          93s
ignition-server-6b689748fc-zdqzk                      1/1     Running   0          95s
ignition-server-proxy-54d4bb9b9b-6zkg7                1/1     Running   0          95s
ingress-operator-6548dc758b-f9gtg                     1/2     Running   0          94s
konnectivity-agent-7767cdc6f5-tw782                   1/1     Running   0          95s
kube-apiserver-7b5799b6c8-9f5bp                       4/4     Running   0          3m7s
kube-controller-manager-5465bc4dd6-zpdlk              1/1     Running   0          44s
kube-scheduler-5dd5f78b94-bbbck                       1/1     Running   0          2m36s
machine-approver-846c69f56-jxvfr                      1/1     Running   0          92s
oauth-openshift-79c7bf44bf-j975g                      2/2     Running   0          62s
olm-operator-767f9584c-4lcl2                          2/2     Running   0          93s
openshift-apiserver-5d469778c6-pl8tj                  3/3     Running   0          2m36s
openshift-controller-manager-6475fdff58-hl4f7         1/1     Running   0          95s
openshift-oauth-apiserver-dbbc5cc5f-98574             2/2     Running   0          95s
openshift-route-controller-manager-5f6997b48f-s9vdc   1/1     Running   0          95s
packageserver-67c87d4d4f-kl7qh                        2/2     Running   0          93s

    When the hosted cluster is available, the output looks like the following example.

    Example output
    NAMESPACE   NAME         VERSION   KUBECONFIG                PROGRESS   AVAILABLE   PROGRESSING   MESSAGE
clusters    hosted-dual            hosted-admin-kubeconfig   Partial    True          False         The hosted control plane is available

Creating a NodePool object for the hosted cluster

A NodePool is a scalable set of worker nodes that is associated with a hosted cluster. NodePool machine architectures remain consistent within a specific pool and are independent of the machine architecture of the control plane.

Procedure

  1. Create a YAML file with the following information about the NodePool object, replacing values as necessary:

    apiVersion: hypershift.openshift.io/v1beta1
kind: NodePool
metadata:
  creationTimestamp: null
  name: <hosted_cluster_name> \
  namespace: <hosted_cluster_namespace> \
spec:
  arch: amd64
  clusterName: <hosted_cluster_name>
  management:
    autoRepair: false \
    upgradeType: InPlace \
  nodeDrainTimeout: 0s
  platform:
    type: Agent
  release:
    image: registry.<dns.base.domain.name>:5000/openshift/release-images:4.x.y-x86_64 \
  replicas: 2 
status:
  replicas: 2
    1. Replace <hosted_cluster_name> with your hosted cluster.
    2. Replace <hosted_cluster_namespace> with the name of your hosted cluster namespace.
    3. The autoRepair field is set to false because the node will not be re-created if it is removed.
    4. The upgradeType is set to InPlace, which indicates that the same bare metal node is reused during an upgrade.
    5. All of the nodes included in this NodePool are based on the following OpenShift Container Platform version: 4.x.y-x86_64. Replace the <dns.base.domain.name> value with your DNS base domain name and the 4.x.y value with the supported OpenShift Container Platform version you want to use.
    6. You can set the replicas value to 2 to create two node pool replicas in your hosted cluster.

  2. Create the NodePool object by entering the following command:

    $ oc apply -f 02-nodepool.yaml
    Example output
    NAMESPACE   NAME          CLUSTER   DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION                              UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
clusters    hosted-dual   hosted    0                               False         False        4.x.y-x86_64

Creating an InfraEnv resource for the hosted cluster

The InfraEnv resource is an Assisted Service object that includes essential details, such as the pullSecretRef and the sshAuthorizedKey. Those details are used to create the Red Hat Enterprise Linux CoreOS (RHCOS) boot image that is customized for the hosted cluster.

You can host more than one InfraEnv resource, and each one can adopt certain types of hosts. For example, you might want to divide your server farm between a host that has greater RAM capacity.

Procedure

  1. Create a YAML file with the following information about the InfraEnv resource, replacing values as necessary:

    apiVersion: agent-install.openshift.io/v1beta1
kind: InfraEnv
metadata:
  name: <hosted_cluster_name>
  namespace: <hosted_cluster_namespace>
spec:
  pullSecretRef:
    name: pull-secret
  sshAuthorizedKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDk7ICaUE+/k4zTpxLk4+xFdHi4ZuDi5qjeF52afsNkw0w/glILHhwpL5gnp5WkRuL8GwJuZ1VqLC9EKrdmegn4MrmUlq7WTsP0VFOZFBfq2XRUxo1wrRdor2z0Bbh93ytR+ZsDbbLlGngXaMa0Vbt+z74FqlcajbHTZ6zBmTpBVq5RHtDPgKITdpE1fongp7+ZXQNBlkaavaqv8bnyrP4BWahLP4iO9/xJF9lQYboYwEEDzmnKLMW1VtCE6nJzEgWCufACTbxpNS7GvKtoHT/OVzw8ArEXhZXQUS1UY8zKsX2iXwmyhw5Sj6YboA8WICs4z+TrFP89LmxXY0j6536TQFyRz1iB4WWvCbH5n6W+ABV2e8ssJB1AmEy8QYNwpJQJNpSxzoKBjI73XxvPYYC/IjPFMySwZqrSZCkJYqQ023ySkaQxWZT7in4KeMu7eS2tC+Kn4deJ7KwwUycx8n6RHMeD8Qg9flTHCv3gmab8JKZJqN3hW1D378JuvmIX4V0=

    where:

    <hosted_cluster_namespace>

    Specifies the name of your hosted cluster namespace.

    pullSecretRef

    Specifies the config map reference in the same namespace as the InfraEnv, where the pull secret is used.

    sshAuthorizedKey

    Specifies the SSH public key that is placed in the boot image. The SSH key allows access to the worker nodes as the core user.

  2. Create the InfraEnv resource by entering the following command:

    $ oc apply -f 03-infraenv.yaml
    Example output
    NAMESPACE              NAME     ISO CREATED AT
clusters-hosted-dual   hosted   2023-09-11T15:14:10Z

Creating bare metal hosts for the hosted cluster

A bare metal host is an openshift-machine-api object that encompasses physical and logical details so that it can be identified by a Metal3 Operator. Those details are associated with other Assisted Service objects, known as agents.

Prerequisites

  • Before you create the bare metal host and destination nodes, you must have the destination machines ready.

  • You have installed a Red Hat Enterprise Linux CoreOS (RHCOS) compute machine on bare-metal infrastructure for use in the cluster.

Procedure

To create a bare metal host, complete the following steps:

  1. Create a YAML file with the following information. For more information about what details to enter for the bare metal host, see "Provisioning new hosts in a user-provisioned cluster by using the BMO".

    Because you have at least one secret that holds the bare metal host credentials, you need to create at least two objects for each worker node.

    apiVersion: v1
kind: Secret
metadata:
  name: <hosted_cluster_name>-worker0-bmc-secret
  namespace: <hosted_cluster_namespace>
data:
  password: YWRtaW4=
  username: YWRtaW4=
type: Opaque
# ...
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: <hosted_cluster_name>-worker0
  namespace: <hosted_cluster_namespace>
  labels:
    infraenvs.agent-install.openshift.io: <hosted_cluster_name>
  annotations:
    inspect.metal3.io: disabled
    bmac.agent-install.openshift.io/hostname: <hosted_cluster_name>-worker0
spec:
  automatedCleaningMode: disabled
  bmc:
    disableCertificateVerification: true
    address: redfish-virtualmedia://[192.168.126.1]:9000/redfish/v1/Systems/local/<hosted_cluster_name>-worker0
    credentialsName: <hosted_cluster_name>-worker0-bmc-secret
  bootMACAddress: aa:aa:aa:aa:02:11
  online: true

    where:

    <hosted_cluster_name>

    Specifies the name of your hosted cluster.

    <hosted_cluster_namespace>

    Specifies the name of your hosted cluster namespace.

    password

    Specifies the password of the baseboard management controller (BMC) in Base64 format.

    username

    Specifies the user name of the BMC in Base64 format.

    infraenvs.agent-install.openshift.io

    Serves as the link between the Assisted Installer and the BareMetalHost objects.

    bmac.agent-install.openshift.io/hostname

    Represents the node name that is adopted during deployment.

    automatedCleaningMode

    Prevents the node from being erased by the Metal3 Operator.

    disableCertificateVerification

    Is set to true to bypass certificate validation from the client.

    address

    Denotes the BMC address of the worker node.

    credentialsName

    Points to the secret where the user and password credentials are stored.

    bootMACAddress

    Indicates the interface MAC address that the node starts from.

    online

    Defines the state of the node after the BareMetalHost object is created.

  2. Deploy the BareMetalHost object by entering the following command:

    $ oc apply -f 04-bmh.yaml

    During the process, you can view the following output:

    • This output indicates that the process is trying to reach the nodes:

      Example output
      NAMESPACE         NAME             STATE         CONSUMER   ONLINE   ERROR   AGE
clusters-hosted   hosted-worker0   registering              true             2s
clusters-hosted   hosted-worker1   registering              true             2s
clusters-hosted   hosted-worker2   registering              true             2s

    • This output indicates that the nodes are starting:

      Example output
      NAMESPACE         NAME             STATE          CONSUMER   ONLINE   ERROR   AGE
clusters-hosted   hosted-worker0   provisioning              true             16s
clusters-hosted   hosted-worker1   provisioning              true             16s
clusters-hosted   hosted-worker2   provisioning              true             16s

    • This output indicates that the nodes started successfully:

      Example output
      NAMESPACE         NAME             STATE         CONSUMER   ONLINE   ERROR   AGE
clusters-hosted   hosted-worker0   provisioned              true             67s
clusters-hosted   hosted-worker1   provisioned              true             67s
clusters-hosted   hosted-worker2   provisioned              true             67s

  3. After the nodes start, notice the agents in the namespace, as shown in this example:

    Example output
    NAMESPACE         NAME                                   CLUSTER   APPROVED   ROLE          STAGE
clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0411             true       auto-assign
clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0412             true       auto-assign
clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0413             true       auto-assign

    The agents represent nodes that are available for installation. To assign the nodes to a hosted cluster, scale up the node pool.

Additional resources

Scaling up the node pool

After you create the bare metal hosts, their statuses change from Registering to Provisioning to Provisioned. The nodes start with the LiveISO of the agent and a default pod that is named agent. That agent is responsible for receiving instructions from the Assisted Service Operator to install the OpenShift Container Platform payload.

Procedure

  1. To scale up the node pool, enter the following command:

    $ oc -n <hosted_cluster_namespace> scale nodepool <hosted_cluster_name> \
  --replicas 3

    where:

    • <hosted_cluster_namespace> is the name of the hosted cluster namespace.

    • <hosted_cluster_name> is the name of the hosted cluster.

  2. After the scaling process is complete, notice that the agents are assigned to a hosted cluster:

    Example output
    NAMESPACE         NAME                                   CLUSTER   APPROVED   ROLE          STAGE
clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0411   hosted    true       auto-assign
clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0412   hosted    true       auto-assign
clusters-hosted   aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaa0413   hosted    true       auto-assign

  3. Also notice that the node pool replicas are set:

    Example output
    NAMESPACE   NAME     CLUSTER   DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION       UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
clusters    hosted   hosted    3                               False         False        <4.x.y>-x86_64                                     Minimum availability requires 3 replicas, current 0 available

    Replace <4.x.y> with the supported OpenShift Container Platform version that you want to use.

  4. Wait until the nodes join the cluster. During the process, the agents provide updates on their stage and status.