Creating secondary networks with other CNI plugins

The specific configuration fields for secondary networks are described in the following sections.

Configuration for a bridge secondary network

The bridge CNI plugin JSON configuration object describes the configuration parameters for the Bridge CNI plugin. The following table details these parameters:

Field Type Description

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The mandatory, unique identifier assigned to this CNI network attachment definition. It is used by the container runtime to select the correct network configuration and serves as the key for persistent resource state management, such as IP address allocations.

type

string

The name of the CNI plugin to configure: bridge.

ipam

object

The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition.

bridge

string

Optional: Specify the name of the virtual bridge to use. If the bridge interface does not exist on the host, the bridge interface gets created. The default value is cni0.

ipMasq

boolean

Optional: Set to true to enable IP masquerading for traffic that leaves the virtual network. The source IP address for all traffic is rewritten to the bridge’s IP address. If the bridge does not have an IP address, this setting has no effect. The default value is false.

isGateway

boolean

Optional: Set to true to assign an IP address to the bridge. The default value is false.

isDefaultGateway

boolean

Optional: Set to true to configure the bridge as the default gateway for the virtual network. The default value is false. If isDefaultGateway is set to true, then isGateway is also set to true automatically.

forceAddress

boolean

Optional: Set to true to allow assignment of a previously assigned IP address to the virtual bridge. When set to false, if an IPv4 address or an IPv6 address from overlapping subsets is assigned to the virtual bridge, an error occurs. The default value is false.

hairpinMode

boolean

Optional: Set to true to allow the virtual bridge to send an Ethernet frame back through the virtual port it was received on. This mode is also known as reflective relay. The default value is false.

promiscMode

boolean

Optional: Set to true to enable promiscuous mode on the bridge. The default value is false.

vlan

string

Optional: Specify a virtual LAN (VLAN) tag as an integer value. By default, no VLAN tag is assigned.

preserveDefaultVlan

string

Optional: Indicates whether the default vlan must be preserved on the veth end connected to the bridge. Defaults to true.

vlanTrunk

list

Optional: Assign a VLAN trunk tag. The default value is none.

mtu

integer

Optional: Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

enabledad

boolean

Optional: Enables duplicate address detection for the container side veth. The default value is false.

macspoofchk

boolean

Optional: Enables mac spoof check, limiting the traffic originating from the container to the mac address of the interface. The default value is false.

Note

The VLAN parameter configures the VLAN tag on the host end of the veth and also enables the vlan_filtering feature on the bridge interface.

Note

To configure an uplink for an L2 network, you must allow the VLAN on the uplink interface by using the following command:

$  bridge vlan add vid VLAN_ID dev DEV

Bridge CNI plugin configuration example

The following example configures a secondary network named bridge-net:

{
  "cniVersion": "0.3.1",
  "name": "bridge-net",
  "type": "bridge",
  "isGateway": true,
  "vlan": 2,
  "ipam": {
    "type": "dhcp"
    }
}

Configuration for a Bond CNI secondary network

The Bond Container Network Interface (Bond CNI) enables the aggregation of multiple network interfaces into a single logical bonded interface within a container, which enhanches network redundancy and fault tolerance. Only SR-IOV Virtual Functions (VFs) are supported for bonding with this plugin.

The following table describes the configuration parameters for the Bond CNI plugin:

Table 1. Bond CNI plugin JSON configuration object
Field	Type	Description
`name`	`string`	The mandatory, unique identifier assigned to this CNI network attachment definition. It is used by the container runtime to select the correct network configuration and serves as the key for persistent resource state management, such as IP address allocations.
`cniVersion`	`string`	The CNI specification version.
`type`	`string`	Specifies the name of the CNI plugin to configure: `bond`.
`miimon`	`string`	Specifies the address resolution protocol (ARP) link monitoring frequency in milliseconds. This parameter defines how often the bond interface sends ARP requests to check the availability of its aggregated interfaces.
`mtu`	`integer`	Optional: Specifies the maximum transmission unit (MTU) of the bond. The default is `1500`.
`failOverMac`	`integer`	Optional: Specifies the `failOverMac` setting for the bond. Default is `0`.
`mode`	`string`	Specifies the bonding policy.
`xmitHashPolicy`	`string`	Specifies the transmit hash policy for load balancing across the aggregated interfaces. This parameter defaults to `layer2` and supports the following values: `layer2`, `layer2+3` and `layer3+4`.
`linksInContainer`	`boolean`	Optional: Specifies whether the network interfaces intended for bonding are expected to be created and available directly within the network namespace of the container when the bond starts. If `false` which is the default, the CNI plugin looks for these interfaces on the host system first before attempting to form the bond.
`links`	`object`	Specifies the interfaces to be bonded.
`ipam`	`object`	The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition.

Important

xmitHashPolicy is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Bond CNI plugin configuration example

The following example configures a secondary network named bond-net1:

{
 "type": "bond",
 "cniVersion": "0.3.1",
 "name": "bond-net1",
 "mode": "active-backup",
 "failOverMac": 1,
 "linksInContainer": true,
 "miimon": "100",
 "mtu": 1500,
 "links": [
       {"name": "net1"},
       {"name": "net2"}
   ],
  "ipam": {
        "type": "host-local",
        "subnet": "10.56.217.0/24",
        "routes": [{
        "dst": "0.0.0.0/0"
        }],
        "gateway": "10.56.217.1"
    }
}

The following example configures a secondary network named bond-tlb-net with the xmitHashPolicy feature enabled:

{
 "type": "bond",
 "cniVersion": "0.3.1",
 "name": "bond-tlb-net",
 "mode": "tlb",
 "xmitHashPolicy": "layer2+3",
 "failOverMac": 0,
 "linksInContainer": true,
 "miimon": "100",
 "mtu": 1500,
 "links": [
       {"name": "net1"},
       {"name": "net2"}
   ],
  "ipam": {
        "type": "host-local",
        "subnet": "10.57.218.0/24",
        "routes": [{
        "dst": "0.0.0.0/0"
        }],
        "gateway": "10.57.218.1"
    }
}

xmitHashPolicy: This parameter dictates how outgoing network traffic is distributed across the net1 and net2 active member interfaces within the bond. The hashing algorithm combines layer 2 information, specifically source and destination MAC addresses, with layer 3 information, which includes source and destination IP addresses.

Additional resources

Configuring a bond interface from two SR-IOV interfaces

Configuration for a host device secondary network

The host device CNI plugin JSON configuration object describes the configuration parameters for the host-device CNI plugin.

Note

Specify your network device by setting only one of the following parameters: device,hwaddr, kernelpath, or pciBusID.

The following table details the configuration parameters:

Field Type Description

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The mandatory, unique identifier assigned to this CNI network attachment definition. It is used by the container runtime to select the correct network configuration and serves as the key for persistent resource state management, such as IP address allocations.

type

string

The name of the CNI plugin to configure: host-device.

device

string

Optional: The name of the device, such as eth0.

hwaddr

string

Optional: The device hardware MAC address.

kernelpath

string

Optional: The Linux kernel device path, such as /sys/devices/pci0000:00/0000:00:1f.6.

pciBusID

string

Optional: The PCI address of the network device, such as 0000:00:1f.6.

host-device configuration example

The following example configures a secondary network named hostdev-net:

{
  "cniVersion": "0.3.1",
  "name": "hostdev-net",
  "type": "host-device",
  "device": "eth1"
}

Configuration for a dummy device additional network

The dummy CNI plugin functions like a loopback device. The plugin is a virtual interface, and you can use the plugin to route the packets to a designated IP address. Unlike a loopback device, the IP address is arbitrary and is not restricted to the 127.0.0.0/8 address range.

The dummy device CNI plugin JSON configuration object describes the configuration parameters for the dummy CNI plugin. The following table details these parameters:

Field Type Description

cniVersion

string

The CNI specification version. The required value is 0.3.1.

name

string

The mandatory, unique identifier assigned to this CNI network attachment definition. It is used by the container runtime to select the correct network configuration and serves as the key for persistent resource state management, such as IP address allocations.

type

string

The name of the CNI plugin that you want to configure. The required value is dummy.

ipam

object

The configuration object for the IPAM CNI plugin. The plugin manages the IP address assignment for the attachment definition.

dummy configuration example

The following example configures an additional network named hostdev-net:

{
  "cniVersion": "0.3.1",
  "name": "dummy-net",
  "type": "dummy",
  "ipam": {
      "type": "host-local",
      "subnet": "10.1.1.0/24"
  }
}

Configuration for a VLAN secondary network

The VLAN CNI plugin JSON configuration object describes the configuration parameters for the VLAN, vlan, CNI plugin. The following table details these parameters:

Field Type Description

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The mandatory, unique identifier assigned to this CNI network attachment definition. It is used by the container runtime to select the correct network configuration and serves as the key for persistent resource state management, such as IP address allocations.

type

string

The name of the CNI plugin to configure: vlan.

master

string

The Ethernet interface to associate with the network attachment. If a master is not specified, the interface for the default network route is used.

vlanId

integer

Set the ID of the vlan.

ipam

object

The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition.

mtu

integer

Optional: Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

dns

integer

Optional: DNS information to return. For example, a priority-ordered list of DNS nameservers.

linkInContainer

boolean

Optional: Specifies whether the master interface is in the container network namespace or the main network namespace. Set the value to true to request the use of a container namespace master interface.

Important

A NetworkAttachmentDefinition custom resource definition (CRD) with a vlan configuration can be used only on a single pod in a node because the CNI plugin cannot create multiple vlan subinterfaces with the same vlanId on the same master interface.

VLAN configuration example

The following example demonstrates a vlan configuration with a secondary network that is named vlan-net:

{
  "name": "vlan-net",
  "cniVersion": "0.3.1",
  "type": "vlan",
  "master": "eth0",
  "mtu": 1500,
  "vlanId": 5,
  "linkInContainer": false,
  "ipam": {
      "type": "host-local",
      "subnet": "10.1.1.0/24"
  },
  "dns": {
      "nameservers": [ "10.1.1.1", "8.8.8.8" ]
  }
}

ipam.type.host-local: Allocates IPv4 and IPv6 IP addresses from a specified set of address ranges. IPAM plugin stores the IP addresses locally on the host filesystem so that the addresses remain unique to the host.

Configuration for an IPVLAN secondary network

The IPVLAN CNI plugin JSON configuration object describes the configuration parameters for the IPVLAN, ipvlan, CNI plugin. The following table details these parameters:

Field Type Description

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The mandatory, unique identifier assigned to this CNI network attachment definition. It is used by the container runtime to select the correct network configuration and serves as the key for persistent resource state management, such as IP address allocations.

type

string

The name of the CNI plugin to configure: ipvlan.

ipam

object

The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition. This is required unless the plugin is chained.

mode

string

Optional: The operating mode for the virtual network. The value must be l2, l3, or l3s. The default value is l2.

master

string

Optional: The Ethernet interface to associate with the network attachment. If a master is not specified, the interface for the default network route is used.

mtu

integer

Optional: Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

linkInContainer

boolean

Optional: Specifies whether the master interface is in the container network namespace or the main network namespace. Set the value to true to request the use of a container namespace master interface.

Important

The ipvlan object does not allow virtual interfaces to communicate with the master interface. Therefore the container is not able to reach the host by using the ipvlan interface. Be sure that the container joins a network that provides connectivity to the host, such as a network supporting the Precision Time Protocol (PTP).
A single master interface cannot simultaneously be configured to use both macvlan and ipvlan.
For IP allocation schemes that cannot be interface agnostic, the ipvlan plugin can be chained with an earlier plugin that handles this logic. If the master is omitted, then the previous result must contain a single interface name for the ipvlan plugin to enslave. If ipam is omitted, then the previous result is used to configure the ipvlan interface.

IPVLAN CNI plugin configuration example

The following example configures a secondary network named ipvlan-net:

{
  "cniVersion": "0.3.1",
  "name": "ipvlan-net",
  "type": "ipvlan",
  "master": "eth1",
  "linkInContainer": false,
  "mode": "l3",
  "ipam": {
    "type": "static",
    "addresses": [
       {
         "address": "192.168.10.10/24"
       }
    ]
  }
}

Configuration for a MACVLAN secondary network

The MACVLAN CNI plugin JSON configuration object describes the configuration parameters for the MAC Virtual LAN (MACVLAN) Container Network Interface (CNI) plugin. The following table describes these parameters:

Field Type Description

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The mandatory, unique identifier assigned to this CNI network attachment definition. It is used by the container runtime to select the correct network configuration and serves as the key for persistent resource state management, such as IP address allocations.

type

string

The name of the CNI plugin to configure: macvlan.

ipam

object

The configuration object for the IPAM CNI plugin. The plugin manages IP address assignment for the attachment definition.

mode

string

Optional: Configures traffic visibility on the virtual network. Must be either bridge, passthru, private, or vepa. If a value is not provided, the default value is bridge.

master

string

Optional: The host network interface to associate with the newly created macvlan interface. If a value is not specified, then the default route interface is used.

mtu

integer

Optional: The maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

linkInContainer

boolean

Optional: Specifies whether the master interface is in the container network namespace or the main network namespace. Set the value to true to request the use of a container namespace master interface.

Note

If you specify the master key for the plugin configuration, use a different physical network interface than the one that is associated with your primary network plugin to avoid possible conflicts.

MACVLAN CNI plugin configuration example

The following example configures a secondary network named macvlan-net:

{
  "cniVersion": "0.3.1",
  "name": "macvlan-net",
  "type": "macvlan",
  "master": "eth1",
  "linkInContainer": false,
  "mode": "bridge",
  "ipam": {
    "type": "dhcp"
    }
}

Configuration for a TAP secondary network

The TAP CNI plugin JSON configuration object describes the configuration parameters for the TAP CNI plugin. The following table describes these parameters:

Field Type Description

cniVersion

string

The CNI specification version. The 0.3.1 value is required.

name

string

The mandatory, unique identifier assigned to this CNI network attachment definition. It is used by the container runtime to select the correct network configuration and serves as the key for persistent resource state management, such as IP address allocations.

type

string

The name of the CNI plugin to configure: tap.

mac

string

Optional: Request the specified MAC address for the interface.

mtu

integer

Optional: Set the maximum transmission unit (MTU) to the specified value. The default value is automatically set by the kernel.

selinuxcontext

string

Optional: The SELinux context to associate with the tap device.

Note

The value system_u:system_r:container_t:s0 is required for OpenShift Container Platform.

multiQueue

boolean

Optional: Set to true to enable multi-queue.

owner

integer

Optional: The user owning the tap device.

group

integer

Optional: The group owning the tap device.

bridge

string

Optional: Set the tap device as a port of an already existing bridge.

Tap configuration example

The following example configures a secondary network named mynet:

{
 "name": "mynet",
 "cniVersion": "0.3.1",
 "type": "tap",
 "mac": "00:11:22:33:44:55",
 "mtu": 1500,
 "selinuxcontext": "system_u:system_r:container_t:s0",
 "multiQueue": true,
 "owner": 0,
 "group": 0
 "bridge": "br1"
}

Setting SELinux boolean for the TAP CNI plugin

To create the tap device with the container_t SELinux context, enable the container_use_devices boolean on the host by using the Machine Config Operator (MCO).

Prerequisites

You have installed the OpenShift CLI (oc).

Procedure

Create a new YAML file with the following details:

Example setsebool-container-use-devices.yaml

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-worker-setsebool
spec:
  config:
    ignition:
      version: 3.2.0
    systemd:
      units:
      - enabled: true
        name: setsebool.service
        contents: |
          [Unit]
          Description=Set SELinux boolean for the TAP CNI plugin
          Before=kubelet.service

          [Service]
          Type=oneshot
          ExecStart=/usr/sbin/setsebool container_use_devices=on
          RemainAfterExit=true

          [Install]
          WantedBy=multi-user.target graphical.target

Create the new MachineConfig object by running the following command:
```
$ oc apply -f setsebool-container-use-devices.yaml
```
Note

Applying any changes to the MachineConfig object causes all affected nodes to gracefully reboot after the change is applied. The MCO might take some time to apply the update.

Verification

Verify that the change is applied by running the following command:

$ oc get machineconfigpools

NAME        CONFIG                                                UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master      rendered-master-e5e0c8e8be9194e7c5a882e047379cfa      True      False      False      3              3                   3                     0                      7d2h
worker      rendered-worker-d6c9ca107fba6cd76cdcbfcedcafa0f2      True      False      False      3              3                   3                     0                      7d

Note

All nodes should be in the Updated and Ready state.

Configuring routes using the route-override plugin on a secondary network

The Route override CNI plugin JSON configuration object describes the configuration parameters for the route-override CNI plugin. The following table details these parameters:

Field Type Description

type

string

The name of the CNI plugin to configure: route-override.

flushroutes

boolean

Optional: Set to true to flush any existing routes.

flushgateway

boolean

Optional: Set to true to flush the default route namely the gateway route.

delroutes

object

Optional: Specify the list of routes to delete from the container namespace.

addroutes

object

Optional: Specify the list of routes to add to the container namespace. Each route is a dictionary with dst and optional gw fields. If gw is omitted, the plugin uses the default gateway value.

skipcheck

boolean

Optional: Set this to true to skip the check command. By default, CNI plugins verify the network setup during the container lifecycle. When modifying routes dynamically with route-override, skipping this check ensures the final configuration reflects the updated routes.

Route-override plugin configuration example

The route-override CNI is a type of CNI that is designed to be used when chained with a parent CNI. The CNI type does not operate independently, but relies on the parent CNI to first create the network interface and assign IP addresses before the CNI type can modify the routing rules.

The following example configures a secondary network named mymacvlan. The parent CNI creates a network interface attached to eth1 and assigns an IP address in the 192.168.1.0/24 range by using host-local IPAM. The route-override CNI is then chained to the parent CNI and modifies the routing rules by flushing existing routes, deleting the route to 192.168.0.0/24, and adding a new route for 192.168.0.0/24 with a custom gateway.

{
    "cniVersion": "0.3.0",
    "name": "mymacvlan",
    "plugins": [
        {
            "type": "macvlan",
            "master": "eth1",
            "mode": "bridge",
            "ipam": {
                "type": "host-local",
                "subnet": "192.168.1.0/24"
            }
        },
        {
            "type": "route-override",
            "flushroutes": true,
            "delroutes": [
                {
                    "dst": "192.168.0.0/24"
                }
            ],
            "addroutes": [
                {
                    "dst": "192.168.0.0/24",
                    "gw": "10.1.254.254"
                }
            ]
        }
    ]
}

where:

"type": "macvlan": The parent CNI creates a network interface attached to eth1.
"type": "route-override": The chained route-override CNI modifies the routing rules.

Additional resources

Setting SELinux booleans