p0f - User contributions [en]

OCP4-IPI-libvirt

2024-02-05T10:31:16Z

Gregab: /* Creating Configuration Manually */ remove moaning about network interface matching, TBFO

= Introduction =

== What I Assume ==

* You know how OpenShift installation works and what the difference between IPI and UPI is.
* You know about the OpenShift Machine API and various underlying mechanisms.
* You understand the different types of network interfaces on Linux and different <code>libvirt</code> networks.
* You are familiar and comfortable with NetworkManager and the <code>nmcli</code> tool.
* You are familiar and comfortable with <code>libvirt</code> CLI and XML.
* You are familiar and comfortable with <code>qemu-img</code> tool.
* We are not talking about any firewall restrictions here - it is your responsibility to ensure traffic is not blocked.

== Outcomes ==

The installation described is for a fully managed IPI running OpenShift Container Platform v4.14, initially with three master and two worker nodes.

''At a later point, I will add a couple of steps needed to grow the cluster by one extra worker node.''

= OpenShift Container Platform IPI Installation Using Libvirt =

== Prerequisites ==

Hardware requirements for the cluster:

* 136 GiB RAM (32 GiB per control plane, 20 GiB per compute node), max overcommit ratio of 1.5 (make sure enough swap is available)
* 52 vCPUs (12 per control plane, 8 per compute node), max overcommit ratio of 1.3 (higher might work, but will slow down the installation horribly and may ultimately fail)
* one physical network interface that will be used for the public bridged network
* a physical or virtual network interface that will be used for the provisioning network bridge

Hardware requirements for the installation client (provisioner) machine:

* a minimum of 8 GiB RAM and 4 CPUs
* a network connection to both the public bridged network and the provisioning network

Due to the fact provisioner needs access to both networks, and the provisioning network in this guide is a virtual one, it might be best if you define the provisioner as a VM, with the same network interface settings as the control/compute nodes.

In the case you want to run the workloads spread across several hypervisor hosts, there are some extra steps, but nothing big. More on that in [[#Installation Spanning Multiple Hypervisors]] below.

Software artifacts needed on the provisioner host:

* <code>oc</code>, the command line client, of the corresponding version - download from https://mirror.openshift.com/pub/openshift-v4/clients/ocp/
* <code>libvirt-client</code> package is required for <code>openshift-baremetal-install</code> to be able to communicate to hypervisor(s)
* <code>ipmitool</code> or some other IPMI client
* a <code>pull-secret</code> file containing authentication credentials for OpenShift Container Platform registries - download from https://console.redhat.com/openshift/
* an SSH keypair that can be used for accessing OpenShift nodes

'''IMPORTANT''': The external IP addresses of cluster nodes must be assigned by your infrastructure DHCP server.

== Host Configuration ==

Beyond the logical requirement of having <code>libvirt</code> installed and started, here are the other configuration details for the hypervisor.

=== Network Settings ===

First thing you definitely need to make sure of, is that IP forwarding is enabled.

<pre>
$ sysctl net.ipv4.ip_forward
1
</pre>

Linux network settings need to be configured to have two '''Linux''' bridges, a public and a private provisioning one.

* public bridge, call it <code>bridge0</code>, needs to have the public network interface enslaved to it
* private bridge, call it <code>provbr0</code>, can be a virtual bridge since it is only needed for the provisioning network, which is supposed to be isolated and without any infrastructure services (such as DHCP, DNS, etc.)

It would be wonderful if the bridges could be OpenVSwitch ones, but unfortunately the Terraform bundled with <code>openshift-baremetal-install</code> currently does not include an OpenVSwitch provider, so there's goodbye to that.

As an example, here is my host configuration.

Public bridge:
<pre>
$ ip addr show bridge0
6: bridge0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff
inet 172.25.35.2/24 brd 172.25.35.255 scope global noprefixroute bridge0
valid_lft forever preferred_lft forever
inet6 fe80::4a21:bff:fe57:e06/64 scope link
valid_lft forever preferred_lft forever

$ ip addr show enp86s0
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master bridge0 state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff

$ bridge link | grep "master bridge0"
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master bridge0 state forwarding priority 32 cost 100
</pre>

Provisioning bridge:
<pre>
$ ip addr show provbr0
5: provbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ce:70:26:9c:88:a4 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.2/24 brd 10.1.1.255 scope global noprefixroute provbr0
valid_lft forever preferred_lft forever
inet6 fe80::cc70:26ff:fe9c:88a4/64 scope link
valid_lft forever preferred_lft forever

$ bridge link | grep "master provbr0"
</pre>

=== Installation Spanning Multiple Hypervisors ===

If you want to have your cluster spanning multiple hypervisors, make sure there is also a VXLAN connection between all the provisioning bridges.

You can do that by creating a <code>vxlan</code> type interface, which is a slave connection of type <code>bridge</code>, and the master is set to <code>provbr0</code>. Choose any unique VXLAN ID, and make sure it is the same on all interconnected hosts.

As an example, here is one VXLAN interface connecting hypervisor A to B.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1703164860
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.2
vxlan.remote: 172.25.35.3
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

And this is the corresponding VXLAN interface definition connecting host B to A.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1697549049
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.3
vxlan.remote: 172.25.35.2
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

In this case, <code>bridge link</code> will of course initially also show the <code>provbr0-vxlan10</code> interface as a slave, and will not show empty as above.

<pre>
$ bridge link | grep "master provbr0"
7: provbr0-vxlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master provbr0 state forwarding priority 32 cost 100
</pre>

=== <code>libvirt</code> Settings ===

Your <code>libvirt</code> will of course need to know about those network bridges in order to be able to attach VMs to them.

For that, you will need two network definitions, looking a bit like the following XML. Make sure they are autostart for least headache.

<pre>
$ sudo virsh net-dumpxml external
<network>
<name>external</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='bridge0'/>
</network>

$ sudo virsh net-dumpxml provisioning
<network>
<name>provisioning</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='provbr0'/>
</network>
</pre>

Additionally, you want to ensure that the storage pool is big enough, but that is not directly related to the subject at hand.

<pre>
$ sudo virsh pool-info default
Name: default
UUID: whatever
State: running
Persistent: yes
Autostart: yes
Capacity: 250.92 GiB
Allocation: 0 GiB
Available: 250.92 GiB
</pre>

=== VirtualBMC ===

The most important part of IPI facilitation is to be able to simulate a baseboard management controller for your VMs. <code>libvirt</code> obviously doesn't do this, but luckily there's a small bit of Python code that does, and it's called <code>virtualbmc</code>.

In most Python environments you can install it using <code>pip3</code>, just make sure <code>pip3</code> is up-to-date first.

<pre>
$ pip3 install --upgrade pip
...

$ pip3 install virtualbmc
...
</pre>

This gives you <code>/usr/local/bin/vbmcd</code> which you can control using the following systemd unit:

<pre>
[Unit]
Description=vbmcd
[Service]
Type=forking
ExecStart=/usr/local/bin/vbmcd
[Install]
WantedBy=multi-user.target
</pre>

Put the above content into <code>/etc/systemd/system/vbmcd.service</code>, reload systemd, and enable/start the service.

<pre>
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now vbmcd
</pre>

You now have the ability to associate a TCP port with a virtual machine defined on the hypervisor host, and have it simulate an IPMI BMC for that VM!

== Virtual Machine Configuration ==

=== VM Definitions ===

The virtual machines need to be configured with sufficient amount of compute resources, as per [[#Prerequisites]] above.

This section ties into the [[#Network Settings]] section above. You need two bridges on your hypervisor(s), <code>bridge0</code> and <code>provbr0</code>.

An example control plane node definition in <code>libvirt XML</code> would look like this:

<pre>
<domain type='kvm'>
<name>controlplane1</name>
<memory unit='GiB'>32</memory>
<currentMemory unit='GiB'>32</currentMemory>
<vcpu placement='static'>12</vcpu>
<os>
<type arch='x86_64' machine='q35'>hvm</type>
<boot dev='hd'/>
<boot dev='network'/>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model' check='partial'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/controlplane1-vda.qcow2'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'/>
<controller type='pci' index='0' model='pcie-root'/>
<interface type='bridge'>
<mac address='52:54:00:00:fb:11'/>
<source bridge='provbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:00:fa:11'/>
<source bridge='bridge0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<console type='pty'/>
<channel type='unix'>
<source mode='bind'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' autoport='yes' listen='0.0.0.0'/>
<video>
<model type='virtio'/>
</video>
<memballoon model='virtio'/>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
</rng>
</devices>
</domain>
</pre>

A couple of things to note:

* the first network interface is attached to <code>provbr0</code>, and its PCI address is ''lower'' (0x03), causing it to be the PXE default device
* the second network interface is attached to <code>bridge0</code>, and its PCI address is ''higher'' (0x09), making it the second interface (for external connections)
* boot order is set to hard disk first, network second, which means the host will only PXE boot if the disk image is unbootable
* the disk image needs to be 64GiB in size at the minimum, but you can make it larger and/or add more disk images if you intend to use the local storage operator

When configuring other nodes, simply remember to change the node name, disk image name, and MAC addresses to be unique. Adjust hardware resources accordingly for compute nodes.

=== IPMI BMC ===

What is important for OCP IPI to be able to perform the installation properly is to register each virtual machine with <code>vbmcd</code> and assign it with a port.

<pre>
$ vbmc add --port=6211 controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | down | :: | 6211 |
+-------------------+---------+---------+------+
$ vbmc start controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | running | :: | 6211 |
+-------------------+---------+---------+------+
$ sudo ss -aunp | grep 6211
UNCONN 0 0 *:6211 *:* users:(("vbmcd",pid=766290,fd=21))
</pre>

There is no need to run the <code>vbmc</code> client as root as it is the daemon that is running as root and can see all the VMs accessible through the <code>qemu:///system</code> URL.

Of course there are options. For any VM you add, you can specify a custom set of IPMI admin credentials (options <code>--username</code> and <code>--password</code>, they default to <code>admin</code> and <code>password</code>), a custom libvirt URL and credentials if necessary (options <code>--libvirt-uri</code>, <code>--libvirt-sasl-username</code>, and <code>--libvirt-sasl-password</code>), and a custom IP address to listen on (defaults to all addresses, use option <code>--address</code> to restrict it).

<pre>
$ vbmc show controlplane1
+-----------------------+-------------------+
| Property | Value |
+-----------------------+-------------------+
| active | True |
| address | :: |
| domain_name | controlplane1 |
| libvirt_sasl_password | *** |
| libvirt_sasl_username | None |
| libvirt_uri | qemu:///system |
| password | *** |
| port | 6211 |
| status | running |
| username | admin |
+-----------------------+-------------------+
</pre>

Once started (which just opens the port) you can test the BMC connection using <code>ipmitool</code> or similar.

<pre>
$ ipmitool -I lanplus -H localhost -p 6211 -U admin -P password chassis status
System Power : off
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : always-off
Last Power Event :
Chassis Intrusion : inactive
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false
</pre>

That's it! We're ready to install OCP!

== Installer Configuration ==

This is where it all comes together. You need to execute the steps in this section (and next) on the ''provisioner'' host, that is, a system that has access to both external and provisioning networks of the OpenShift cluster-to-be. The access need not be direct, it can be routed, but if you, as in our example, configured the provisioning network to be an isolated virtual bridge, you will be best off by creating an additional VM that is directly connected to both bridges.

=== Gathering the Bits Together ===

First step is to make sure the following artifacts are available:

* <code>oc</code>
* <code>pull-secret</code>
* an SSH public key for compute node access

The global cluster network settings that we will need to configure are:

* the parent DNS domain of the cluster (such as <code>example.com</code>)
* the name of the cluster (concatenated with DNS domain, for example <code>mycluster</code> will become <code>mycluster.example.com</code>)
* the provisioning network CIDR (in our case, it can be any IP address block not overlapping with the external network as the provisioning network is isolated)
* from the external network address space, a designated:
** API server VIP (<code>api.mycluster.example.com</code> should point to it)
** ingress load balancer VIP (any host within <code>apps.mycluster.example.com</code> should resolve to it, usually via a wildcard record)

Additionally, you will need, for each node its:

* node name
* provisioning interface MAC address
* IPMI BMC address and port
* IPMI BMC credentials
* the name of the provisioning interface as seen from within the VM (re [[#Virtual Machine Configuration]] above - since the PCI address of the interface is ''bus 0x0 slot 0x3'' it will be named <code>enp0s3</code>)

As already said, DHCP address assignment to external interfaces is not managed by the installer. It must be handled by your infrastructure.

=== Preparing the Software ===

First, make sure your <code>pull-secret</code> file is on the provisioner.

<pre>
$ ls -l pull-secret
-rw-r-----. 1 provisioner provisioner 2734 Oct 27 12:21 pull-secret
</pre>

Then after downloading <code>oc</code> (if you didn't do that already)...

<pre>
$ curl -OL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/openshift-client-linux.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 60.9M 100 60.9M 0 0 4287k 0 0:00:14 0:00:14 --:--:-- 5642k

$ tar xf openshift-client-linux.tar.gz oc

$ ./oc version
Client Version: 4.14.9
Kustomize Version: v5.0.1

$ sudo cp ./oc /usr/local/bin/oc
$ rm -f ./oc ./openshift-client-linux.tar.gz
</pre>

...and optionally generating a bash completion file...

<pre>
$ oc completion bash > oc.completion
$ sudo cp oc.completion /etc/bash_completion.d/oc
$ rm -f oc.completion
$ source /etc/bash_completion.d/oc
</pre>

...you can use it to extract <code>openshift-baremetal-install</code> from the release image you intend to use.

<pre>
$ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/release.txt | grep "Pull From:"
Pull From: quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ oc adm release extract --registry-config=pull-secret --command=openshift-baremetal-install --to=. \
quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ ./openshift-baremetal-install version
./openshift-baremetal-install 4.14.9
built from commit dfafb5ca972a6ed4677257aebfe4f284ac020830
release image quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44
release architecture amd64

$ sudo cp ./openshift-baremetal-install /usr/local/bin/
$ rm -f ./openshift-baremetal-install

$ openshift-baremetal-install completion bash > oinst.completion
$ sudo cp oinst.completion /etc/bash_completion.d/oinst
$ rm -f oinst.completion
</pre>

'''NOTE''': The extraction process can take up to 5 minutes, depending on your network speed and Quay.io responsiveness.

=== Creating Installer Configuration ===

Installer configuration file can be created interactively, using <code>openshift-baremetal-install</code>, or you can simply write it out with an editor.

==== Using Interactive Mode ====

'''IMPORTANT''': In interactive mode, machine network (the network segment the nodes have their IPs allocated from) is expected to be <code>10.0.0.0/16</code>. If your VIPs are not on that network, the installer will fail. If your machine network is not <code>10.0.0.0/16</code>, the only way to install is to create <code>install-config.yaml</code> manually.

'''IMPORTANT''': Interactive installer expects three control plane nodes and three workers. It will fail if you add less than three of each type of nodes to the cluster.

The interactive mode will ask you a series of questions and generate <code>install-config.yaml</code> at the end, but there are very few customisation options using this method.

Initially, you must select the SSH public key to be published on cluster nodes, and specify some general settings:

<pre>
$ openshift-baremetal-install --dir=cluster create install-config
? SSH Public Key /home/provisioner/.ssh/id_rsa.pub
? Platform baremetal
? Provisioning Network Managed
? Provisioning Network CIDR 172.22.0.0/24
? Provisioning bridge provbr0
? Provisioning Network Interface enp0s3
? External bridge bridge0
? Add a Host: [Use arrows to move, type to filter]
> control plane
worker
</pre>

As you can see above, after those have been answered, you may add any number of nodes, either control plane or worker.

For both type of nodes, the questions are the same:

<pre>
? Add a Host: control plane
? Name controlplane1
? BMC Address ipmi://192.168.1.1:6211
? BMC Username admin
? BMC Password ********
? Boot MAC Address 52:54:00:00:fb:11
? Add another host? (y/N) y
</pre>

After all the hosts have been added, there is a final series of cluster-related questions:

<pre>
? Add another host? No
? Base Domain example.com
? Cluster Name mycluster
? Pull Secret [? for help] ****************************...
</pre>

At this point, some checks are performed against DNS server to see if <code>api.mycluster.example.com</code> resolves, and if the IP is a part of the machine network. The same check is performed for the ingress VIP.

'''NOTE''': If your VIPs are not on the default machine network, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [platform.baremetal.apiVIPs: Invalid value: "172.25.3.10": IP expected to be in one of the machine networks: 10.0.0.0/16, platform.baremetal.ingressVIPs: Invalid value: "172.25.3.11": IP expected to be in one of the machine networks: 10.0.0.0/16]
</pre>

'''NOTE''': If you added fewer than three control plane nodes, or fewer than three workers, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [...]
</pre>

==== Creating Configuration Manually ====

The installer configuration file consists of a number of mandatory sections:

* cluster domain and name
* network settings (such as machine and cluster IP ranges)
* control plane and compute node settings
* infrastructure platform settings
* pull secret and ssh key

There are also some optional sections which are useful in special cases such as disconnected installation or special install modes.

'''NOTE''': There is an <code>explain</code> subcommand in the installer binary that explains the structure of the <code>installconfig</code> resource.

<pre>
$ openshift-baremetal-install explain installconfig
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
InstallConfig is the configuration for an OpenShift install.

FIELDS:
additionalTrustBundle <string>
AdditionalTrustBundle is a PEM-encoded X.509 certificate bundle that will be added to the nodes' trusted certificate store.
...

$ openshift-baremetal-install explain installconfig.platform.baremetal
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
BareMetal is the configuration used when installing on bare metal.

FIELDS:
apiVIP <string>
Format: ip
DeprecatedAPIVIP is the VIP to use for internal API communication Deprecated: Use APIVIPs

apiVIPs <[]string>
Format: ip
APIVIPs contains the VIP(s) to use for internal API communication. In dual stack clusters it contains an IPv4 and IPv6 address, otherwise only one VIP
...
</pre>

An example <code>install-config.yaml</code> for baremetal IPI looks like this:

<pre>
apiVersion: v1
baseDomain: example.com
metadata:
name: mycluster
networking:
networkType: OVNKubernetes
# These are the networks external IPs will be allocated from.
machineNetwork:
- cidr: 172.25.3.0/24
# This is the pod network.
clusterNetworks:
- cidr: 10.200.0.0/14
hostPrefix: 23
# Only one entry is supported.
serviceNetwork:
- 172.30.0.0/16
compute:
- name: worker
replicas: 2
controlPlane:
name: master
replicas: 3
platform:
baremetal: {}
platform:
baremetal:
apiVIPs:
- 172.25.3.10
ingressVIPs:
- 172.25.3.11
provisioningNetwork: Managed
provisioningNetworkCIDR: 10.1.1.0/24
provisioningDHCPRange: 10.1.1.200,10.1.1.210
# These settings are to configure the temporary bootstrap node as a VM
externalBridge: bridge0
externalMACAddress: '52:54:00:00:fa:0f'
bootstrapProvisioningIP: 10.1.1.9
provisioningBridge: provbr0
provisioningMACAddress: '52:54:00:00:fb:0f'
# This needs to be done to avoid nested virtualisation if provisioner is a VM.
libvirtURI: qemu+ssh://root@hypervisor.example.com/system
hosts:
- name: controlplane1
role: master
bmc:
address: ipmi://hypervisor.example.com:6211
disableCertificateVerification: true
username: admin
password: password
# This is the provisioning network interface.
bootMACAddress: 52:54:00:00:fb:11
bootMode: legacy
# We need this for proper targetting of root device (vda).
hardwareProfile: libvirt
- name: controlplane2
role: master
bmc:
address: ipmi://hypervisor.example.com:6212
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:12
bootMode: legacy
hardwareProfile: libvirt
- name: controlplane3
role: master
bmc:
address: ipmi://hypervisor.example.com:6213
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:13
bootMode: legacy
hardwareProfile: libvirt
- name: worker1
role: master
bmc:
address: ipmi://hypervisor.example.com:6221
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:21
bootMode: legacy
hardwareProfile: libvirt
- name: worker2
role: master
bmc:
address: ipmi://hypervisor.example.com:6222
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:22
bootMode: legacy
hardwareProfile: libvirt
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"...","email":"..."},"quay.io":{"auth":"...","email":"..."},...}}'
sshKey: "ssh-rsa ..."
</pre>

== Installation ==

With the above installation configuration file created, place a copy of it in a subdirectory, such as <code>./mycluster/</code> and run the installer.

<pre>
$ mkdir mycluster
$ cp install-config.yaml ./mycluster/

$ openshift-baremetal-install --dir=./mycluster/ --log-level=debug create cluster
DEBUG OpenShift Installer 4.14.9
DEBUG Built from commit dfafb5ca972a6ed4677257aebfe4f284ac020830
DEBUG Fetching Metadata...
DEBUG Loading Metadata...
...
DEBUG Loading Install Config...
DEBUG Loading Bootstrap Ignition Config...
...
INFO Consuming Install Config from target directory
...
INFO Obtaining RHCOS image file from 'https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.14-9.2/builds/414.92.202310210434-0/x86_64/rhcos-414.92.202310210434-0-qemu.x86_64.qcow2.gz?sha256=aab55f3ee088b88562f8fdcde5be78ace023e06fa01263e7cb9de2edc7131d6f'
...
INFO Creating infrastructure resources...
...
</pre>

When you see the above message, check the hypervisor for the presence of the temporary bootstrap VM.

<pre>
$ virsh list
Id Name State
---------------------------------------
4 provisioner running
5 mycluster-tmkmv-bootstrap running
</pre>

Once you see the VM running, you can inspect any containers on it by using the SSH key configured in install config to log into it and having a look around.

<pre>
$ ssh -i ~/.ssh/id_rsa core@bootstrap.mycluster.example.com
The authenticity of host 'bootstrap.mycluster.example.com (172.25.3.9)' can't be established.
...
Red Hat Enterprise Linux CoreOS 414.92.202310210434-0
Part of OpenShift 4.14, RHCOS is a Kubernetes native operating system
managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
...
[core@localhost ~]$ sudo -i

[root@localhost ~]# systemctl is-active crio
active

[root@localhost ~]# systemctl is-active bootkube
active

[root@localhost ~]# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
da3b3e74fc7f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:... /bin/rundnsmasq About a minute ago Up About a minute dnsmasq

[root@localhost ~]# crictl pods
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
af9df933d3b91 49 seconds ago Ready etcd-bootstrap-member-localhost.localdomain openshift-etcd 0 (default)

[root@localhost ~]# crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
9ad74f6433a33 753b0a16ba606f4c579690ed0035.. 6 seconds ago Running etcd 0 af9df933d3b91 etcd-bootstrap-member-localhost.localdomain
5c9371b25e646 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:... 6 seconds ago Running etcdctl 0 af9df933d3b91 etcd-bootstrap-member-localhost.localdomain
</pre>

Once you see pods like <code>kube-system</code>, <code>openshift-kube-apiserver</code>, and <code>openshift-cluster-version</code> show up in ready state, you can leave the shell and use the generated <code>kubeconfig</code> file to track the progress of the installation from the provisioner machine.

<pre>
$ export KUBECONFIG=$(pwd)/cluster/auth/kubeconfig

$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version False True 15m Unable to apply 4.14.9: an unknown error has occurred: MultipleErrors

$ oc get clusteroperators
$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication
baremetal
cloud-controller-manager
cloud-credential True False False 15m
cluster-autoscaler
config-operator
console
control-plane-machine-set
csi-snapshot-controller
dns
etcd
image-registry
ingress
insights
kube-apiserver
kube-controller-manager
kube-scheduler
kube-storage-version-migrator
machine-api
machine-approver
machine-config
marketplace
monitoring
network
node-tuning
openshift-apiserver
openshift-controller-manager
openshift-samples
operator-lifecycle-manager
operator-lifecycle-manager-catalog
operator-lifecycle-manager-packageserver
service-ca
storage
</pre>

'''NOTE''': It is normal for cluster version and various cluster operators to report transient error states as the progress of one impacts the progress of others. Eventually all these errors should go away.

== Post-Install Smoke Tests ==

= Conclusion =

OCP4-IPI-libvirt

2024-02-05T06:01:32Z

Gregab: /* Creating Configuration Manually */ refine netdev name complaint

= Introduction =

== What I Assume ==

* You know how OpenShift installation works and what the difference between IPI and UPI is.
* You know about the OpenShift Machine API and various underlying mechanisms.
* You understand the different types of network interfaces on Linux and different <code>libvirt</code> networks.
* You are familiar and comfortable with NetworkManager and the <code>nmcli</code> tool.
* You are familiar and comfortable with <code>libvirt</code> CLI and XML.
* You are familiar and comfortable with <code>qemu-img</code> tool.
* We are not talking about any firewall restrictions here - it is your responsibility to ensure traffic is not blocked.

== Outcomes ==

The installation described is for a fully managed IPI running OpenShift Container Platform v4.14, initially with three master and two worker nodes.

''At a later point, I will add a couple of steps needed to grow the cluster by one extra worker node.''

= OpenShift Container Platform IPI Installation Using Libvirt =

== Prerequisites ==

Hardware requirements for the cluster:

* 136 GiB RAM (32 GiB per control plane, 20 GiB per compute node), max overcommit ratio of 1.5 (make sure enough swap is available)
* 52 vCPUs (12 per control plane, 8 per compute node), max overcommit ratio of 1.3 (higher might work, but will slow down the installation horribly and may ultimately fail)
* one physical network interface that will be used for the public bridged network
* a physical or virtual network interface that will be used for the provisioning network bridge

Hardware requirements for the installation client (provisioner) machine:

* a minimum of 8 GiB RAM and 4 CPUs
* a network connection to both the public bridged network and the provisioning network

Due to the fact provisioner needs access to both networks, and the provisioning network in this guide is a virtual one, it might be best if you define the provisioner as a VM, with the same network interface settings as the control/compute nodes.

In the case you want to run the workloads spread across several hypervisor hosts, there are some extra steps, but nothing big. More on that in [[#Installation Spanning Multiple Hypervisors]] below.

Software artifacts needed on the provisioner host:

* <code>oc</code>, the command line client, of the corresponding version - download from https://mirror.openshift.com/pub/openshift-v4/clients/ocp/
* <code>libvirt-client</code> package is required for <code>openshift-baremetal-install</code> to be able to communicate to hypervisor(s)
* <code>ipmitool</code> or some other IPMI client
* a <code>pull-secret</code> file containing authentication credentials for OpenShift Container Platform registries - download from https://console.redhat.com/openshift/
* an SSH keypair that can be used for accessing OpenShift nodes

'''IMPORTANT''': The external IP addresses of cluster nodes must be assigned by your infrastructure DHCP server.

== Host Configuration ==

Beyond the logical requirement of having <code>libvirt</code> installed and started, here are the other configuration details for the hypervisor.

=== Network Settings ===

First thing you definitely need to make sure of, is that IP forwarding is enabled.

<pre>
$ sysctl net.ipv4.ip_forward
1
</pre>

Linux network settings need to be configured to have two '''Linux''' bridges, a public and a private provisioning one.

* public bridge, call it <code>bridge0</code>, needs to have the public network interface enslaved to it
* private bridge, call it <code>provbr0</code>, can be a virtual bridge since it is only needed for the provisioning network, which is supposed to be isolated and without any infrastructure services (such as DHCP, DNS, etc.)

It would be wonderful if the bridges could be OpenVSwitch ones, but unfortunately the Terraform bundled with <code>openshift-baremetal-install</code> currently does not include an OpenVSwitch provider, so there's goodbye to that.

As an example, here is my host configuration.

Public bridge:
<pre>
$ ip addr show bridge0
6: bridge0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff
inet 172.25.35.2/24 brd 172.25.35.255 scope global noprefixroute bridge0
valid_lft forever preferred_lft forever
inet6 fe80::4a21:bff:fe57:e06/64 scope link
valid_lft forever preferred_lft forever

$ ip addr show enp86s0
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master bridge0 state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff

$ bridge link | grep "master bridge0"
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master bridge0 state forwarding priority 32 cost 100
</pre>

Provisioning bridge:
<pre>
$ ip addr show provbr0
5: provbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ce:70:26:9c:88:a4 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.2/24 brd 10.1.1.255 scope global noprefixroute provbr0
valid_lft forever preferred_lft forever
inet6 fe80::cc70:26ff:fe9c:88a4/64 scope link
valid_lft forever preferred_lft forever

$ bridge link | grep "master provbr0"
</pre>

=== Installation Spanning Multiple Hypervisors ===

If you want to have your cluster spanning multiple hypervisors, make sure there is also a VXLAN connection between all the provisioning bridges.

You can do that by creating a <code>vxlan</code> type interface, which is a slave connection of type <code>bridge</code>, and the master is set to <code>provbr0</code>. Choose any unique VXLAN ID, and make sure it is the same on all interconnected hosts.

As an example, here is one VXLAN interface connecting hypervisor A to B.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1703164860
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.2
vxlan.remote: 172.25.35.3
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

And this is the corresponding VXLAN interface definition connecting host B to A.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1697549049
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.3
vxlan.remote: 172.25.35.2
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

In this case, <code>bridge link</code> will of course initially also show the <code>provbr0-vxlan10</code> interface as a slave, and will not show empty as above.

<pre>
$ bridge link | grep "master provbr0"
7: provbr0-vxlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master provbr0 state forwarding priority 32 cost 100
</pre>

=== <code>libvirt</code> Settings ===

Your <code>libvirt</code> will of course need to know about those network bridges in order to be able to attach VMs to them.

For that, you will need two network definitions, looking a bit like the following XML. Make sure they are autostart for least headache.

<pre>
$ sudo virsh net-dumpxml external
<network>
<name>external</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='bridge0'/>
</network>

$ sudo virsh net-dumpxml provisioning
<network>
<name>provisioning</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='provbr0'/>
</network>
</pre>

Additionally, you want to ensure that the storage pool is big enough, but that is not directly related to the subject at hand.

<pre>
$ sudo virsh pool-info default
Name: default
UUID: whatever
State: running
Persistent: yes
Autostart: yes
Capacity: 250.92 GiB
Allocation: 0 GiB
Available: 250.92 GiB
</pre>

=== VirtualBMC ===

The most important part of IPI facilitation is to be able to simulate a baseboard management controller for your VMs. <code>libvirt</code> obviously doesn't do this, but luckily there's a small bit of Python code that does, and it's called <code>virtualbmc</code>.

In most Python environments you can install it using <code>pip3</code>, just make sure <code>pip3</code> is up-to-date first.

<pre>
$ pip3 install --upgrade pip
...

$ pip3 install virtualbmc
...
</pre>

This gives you <code>/usr/local/bin/vbmcd</code> which you can control using the following systemd unit:

<pre>
[Unit]
Description=vbmcd
[Service]
Type=forking
ExecStart=/usr/local/bin/vbmcd
[Install]
WantedBy=multi-user.target
</pre>

Put the above content into <code>/etc/systemd/system/vbmcd.service</code>, reload systemd, and enable/start the service.

<pre>
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now vbmcd
</pre>

You now have the ability to associate a TCP port with a virtual machine defined on the hypervisor host, and have it simulate an IPMI BMC for that VM!

== Virtual Machine Configuration ==

=== VM Definitions ===

The virtual machines need to be configured with sufficient amount of compute resources, as per [[#Prerequisites]] above.

This section ties into the [[#Network Settings]] section above. You need two bridges on your hypervisor(s), <code>bridge0</code> and <code>provbr0</code>.

An example control plane node definition in <code>libvirt XML</code> would look like this:

<pre>
<domain type='kvm'>
<name>controlplane1</name>
<memory unit='GiB'>32</memory>
<currentMemory unit='GiB'>32</currentMemory>
<vcpu placement='static'>12</vcpu>
<os>
<type arch='x86_64' machine='q35'>hvm</type>
<boot dev='hd'/>
<boot dev='network'/>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model' check='partial'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/controlplane1-vda.qcow2'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'/>
<controller type='pci' index='0' model='pcie-root'/>
<interface type='bridge'>
<mac address='52:54:00:00:fb:11'/>
<source bridge='provbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:00:fa:11'/>
<source bridge='bridge0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<console type='pty'/>
<channel type='unix'>
<source mode='bind'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' autoport='yes' listen='0.0.0.0'/>
<video>
<model type='virtio'/>
</video>
<memballoon model='virtio'/>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
</rng>
</devices>
</domain>
</pre>

A couple of things to note:

* the first network interface is attached to <code>provbr0</code>, and its PCI address is ''lower'' (0x03), causing it to be the PXE default device
* the second network interface is attached to <code>bridge0</code>, and its PCI address is ''higher'' (0x09), making it the second interface (for external connections)
* boot order is set to hard disk first, network second, which means the host will only PXE boot if the disk image is unbootable
* the disk image needs to be 64GiB in size at the minimum, but you can make it larger and/or add more disk images if you intend to use the local storage operator

When configuring other nodes, simply remember to change the node name, disk image name, and MAC addresses to be unique. Adjust hardware resources accordingly for compute nodes.

=== IPMI BMC ===

What is important for OCP IPI to be able to perform the installation properly is to register each virtual machine with <code>vbmcd</code> and assign it with a port.

<pre>
$ vbmc add --port=6211 controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | down | :: | 6211 |
+-------------------+---------+---------+------+
$ vbmc start controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | running | :: | 6211 |
+-------------------+---------+---------+------+
$ sudo ss -aunp | grep 6211
UNCONN 0 0 *:6211 *:* users:(("vbmcd",pid=766290,fd=21))
</pre>

There is no need to run the <code>vbmc</code> client as root as it is the daemon that is running as root and can see all the VMs accessible through the <code>qemu:///system</code> URL.

Of course there are options. For any VM you add, you can specify a custom set of IPMI admin credentials (options <code>--username</code> and <code>--password</code>, they default to <code>admin</code> and <code>password</code>), a custom libvirt URL and credentials if necessary (options <code>--libvirt-uri</code>, <code>--libvirt-sasl-username</code>, and <code>--libvirt-sasl-password</code>), and a custom IP address to listen on (defaults to all addresses, use option <code>--address</code> to restrict it).

<pre>
$ vbmc show controlplane1
+-----------------------+-------------------+
| Property | Value |
+-----------------------+-------------------+
| active | True |
| address | :: |
| domain_name | controlplane1 |
| libvirt_sasl_password | *** |
| libvirt_sasl_username | None |
| libvirt_uri | qemu:///system |
| password | *** |
| port | 6211 |
| status | running |
| username | admin |
+-----------------------+-------------------+
</pre>

Once started (which just opens the port) you can test the BMC connection using <code>ipmitool</code> or similar.

<pre>
$ ipmitool -I lanplus -H localhost -p 6211 -U admin -P password chassis status
System Power : off
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : always-off
Last Power Event :
Chassis Intrusion : inactive
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false
</pre>

That's it! We're ready to install OCP!

== Installer Configuration ==

This is where it all comes together. You need to execute the steps in this section (and next) on the ''provisioner'' host, that is, a system that has access to both external and provisioning networks of the OpenShift cluster-to-be. The access need not be direct, it can be routed, but if you, as in our example, configured the provisioning network to be an isolated virtual bridge, you will be best off by creating an additional VM that is directly connected to both bridges.

=== Gathering the Bits Together ===

First step is to make sure the following artifacts are available:

* <code>oc</code>
* <code>pull-secret</code>
* an SSH public key for compute node access

The global cluster network settings that we will need to configure are:

* the parent DNS domain of the cluster (such as <code>example.com</code>)
* the name of the cluster (concatenated with DNS domain, for example <code>mycluster</code> will become <code>mycluster.example.com</code>)
* the provisioning network CIDR (in our case, it can be any IP address block not overlapping with the external network as the provisioning network is isolated)
* from the external network address space, a designated:
** API server VIP (<code>api.mycluster.example.com</code> should point to it)
** ingress load balancer VIP (any host within <code>apps.mycluster.example.com</code> should resolve to it, usually via a wildcard record)

Additionally, you will need, for each node its:

* node name
* provisioning interface MAC address
* IPMI BMC address and port
* IPMI BMC credentials
* the name of the provisioning interface as seen from within the VM (re [[#Virtual Machine Configuration]] above - since the PCI address of the interface is ''bus 0x0 slot 0x3'' it will be named <code>enp0s3</code>)

As already said, DHCP address assignment to external interfaces is not managed by the installer. It must be handled by your infrastructure.

=== Preparing the Software ===

First, make sure your <code>pull-secret</code> file is on the provisioner.

<pre>
$ ls -l pull-secret
-rw-r-----. 1 provisioner provisioner 2734 Oct 27 12:21 pull-secret
</pre>

Then after downloading <code>oc</code> (if you didn't do that already)...

<pre>
$ curl -OL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/openshift-client-linux.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 60.9M 100 60.9M 0 0 4287k 0 0:00:14 0:00:14 --:--:-- 5642k

$ tar xf openshift-client-linux.tar.gz oc

$ ./oc version
Client Version: 4.14.9
Kustomize Version: v5.0.1

$ sudo cp ./oc /usr/local/bin/oc
$ rm -f ./oc ./openshift-client-linux.tar.gz
</pre>

...and optionally generating a bash completion file...

<pre>
$ oc completion bash > oc.completion
$ sudo cp oc.completion /etc/bash_completion.d/oc
$ rm -f oc.completion
$ source /etc/bash_completion.d/oc
</pre>

...you can use it to extract <code>openshift-baremetal-install</code> from the release image you intend to use.

<pre>
$ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/release.txt | grep "Pull From:"
Pull From: quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ oc adm release extract --registry-config=pull-secret --command=openshift-baremetal-install --to=. \
quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ ./openshift-baremetal-install version
./openshift-baremetal-install 4.14.9
built from commit dfafb5ca972a6ed4677257aebfe4f284ac020830
release image quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44
release architecture amd64

$ sudo cp ./openshift-baremetal-install /usr/local/bin/
$ rm -f ./openshift-baremetal-install

$ openshift-baremetal-install completion bash > oinst.completion
$ sudo cp oinst.completion /etc/bash_completion.d/oinst
$ rm -f oinst.completion
</pre>

'''NOTE''': The extraction process can take up to 5 minutes, depending on your network speed and Quay.io responsiveness.

=== Creating Installer Configuration ===

Installer configuration file can be created interactively, using <code>openshift-baremetal-install</code>, or you can simply write it out with an editor.

==== Using Interactive Mode ====

'''IMPORTANT''': In interactive mode, machine network (the network segment the nodes have their IPs allocated from) is expected to be <code>10.0.0.0/16</code>. If your VIPs are not on that network, the installer will fail. If your machine network is not <code>10.0.0.0/16</code>, the only way to install is to create <code>install-config.yaml</code> manually.

'''IMPORTANT''': Interactive installer expects three control plane nodes and three workers. It will fail if you add less than three of each type of nodes to the cluster.

The interactive mode will ask you a series of questions and generate <code>install-config.yaml</code> at the end, but there are very few customisation options using this method.

Initially, you must select the SSH public key to be published on cluster nodes, and specify some general settings:

<pre>
$ openshift-baremetal-install --dir=cluster create install-config
? SSH Public Key /home/provisioner/.ssh/id_rsa.pub
? Platform baremetal
? Provisioning Network Managed
? Provisioning Network CIDR 172.22.0.0/24
? Provisioning bridge provbr0
? Provisioning Network Interface enp0s3
? External bridge bridge0
? Add a Host: [Use arrows to move, type to filter]
> control plane
worker
</pre>

As you can see above, after those have been answered, you may add any number of nodes, either control plane or worker.

For both type of nodes, the questions are the same:

<pre>
? Add a Host: control plane
? Name controlplane1
? BMC Address ipmi://192.168.1.1:6211
? BMC Username admin
? BMC Password ********
? Boot MAC Address 52:54:00:00:fb:11
? Add another host? (y/N) y
</pre>

After all the hosts have been added, there is a final series of cluster-related questions:

<pre>
? Add another host? No
? Base Domain example.com
? Cluster Name mycluster
? Pull Secret [? for help] ****************************...
</pre>

At this point, some checks are performed against DNS server to see if <code>api.mycluster.example.com</code> resolves, and if the IP is a part of the machine network. The same check is performed for the ingress VIP.

'''NOTE''': If your VIPs are not on the default machine network, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [platform.baremetal.apiVIPs: Invalid value: "172.25.3.10": IP expected to be in one of the machine networks: 10.0.0.0/16, platform.baremetal.ingressVIPs: Invalid value: "172.25.3.11": IP expected to be in one of the machine networks: 10.0.0.0/16]
</pre>

'''NOTE''': If you added fewer than three control plane nodes, or fewer than three workers, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [...]
</pre>

==== Creating Configuration Manually ====

The installer configuration file consists of a number of mandatory sections:

* cluster domain and name
* network settings (such as machine and cluster IP ranges)
* control plane and compute node settings
* infrastructure platform settings
* pull secret and ssh key

There are also some optional sections which are useful in special cases such as disconnected installation or special install modes.

'''NOTE''': There is an <code>explain</code> subcommand in the installer binary that explains the structure of the <code>installconfig</code> resource.

<pre>
$ openshift-baremetal-install explain installconfig
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
InstallConfig is the configuration for an OpenShift install.

FIELDS:
additionalTrustBundle <string>
AdditionalTrustBundle is a PEM-encoded X.509 certificate bundle that will be added to the nodes' trusted certificate store.
...

$ openshift-baremetal-install explain installconfig.platform.baremetal
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
BareMetal is the configuration used when installing on bare metal.

FIELDS:
apiVIP <string>
Format: ip
DeprecatedAPIVIP is the VIP to use for internal API communication Deprecated: Use APIVIPs

apiVIPs <[]string>
Format: ip
APIVIPs contains the VIP(s) to use for internal API communication. In dual stack clusters it contains an IPv4 and IPv6 address, otherwise only one VIP
...
</pre>

An example <code>install-config.yaml</code> for baremetal IPI looks like this:

<pre>
apiVersion: v1
baseDomain: example.com
metadata:
name: mycluster
networking:
networkType: OVNKubernetes
# These are the networks external IPs will be allocated from.
machineNetwork:
- cidr: 172.25.3.0/24
# This is the pod network.
clusterNetworks:
- cidr: 10.200.0.0/14
hostPrefix: 23
# Only one entry is supported.
serviceNetwork:
- 172.30.0.0/16
compute:
- name: worker
replicas: 2
controlPlane:
name: master
replicas: 3
platform:
baremetal: {}
platform:
baremetal:
apiVIPs:
- 172.25.3.10
ingressVIPs:
- 172.25.3.11
provisioningNetwork: Managed
provisioningNetworkCIDR: 10.1.1.0/24
provisioningDHCPRange: 10.1.1.200,10.1.1.210
# These settings are to configure the temporary bootstrap node as a VM
externalBridge: bridge0
externalMACAddress: '52:54:00:00:fa:0f'
bootstrapProvisioningIP: 10.1.1.9
provisioningBridge: provbr0
provisioningNetworkInterface: ens4
provisioningMACAddress: '52:54:00:00:fb:0f'
# This needs to be done to avoid nested virtualisation if provisioner is a VM.
libvirtURI: qemu+ssh://root@hypervisor.example.com/system
hosts:
- name: controlplane1
role: master
bmc:
address: ipmi://hypervisor.example.com:6211
disableCertificateVerification: true
username: admin
password: password
# This is the provisioning network interface.
bootMACAddress: 52:54:00:00:fb:11
bootMode: legacy
# We need this for proper targetting of root device (vda).
hardwareProfile: libvirt
- name: controlplane2
role: master
bmc:
address: ipmi://hypervisor.example.com:6212
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:12
bootMode: legacy
hardwareProfile: libvirt
- name: controlplane3
role: master
bmc:
address: ipmi://hypervisor.example.com:6213
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:13
bootMode: legacy
hardwareProfile: libvirt
- name: worker1
role: master
bmc:
address: ipmi://hypervisor.example.com:6221
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:21
bootMode: legacy
hardwareProfile: libvirt
- name: worker2
role: master
bmc:
address: ipmi://hypervisor.example.com:6222
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:22
bootMode: legacy
hardwareProfile: libvirt
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"...","email":"..."},"quay.io":{"auth":"...","email":"..."},...}}'
sshKey: "ssh-rsa ..."
</pre>

'''''WARNING'''''
<blockquote>
Somewhere between RHCOS 4.12 and 4.14 the network devices stopped using the PCI naming scheme (in VMs only?). Consequently, <code>NetworkManager-wait-online</code> will fail because it's looking for a device (<code>enp0s3</code>) that isn't there (it's now called <code>ens4</code>, go figure). Sure, it's got an ''altname'' of <code>enp0s4</code> but that is incorrect and <code>NetworkManager</code> don't care.

<pre>
$ ip ad sh ens4
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 52:54:00:00:fb:13 brd ff:ff:ff:ff:ff:ff
altname enp0s4
inet 10.1.1.13/24 brd 10.1.1.255 scope global noprefixroute ens4
valid_lft forever preferred_lft forever
inet6 fe80::3abb:b5:9c4f:c94d/64 scope link noprefixroute
valid_lft forever preferred_lft forever
</pre>

In turn, that will break most of the rest of the installation. Go figure. Just configure the interface to be <code>ens4</code> in install config and move on.
</blockquote>

== Installation ==

With the above installation configuration file created, place a copy of it in a subdirectory, such as <code>./mycluster/</code> and run the installer.

<pre>
$ mkdir mycluster
$ cp install-config.yaml ./mycluster/

$ openshift-baremetal-install --dir=./mycluster/ --log-level=debug create cluster
DEBUG OpenShift Installer 4.14.9
DEBUG Built from commit dfafb5ca972a6ed4677257aebfe4f284ac020830
DEBUG Fetching Metadata...
DEBUG Loading Metadata...
...
DEBUG Loading Install Config...
DEBUG Loading Bootstrap Ignition Config...
...
INFO Consuming Install Config from target directory
...
INFO Obtaining RHCOS image file from 'https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.14-9.2/builds/414.92.202310210434-0/x86_64/rhcos-414.92.202310210434-0-qemu.x86_64.qcow2.gz?sha256=aab55f3ee088b88562f8fdcde5be78ace023e06fa01263e7cb9de2edc7131d6f'
...
INFO Creating infrastructure resources...
...
</pre>

When you see the above message, check the hypervisor for the presence of the temporary bootstrap VM.

<pre>
$ virsh list
Id Name State
---------------------------------------
4 provisioner running
5 mycluster-tmkmv-bootstrap running
</pre>

Once you see the VM running, you can inspect any containers on it by using the SSH key configured in install config to log into it and having a look around.

<pre>
$ ssh -i ~/.ssh/id_rsa core@bootstrap.mycluster.example.com
The authenticity of host 'bootstrap.mycluster.example.com (172.25.3.9)' can't be established.
...
Red Hat Enterprise Linux CoreOS 414.92.202310210434-0
Part of OpenShift 4.14, RHCOS is a Kubernetes native operating system
managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
...
[core@localhost ~]$ sudo -i

[root@localhost ~]# systemctl is-active crio
active

[root@localhost ~]# systemctl is-active bootkube
active

[root@localhost ~]# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
da3b3e74fc7f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:... /bin/rundnsmasq About a minute ago Up About a minute dnsmasq

[root@localhost ~]# crictl pods
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
af9df933d3b91 49 seconds ago Ready etcd-bootstrap-member-localhost.localdomain openshift-etcd 0 (default)

[root@localhost ~]# crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
9ad74f6433a33 753b0a16ba606f4c579690ed0035.. 6 seconds ago Running etcd 0 af9df933d3b91 etcd-bootstrap-member-localhost.localdomain
5c9371b25e646 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:... 6 seconds ago Running etcdctl 0 af9df933d3b91 etcd-bootstrap-member-localhost.localdomain
</pre>

Once you see pods like <code>kube-system</code>, <code>openshift-kube-apiserver</code>, and <code>openshift-cluster-version</code> show up in ready state, you can leave the shell and use the generated <code>kubeconfig</code> file to track the progress of the installation from the provisioner machine.

<pre>
$ export KUBECONFIG=$(pwd)/cluster/auth/kubeconfig

$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version False True 15m Unable to apply 4.14.9: an unknown error has occurred: MultipleErrors

$ oc get clusteroperators
$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication
baremetal
cloud-controller-manager
cloud-credential True False False 15m
cluster-autoscaler
config-operator
console
control-plane-machine-set
csi-snapshot-controller
dns
etcd
image-registry
ingress
insights
kube-apiserver
kube-controller-manager
kube-scheduler
kube-storage-version-migrator
machine-api
machine-approver
machine-config
marketplace
monitoring
network
node-tuning
openshift-apiserver
openshift-controller-manager
openshift-samples
operator-lifecycle-manager
operator-lifecycle-manager-catalog
operator-lifecycle-manager-packageserver
service-ca
storage
</pre>

'''NOTE''': It is normal for cluster version and various cluster operators to report transient error states as the progress of one impacts the progress of others. Eventually all these errors should go away.

== Post-Install Smoke Tests ==

= Conclusion =

OCP4-IPI-libvirt

2024-02-02T20:49:30Z

Gregab: /* Creating Configuration Manually */ ens3 vs enp0s3

= Introduction =

== What I Assume ==

* You know how OpenShift installation works and what the difference between IPI and UPI is.
* You know about the OpenShift Machine API and various underlying mechanisms.
* You understand the different types of network interfaces on Linux and different <code>libvirt</code> networks.
* You are familiar and comfortable with NetworkManager and the <code>nmcli</code> tool.
* You are familiar and comfortable with <code>libvirt</code> CLI and XML.
* You are familiar and comfortable with <code>qemu-img</code> tool.
* We are not talking about any firewall restrictions here - it is your responsibility to ensure traffic is not blocked.

== Outcomes ==

The installation described is for a fully managed IPI running OpenShift Container Platform v4.14, initially with three master and two worker nodes.

''At a later point, I will add a couple of steps needed to grow the cluster by one extra worker node.''

= OpenShift Container Platform IPI Installation Using Libvirt =

== Prerequisites ==

Hardware requirements for the cluster:

* 136 GiB RAM (32 GiB per control plane, 20 GiB per compute node), max overcommit ratio of 1.5 (make sure enough swap is available)
* 52 vCPUs (12 per control plane, 8 per compute node), max overcommit ratio of 1.3 (higher might work, but will slow down the installation horribly and may ultimately fail)
* one physical network interface that will be used for the public bridged network
* a physical or virtual network interface that will be used for the provisioning network bridge

Hardware requirements for the installation client (provisioner) machine:

* a minimum of 8 GiB RAM and 4 CPUs
* a network connection to both the public bridged network and the provisioning network

Due to the fact provisioner needs access to both networks, and the provisioning network in this guide is a virtual one, it might be best if you define the provisioner as a VM, with the same network interface settings as the control/compute nodes.

In the case you want to run the workloads spread across several hypervisor hosts, there are some extra steps, but nothing big. More on that in [[#Installation Spanning Multiple Hypervisors]] below.

Software artifacts needed on the provisioner host:

* <code>oc</code>, the command line client, of the corresponding version - download from https://mirror.openshift.com/pub/openshift-v4/clients/ocp/
* <code>libvirt-client</code> package is required for <code>openshift-baremetal-install</code> to be able to communicate to hypervisor(s)
* <code>ipmitool</code> or some other IPMI client
* a <code>pull-secret</code> file containing authentication credentials for OpenShift Container Platform registries - download from https://console.redhat.com/openshift/
* an SSH keypair that can be used for accessing OpenShift nodes

'''IMPORTANT''': The external IP addresses of cluster nodes must be assigned by your infrastructure DHCP server.

== Host Configuration ==

Beyond the logical requirement of having <code>libvirt</code> installed and started, here are the other configuration details for the hypervisor.

=== Network Settings ===

First thing you definitely need to make sure of, is that IP forwarding is enabled.

<pre>
$ sysctl net.ipv4.ip_forward
1
</pre>

Linux network settings need to be configured to have two '''Linux''' bridges, a public and a private provisioning one.

* public bridge, call it <code>bridge0</code>, needs to have the public network interface enslaved to it
* private bridge, call it <code>provbr0</code>, can be a virtual bridge since it is only needed for the provisioning network, which is supposed to be isolated and without any infrastructure services (such as DHCP, DNS, etc.)

It would be wonderful if the bridges could be OpenVSwitch ones, but unfortunately the Terraform bundled with <code>openshift-baremetal-install</code> currently does not include an OpenVSwitch provider, so there's goodbye to that.

As an example, here is my host configuration.

Public bridge:
<pre>
$ ip addr show bridge0
6: bridge0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff
inet 172.25.35.2/24 brd 172.25.35.255 scope global noprefixroute bridge0
valid_lft forever preferred_lft forever
inet6 fe80::4a21:bff:fe57:e06/64 scope link
valid_lft forever preferred_lft forever

$ ip addr show enp86s0
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master bridge0 state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff

$ bridge link | grep "master bridge0"
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master bridge0 state forwarding priority 32 cost 100
</pre>

Provisioning bridge:
<pre>
$ ip addr show provbr0
5: provbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ce:70:26:9c:88:a4 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.2/24 brd 10.1.1.255 scope global noprefixroute provbr0
valid_lft forever preferred_lft forever
inet6 fe80::cc70:26ff:fe9c:88a4/64 scope link
valid_lft forever preferred_lft forever

$ bridge link | grep "master provbr0"
</pre>

=== Installation Spanning Multiple Hypervisors ===

If you want to have your cluster spanning multiple hypervisors, make sure there is also a VXLAN connection between all the provisioning bridges.

You can do that by creating a <code>vxlan</code> type interface, which is a slave connection of type <code>bridge</code>, and the master is set to <code>provbr0</code>. Choose any unique VXLAN ID, and make sure it is the same on all interconnected hosts.

As an example, here is one VXLAN interface connecting hypervisor A to B.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1703164860
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.2
vxlan.remote: 172.25.35.3
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

And this is the corresponding VXLAN interface definition connecting host B to A.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1697549049
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.3
vxlan.remote: 172.25.35.2
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

In this case, <code>bridge link</code> will of course initially also show the <code>provbr0-vxlan10</code> interface as a slave, and will not show empty as above.

<pre>
$ bridge link | grep "master provbr0"
7: provbr0-vxlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master provbr0 state forwarding priority 32 cost 100
</pre>

=== <code>libvirt</code> Settings ===

Your <code>libvirt</code> will of course need to know about those network bridges in order to be able to attach VMs to them.

For that, you will need two network definitions, looking a bit like the following XML. Make sure they are autostart for least headache.

<pre>
$ sudo virsh net-dumpxml external
<network>
<name>external</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='bridge0'/>
</network>

$ sudo virsh net-dumpxml provisioning
<network>
<name>provisioning</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='provbr0'/>
</network>
</pre>

Additionally, you want to ensure that the storage pool is big enough, but that is not directly related to the subject at hand.

<pre>
$ sudo virsh pool-info default
Name: default
UUID: whatever
State: running
Persistent: yes
Autostart: yes
Capacity: 250.92 GiB
Allocation: 0 GiB
Available: 250.92 GiB
</pre>

=== VirtualBMC ===

The most important part of IPI facilitation is to be able to simulate a baseboard management controller for your VMs. <code>libvirt</code> obviously doesn't do this, but luckily there's a small bit of Python code that does, and it's called <code>virtualbmc</code>.

In most Python environments you can install it using <code>pip3</code>, just make sure <code>pip3</code> is up-to-date first.

<pre>
$ pip3 install --upgrade pip
...

$ pip3 install virtualbmc
...
</pre>

This gives you <code>/usr/local/bin/vbmcd</code> which you can control using the following systemd unit:

<pre>
[Unit]
Description=vbmcd
[Service]
Type=forking
ExecStart=/usr/local/bin/vbmcd
[Install]
WantedBy=multi-user.target
</pre>

Put the above content into <code>/etc/systemd/system/vbmcd.service</code>, reload systemd, and enable/start the service.

<pre>
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now vbmcd
</pre>

You now have the ability to associate a TCP port with a virtual machine defined on the hypervisor host, and have it simulate an IPMI BMC for that VM!

== Virtual Machine Configuration ==

=== VM Definitions ===

The virtual machines need to be configured with sufficient amount of compute resources, as per [[#Prerequisites]] above.

This section ties into the [[#Network Settings]] section above. You need two bridges on your hypervisor(s), <code>bridge0</code> and <code>provbr0</code>.

An example control plane node definition in <code>libvirt XML</code> would look like this:

<pre>
<domain type='kvm'>
<name>controlplane1</name>
<memory unit='GiB'>32</memory>
<currentMemory unit='GiB'>32</currentMemory>
<vcpu placement='static'>12</vcpu>
<os>
<type arch='x86_64' machine='q35'>hvm</type>
<boot dev='hd'/>
<boot dev='network'/>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model' check='partial'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/controlplane1-vda.qcow2'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'/>
<controller type='pci' index='0' model='pcie-root'/>
<interface type='bridge'>
<mac address='52:54:00:00:fb:11'/>
<source bridge='provbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:00:fa:11'/>
<source bridge='bridge0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<console type='pty'/>
<channel type='unix'>
<source mode='bind'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' autoport='yes' listen='0.0.0.0'/>
<video>
<model type='virtio'/>
</video>
<memballoon model='virtio'/>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
</rng>
</devices>
</domain>
</pre>

A couple of things to note:

* the first network interface is attached to <code>provbr0</code>, and its PCI address is ''lower'' (0x03), causing it to be the PXE default device
* the second network interface is attached to <code>bridge0</code>, and its PCI address is ''higher'' (0x09), making it the second interface (for external connections)
* boot order is set to hard disk first, network second, which means the host will only PXE boot if the disk image is unbootable
* the disk image needs to be 64GiB in size at the minimum, but you can make it larger and/or add more disk images if you intend to use the local storage operator

When configuring other nodes, simply remember to change the node name, disk image name, and MAC addresses to be unique. Adjust hardware resources accordingly for compute nodes.

=== IPMI BMC ===

What is important for OCP IPI to be able to perform the installation properly is to register each virtual machine with <code>vbmcd</code> and assign it with a port.

<pre>
$ vbmc add --port=6211 controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | down | :: | 6211 |
+-------------------+---------+---------+------+
$ vbmc start controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | running | :: | 6211 |
+-------------------+---------+---------+------+
$ sudo ss -aunp | grep 6211
UNCONN 0 0 *:6211 *:* users:(("vbmcd",pid=766290,fd=21))
</pre>

There is no need to run the <code>vbmc</code> client as root as it is the daemon that is running as root and can see all the VMs accessible through the <code>qemu:///system</code> URL.

Of course there are options. For any VM you add, you can specify a custom set of IPMI admin credentials (options <code>--username</code> and <code>--password</code>, they default to <code>admin</code> and <code>password</code>), a custom libvirt URL and credentials if necessary (options <code>--libvirt-uri</code>, <code>--libvirt-sasl-username</code>, and <code>--libvirt-sasl-password</code>), and a custom IP address to listen on (defaults to all addresses, use option <code>--address</code> to restrict it).

<pre>
$ vbmc show controlplane1
+-----------------------+-------------------+
| Property | Value |
+-----------------------+-------------------+
| active | True |
| address | :: |
| domain_name | controlplane1 |
| libvirt_sasl_password | *** |
| libvirt_sasl_username | None |
| libvirt_uri | qemu:///system |
| password | *** |
| port | 6211 |
| status | running |
| username | admin |
+-----------------------+-------------------+
</pre>

Once started (which just opens the port) you can test the BMC connection using <code>ipmitool</code> or similar.

<pre>
$ ipmitool -I lanplus -H localhost -p 6211 -U admin -P password chassis status
System Power : off
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : always-off
Last Power Event :
Chassis Intrusion : inactive
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false
</pre>

That's it! We're ready to install OCP!

== Installer Configuration ==

This is where it all comes together. You need to execute the steps in this section (and next) on the ''provisioner'' host, that is, a system that has access to both external and provisioning networks of the OpenShift cluster-to-be. The access need not be direct, it can be routed, but if you, as in our example, configured the provisioning network to be an isolated virtual bridge, you will be best off by creating an additional VM that is directly connected to both bridges.

=== Gathering the Bits Together ===

First step is to make sure the following artifacts are available:

* <code>oc</code>
* <code>pull-secret</code>
* an SSH public key for compute node access

The global cluster network settings that we will need to configure are:

* the parent DNS domain of the cluster (such as <code>example.com</code>)
* the name of the cluster (concatenated with DNS domain, for example <code>mycluster</code> will become <code>mycluster.example.com</code>)
* the provisioning network CIDR (in our case, it can be any IP address block not overlapping with the external network as the provisioning network is isolated)
* from the external network address space, a designated:
** API server VIP (<code>api.mycluster.example.com</code> should point to it)
** ingress load balancer VIP (any host within <code>apps.mycluster.example.com</code> should resolve to it, usually via a wildcard record)

Additionally, you will need, for each node its:

* node name
* provisioning interface MAC address
* IPMI BMC address and port
* IPMI BMC credentials
* the name of the provisioning interface as seen from within the VM (re [[#Virtual Machine Configuration]] above - since the PCI address of the interface is ''bus 0x0 slot 0x3'' it will be named <code>enp0s3</code>)

As already said, DHCP address assignment to external interfaces is not managed by the installer. It must be handled by your infrastructure.

=== Preparing the Software ===

First, make sure your <code>pull-secret</code> file is on the provisioner.

<pre>
$ ls -l pull-secret
-rw-r-----. 1 provisioner provisioner 2734 Oct 27 12:21 pull-secret
</pre>

Then after downloading <code>oc</code> (if you didn't do that already)...

<pre>
$ curl -OL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/openshift-client-linux.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 60.9M 100 60.9M 0 0 4287k 0 0:00:14 0:00:14 --:--:-- 5642k

$ tar xf openshift-client-linux.tar.gz oc

$ ./oc version
Client Version: 4.14.9
Kustomize Version: v5.0.1

$ sudo cp ./oc /usr/local/bin/oc
$ rm -f ./oc ./openshift-client-linux.tar.gz
</pre>

...and optionally generating a bash completion file...

<pre>
$ oc completion bash > oc.completion
$ sudo cp oc.completion /etc/bash_completion.d/oc
$ rm -f oc.completion
$ source /etc/bash_completion.d/oc
</pre>

...you can use it to extract <code>openshift-baremetal-install</code> from the release image you intend to use.

<pre>
$ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/release.txt | grep "Pull From:"
Pull From: quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ oc adm release extract --registry-config=pull-secret --command=openshift-baremetal-install --to=. \
quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ ./openshift-baremetal-install version
./openshift-baremetal-install 4.14.9
built from commit dfafb5ca972a6ed4677257aebfe4f284ac020830
release image quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44
release architecture amd64

$ sudo cp ./openshift-baremetal-install /usr/local/bin/
$ rm -f ./openshift-baremetal-install

$ openshift-baremetal-install completion bash > oinst.completion
$ sudo cp oinst.completion /etc/bash_completion.d/oinst
$ rm -f oinst.completion
</pre>

'''NOTE''': The extraction process can take up to 5 minutes, depending on your network speed and Quay.io responsiveness.

=== Creating Installer Configuration ===

Installer configuration file can be created interactively, using <code>openshift-baremetal-install</code>, or you can simply write it out with an editor.

==== Using Interactive Mode ====

'''IMPORTANT''': In interactive mode, machine network (the network segment the nodes have their IPs allocated from) is expected to be <code>10.0.0.0/16</code>. If your VIPs are not on that network, the installer will fail. If your machine network is not <code>10.0.0.0/16</code>, the only way to install is to create <code>install-config.yaml</code> manually.

'''IMPORTANT''': Interactive installer expects three control plane nodes and three workers. It will fail if you add less than three of each type of nodes to the cluster.

The interactive mode will ask you a series of questions and generate <code>install-config.yaml</code> at the end, but there are very few customisation options using this method.

Initially, you must select the SSH public key to be published on cluster nodes, and specify some general settings:

<pre>
$ openshift-baremetal-install --dir=cluster create install-config
? SSH Public Key /home/provisioner/.ssh/id_rsa.pub
? Platform baremetal
? Provisioning Network Managed
? Provisioning Network CIDR 172.22.0.0/24
? Provisioning bridge provbr0
? Provisioning Network Interface enp0s3
? External bridge bridge0
? Add a Host: [Use arrows to move, type to filter]
> control plane
worker
</pre>

As you can see above, after those have been answered, you may add any number of nodes, either control plane or worker.

For both type of nodes, the questions are the same:

<pre>
? Add a Host: control plane
? Name controlplane1
? BMC Address ipmi://192.168.1.1:6211
? BMC Username admin
? BMC Password ********
? Boot MAC Address 52:54:00:00:fb:11
? Add another host? (y/N) y
</pre>

After all the hosts have been added, there is a final series of cluster-related questions:

<pre>
? Add another host? No
? Base Domain example.com
? Cluster Name mycluster
? Pull Secret [? for help] ****************************...
</pre>

At this point, some checks are performed against DNS server to see if <code>api.mycluster.example.com</code> resolves, and if the IP is a part of the machine network. The same check is performed for the ingress VIP.

'''NOTE''': If your VIPs are not on the default machine network, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [platform.baremetal.apiVIPs: Invalid value: "172.25.3.10": IP expected to be in one of the machine networks: 10.0.0.0/16, platform.baremetal.ingressVIPs: Invalid value: "172.25.3.11": IP expected to be in one of the machine networks: 10.0.0.0/16]
</pre>

'''NOTE''': If you added fewer than three control plane nodes, or fewer than three workers, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [...]
</pre>

==== Creating Configuration Manually ====

The installer configuration file consists of a number of mandatory sections:

* cluster domain and name
* network settings (such as machine and cluster IP ranges)
* control plane and compute node settings
* infrastructure platform settings
* pull secret and ssh key

There are also some optional sections which are useful in special cases such as disconnected installation or special install modes.

'''NOTE''': There is an <code>explain</code> subcommand in the installer binary that explains the structure of the <code>installconfig</code> resource.

<pre>
$ openshift-baremetal-install explain installconfig
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
InstallConfig is the configuration for an OpenShift install.

FIELDS:
additionalTrustBundle <string>
AdditionalTrustBundle is a PEM-encoded X.509 certificate bundle that will be added to the nodes' trusted certificate store.
...

$ openshift-baremetal-install explain installconfig.platform.baremetal
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
BareMetal is the configuration used when installing on bare metal.

FIELDS:
apiVIP <string>
Format: ip
DeprecatedAPIVIP is the VIP to use for internal API communication Deprecated: Use APIVIPs

apiVIPs <[]string>
Format: ip
APIVIPs contains the VIP(s) to use for internal API communication. In dual stack clusters it contains an IPv4 and IPv6 address, otherwise only one VIP
...
</pre>

An example <code>install-config.yaml</code> for baremetal IPI looks like this:

<pre>
apiVersion: v1
baseDomain: example.com
metadata:
name: mycluster
networking:
networkType: OVNKubernetes
# These are the networks external IPs will be allocated from.
machineNetwork:
- cidr: 172.25.3.0/24
# This is the pod network.
clusterNetworks:
- cidr: 10.200.0.0/14
hostPrefix: 23
# Only one entry is supported.
serviceNetwork:
- 172.30.0.0/16
compute:
- name: worker
replicas: 2
controlPlane:
name: master
replicas: 3
platform:
baremetal: {}
platform:
baremetal:
apiVIPs:
- 172.25.3.10
ingressVIPs:
- 172.25.3.11
provisioningNetwork: Managed
provisioningNetworkCIDR: 10.1.1.0/24
provisioningDHCPRange: 10.1.1.200,10.1.1.210
# These settings are to configure the temporary bootstrap node as a VM
externalBridge: bridge0
externalMACAddress: '52:54:00:00:fa:0f'
bootstrapProvisioningIP: 10.1.1.9
provisioningBridge: provbr0
provisioningNetworkInterface: ens3
provisioningMACAddress: '52:54:00:00:fb:0f'
# This needs to be done to avoid nested virtualisation if provisioner is a VM.
libvirtURI: qemu+ssh://root@hypervisor.example.com/system
hosts:
- name: controlplane1
role: master
bmc:
address: ipmi://hypervisor.example.com:6211
disableCertificateVerification: true
username: admin
password: password
# This is the provisioning network interface.
bootMACAddress: 52:54:00:00:fb:11
bootMode: legacy
# We need this for proper targetting of root device (vda).
hardwareProfile: libvirt
- name: controlplane2
role: master
bmc:
address: ipmi://hypervisor.example.com:6212
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:12
bootMode: legacy
hardwareProfile: libvirt
- name: controlplane3
role: master
bmc:
address: ipmi://hypervisor.example.com:6213
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:13
bootMode: legacy
hardwareProfile: libvirt
- name: worker1
role: master
bmc:
address: ipmi://hypervisor.example.com:6221
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:21
bootMode: legacy
hardwareProfile: libvirt
- name: worker2
role: master
bmc:
address: ipmi://hypervisor.example.com:6222
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:22
bootMode: legacy
hardwareProfile: libvirt
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"...","email":"..."},"quay.io":{"auth":"...","email":"..."},...}}'
sshKey: "ssh-rsa ..."
</pre>

Somewhere between RHCOS 4.12 and 4.14 the network devices stopped using the PCI naming scheme (in VMs only?). Consequently, <code>NetworkManager-wait-online</code> will fail because it's looking for a device (<code>enp0s3</code>) that isn't there (it's now called <code>ens3</code>). Sure, it's got an ''altname'' of <code>enp0s3</code> but <code>NetworkManager</code> don't care.

<pre>
$ ip ad sh enp0s3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 52:54:00:00:fa:13 brd ff:ff:ff:ff:ff:ff
altname enp0s3
inet 172.25.3.13/24 brd 172.25.3.255 scope global dynamic noprefixroute ens3
valid_lft 86083sec preferred_lft 86083sec
inet6 fe80::a30:57dd:ee31:e42e/64 scope link noprefixroute
valid_lft forever preferred_lft forever
</pre>

In turn, that will break most of the rest of the installation. Go figure. Just call the interface <code>ens3</code> and move on.

== Installation ==

With the above installation configuration file created, place a copy of it in a subdirectory, such as <code>./mycluster/</code> and run the installer.

<pre>
$ mkdir mycluster
$ cp install-config.yaml ./mycluster/

$ openshift-baremetal-install --dir=./mycluster/ --log-level=debug create cluster
DEBUG OpenShift Installer 4.14.9
DEBUG Built from commit dfafb5ca972a6ed4677257aebfe4f284ac020830
DEBUG Fetching Metadata...
DEBUG Loading Metadata...
...
DEBUG Loading Install Config...
DEBUG Loading Bootstrap Ignition Config...
...
INFO Consuming Install Config from target directory
...
INFO Obtaining RHCOS image file from 'https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.14-9.2/builds/414.92.202310210434-0/x86_64/rhcos-414.92.202310210434-0-qemu.x86_64.qcow2.gz?sha256=aab55f3ee088b88562f8fdcde5be78ace023e06fa01263e7cb9de2edc7131d6f'
...
INFO Creating infrastructure resources...
...
</pre>

When you see the above message, check the hypervisor for the presence of the temporary bootstrap VM.

<pre>
$ virsh list
Id Name State
---------------------------------------
4 provisioner running
5 mycluster-tmkmv-bootstrap running
</pre>

Once you see the VM running, you can inspect any containers on it by using the SSH key configured in install config to log into it and having a look around.

<pre>
$ ssh -i ~/.ssh/id_rsa core@bootstrap.mycluster.example.com
The authenticity of host 'bootstrap.mycluster.example.com (172.25.3.9)' can't be established.
...
Red Hat Enterprise Linux CoreOS 414.92.202310210434-0
Part of OpenShift 4.14, RHCOS is a Kubernetes native operating system
managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
...
[core@localhost ~]$ sudo -i

[root@localhost ~]# systemctl is-active crio
active

[root@localhost ~]# systemctl is-active bootkube
active

[root@localhost ~]# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
da3b3e74fc7f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:... /bin/rundnsmasq About a minute ago Up About a minute dnsmasq

[root@localhost ~]# crictl pods
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
af9df933d3b91 49 seconds ago Ready etcd-bootstrap-member-localhost.localdomain openshift-etcd 0 (default)

[root@localhost ~]# crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
9ad74f6433a33 753b0a16ba606f4c579690ed0035.. 6 seconds ago Running etcd 0 af9df933d3b91 etcd-bootstrap-member-localhost.localdomain
5c9371b25e646 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:... 6 seconds ago Running etcdctl 0 af9df933d3b91 etcd-bootstrap-member-localhost.localdomain
</pre>

Once you see pods like <code>kube-system</code>, <code>openshift-kube-apiserver</code>, and <code>openshift-cluster-version</code> show up in ready state, you can leave the shell and use the generated <code>kubeconfig</code> file to track the progress of the installation from the provisioner machine.

<pre>
$ export KUBECONFIG=$(pwd)/cluster/auth/kubeconfig

$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version False True 15m Unable to apply 4.14.9: an unknown error has occurred: MultipleErrors

$ oc get clusteroperators
$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication
baremetal
cloud-controller-manager
cloud-credential True False False 15m
cluster-autoscaler
config-operator
console
control-plane-machine-set
csi-snapshot-controller
dns
etcd
image-registry
ingress
insights
kube-apiserver
kube-controller-manager
kube-scheduler
kube-storage-version-migrator
machine-api
machine-approver
machine-config
marketplace
monitoring
network
node-tuning
openshift-apiserver
openshift-controller-manager
openshift-samples
operator-lifecycle-manager
operator-lifecycle-manager-catalog
operator-lifecycle-manager-packageserver
service-ca
storage
</pre>

'''NOTE''': It is normal for cluster version and various cluster operators to report transient error states as the progress of one impacts the progress of others. Eventually all these errors should go away.

== Post-Install Smoke Tests ==

= Conclusion =

OCP4-IPI-libvirt

2024-02-02T18:58:17Z

Gregab: /* Preparing the Software */ cleanup steps

= Introduction =

== What I Assume ==

* You know how OpenShift installation works and what the difference between IPI and UPI is.
* You know about the OpenShift Machine API and various underlying mechanisms.
* You understand the different types of network interfaces on Linux and different <code>libvirt</code> networks.
* You are familiar and comfortable with NetworkManager and the <code>nmcli</code> tool.
* You are familiar and comfortable with <code>libvirt</code> CLI and XML.
* You are familiar and comfortable with <code>qemu-img</code> tool.
* We are not talking about any firewall restrictions here - it is your responsibility to ensure traffic is not blocked.

== Outcomes ==

The installation described is for a fully managed IPI running OpenShift Container Platform v4.14, initially with three master and two worker nodes.

''At a later point, I will add a couple of steps needed to grow the cluster by one extra worker node.''

= OpenShift Container Platform IPI Installation Using Libvirt =

== Prerequisites ==

Hardware requirements for the cluster:

* 136 GiB RAM (32 GiB per control plane, 20 GiB per compute node), max overcommit ratio of 1.5 (make sure enough swap is available)
* 52 vCPUs (12 per control plane, 8 per compute node), max overcommit ratio of 1.3 (higher might work, but will slow down the installation horribly and may ultimately fail)
* one physical network interface that will be used for the public bridged network
* a physical or virtual network interface that will be used for the provisioning network bridge

Hardware requirements for the installation client (provisioner) machine:

* a minimum of 8 GiB RAM and 4 CPUs
* a network connection to both the public bridged network and the provisioning network

Due to the fact provisioner needs access to both networks, and the provisioning network in this guide is a virtual one, it might be best if you define the provisioner as a VM, with the same network interface settings as the control/compute nodes.

In the case you want to run the workloads spread across several hypervisor hosts, there are some extra steps, but nothing big. More on that in [[#Installation Spanning Multiple Hypervisors]] below.

Software artifacts needed on the provisioner host:

* <code>oc</code>, the command line client, of the corresponding version - download from https://mirror.openshift.com/pub/openshift-v4/clients/ocp/
* <code>libvirt-client</code> package is required for <code>openshift-baremetal-install</code> to be able to communicate to hypervisor(s)
* <code>ipmitool</code> or some other IPMI client
* a <code>pull-secret</code> file containing authentication credentials for OpenShift Container Platform registries - download from https://console.redhat.com/openshift/
* an SSH keypair that can be used for accessing OpenShift nodes

'''IMPORTANT''': The external IP addresses of cluster nodes must be assigned by your infrastructure DHCP server.

== Host Configuration ==

Beyond the logical requirement of having <code>libvirt</code> installed and started, here are the other configuration details for the hypervisor.

=== Network Settings ===

First thing you definitely need to make sure of, is that IP forwarding is enabled.

<pre>
$ sysctl net.ipv4.ip_forward
1
</pre>

Linux network settings need to be configured to have two '''Linux''' bridges, a public and a private provisioning one.

* public bridge, call it <code>bridge0</code>, needs to have the public network interface enslaved to it
* private bridge, call it <code>provbr0</code>, can be a virtual bridge since it is only needed for the provisioning network, which is supposed to be isolated and without any infrastructure services (such as DHCP, DNS, etc.)

It would be wonderful if the bridges could be OpenVSwitch ones, but unfortunately the Terraform bundled with <code>openshift-baremetal-install</code> currently does not include an OpenVSwitch provider, so there's goodbye to that.

As an example, here is my host configuration.

Public bridge:
<pre>
$ ip addr show bridge0
6: bridge0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff
inet 172.25.35.2/24 brd 172.25.35.255 scope global noprefixroute bridge0
valid_lft forever preferred_lft forever
inet6 fe80::4a21:bff:fe57:e06/64 scope link
valid_lft forever preferred_lft forever

$ ip addr show enp86s0
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master bridge0 state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff

$ bridge link | grep "master bridge0"
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master bridge0 state forwarding priority 32 cost 100
</pre>

Provisioning bridge:
<pre>
$ ip addr show provbr0
5: provbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ce:70:26:9c:88:a4 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.2/24 brd 10.1.1.255 scope global noprefixroute provbr0
valid_lft forever preferred_lft forever
inet6 fe80::cc70:26ff:fe9c:88a4/64 scope link
valid_lft forever preferred_lft forever

$ bridge link | grep "master provbr0"
</pre>

=== Installation Spanning Multiple Hypervisors ===

If you want to have your cluster spanning multiple hypervisors, make sure there is also a VXLAN connection between all the provisioning bridges.

You can do that by creating a <code>vxlan</code> type interface, which is a slave connection of type <code>bridge</code>, and the master is set to <code>provbr0</code>. Choose any unique VXLAN ID, and make sure it is the same on all interconnected hosts.

As an example, here is one VXLAN interface connecting hypervisor A to B.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1703164860
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.2
vxlan.remote: 172.25.35.3
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

And this is the corresponding VXLAN interface definition connecting host B to A.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1697549049
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.3
vxlan.remote: 172.25.35.2
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

In this case, <code>bridge link</code> will of course initially also show the <code>provbr0-vxlan10</code> interface as a slave, and will not show empty as above.

<pre>
$ bridge link | grep "master provbr0"
7: provbr0-vxlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master provbr0 state forwarding priority 32 cost 100
</pre>

=== <code>libvirt</code> Settings ===

Your <code>libvirt</code> will of course need to know about those network bridges in order to be able to attach VMs to them.

For that, you will need two network definitions, looking a bit like the following XML. Make sure they are autostart for least headache.

<pre>
$ sudo virsh net-dumpxml external
<network>
<name>external</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='bridge0'/>
</network>

$ sudo virsh net-dumpxml provisioning
<network>
<name>provisioning</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='provbr0'/>
</network>
</pre>

Additionally, you want to ensure that the storage pool is big enough, but that is not directly related to the subject at hand.

<pre>
$ sudo virsh pool-info default
Name: default
UUID: whatever
State: running
Persistent: yes
Autostart: yes
Capacity: 250.92 GiB
Allocation: 0 GiB
Available: 250.92 GiB
</pre>

=== VirtualBMC ===

The most important part of IPI facilitation is to be able to simulate a baseboard management controller for your VMs. <code>libvirt</code> obviously doesn't do this, but luckily there's a small bit of Python code that does, and it's called <code>virtualbmc</code>.

In most Python environments you can install it using <code>pip3</code>, just make sure <code>pip3</code> is up-to-date first.

<pre>
$ pip3 install --upgrade pip
...

$ pip3 install virtualbmc
...
</pre>

This gives you <code>/usr/local/bin/vbmcd</code> which you can control using the following systemd unit:

<pre>
[Unit]
Description=vbmcd
[Service]
Type=forking
ExecStart=/usr/local/bin/vbmcd
[Install]
WantedBy=multi-user.target
</pre>

Put the above content into <code>/etc/systemd/system/vbmcd.service</code>, reload systemd, and enable/start the service.

<pre>
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now vbmcd
</pre>

You now have the ability to associate a TCP port with a virtual machine defined on the hypervisor host, and have it simulate an IPMI BMC for that VM!

== Virtual Machine Configuration ==

=== VM Definitions ===

The virtual machines need to be configured with sufficient amount of compute resources, as per [[#Prerequisites]] above.

This section ties into the [[#Network Settings]] section above. You need two bridges on your hypervisor(s), <code>bridge0</code> and <code>provbr0</code>.

An example control plane node definition in <code>libvirt XML</code> would look like this:

<pre>
<domain type='kvm'>
<name>controlplane1</name>
<memory unit='GiB'>32</memory>
<currentMemory unit='GiB'>32</currentMemory>
<vcpu placement='static'>12</vcpu>
<os>
<type arch='x86_64' machine='q35'>hvm</type>
<boot dev='hd'/>
<boot dev='network'/>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model' check='partial'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/controlplane1-vda.qcow2'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'/>
<controller type='pci' index='0' model='pcie-root'/>
<interface type='bridge'>
<mac address='52:54:00:00:fb:11'/>
<source bridge='provbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:00:fa:11'/>
<source bridge='bridge0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<console type='pty'/>
<channel type='unix'>
<source mode='bind'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' autoport='yes' listen='0.0.0.0'/>
<video>
<model type='virtio'/>
</video>
<memballoon model='virtio'/>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
</rng>
</devices>
</domain>
</pre>

A couple of things to note:

* the first network interface is attached to <code>provbr0</code>, and its PCI address is ''lower'' (0x03), causing it to be the PXE default device
* the second network interface is attached to <code>bridge0</code>, and its PCI address is ''higher'' (0x09), making it the second interface (for external connections)
* boot order is set to hard disk first, network second, which means the host will only PXE boot if the disk image is unbootable
* the disk image needs to be 64GiB in size at the minimum, but you can make it larger and/or add more disk images if you intend to use the local storage operator

When configuring other nodes, simply remember to change the node name, disk image name, and MAC addresses to be unique. Adjust hardware resources accordingly for compute nodes.

=== IPMI BMC ===

What is important for OCP IPI to be able to perform the installation properly is to register each virtual machine with <code>vbmcd</code> and assign it with a port.

<pre>
$ vbmc add --port=6211 controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | down | :: | 6211 |
+-------------------+---------+---------+------+
$ vbmc start controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | running | :: | 6211 |
+-------------------+---------+---------+------+
$ sudo ss -aunp | grep 6211
UNCONN 0 0 *:6211 *:* users:(("vbmcd",pid=766290,fd=21))
</pre>

There is no need to run the <code>vbmc</code> client as root as it is the daemon that is running as root and can see all the VMs accessible through the <code>qemu:///system</code> URL.

Of course there are options. For any VM you add, you can specify a custom set of IPMI admin credentials (options <code>--username</code> and <code>--password</code>, they default to <code>admin</code> and <code>password</code>), a custom libvirt URL and credentials if necessary (options <code>--libvirt-uri</code>, <code>--libvirt-sasl-username</code>, and <code>--libvirt-sasl-password</code>), and a custom IP address to listen on (defaults to all addresses, use option <code>--address</code> to restrict it).

<pre>
$ vbmc show controlplane1
+-----------------------+-------------------+
| Property | Value |
+-----------------------+-------------------+
| active | True |
| address | :: |
| domain_name | controlplane1 |
| libvirt_sasl_password | *** |
| libvirt_sasl_username | None |
| libvirt_uri | qemu:///system |
| password | *** |
| port | 6211 |
| status | running |
| username | admin |
+-----------------------+-------------------+
</pre>

Once started (which just opens the port) you can test the BMC connection using <code>ipmitool</code> or similar.

<pre>
$ ipmitool -I lanplus -H localhost -p 6211 -U admin -P password chassis status
System Power : off
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : always-off
Last Power Event :
Chassis Intrusion : inactive
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false
</pre>

That's it! We're ready to install OCP!

== Installer Configuration ==

This is where it all comes together. You need to execute the steps in this section (and next) on the ''provisioner'' host, that is, a system that has access to both external and provisioning networks of the OpenShift cluster-to-be. The access need not be direct, it can be routed, but if you, as in our example, configured the provisioning network to be an isolated virtual bridge, you will be best off by creating an additional VM that is directly connected to both bridges.

=== Gathering the Bits Together ===

First step is to make sure the following artifacts are available:

* <code>oc</code>
* <code>pull-secret</code>
* an SSH public key for compute node access

The global cluster network settings that we will need to configure are:

* the parent DNS domain of the cluster (such as <code>example.com</code>)
* the name of the cluster (concatenated with DNS domain, for example <code>mycluster</code> will become <code>mycluster.example.com</code>)
* the provisioning network CIDR (in our case, it can be any IP address block not overlapping with the external network as the provisioning network is isolated)
* from the external network address space, a designated:
** API server VIP (<code>api.mycluster.example.com</code> should point to it)
** ingress load balancer VIP (any host within <code>apps.mycluster.example.com</code> should resolve to it, usually via a wildcard record)

Additionally, you will need, for each node its:

* node name
* provisioning interface MAC address
* IPMI BMC address and port
* IPMI BMC credentials
* the name of the provisioning interface as seen from within the VM (re [[#Virtual Machine Configuration]] above - since the PCI address of the interface is ''bus 0x0 slot 0x3'' it will be named <code>enp0s3</code>)

As already said, DHCP address assignment to external interfaces is not managed by the installer. It must be handled by your infrastructure.

=== Preparing the Software ===

First, make sure your <code>pull-secret</code> file is on the provisioner.

<pre>
$ ls -l pull-secret
-rw-r-----. 1 provisioner provisioner 2734 Oct 27 12:21 pull-secret
</pre>

Then after downloading <code>oc</code> (if you didn't do that already)...

<pre>
$ curl -OL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/openshift-client-linux.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 60.9M 100 60.9M 0 0 4287k 0 0:00:14 0:00:14 --:--:-- 5642k

$ tar xf openshift-client-linux.tar.gz oc

$ ./oc version
Client Version: 4.14.9
Kustomize Version: v5.0.1

$ sudo cp ./oc /usr/local/bin/oc
$ rm -f ./oc ./openshift-client-linux.tar.gz
</pre>

...and optionally generating a bash completion file...

<pre>
$ oc completion bash > oc.completion
$ sudo cp oc.completion /etc/bash_completion.d/oc
$ rm -f oc.completion
$ source /etc/bash_completion.d/oc
</pre>

...you can use it to extract <code>openshift-baremetal-install</code> from the release image you intend to use.

<pre>
$ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/release.txt | grep "Pull From:"
Pull From: quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ oc adm release extract --registry-config=pull-secret --command=openshift-baremetal-install --to=. \
quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ ./openshift-baremetal-install version
./openshift-baremetal-install 4.14.9
built from commit dfafb5ca972a6ed4677257aebfe4f284ac020830
release image quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44
release architecture amd64

$ sudo cp ./openshift-baremetal-install /usr/local/bin/
$ rm -f ./openshift-baremetal-install

$ openshift-baremetal-install completion bash > oinst.completion
$ sudo cp oinst.completion /etc/bash_completion.d/oinst
$ rm -f oinst.completion
</pre>

'''NOTE''': The extraction process can take up to 5 minutes, depending on your network speed and Quay.io responsiveness.

=== Creating Installer Configuration ===

Installer configuration file can be created interactively, using <code>openshift-baremetal-install</code>, or you can simply write it out with an editor.

==== Using Interactive Mode ====

'''IMPORTANT''': In interactive mode, machine network (the network segment the nodes have their IPs allocated from) is expected to be <code>10.0.0.0/16</code>. If your VIPs are not on that network, the installer will fail. If your machine network is not <code>10.0.0.0/16</code>, the only way to install is to create <code>install-config.yaml</code> manually.

'''IMPORTANT''': Interactive installer expects three control plane nodes and three workers. It will fail if you add less than three of each type of nodes to the cluster.

The interactive mode will ask you a series of questions and generate <code>install-config.yaml</code> at the end, but there are very few customisation options using this method.

Initially, you must select the SSH public key to be published on cluster nodes, and specify some general settings:

<pre>
$ openshift-baremetal-install --dir=cluster create install-config
? SSH Public Key /home/provisioner/.ssh/id_rsa.pub
? Platform baremetal
? Provisioning Network Managed
? Provisioning Network CIDR 172.22.0.0/24
? Provisioning bridge provbr0
? Provisioning Network Interface enp0s3
? External bridge bridge0
? Add a Host: [Use arrows to move, type to filter]
> control plane
worker
</pre>

As you can see above, after those have been answered, you may add any number of nodes, either control plane or worker.

For both type of nodes, the questions are the same:

<pre>
? Add a Host: control plane
? Name controlplane1
? BMC Address ipmi://192.168.1.1:6211
? BMC Username admin
? BMC Password ********
? Boot MAC Address 52:54:00:00:fb:11
? Add another host? (y/N) y
</pre>

After all the hosts have been added, there is a final series of cluster-related questions:

<pre>
? Add another host? No
? Base Domain example.com
? Cluster Name mycluster
? Pull Secret [? for help] ****************************...
</pre>

At this point, some checks are performed against DNS server to see if <code>api.mycluster.example.com</code> resolves, and if the IP is a part of the machine network. The same check is performed for the ingress VIP.

'''NOTE''': If your VIPs are not on the default machine network, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [platform.baremetal.apiVIPs: Invalid value: "172.25.3.10": IP expected to be in one of the machine networks: 10.0.0.0/16, platform.baremetal.ingressVIPs: Invalid value: "172.25.3.11": IP expected to be in one of the machine networks: 10.0.0.0/16]
</pre>

'''NOTE''': If you added fewer than three control plane nodes, or fewer than three workers, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [...]
</pre>

==== Creating Configuration Manually ====

The installer configuration file consists of a number of mandatory sections:

* cluster domain and name
* network settings (such as machine and cluster IP ranges)
* control plane and compute node settings
* infrastructure platform settings
* pull secret and ssh key

There are also some optional sections which are useful in special cases such as disconnected installation or special install modes.

'''NOTE''': There is an <code>explain</code> subcommand in the installer binary that explains the structure of the <code>installconfig</code> resource.

<pre>
$ openshift-baremetal-install explain installconfig
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
InstallConfig is the configuration for an OpenShift install.

FIELDS:
additionalTrustBundle <string>
AdditionalTrustBundle is a PEM-encoded X.509 certificate bundle that will be added to the nodes' trusted certificate store.
...

$ openshift-baremetal-install explain installconfig.platform.baremetal
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
BareMetal is the configuration used when installing on bare metal.

FIELDS:
apiVIP <string>
Format: ip
DeprecatedAPIVIP is the VIP to use for internal API communication Deprecated: Use APIVIPs

apiVIPs <[]string>
Format: ip
APIVIPs contains the VIP(s) to use for internal API communication. In dual stack clusters it contains an IPv4 and IPv6 address, otherwise only one VIP
...
</pre>

An example <code>install-config.yaml</code> for baremetal IPI looks like this:

<pre>
apiVersion: v1
baseDomain: example.com
metadata:
name: mycluster
networking:
networkType: OVNKubernetes
# These are the networks external IPs will be allocated from.
machineNetwork:
- cidr: 172.25.3.0/24
# This is the pod network.
clusterNetworks:
- cidr: 10.200.0.0/14
hostPrefix: 23
# Only one entry is supported.
serviceNetwork:
- 172.30.0.0/16
compute:
- name: worker
replicas: 2
controlPlane:
name: master
replicas: 3
platform:
baremetal: {}
platform:
baremetal:
apiVIPs:
- 172.25.3.10
ingressVIPs:
- 172.25.3.11
provisioningNetwork: Managed
provisioningNetworkCIDR: 10.1.1.0/24
provisioningDHCPRange: 10.1.1.200,10.1.1.210
# These settings are to configure the temporary bootstrap node as a VM
externalBridge: bridge0
externalMACAddress: '52:54:00:00:fa:0f'
bootstrapProvisioningIP: 10.1.1.9
provisioningBridge: provbr0
provisioningNetworkInterface: enp0s3
provisioningMACAddress: '52:54:00:00:fb:0f'
# This needs to be done to avoid nested virtualisation if provisioner is a VM.
libvirtURI: qemu+ssh://root@hypervisor.example.com/system
hosts:
- name: controlplane1
role: master
bmc:
address: ipmi://hypervisor.example.com:6211
disableCertificateVerification: true
username: admin
password: password
# This is the provisioning network interface.
bootMACAddress: 52:54:00:00:fb:11
bootMode: legacy
# We need this for proper targetting of root device (vda).
hardwareProfile: libvirt
- name: controlplane2
role: master
bmc:
address: ipmi://hypervisor.example.com:6212
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:12
bootMode: legacy
hardwareProfile: libvirt
- name: controlplane3
role: master
bmc:
address: ipmi://hypervisor.example.com:6213
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:13
bootMode: legacy
hardwareProfile: libvirt
- name: worker1
role: master
bmc:
address: ipmi://hypervisor.example.com:6221
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:21
bootMode: legacy
hardwareProfile: libvirt
- name: worker2
role: master
bmc:
address: ipmi://hypervisor.example.com:6222
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:22
bootMode: legacy
hardwareProfile: libvirt
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"...","email":"..."},"quay.io":{"auth":"...","email":"..."},...}}'
sshKey: "ssh-rsa ..."
</pre>

== Installation ==

With the above installation configuration file created, place a copy of it in a subdirectory, such as <code>./mycluster/</code> and run the installer.

<pre>
$ mkdir mycluster
$ cp install-config.yaml ./mycluster/

$ openshift-baremetal-install --dir=./mycluster/ --log-level=debug create cluster
DEBUG OpenShift Installer 4.14.9
DEBUG Built from commit dfafb5ca972a6ed4677257aebfe4f284ac020830
DEBUG Fetching Metadata...
DEBUG Loading Metadata...
...
DEBUG Loading Install Config...
DEBUG Loading Bootstrap Ignition Config...
...
INFO Consuming Install Config from target directory
...
INFO Obtaining RHCOS image file from 'https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.14-9.2/builds/414.92.202310210434-0/x86_64/rhcos-414.92.202310210434-0-qemu.x86_64.qcow2.gz?sha256=aab55f3ee088b88562f8fdcde5be78ace023e06fa01263e7cb9de2edc7131d6f'
...
INFO Creating infrastructure resources...
...
</pre>

When you see the above message, check the hypervisor for the presence of the temporary bootstrap VM.

<pre>
$ virsh list
Id Name State
---------------------------------------
4 provisioner running
5 mycluster-tmkmv-bootstrap running
</pre>

Once you see the VM running, you can inspect any containers on it by using the SSH key configured in install config to log into it and having a look around.

<pre>
$ ssh -i ~/.ssh/id_rsa core@bootstrap.mycluster.example.com
The authenticity of host 'bootstrap.mycluster.example.com (172.25.3.9)' can't be established.
...
Red Hat Enterprise Linux CoreOS 414.92.202310210434-0
Part of OpenShift 4.14, RHCOS is a Kubernetes native operating system
managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
...
[core@localhost ~]$ sudo -i

[root@localhost ~]# systemctl is-active crio
active

[root@localhost ~]# systemctl is-active bootkube
active

[root@localhost ~]# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
da3b3e74fc7f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:... /bin/rundnsmasq About a minute ago Up About a minute dnsmasq

[root@localhost ~]# crictl pods
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
af9df933d3b91 49 seconds ago Ready etcd-bootstrap-member-localhost.localdomain openshift-etcd 0 (default)

[root@localhost ~]# crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
9ad74f6433a33 753b0a16ba606f4c579690ed0035.. 6 seconds ago Running etcd 0 af9df933d3b91 etcd-bootstrap-member-localhost.localdomain
5c9371b25e646 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:... 6 seconds ago Running etcdctl 0 af9df933d3b91 etcd-bootstrap-member-localhost.localdomain
</pre>

Once you see pods like <code>kube-system</code>, <code>openshift-kube-apiserver</code>, and <code>openshift-cluster-version</code> show up in ready state, you can leave the shell and use the generated <code>kubeconfig</code> file to track the progress of the installation from the provisioner machine.

<pre>
$ export KUBECONFIG=$(pwd)/cluster/auth/kubeconfig

$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version False True 15m Unable to apply 4.14.9: an unknown error has occurred: MultipleErrors

$ oc get clusteroperators
$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication
baremetal
cloud-controller-manager
cloud-credential True False False 15m
cluster-autoscaler
config-operator
console
control-plane-machine-set
csi-snapshot-controller
dns
etcd
image-registry
ingress
insights
kube-apiserver
kube-controller-manager
kube-scheduler
kube-storage-version-migrator
machine-api
machine-approver
machine-config
marketplace
monitoring
network
node-tuning
openshift-apiserver
openshift-controller-manager
openshift-samples
operator-lifecycle-manager
operator-lifecycle-manager-catalog
operator-lifecycle-manager-packageserver
service-ca
storage
</pre>

'''NOTE''': It is normal for cluster version and various cluster operators to report transient error states as the progress of one impacts the progress of others. Eventually all these errors should go away.

== Post-Install Smoke Tests ==

= Conclusion =

OCP4-IPI-libvirt

2024-01-26T12:49:46Z

Gregab: /* Installation */ add debugging through API

= Introduction =

== What I Assume ==

* You know how OpenShift installation works and what the difference between IPI and UPI is.
* You know about the OpenShift Machine API and various underlying mechanisms.
* You understand the different types of network interfaces on Linux and different <code>libvirt</code> networks.
* You are familiar and comfortable with NetworkManager and the <code>nmcli</code> tool.
* You are familiar and comfortable with <code>libvirt</code> CLI and XML.
* You are familiar and comfortable with <code>qemu-img</code> tool.
* We are not talking about any firewall restrictions here - it is your responsibility to ensure traffic is not blocked.

== Outcomes ==

The installation described is for a fully managed IPI running OpenShift Container Platform v4.14, initially with three master and two worker nodes.

''At a later point, I will add a couple of steps needed to grow the cluster by one extra worker node.''

= OpenShift Container Platform IPI Installation Using Libvirt =

== Prerequisites ==

Hardware requirements for the cluster:

* 136 GiB RAM (32 GiB per control plane, 20 GiB per compute node), max overcommit ratio of 1.5 (make sure enough swap is available)
* 52 vCPUs (12 per control plane, 8 per compute node), max overcommit ratio of 1.3 (higher might work, but will slow down the installation horribly and may ultimately fail)
* one physical network interface that will be used for the public bridged network
* a physical or virtual network interface that will be used for the provisioning network bridge

Hardware requirements for the installation client (provisioner) machine:

* a minimum of 8 GiB RAM and 4 CPUs
* a network connection to both the public bridged network and the provisioning network

Due to the fact provisioner needs access to both networks, and the provisioning network in this guide is a virtual one, it might be best if you define the provisioner as a VM, with the same network interface settings as the control/compute nodes.

In the case you want to run the workloads spread across several hypervisor hosts, there are some extra steps, but nothing big. More on that in [[#Installation Spanning Multiple Hypervisors]] below.

Software artifacts needed on the provisioner host:

* <code>oc</code>, the command line client, of the corresponding version - download from https://mirror.openshift.com/pub/openshift-v4/clients/ocp/
* <code>libvirt-client</code> package is required for <code>openshift-baremetal-install</code> to be able to communicate to hypervisor(s)
* <code>ipmitool</code> or some other IPMI client
* a <code>pull-secret</code> file containing authentication credentials for OpenShift Container Platform registries - download from https://console.redhat.com/openshift/
* an SSH keypair that can be used for accessing OpenShift nodes

'''IMPORTANT''': The external IP addresses of cluster nodes must be assigned by your infrastructure DHCP server.

== Host Configuration ==

Beyond the logical requirement of having <code>libvirt</code> installed and started, here are the other configuration details for the hypervisor.

=== Network Settings ===

First thing you definitely need to make sure of, is that IP forwarding is enabled.

<pre>
$ sysctl net.ipv4.ip_forward
1
</pre>

Linux network settings need to be configured to have two '''Linux''' bridges, a public and a private provisioning one.

* public bridge, call it <code>bridge0</code>, needs to have the public network interface enslaved to it
* private bridge, call it <code>provbr0</code>, can be a virtual bridge since it is only needed for the provisioning network, which is supposed to be isolated and without any infrastructure services (such as DHCP, DNS, etc.)

It would be wonderful if the bridges could be OpenVSwitch ones, but unfortunately the Terraform bundled with <code>openshift-baremetal-install</code> currently does not include an OpenVSwitch provider, so there's goodbye to that.

As an example, here is my host configuration.

Public bridge:
<pre>
$ ip addr show bridge0
6: bridge0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff
inet 172.25.35.2/24 brd 172.25.35.255 scope global noprefixroute bridge0
valid_lft forever preferred_lft forever
inet6 fe80::4a21:bff:fe57:e06/64 scope link
valid_lft forever preferred_lft forever

$ ip addr show enp86s0
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master bridge0 state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff

$ bridge link | grep "master bridge0"
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master bridge0 state forwarding priority 32 cost 100
</pre>

Provisioning bridge:
<pre>
$ ip addr show provbr0
5: provbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ce:70:26:9c:88:a4 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.2/24 brd 10.1.1.255 scope global noprefixroute provbr0
valid_lft forever preferred_lft forever
inet6 fe80::cc70:26ff:fe9c:88a4/64 scope link
valid_lft forever preferred_lft forever

$ bridge link | grep "master provbr0"
</pre>

=== Installation Spanning Multiple Hypervisors ===

If you want to have your cluster spanning multiple hypervisors, make sure there is also a VXLAN connection between all the provisioning bridges.

You can do that by creating a <code>vxlan</code> type interface, which is a slave connection of type <code>bridge</code>, and the master is set to <code>provbr0</code>. Choose any unique VXLAN ID, and make sure it is the same on all interconnected hosts.

As an example, here is one VXLAN interface connecting hypervisor A to B.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1703164860
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.2
vxlan.remote: 172.25.35.3
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

And this is the corresponding VXLAN interface definition connecting host B to A.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1697549049
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.3
vxlan.remote: 172.25.35.2
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

In this case, <code>bridge link</code> will of course initially also show the <code>provbr0-vxlan10</code> interface as a slave, and will not show empty as above.

<pre>
$ bridge link | grep "master provbr0"
7: provbr0-vxlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master provbr0 state forwarding priority 32 cost 100
</pre>

=== <code>libvirt</code> Settings ===

Your <code>libvirt</code> will of course need to know about those network bridges in order to be able to attach VMs to them.

For that, you will need two network definitions, looking a bit like the following XML. Make sure they are autostart for least headache.

<pre>
$ sudo virsh net-dumpxml external
<network>
<name>external</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='bridge0'/>
</network>

$ sudo virsh net-dumpxml provisioning
<network>
<name>provisioning</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='provbr0'/>
</network>
</pre>

Additionally, you want to ensure that the storage pool is big enough, but that is not directly related to the subject at hand.

<pre>
$ sudo virsh pool-info default
Name: default
UUID: whatever
State: running
Persistent: yes
Autostart: yes
Capacity: 250.92 GiB
Allocation: 0 GiB
Available: 250.92 GiB
</pre>

=== VirtualBMC ===

The most important part of IPI facilitation is to be able to simulate a baseboard management controller for your VMs. <code>libvirt</code> obviously doesn't do this, but luckily there's a small bit of Python code that does, and it's called <code>virtualbmc</code>.

In most Python environments you can install it using <code>pip3</code>, just make sure <code>pip3</code> is up-to-date first.

<pre>
$ pip3 install --upgrade pip
...

$ pip3 install virtualbmc
...
</pre>

This gives you <code>/usr/local/bin/vbmcd</code> which you can control using the following systemd unit:

<pre>
[Unit]
Description=vbmcd
[Service]
Type=forking
ExecStart=/usr/local/bin/vbmcd
[Install]
WantedBy=multi-user.target
</pre>

Put the above content into <code>/etc/systemd/system/vbmcd.service</code>, reload systemd, and enable/start the service.

<pre>
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now vbmcd
</pre>

You now have the ability to associate a TCP port with a virtual machine defined on the hypervisor host, and have it simulate an IPMI BMC for that VM!

== Virtual Machine Configuration ==

=== VM Definitions ===

The virtual machines need to be configured with sufficient amount of compute resources, as per [[#Prerequisites]] above.

This section ties into the [[#Network Settings]] section above. You need two bridges on your hypervisor(s), <code>bridge0</code> and <code>provbr0</code>.

An example control plane node definition in <code>libvirt XML</code> would look like this:

<pre>
<domain type='kvm'>
<name>controlplane1</name>
<memory unit='GiB'>32</memory>
<currentMemory unit='GiB'>32</currentMemory>
<vcpu placement='static'>12</vcpu>
<os>
<type arch='x86_64' machine='q35'>hvm</type>
<boot dev='hd'/>
<boot dev='network'/>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model' check='partial'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/controlplane1-vda.qcow2'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'/>
<controller type='pci' index='0' model='pcie-root'/>
<interface type='bridge'>
<mac address='52:54:00:00:fb:11'/>
<source bridge='provbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:00:fa:11'/>
<source bridge='bridge0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<console type='pty'/>
<channel type='unix'>
<source mode='bind'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' autoport='yes' listen='0.0.0.0'/>
<video>
<model type='virtio'/>
</video>
<memballoon model='virtio'/>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
</rng>
</devices>
</domain>
</pre>

A couple of things to note:

* the first network interface is attached to <code>provbr0</code>, and its PCI address is ''lower'' (0x03), causing it to be the PXE default device
* the second network interface is attached to <code>bridge0</code>, and its PCI address is ''higher'' (0x09), making it the second interface (for external connections)
* boot order is set to hard disk first, network second, which means the host will only PXE boot if the disk image is unbootable
* the disk image needs to be 64GiB in size at the minimum, but you can make it larger and/or add more disk images if you intend to use the local storage operator

When configuring other nodes, simply remember to change the node name, disk image name, and MAC addresses to be unique. Adjust hardware resources accordingly for compute nodes.

=== IPMI BMC ===

What is important for OCP IPI to be able to perform the installation properly is to register each virtual machine with <code>vbmcd</code> and assign it with a port.

<pre>
$ vbmc add --port=6211 controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | down | :: | 6211 |
+-------------------+---------+---------+------+
$ vbmc start controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | running | :: | 6211 |
+-------------------+---------+---------+------+
$ sudo ss -aunp | grep 6211
UNCONN 0 0 *:6211 *:* users:(("vbmcd",pid=766290,fd=21))
</pre>

There is no need to run the <code>vbmc</code> client as root as it is the daemon that is running as root and can see all the VMs accessible through the <code>qemu:///system</code> URL.

Of course there are options. For any VM you add, you can specify a custom set of IPMI admin credentials (options <code>--username</code> and <code>--password</code>, they default to <code>admin</code> and <code>password</code>), a custom libvirt URL and credentials if necessary (options <code>--libvirt-uri</code>, <code>--libvirt-sasl-username</code>, and <code>--libvirt-sasl-password</code>), and a custom IP address to listen on (defaults to all addresses, use option <code>--address</code> to restrict it).

<pre>
$ vbmc show controlplane1
+-----------------------+-------------------+
| Property | Value |
+-----------------------+-------------------+
| active | True |
| address | :: |
| domain_name | controlplane1 |
| libvirt_sasl_password | *** |
| libvirt_sasl_username | None |
| libvirt_uri | qemu:///system |
| password | *** |
| port | 6211 |
| status | running |
| username | admin |
+-----------------------+-------------------+
</pre>

Once started (which just opens the port) you can test the BMC connection using <code>ipmitool</code> or similar.

<pre>
$ ipmitool -I lanplus -H localhost -p 6211 -U admin -P password chassis status
System Power : off
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : always-off
Last Power Event :
Chassis Intrusion : inactive
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false
</pre>

That's it! We're ready to install OCP!

== Installer Configuration ==

This is where it all comes together. You need to execute the steps in this section (and next) on the ''provisioner'' host, that is, a system that has access to both external and provisioning networks of the OpenShift cluster-to-be. The access need not be direct, it can be routed, but if you, as in our example, configured the provisioning network to be an isolated virtual bridge, you will be best off by creating an additional VM that is directly connected to both bridges.

=== Gathering the Bits Together ===

First step is to make sure the following artifacts are available:

* <code>oc</code>
* <code>pull-secret</code>
* an SSH public key for compute node access

The global cluster network settings that we will need to configure are:

* the parent DNS domain of the cluster (such as <code>example.com</code>)
* the name of the cluster (concatenated with DNS domain, for example <code>mycluster</code> will become <code>mycluster.example.com</code>)
* the provisioning network CIDR (in our case, it can be any IP address block not overlapping with the external network as the provisioning network is isolated)
* from the external network address space, a designated:
** API server VIP (<code>api.mycluster.example.com</code> should point to it)
** ingress load balancer VIP (any host within <code>apps.mycluster.example.com</code> should resolve to it, usually via a wildcard record)

Additionally, you will need, for each node its:

* node name
* provisioning interface MAC address
* IPMI BMC address and port
* IPMI BMC credentials
* the name of the provisioning interface as seen from within the VM (re [[#Virtual Machine Configuration]] above - since the PCI address of the interface is ''bus 0x0 slot 0x3'' it will be named <code>enp0s3</code>)

As already said, DHCP address assignment to external interfaces is not managed by the installer. It must be handled by your infrastructure.

=== Preparing the Software ===

First, make sure your <code>pull-secret</code> file is on the provisioner.

<pre>
$ ls -l pull-secret
-rw-r-----. 1 provisioner provisioner 2734 Oct 27 12:21 pull-secret
</pre>

Then after downloading <code>oc</code> (if you didn't do that already)...

<pre>
$ curl -OL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/openshift-client-linux.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 60.9M 100 60.9M 0 0 4287k 0 0:00:14 0:00:14 --:--:-- 5642k

$ tar xf openshift-client-linux.tar.gz oc

$ ./oc version
Client Version: 4.14.9
Kustomize Version: v5.0.1

$ sudo cp ./oc /usr/local/bin/oc
$ rm -f ./oc
</pre>

...and optionally generating a bash completion file...

<pre>
$ oc completion bash > oc.completion
$ sudo cp oc.completion /etc/bash_completion.d/oc
$ rm -f oc.completion
$ source /etc/bash_completion.d/oc
</pre>

...you can use it to extract <code>openshift-baremetal-install</code> from the release image you intend to use.

<pre>
$ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/release.txt | grep "Pull From:"
Pull From: quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ oc adm release extract --registry-config=pull-secret --command=openshift-baremetal-install --to=. \
quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ ./openshift-baremetal-install version
./openshift-baremetal-install 4.14.9
built from commit dfafb5ca972a6ed4677257aebfe4f284ac020830
release image quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44
release architecture amd64

$ sudo cp ./openshift-baremetal-install /usr/local/bin/

$ openshift-baremetal-install completion bash > oinst.completion
$ sudo cp oinst.completion /etc/bash_completion.d/oinst
$ rm -f oinst.completion
</pre>

'''NOTE''': The extraction process can take up to 5 minutes, depending on your network speed and Quay.io responsiveness.

=== Creating Installer Configuration ===

Installer configuration file can be created interactively, using <code>openshift-baremetal-install</code>, or you can simply write it out with an editor.

==== Using Interactive Mode ====

'''IMPORTANT''': In interactive mode, machine network (the network segment the nodes have their IPs allocated from) is expected to be <code>10.0.0.0/16</code>. If your VIPs are not on that network, the installer will fail. If your machine network is not <code>10.0.0.0/16</code>, the only way to install is to create <code>install-config.yaml</code> manually.

'''IMPORTANT''': Interactive installer expects three control plane nodes and three workers. It will fail if you add less than three of each type of nodes to the cluster.

The interactive mode will ask you a series of questions and generate <code>install-config.yaml</code> at the end, but there are very few customisation options using this method.

Initially, you must select the SSH public key to be published on cluster nodes, and specify some general settings:

<pre>
$ openshift-baremetal-install --dir=cluster create install-config
? SSH Public Key /home/provisioner/.ssh/id_rsa.pub
? Platform baremetal
? Provisioning Network Managed
? Provisioning Network CIDR 172.22.0.0/24
? Provisioning bridge provbr0
? Provisioning Network Interface enp0s3
? External bridge bridge0
? Add a Host: [Use arrows to move, type to filter]
> control plane
worker
</pre>

As you can see above, after those have been answered, you may add any number of nodes, either control plane or worker.

For both type of nodes, the questions are the same:

<pre>
? Add a Host: control plane
? Name controlplane1
? BMC Address ipmi://192.168.1.1:6211
? BMC Username admin
? BMC Password ********
? Boot MAC Address 52:54:00:00:fb:11
? Add another host? (y/N) y
</pre>

After all the hosts have been added, there is a final series of cluster-related questions:

<pre>
? Add another host? No
? Base Domain example.com
? Cluster Name mycluster
? Pull Secret [? for help] ****************************...
</pre>

At this point, some checks are performed against DNS server to see if <code>api.mycluster.example.com</code> resolves, and if the IP is a part of the machine network. The same check is performed for the ingress VIP.

'''NOTE''': If your VIPs are not on the default machine network, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [platform.baremetal.apiVIPs: Invalid value: "172.25.3.10": IP expected to be in one of the machine networks: 10.0.0.0/16, platform.baremetal.ingressVIPs: Invalid value: "172.25.3.11": IP expected to be in one of the machine networks: 10.0.0.0/16]
</pre>

'''NOTE''': If you added fewer than three control plane nodes, or fewer than three workers, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [...]
</pre>

==== Creating Configuration Manually ====

The installer configuration file consists of a number of mandatory sections:

* cluster domain and name
* network settings (such as machine and cluster IP ranges)
* control plane and compute node settings
* infrastructure platform settings
* pull secret and ssh key

There are also some optional sections which are useful in special cases such as disconnected installation or special install modes.

'''NOTE''': There is an <code>explain</code> subcommand in the installer binary that explains the structure of the <code>installconfig</code> resource.

<pre>
$ openshift-baremetal-install explain installconfig
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
InstallConfig is the configuration for an OpenShift install.

FIELDS:
additionalTrustBundle <string>
AdditionalTrustBundle is a PEM-encoded X.509 certificate bundle that will be added to the nodes' trusted certificate store.
...

$ openshift-baremetal-install explain installconfig.platform.baremetal
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
BareMetal is the configuration used when installing on bare metal.

FIELDS:
apiVIP <string>
Format: ip
DeprecatedAPIVIP is the VIP to use for internal API communication Deprecated: Use APIVIPs

apiVIPs <[]string>
Format: ip
APIVIPs contains the VIP(s) to use for internal API communication. In dual stack clusters it contains an IPv4 and IPv6 address, otherwise only one VIP
...
</pre>

An example <code>install-config.yaml</code> for baremetal IPI looks like this:

<pre>
apiVersion: v1
baseDomain: example.com
metadata:
name: mycluster
networking:
networkType: OVNKubernetes
# These are the networks external IPs will be allocated from.
machineNetwork:
- cidr: 172.25.3.0/24
# This is the pod network.
clusterNetworks:
- cidr: 10.200.0.0/14
hostPrefix: 23
# Only one entry is supported.
serviceNetwork:
- 172.30.0.0/16
compute:
- name: worker
replicas: 2
controlPlane:
name: master
replicas: 3
platform:
baremetal: {}
platform:
baremetal:
apiVIPs:
- 172.25.3.10
ingressVIPs:
- 172.25.3.11
provisioningNetwork: Managed
provisioningNetworkCIDR: 10.1.1.0/24
provisioningDHCPRange: 10.1.1.200,10.1.1.210
# These settings are to configure the temporary bootstrap node as a VM
externalBridge: bridge0
externalMACAddress: '52:54:00:00:fa:0f'
bootstrapProvisioningIP: 10.1.1.9
provisioningBridge: provbr0
provisioningNetworkInterface: enp0s3
provisioningMACAddress: '52:54:00:00:fb:0f'
# This needs to be done to avoid nested virtualisation if provisioner is a VM.
libvirtURI: qemu+ssh://root@hypervisor.example.com/system
hosts:
- name: controlplane1
role: master
bmc:
address: ipmi://hypervisor.example.com:6211
disableCertificateVerification: true
username: admin
password: password
# This is the provisioning network interface.
bootMACAddress: 52:54:00:00:fb:11
bootMode: legacy
# We need this for proper targetting of root device (vda).
hardwareProfile: libvirt
- name: controlplane2
role: master
bmc:
address: ipmi://hypervisor.example.com:6212
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:12
bootMode: legacy
hardwareProfile: libvirt
- name: controlplane3
role: master
bmc:
address: ipmi://hypervisor.example.com:6213
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:13
bootMode: legacy
hardwareProfile: libvirt
- name: worker1
role: master
bmc:
address: ipmi://hypervisor.example.com:6221
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:21
bootMode: legacy
hardwareProfile: libvirt
- name: worker2
role: master
bmc:
address: ipmi://hypervisor.example.com:6222
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:22
bootMode: legacy
hardwareProfile: libvirt
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"...","email":"..."},"quay.io":{"auth":"...","email":"..."},...}}'
sshKey: "ssh-rsa ..."
</pre>

== Installation ==

With the above installation configuration file created, place a copy of it in a subdirectory, such as <code>./mycluster/</code> and run the installer.

<pre>
$ mkdir mycluster
$ cp install-config.yaml ./mycluster/

$ openshift-baremetal-install --dir=./mycluster/ --log-level=debug create cluster
DEBUG OpenShift Installer 4.14.9
DEBUG Built from commit dfafb5ca972a6ed4677257aebfe4f284ac020830
DEBUG Fetching Metadata...
DEBUG Loading Metadata...
...
DEBUG Loading Install Config...
DEBUG Loading Bootstrap Ignition Config...
...
INFO Consuming Install Config from target directory
...
INFO Obtaining RHCOS image file from 'https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.14-9.2/builds/414.92.202310210434-0/x86_64/rhcos-414.92.202310210434-0-qemu.x86_64.qcow2.gz?sha256=aab55f3ee088b88562f8fdcde5be78ace023e06fa01263e7cb9de2edc7131d6f'
...
INFO Creating infrastructure resources...
...
</pre>

When you see the above message, check the hypervisor for the presence of the temporary bootstrap VM.

<pre>
$ virsh list
Id Name State
---------------------------------------
4 provisioner running
5 mycluster-tmkmv-bootstrap running
</pre>

Once you see the VM running, you can inspect any containers on it by using the SSH key configured in install config to log into it and having a look around.

<pre>
$ ssh -i ~/.ssh/id_rsa core@bootstrap.mycluster.example.com
The authenticity of host 'bootstrap.mycluster.example.com (172.25.3.9)' can't be established.
...
Red Hat Enterprise Linux CoreOS 414.92.202310210434-0
Part of OpenShift 4.14, RHCOS is a Kubernetes native operating system
managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
...
[core@localhost ~]$ sudo -i

[root@localhost ~]# systemctl is-active crio
active

[root@localhost ~]# systemctl is-active bootkube
active

[root@localhost ~]# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
da3b3e74fc7f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:... /bin/rundnsmasq About a minute ago Up About a minute dnsmasq

[root@localhost ~]# crictl pods
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
af9df933d3b91 49 seconds ago Ready etcd-bootstrap-member-localhost.localdomain openshift-etcd 0 (default)

[root@localhost ~]# crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
9ad74f6433a33 753b0a16ba606f4c579690ed0035.. 6 seconds ago Running etcd 0 af9df933d3b91 etcd-bootstrap-member-localhost.localdomain
5c9371b25e646 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:... 6 seconds ago Running etcdctl 0 af9df933d3b91 etcd-bootstrap-member-localhost.localdomain
</pre>

Once you see pods like <code>kube-system</code>, <code>openshift-kube-apiserver</code>, and <code>openshift-cluster-version</code> show up in ready state, you can leave the shell and use the generated <code>kubeconfig</code> file to track the progress of the installation from the provisioner machine.

<pre>
$ export KUBECONFIG=$(pwd)/cluster/auth/kubeconfig

$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version False True 15m Unable to apply 4.14.9: an unknown error has occurred: MultipleErrors

$ oc get clusteroperators
$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication
baremetal
cloud-controller-manager
cloud-credential True False False 15m
cluster-autoscaler
config-operator
console
control-plane-machine-set
csi-snapshot-controller
dns
etcd
image-registry
ingress
insights
kube-apiserver
kube-controller-manager
kube-scheduler
kube-storage-version-migrator
machine-api
machine-approver
machine-config
marketplace
monitoring
network
node-tuning
openshift-apiserver
openshift-controller-manager
openshift-samples
operator-lifecycle-manager
operator-lifecycle-manager-catalog
operator-lifecycle-manager-packageserver
service-ca
storage
</pre>

'''NOTE''': It is normal for cluster version and various cluster operators to report transient error states as the progress of one impacts the progress of others. Eventually all these errors should go away.

== Post-Install Smoke Tests ==

= Conclusion =

OCP4-IPI-libvirt

2024-01-26T12:21:41Z

Gregab: /* Installation */ done up to bootstrap

= Introduction =

== What I Assume ==

* You know how OpenShift installation works and what the difference between IPI and UPI is.
* You know about the OpenShift Machine API and various underlying mechanisms.
* You understand the different types of network interfaces on Linux and different <code>libvirt</code> networks.
* You are familiar and comfortable with NetworkManager and the <code>nmcli</code> tool.
* You are familiar and comfortable with <code>libvirt</code> CLI and XML.
* You are familiar and comfortable with <code>qemu-img</code> tool.
* We are not talking about any firewall restrictions here - it is your responsibility to ensure traffic is not blocked.

== Outcomes ==

The installation described is for a fully managed IPI running OpenShift Container Platform v4.14, initially with three master and two worker nodes.

''At a later point, I will add a couple of steps needed to grow the cluster by one extra worker node.''

= OpenShift Container Platform IPI Installation Using Libvirt =

== Prerequisites ==

Hardware requirements for the cluster:

* 136 GiB RAM (32 GiB per control plane, 20 GiB per compute node), max overcommit ratio of 1.5 (make sure enough swap is available)
* 52 vCPUs (12 per control plane, 8 per compute node), max overcommit ratio of 1.3 (higher might work, but will slow down the installation horribly and may ultimately fail)
* one physical network interface that will be used for the public bridged network
* a physical or virtual network interface that will be used for the provisioning network bridge

Hardware requirements for the installation client (provisioner) machine:

* a minimum of 8 GiB RAM and 4 CPUs
* a network connection to both the public bridged network and the provisioning network

Due to the fact provisioner needs access to both networks, and the provisioning network in this guide is a virtual one, it might be best if you define the provisioner as a VM, with the same network interface settings as the control/compute nodes.

In the case you want to run the workloads spread across several hypervisor hosts, there are some extra steps, but nothing big. More on that in [[#Installation Spanning Multiple Hypervisors]] below.

Software artifacts needed on the provisioner host:

* <code>oc</code>, the command line client, of the corresponding version - download from https://mirror.openshift.com/pub/openshift-v4/clients/ocp/
* <code>libvirt-client</code> package is required for <code>openshift-baremetal-install</code> to be able to communicate to hypervisor(s)
* <code>ipmitool</code> or some other IPMI client
* a <code>pull-secret</code> file containing authentication credentials for OpenShift Container Platform registries - download from https://console.redhat.com/openshift/
* an SSH keypair that can be used for accessing OpenShift nodes

'''IMPORTANT''': The external IP addresses of cluster nodes must be assigned by your infrastructure DHCP server.

== Host Configuration ==

Beyond the logical requirement of having <code>libvirt</code> installed and started, here are the other configuration details for the hypervisor.

=== Network Settings ===

First thing you definitely need to make sure of, is that IP forwarding is enabled.

<pre>
$ sysctl net.ipv4.ip_forward
1
</pre>

Linux network settings need to be configured to have two '''Linux''' bridges, a public and a private provisioning one.

* public bridge, call it <code>bridge0</code>, needs to have the public network interface enslaved to it
* private bridge, call it <code>provbr0</code>, can be a virtual bridge since it is only needed for the provisioning network, which is supposed to be isolated and without any infrastructure services (such as DHCP, DNS, etc.)

It would be wonderful if the bridges could be OpenVSwitch ones, but unfortunately the Terraform bundled with <code>openshift-baremetal-install</code> currently does not include an OpenVSwitch provider, so there's goodbye to that.

As an example, here is my host configuration.

Public bridge:
<pre>
$ ip addr show bridge0
6: bridge0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff
inet 172.25.35.2/24 brd 172.25.35.255 scope global noprefixroute bridge0
valid_lft forever preferred_lft forever
inet6 fe80::4a21:bff:fe57:e06/64 scope link
valid_lft forever preferred_lft forever

$ ip addr show enp86s0
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master bridge0 state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff

$ bridge link | grep "master bridge0"
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master bridge0 state forwarding priority 32 cost 100
</pre>

Provisioning bridge:
<pre>
$ ip addr show provbr0
5: provbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ce:70:26:9c:88:a4 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.2/24 brd 10.1.1.255 scope global noprefixroute provbr0
valid_lft forever preferred_lft forever
inet6 fe80::cc70:26ff:fe9c:88a4/64 scope link
valid_lft forever preferred_lft forever

$ bridge link | grep "master provbr0"
</pre>

=== Installation Spanning Multiple Hypervisors ===

If you want to have your cluster spanning multiple hypervisors, make sure there is also a VXLAN connection between all the provisioning bridges.

You can do that by creating a <code>vxlan</code> type interface, which is a slave connection of type <code>bridge</code>, and the master is set to <code>provbr0</code>. Choose any unique VXLAN ID, and make sure it is the same on all interconnected hosts.

As an example, here is one VXLAN interface connecting hypervisor A to B.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1703164860
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.2
vxlan.remote: 172.25.35.3
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

And this is the corresponding VXLAN interface definition connecting host B to A.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1697549049
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.3
vxlan.remote: 172.25.35.2
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

In this case, <code>bridge link</code> will of course initially also show the <code>provbr0-vxlan10</code> interface as a slave, and will not show empty as above.

<pre>
$ bridge link | grep "master provbr0"
7: provbr0-vxlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master provbr0 state forwarding priority 32 cost 100
</pre>

=== <code>libvirt</code> Settings ===

Your <code>libvirt</code> will of course need to know about those network bridges in order to be able to attach VMs to them.

For that, you will need two network definitions, looking a bit like the following XML. Make sure they are autostart for least headache.

<pre>
$ sudo virsh net-dumpxml external
<network>
<name>external</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='bridge0'/>
</network>

$ sudo virsh net-dumpxml provisioning
<network>
<name>provisioning</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='provbr0'/>
</network>
</pre>

Additionally, you want to ensure that the storage pool is big enough, but that is not directly related to the subject at hand.

<pre>
$ sudo virsh pool-info default
Name: default
UUID: whatever
State: running
Persistent: yes
Autostart: yes
Capacity: 250.92 GiB
Allocation: 0 GiB
Available: 250.92 GiB
</pre>

=== VirtualBMC ===

The most important part of IPI facilitation is to be able to simulate a baseboard management controller for your VMs. <code>libvirt</code> obviously doesn't do this, but luckily there's a small bit of Python code that does, and it's called <code>virtualbmc</code>.

In most Python environments you can install it using <code>pip3</code>, just make sure <code>pip3</code> is up-to-date first.

<pre>
$ pip3 install --upgrade pip
...

$ pip3 install virtualbmc
...
</pre>

This gives you <code>/usr/local/bin/vbmcd</code> which you can control using the following systemd unit:

<pre>
[Unit]
Description=vbmcd
[Service]
Type=forking
ExecStart=/usr/local/bin/vbmcd
[Install]
WantedBy=multi-user.target
</pre>

Put the above content into <code>/etc/systemd/system/vbmcd.service</code>, reload systemd, and enable/start the service.

<pre>
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now vbmcd
</pre>

You now have the ability to associate a TCP port with a virtual machine defined on the hypervisor host, and have it simulate an IPMI BMC for that VM!

== Virtual Machine Configuration ==

=== VM Definitions ===

The virtual machines need to be configured with sufficient amount of compute resources, as per [[#Prerequisites]] above.

This section ties into the [[#Network Settings]] section above. You need two bridges on your hypervisor(s), <code>bridge0</code> and <code>provbr0</code>.

An example control plane node definition in <code>libvirt XML</code> would look like this:

<pre>
<domain type='kvm'>
<name>controlplane1</name>
<memory unit='GiB'>32</memory>
<currentMemory unit='GiB'>32</currentMemory>
<vcpu placement='static'>12</vcpu>
<os>
<type arch='x86_64' machine='q35'>hvm</type>
<boot dev='hd'/>
<boot dev='network'/>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model' check='partial'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/controlplane1-vda.qcow2'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'/>
<controller type='pci' index='0' model='pcie-root'/>
<interface type='bridge'>
<mac address='52:54:00:00:fb:11'/>
<source bridge='provbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:00:fa:11'/>
<source bridge='bridge0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<console type='pty'/>
<channel type='unix'>
<source mode='bind'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' autoport='yes' listen='0.0.0.0'/>
<video>
<model type='virtio'/>
</video>
<memballoon model='virtio'/>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
</rng>
</devices>
</domain>
</pre>

A couple of things to note:

* the first network interface is attached to <code>provbr0</code>, and its PCI address is ''lower'' (0x03), causing it to be the PXE default device
* the second network interface is attached to <code>bridge0</code>, and its PCI address is ''higher'' (0x09), making it the second interface (for external connections)
* boot order is set to hard disk first, network second, which means the host will only PXE boot if the disk image is unbootable
* the disk image needs to be 64GiB in size at the minimum, but you can make it larger and/or add more disk images if you intend to use the local storage operator

When configuring other nodes, simply remember to change the node name, disk image name, and MAC addresses to be unique. Adjust hardware resources accordingly for compute nodes.

=== IPMI BMC ===

What is important for OCP IPI to be able to perform the installation properly is to register each virtual machine with <code>vbmcd</code> and assign it with a port.

<pre>
$ vbmc add --port=6211 controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | down | :: | 6211 |
+-------------------+---------+---------+------+
$ vbmc start controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | running | :: | 6211 |
+-------------------+---------+---------+------+
$ sudo ss -aunp | grep 6211
UNCONN 0 0 *:6211 *:* users:(("vbmcd",pid=766290,fd=21))
</pre>

There is no need to run the <code>vbmc</code> client as root as it is the daemon that is running as root and can see all the VMs accessible through the <code>qemu:///system</code> URL.

Of course there are options. For any VM you add, you can specify a custom set of IPMI admin credentials (options <code>--username</code> and <code>--password</code>, they default to <code>admin</code> and <code>password</code>), a custom libvirt URL and credentials if necessary (options <code>--libvirt-uri</code>, <code>--libvirt-sasl-username</code>, and <code>--libvirt-sasl-password</code>), and a custom IP address to listen on (defaults to all addresses, use option <code>--address</code> to restrict it).

<pre>
$ vbmc show controlplane1
+-----------------------+-------------------+
| Property | Value |
+-----------------------+-------------------+
| active | True |
| address | :: |
| domain_name | controlplane1 |
| libvirt_sasl_password | *** |
| libvirt_sasl_username | None |
| libvirt_uri | qemu:///system |
| password | *** |
| port | 6211 |
| status | running |
| username | admin |
+-----------------------+-------------------+
</pre>

Once started (which just opens the port) you can test the BMC connection using <code>ipmitool</code> or similar.

<pre>
$ ipmitool -I lanplus -H localhost -p 6211 -U admin -P password chassis status
System Power : off
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : always-off
Last Power Event :
Chassis Intrusion : inactive
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false
</pre>

That's it! We're ready to install OCP!

== Installer Configuration ==

This is where it all comes together. You need to execute the steps in this section (and next) on the ''provisioner'' host, that is, a system that has access to both external and provisioning networks of the OpenShift cluster-to-be. The access need not be direct, it can be routed, but if you, as in our example, configured the provisioning network to be an isolated virtual bridge, you will be best off by creating an additional VM that is directly connected to both bridges.

=== Gathering the Bits Together ===

First step is to make sure the following artifacts are available:

* <code>oc</code>
* <code>pull-secret</code>
* an SSH public key for compute node access

The global cluster network settings that we will need to configure are:

* the parent DNS domain of the cluster (such as <code>example.com</code>)
* the name of the cluster (concatenated with DNS domain, for example <code>mycluster</code> will become <code>mycluster.example.com</code>)
* the provisioning network CIDR (in our case, it can be any IP address block not overlapping with the external network as the provisioning network is isolated)
* from the external network address space, a designated:
** API server VIP (<code>api.mycluster.example.com</code> should point to it)
** ingress load balancer VIP (any host within <code>apps.mycluster.example.com</code> should resolve to it, usually via a wildcard record)

Additionally, you will need, for each node its:

* node name
* provisioning interface MAC address
* IPMI BMC address and port
* IPMI BMC credentials
* the name of the provisioning interface as seen from within the VM (re [[#Virtual Machine Configuration]] above - since the PCI address of the interface is ''bus 0x0 slot 0x3'' it will be named <code>enp0s3</code>)

As already said, DHCP address assignment to external interfaces is not managed by the installer. It must be handled by your infrastructure.

=== Preparing the Software ===

First, make sure your <code>pull-secret</code> file is on the provisioner.

<pre>
$ ls -l pull-secret
-rw-r-----. 1 provisioner provisioner 2734 Oct 27 12:21 pull-secret
</pre>

Then after downloading <code>oc</code> (if you didn't do that already)...

<pre>
$ curl -OL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/openshift-client-linux.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 60.9M 100 60.9M 0 0 4287k 0 0:00:14 0:00:14 --:--:-- 5642k

$ tar xf openshift-client-linux.tar.gz oc

$ ./oc version
Client Version: 4.14.9
Kustomize Version: v5.0.1

$ sudo cp ./oc /usr/local/bin/oc
$ rm -f ./oc
</pre>

...and optionally generating a bash completion file...

<pre>
$ oc completion bash > oc.completion
$ sudo cp oc.completion /etc/bash_completion.d/oc
$ rm -f oc.completion
$ source /etc/bash_completion.d/oc
</pre>

...you can use it to extract <code>openshift-baremetal-install</code> from the release image you intend to use.

<pre>
$ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/release.txt | grep "Pull From:"
Pull From: quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ oc adm release extract --registry-config=pull-secret --command=openshift-baremetal-install --to=. \
quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ ./openshift-baremetal-install version
./openshift-baremetal-install 4.14.9
built from commit dfafb5ca972a6ed4677257aebfe4f284ac020830
release image quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44
release architecture amd64

$ sudo cp ./openshift-baremetal-install /usr/local/bin/

$ openshift-baremetal-install completion bash > oinst.completion
$ sudo cp oinst.completion /etc/bash_completion.d/oinst
$ rm -f oinst.completion
</pre>

'''NOTE''': The extraction process can take up to 5 minutes, depending on your network speed and Quay.io responsiveness.

=== Creating Installer Configuration ===

Installer configuration file can be created interactively, using <code>openshift-baremetal-install</code>, or you can simply write it out with an editor.

==== Using Interactive Mode ====

'''IMPORTANT''': In interactive mode, machine network (the network segment the nodes have their IPs allocated from) is expected to be <code>10.0.0.0/16</code>. If your VIPs are not on that network, the installer will fail. If your machine network is not <code>10.0.0.0/16</code>, the only way to install is to create <code>install-config.yaml</code> manually.

'''IMPORTANT''': Interactive installer expects three control plane nodes and three workers. It will fail if you add less than three of each type of nodes to the cluster.

The interactive mode will ask you a series of questions and generate <code>install-config.yaml</code> at the end, but there are very few customisation options using this method.

Initially, you must select the SSH public key to be published on cluster nodes, and specify some general settings:

<pre>
$ openshift-baremetal-install --dir=cluster create install-config
? SSH Public Key /home/provisioner/.ssh/id_rsa.pub
? Platform baremetal
? Provisioning Network Managed
? Provisioning Network CIDR 172.22.0.0/24
? Provisioning bridge provbr0
? Provisioning Network Interface enp0s3
? External bridge bridge0
? Add a Host: [Use arrows to move, type to filter]
> control plane
worker
</pre>

As you can see above, after those have been answered, you may add any number of nodes, either control plane or worker.

For both type of nodes, the questions are the same:

<pre>
? Add a Host: control plane
? Name controlplane1
? BMC Address ipmi://192.168.1.1:6211
? BMC Username admin
? BMC Password ********
? Boot MAC Address 52:54:00:00:fb:11
? Add another host? (y/N) y
</pre>

After all the hosts have been added, there is a final series of cluster-related questions:

<pre>
? Add another host? No
? Base Domain example.com
? Cluster Name mycluster
? Pull Secret [? for help] ****************************...
</pre>

At this point, some checks are performed against DNS server to see if <code>api.mycluster.example.com</code> resolves, and if the IP is a part of the machine network. The same check is performed for the ingress VIP.

'''NOTE''': If your VIPs are not on the default machine network, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [platform.baremetal.apiVIPs: Invalid value: "172.25.3.10": IP expected to be in one of the machine networks: 10.0.0.0/16, platform.baremetal.ingressVIPs: Invalid value: "172.25.3.11": IP expected to be in one of the machine networks: 10.0.0.0/16]
</pre>

'''NOTE''': If you added fewer than three control plane nodes, or fewer than three workers, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [...]
</pre>

==== Creating Configuration Manually ====

The installer configuration file consists of a number of mandatory sections:

* cluster domain and name
* network settings (such as machine and cluster IP ranges)
* control plane and compute node settings
* infrastructure platform settings
* pull secret and ssh key

There are also some optional sections which are useful in special cases such as disconnected installation or special install modes.

'''NOTE''': There is an <code>explain</code> subcommand in the installer binary that explains the structure of the <code>installconfig</code> resource.

<pre>
$ openshift-baremetal-install explain installconfig
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
InstallConfig is the configuration for an OpenShift install.

FIELDS:
additionalTrustBundle <string>
AdditionalTrustBundle is a PEM-encoded X.509 certificate bundle that will be added to the nodes' trusted certificate store.
...

$ openshift-baremetal-install explain installconfig.platform.baremetal
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
BareMetal is the configuration used when installing on bare metal.

FIELDS:
apiVIP <string>
Format: ip
DeprecatedAPIVIP is the VIP to use for internal API communication Deprecated: Use APIVIPs

apiVIPs <[]string>
Format: ip
APIVIPs contains the VIP(s) to use for internal API communication. In dual stack clusters it contains an IPv4 and IPv6 address, otherwise only one VIP
...
</pre>

An example <code>install-config.yaml</code> for baremetal IPI looks like this:

<pre>
apiVersion: v1
baseDomain: example.com
metadata:
name: mycluster
networking:
networkType: OVNKubernetes
# These are the networks external IPs will be allocated from.
machineNetwork:
- cidr: 172.25.3.0/24
# This is the pod network.
clusterNetworks:
- cidr: 10.200.0.0/14
hostPrefix: 23
# Only one entry is supported.
serviceNetwork:
- 172.30.0.0/16
compute:
- name: worker
replicas: 2
controlPlane:
name: master
replicas: 3
platform:
baremetal: {}
platform:
baremetal:
apiVIPs:
- 172.25.3.10
ingressVIPs:
- 172.25.3.11
provisioningNetwork: Managed
provisioningNetworkCIDR: 10.1.1.0/24
provisioningDHCPRange: 10.1.1.200,10.1.1.210
# These settings are to configure the temporary bootstrap node as a VM
externalBridge: bridge0
externalMACAddress: '52:54:00:00:fa:0f'
bootstrapProvisioningIP: 10.1.1.9
provisioningBridge: provbr0
provisioningNetworkInterface: enp0s3
provisioningMACAddress: '52:54:00:00:fb:0f'
# This needs to be done to avoid nested virtualisation if provisioner is a VM.
libvirtURI: qemu+ssh://root@hypervisor.example.com/system
hosts:
- name: controlplane1
role: master
bmc:
address: ipmi://hypervisor.example.com:6211
disableCertificateVerification: true
username: admin
password: password
# This is the provisioning network interface.
bootMACAddress: 52:54:00:00:fb:11
bootMode: legacy
# We need this for proper targetting of root device (vda).
hardwareProfile: libvirt
- name: controlplane2
role: master
bmc:
address: ipmi://hypervisor.example.com:6212
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:12
bootMode: legacy
hardwareProfile: libvirt
- name: controlplane3
role: master
bmc:
address: ipmi://hypervisor.example.com:6213
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:13
bootMode: legacy
hardwareProfile: libvirt
- name: worker1
role: master
bmc:
address: ipmi://hypervisor.example.com:6221
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:21
bootMode: legacy
hardwareProfile: libvirt
- name: worker2
role: master
bmc:
address: ipmi://hypervisor.example.com:6222
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:22
bootMode: legacy
hardwareProfile: libvirt
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"...","email":"..."},"quay.io":{"auth":"...","email":"..."},...}}'
sshKey: "ssh-rsa ..."
</pre>

== Installation ==

With the above installation configuration file created, place a copy of it in a subdirectory, such as <code>./mycluster/</code> and run the installer.

<pre>
$ mkdir mycluster
$ cp install-config.yaml ./mycluster/

$ openshift-baremetal-install --dir=./mycluster/ --log-level=debug create cluster
DEBUG OpenShift Installer 4.14.9
DEBUG Built from commit dfafb5ca972a6ed4677257aebfe4f284ac020830
DEBUG Fetching Metadata...
DEBUG Loading Metadata...
...
DEBUG Loading Install Config...
DEBUG Loading Bootstrap Ignition Config...
...
INFO Consuming Install Config from target directory
...
INFO Obtaining RHCOS image file from 'https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.14-9.2/builds/414.92.202310210434-0/x86_64/rhcos-414.92.202310210434-0-qemu.x86_64.qcow2.gz?sha256=aab55f3ee088b88562f8fdcde5be78ace023e06fa01263e7cb9de2edc7131d6f'
...
INFO Creating infrastructure resources...
...
</pre>

When you see the above message, check the hypervisor for the presence of the temporary bootstrap VM.

<pre>
$ virsh list
Id Name State
---------------------------------------
4 provisioner running
5 mycluster-tmkmv-bootstrap running
</pre>

Once you see the VM running, you can inspect any containers on it by using the SSH key configured in install config to log into it and having a look around.

<pre>
$ ssh -i ~/.ssh/id_rsa core@bootstrap.mycluster.example.com
The authenticity of host 'bootstrap.mycluster.example.com (172.25.3.9)' can't be established.
...
Red Hat Enterprise Linux CoreOS 414.92.202310210434-0
Part of OpenShift 4.14, RHCOS is a Kubernetes native operating system
managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
...
[core@localhost ~]$ sudo -i

[root@localhost ~]# systemctl is-active crio
active

[root@localhost ~]# systemctl is-active bootkube
active

[root@localhost ~]# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
da3b3e74fc7f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:... /bin/rundnsmasq About a minute ago Up About a minute dnsmasq

[root@localhost ~]# crictl pods
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
af9df933d3b91 49 seconds ago Ready etcd-bootstrap-member-localhost.localdomain openshift-etcd 0 (default)

[root@localhost ~]# crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
9ad74f6433a33 753b0a16ba606f4c579690ed0035.. 6 seconds ago Running etcd 0 af9df933d3b91 etcd-bootstrap-member-localhost.localdomain
5c9371b25e646 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:... 6 seconds ago Running etcdctl 0 af9df933d3b91 etcd-bootstrap-member-localhost.localdomain
</pre>

== Post-Install Smoke Tests ==

= Conclusion =

OCP4-IPI-libvirt

2024-01-26T12:09:18Z

Gregab: /* Creating Configuration Manually */ fix typo

= Introduction =

== What I Assume ==

* You know how OpenShift installation works and what the difference between IPI and UPI is.
* You know about the OpenShift Machine API and various underlying mechanisms.
* You understand the different types of network interfaces on Linux and different <code>libvirt</code> networks.
* You are familiar and comfortable with NetworkManager and the <code>nmcli</code> tool.
* You are familiar and comfortable with <code>libvirt</code> CLI and XML.
* You are familiar and comfortable with <code>qemu-img</code> tool.
* We are not talking about any firewall restrictions here - it is your responsibility to ensure traffic is not blocked.

== Outcomes ==

The installation described is for a fully managed IPI running OpenShift Container Platform v4.14, initially with three master and two worker nodes.

''At a later point, I will add a couple of steps needed to grow the cluster by one extra worker node.''

= OpenShift Container Platform IPI Installation Using Libvirt =

== Prerequisites ==

Hardware requirements for the cluster:

* 136 GiB RAM (32 GiB per control plane, 20 GiB per compute node), max overcommit ratio of 1.5 (make sure enough swap is available)
* 52 vCPUs (12 per control plane, 8 per compute node), max overcommit ratio of 1.3 (higher might work, but will slow down the installation horribly and may ultimately fail)
* one physical network interface that will be used for the public bridged network
* a physical or virtual network interface that will be used for the provisioning network bridge

Hardware requirements for the installation client (provisioner) machine:

* a minimum of 8 GiB RAM and 4 CPUs
* a network connection to both the public bridged network and the provisioning network

Due to the fact provisioner needs access to both networks, and the provisioning network in this guide is a virtual one, it might be best if you define the provisioner as a VM, with the same network interface settings as the control/compute nodes.

In the case you want to run the workloads spread across several hypervisor hosts, there are some extra steps, but nothing big. More on that in [[#Installation Spanning Multiple Hypervisors]] below.

Software artifacts needed on the provisioner host:

* <code>oc</code>, the command line client, of the corresponding version - download from https://mirror.openshift.com/pub/openshift-v4/clients/ocp/
* <code>libvirt-client</code> package is required for <code>openshift-baremetal-install</code> to be able to communicate to hypervisor(s)
* <code>ipmitool</code> or some other IPMI client
* a <code>pull-secret</code> file containing authentication credentials for OpenShift Container Platform registries - download from https://console.redhat.com/openshift/
* an SSH keypair that can be used for accessing OpenShift nodes

'''IMPORTANT''': The external IP addresses of cluster nodes must be assigned by your infrastructure DHCP server.

== Host Configuration ==

Beyond the logical requirement of having <code>libvirt</code> installed and started, here are the other configuration details for the hypervisor.

=== Network Settings ===

First thing you definitely need to make sure of, is that IP forwarding is enabled.

<pre>
$ sysctl net.ipv4.ip_forward
1
</pre>

Linux network settings need to be configured to have two '''Linux''' bridges, a public and a private provisioning one.

* public bridge, call it <code>bridge0</code>, needs to have the public network interface enslaved to it
* private bridge, call it <code>provbr0</code>, can be a virtual bridge since it is only needed for the provisioning network, which is supposed to be isolated and without any infrastructure services (such as DHCP, DNS, etc.)

It would be wonderful if the bridges could be OpenVSwitch ones, but unfortunately the Terraform bundled with <code>openshift-baremetal-install</code> currently does not include an OpenVSwitch provider, so there's goodbye to that.

As an example, here is my host configuration.

Public bridge:
<pre>
$ ip addr show bridge0
6: bridge0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff
inet 172.25.35.2/24 brd 172.25.35.255 scope global noprefixroute bridge0
valid_lft forever preferred_lft forever
inet6 fe80::4a21:bff:fe57:e06/64 scope link
valid_lft forever preferred_lft forever

$ ip addr show enp86s0
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master bridge0 state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff

$ bridge link | grep "master bridge0"
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master bridge0 state forwarding priority 32 cost 100
</pre>

Provisioning bridge:
<pre>
$ ip addr show provbr0
5: provbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ce:70:26:9c:88:a4 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.2/24 brd 10.1.1.255 scope global noprefixroute provbr0
valid_lft forever preferred_lft forever
inet6 fe80::cc70:26ff:fe9c:88a4/64 scope link
valid_lft forever preferred_lft forever

$ bridge link | grep "master provbr0"
</pre>

=== Installation Spanning Multiple Hypervisors ===

If you want to have your cluster spanning multiple hypervisors, make sure there is also a VXLAN connection between all the provisioning bridges.

You can do that by creating a <code>vxlan</code> type interface, which is a slave connection of type <code>bridge</code>, and the master is set to <code>provbr0</code>. Choose any unique VXLAN ID, and make sure it is the same on all interconnected hosts.

As an example, here is one VXLAN interface connecting hypervisor A to B.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1703164860
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.2
vxlan.remote: 172.25.35.3
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

And this is the corresponding VXLAN interface definition connecting host B to A.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1697549049
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.3
vxlan.remote: 172.25.35.2
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

In this case, <code>bridge link</code> will of course initially also show the <code>provbr0-vxlan10</code> interface as a slave, and will not show empty as above.

<pre>
$ bridge link | grep "master provbr0"
7: provbr0-vxlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master provbr0 state forwarding priority 32 cost 100
</pre>

=== <code>libvirt</code> Settings ===

Your <code>libvirt</code> will of course need to know about those network bridges in order to be able to attach VMs to them.

For that, you will need two network definitions, looking a bit like the following XML. Make sure they are autostart for least headache.

<pre>
$ sudo virsh net-dumpxml external
<network>
<name>external</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='bridge0'/>
</network>

$ sudo virsh net-dumpxml provisioning
<network>
<name>provisioning</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='provbr0'/>
</network>
</pre>

Additionally, you want to ensure that the storage pool is big enough, but that is not directly related to the subject at hand.

<pre>
$ sudo virsh pool-info default
Name: default
UUID: whatever
State: running
Persistent: yes
Autostart: yes
Capacity: 250.92 GiB
Allocation: 0 GiB
Available: 250.92 GiB
</pre>

=== VirtualBMC ===

The most important part of IPI facilitation is to be able to simulate a baseboard management controller for your VMs. <code>libvirt</code> obviously doesn't do this, but luckily there's a small bit of Python code that does, and it's called <code>virtualbmc</code>.

In most Python environments you can install it using <code>pip3</code>, just make sure <code>pip3</code> is up-to-date first.

<pre>
$ pip3 install --upgrade pip
...

$ pip3 install virtualbmc
...
</pre>

This gives you <code>/usr/local/bin/vbmcd</code> which you can control using the following systemd unit:

<pre>
[Unit]
Description=vbmcd
[Service]
Type=forking
ExecStart=/usr/local/bin/vbmcd
[Install]
WantedBy=multi-user.target
</pre>

Put the above content into <code>/etc/systemd/system/vbmcd.service</code>, reload systemd, and enable/start the service.

<pre>
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now vbmcd
</pre>

You now have the ability to associate a TCP port with a virtual machine defined on the hypervisor host, and have it simulate an IPMI BMC for that VM!

== Virtual Machine Configuration ==

=== VM Definitions ===

The virtual machines need to be configured with sufficient amount of compute resources, as per [[#Prerequisites]] above.

This section ties into the [[#Network Settings]] section above. You need two bridges on your hypervisor(s), <code>bridge0</code> and <code>provbr0</code>.

An example control plane node definition in <code>libvirt XML</code> would look like this:

<pre>
<domain type='kvm'>
<name>controlplane1</name>
<memory unit='GiB'>32</memory>
<currentMemory unit='GiB'>32</currentMemory>
<vcpu placement='static'>12</vcpu>
<os>
<type arch='x86_64' machine='q35'>hvm</type>
<boot dev='hd'/>
<boot dev='network'/>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model' check='partial'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/controlplane1-vda.qcow2'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'/>
<controller type='pci' index='0' model='pcie-root'/>
<interface type='bridge'>
<mac address='52:54:00:00:fb:11'/>
<source bridge='provbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:00:fa:11'/>
<source bridge='bridge0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<console type='pty'/>
<channel type='unix'>
<source mode='bind'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' autoport='yes' listen='0.0.0.0'/>
<video>
<model type='virtio'/>
</video>
<memballoon model='virtio'/>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
</rng>
</devices>
</domain>
</pre>

A couple of things to note:

* the first network interface is attached to <code>provbr0</code>, and its PCI address is ''lower'' (0x03), causing it to be the PXE default device
* the second network interface is attached to <code>bridge0</code>, and its PCI address is ''higher'' (0x09), making it the second interface (for external connections)
* boot order is set to hard disk first, network second, which means the host will only PXE boot if the disk image is unbootable
* the disk image needs to be 64GiB in size at the minimum, but you can make it larger and/or add more disk images if you intend to use the local storage operator

When configuring other nodes, simply remember to change the node name, disk image name, and MAC addresses to be unique. Adjust hardware resources accordingly for compute nodes.

=== IPMI BMC ===

What is important for OCP IPI to be able to perform the installation properly is to register each virtual machine with <code>vbmcd</code> and assign it with a port.

<pre>
$ vbmc add --port=6211 controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | down | :: | 6211 |
+-------------------+---------+---------+------+
$ vbmc start controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | running | :: | 6211 |
+-------------------+---------+---------+------+
$ sudo ss -aunp | grep 6211
UNCONN 0 0 *:6211 *:* users:(("vbmcd",pid=766290,fd=21))
</pre>

There is no need to run the <code>vbmc</code> client as root as it is the daemon that is running as root and can see all the VMs accessible through the <code>qemu:///system</code> URL.

Of course there are options. For any VM you add, you can specify a custom set of IPMI admin credentials (options <code>--username</code> and <code>--password</code>, they default to <code>admin</code> and <code>password</code>), a custom libvirt URL and credentials if necessary (options <code>--libvirt-uri</code>, <code>--libvirt-sasl-username</code>, and <code>--libvirt-sasl-password</code>), and a custom IP address to listen on (defaults to all addresses, use option <code>--address</code> to restrict it).

<pre>
$ vbmc show controlplane1
+-----------------------+-------------------+
| Property | Value |
+-----------------------+-------------------+
| active | True |
| address | :: |
| domain_name | controlplane1 |
| libvirt_sasl_password | *** |
| libvirt_sasl_username | None |
| libvirt_uri | qemu:///system |
| password | *** |
| port | 6211 |
| status | running |
| username | admin |
+-----------------------+-------------------+
</pre>

Once started (which just opens the port) you can test the BMC connection using <code>ipmitool</code> or similar.

<pre>
$ ipmitool -I lanplus -H localhost -p 6211 -U admin -P password chassis status
System Power : off
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : always-off
Last Power Event :
Chassis Intrusion : inactive
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false
</pre>

That's it! We're ready to install OCP!

== Installer Configuration ==

This is where it all comes together. You need to execute the steps in this section (and next) on the ''provisioner'' host, that is, a system that has access to both external and provisioning networks of the OpenShift cluster-to-be. The access need not be direct, it can be routed, but if you, as in our example, configured the provisioning network to be an isolated virtual bridge, you will be best off by creating an additional VM that is directly connected to both bridges.

=== Gathering the Bits Together ===

First step is to make sure the following artifacts are available:

* <code>oc</code>
* <code>pull-secret</code>
* an SSH public key for compute node access

The global cluster network settings that we will need to configure are:

* the parent DNS domain of the cluster (such as <code>example.com</code>)
* the name of the cluster (concatenated with DNS domain, for example <code>mycluster</code> will become <code>mycluster.example.com</code>)
* the provisioning network CIDR (in our case, it can be any IP address block not overlapping with the external network as the provisioning network is isolated)
* from the external network address space, a designated:
** API server VIP (<code>api.mycluster.example.com</code> should point to it)
** ingress load balancer VIP (any host within <code>apps.mycluster.example.com</code> should resolve to it, usually via a wildcard record)

Additionally, you will need, for each node its:

* node name
* provisioning interface MAC address
* IPMI BMC address and port
* IPMI BMC credentials
* the name of the provisioning interface as seen from within the VM (re [[#Virtual Machine Configuration]] above - since the PCI address of the interface is ''bus 0x0 slot 0x3'' it will be named <code>enp0s3</code>)

As already said, DHCP address assignment to external interfaces is not managed by the installer. It must be handled by your infrastructure.

=== Preparing the Software ===

First, make sure your <code>pull-secret</code> file is on the provisioner.

<pre>
$ ls -l pull-secret
-rw-r-----. 1 provisioner provisioner 2734 Oct 27 12:21 pull-secret
</pre>

Then after downloading <code>oc</code> (if you didn't do that already)...

<pre>
$ curl -OL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/openshift-client-linux.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 60.9M 100 60.9M 0 0 4287k 0 0:00:14 0:00:14 --:--:-- 5642k

$ tar xf openshift-client-linux.tar.gz oc

$ ./oc version
Client Version: 4.14.9
Kustomize Version: v5.0.1

$ sudo cp ./oc /usr/local/bin/oc
$ rm -f ./oc
</pre>

...and optionally generating a bash completion file...

<pre>
$ oc completion bash > oc.completion
$ sudo cp oc.completion /etc/bash_completion.d/oc
$ rm -f oc.completion
$ source /etc/bash_completion.d/oc
</pre>

...you can use it to extract <code>openshift-baremetal-install</code> from the release image you intend to use.

<pre>
$ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/release.txt | grep "Pull From:"
Pull From: quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ oc adm release extract --registry-config=pull-secret --command=openshift-baremetal-install --to=. \
quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ ./openshift-baremetal-install version
./openshift-baremetal-install 4.14.9
built from commit dfafb5ca972a6ed4677257aebfe4f284ac020830
release image quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44
release architecture amd64

$ sudo cp ./openshift-baremetal-install /usr/local/bin/

$ openshift-baremetal-install completion bash > oinst.completion
$ sudo cp oinst.completion /etc/bash_completion.d/oinst
$ rm -f oinst.completion
</pre>

'''NOTE''': The extraction process can take up to 5 minutes, depending on your network speed and Quay.io responsiveness.

=== Creating Installer Configuration ===

Installer configuration file can be created interactively, using <code>openshift-baremetal-install</code>, or you can simply write it out with an editor.

==== Using Interactive Mode ====

'''IMPORTANT''': In interactive mode, machine network (the network segment the nodes have their IPs allocated from) is expected to be <code>10.0.0.0/16</code>. If your VIPs are not on that network, the installer will fail. If your machine network is not <code>10.0.0.0/16</code>, the only way to install is to create <code>install-config.yaml</code> manually.

'''IMPORTANT''': Interactive installer expects three control plane nodes and three workers. It will fail if you add less than three of each type of nodes to the cluster.

The interactive mode will ask you a series of questions and generate <code>install-config.yaml</code> at the end, but there are very few customisation options using this method.

Initially, you must select the SSH public key to be published on cluster nodes, and specify some general settings:

<pre>
$ openshift-baremetal-install --dir=cluster create install-config
? SSH Public Key /home/provisioner/.ssh/id_rsa.pub
? Platform baremetal
? Provisioning Network Managed
? Provisioning Network CIDR 172.22.0.0/24
? Provisioning bridge provbr0
? Provisioning Network Interface enp0s3
? External bridge bridge0
? Add a Host: [Use arrows to move, type to filter]
> control plane
worker
</pre>

As you can see above, after those have been answered, you may add any number of nodes, either control plane or worker.

For both type of nodes, the questions are the same:

<pre>
? Add a Host: control plane
? Name controlplane1
? BMC Address ipmi://192.168.1.1:6211
? BMC Username admin
? BMC Password ********
? Boot MAC Address 52:54:00:00:fb:11
? Add another host? (y/N) y
</pre>

After all the hosts have been added, there is a final series of cluster-related questions:

<pre>
? Add another host? No
? Base Domain example.com
? Cluster Name mycluster
? Pull Secret [? for help] ****************************...
</pre>

At this point, some checks are performed against DNS server to see if <code>api.mycluster.example.com</code> resolves, and if the IP is a part of the machine network. The same check is performed for the ingress VIP.

'''NOTE''': If your VIPs are not on the default machine network, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [platform.baremetal.apiVIPs: Invalid value: "172.25.3.10": IP expected to be in one of the machine networks: 10.0.0.0/16, platform.baremetal.ingressVIPs: Invalid value: "172.25.3.11": IP expected to be in one of the machine networks: 10.0.0.0/16]
</pre>

'''NOTE''': If you added fewer than three control plane nodes, or fewer than three workers, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [...]
</pre>

==== Creating Configuration Manually ====

The installer configuration file consists of a number of mandatory sections:

* cluster domain and name
* network settings (such as machine and cluster IP ranges)
* control plane and compute node settings
* infrastructure platform settings
* pull secret and ssh key

There are also some optional sections which are useful in special cases such as disconnected installation or special install modes.

'''NOTE''': There is an <code>explain</code> subcommand in the installer binary that explains the structure of the <code>installconfig</code> resource.

<pre>
$ openshift-baremetal-install explain installconfig
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
InstallConfig is the configuration for an OpenShift install.

FIELDS:
additionalTrustBundle <string>
AdditionalTrustBundle is a PEM-encoded X.509 certificate bundle that will be added to the nodes' trusted certificate store.
...

$ openshift-baremetal-install explain installconfig.platform.baremetal
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
BareMetal is the configuration used when installing on bare metal.

FIELDS:
apiVIP <string>
Format: ip
DeprecatedAPIVIP is the VIP to use for internal API communication Deprecated: Use APIVIPs

apiVIPs <[]string>
Format: ip
APIVIPs contains the VIP(s) to use for internal API communication. In dual stack clusters it contains an IPv4 and IPv6 address, otherwise only one VIP
...
</pre>

An example <code>install-config.yaml</code> for baremetal IPI looks like this:

<pre>
apiVersion: v1
baseDomain: example.com
metadata:
name: mycluster
networking:
networkType: OVNKubernetes
# These are the networks external IPs will be allocated from.
machineNetwork:
- cidr: 172.25.3.0/24
# This is the pod network.
clusterNetworks:
- cidr: 10.200.0.0/14
hostPrefix: 23
# Only one entry is supported.
serviceNetwork:
- 172.30.0.0/16
compute:
- name: worker
replicas: 2
controlPlane:
name: master
replicas: 3
platform:
baremetal: {}
platform:
baremetal:
apiVIPs:
- 172.25.3.10
ingressVIPs:
- 172.25.3.11
provisioningNetwork: Managed
provisioningNetworkCIDR: 10.1.1.0/24
provisioningDHCPRange: 10.1.1.200,10.1.1.210
# These settings are to configure the temporary bootstrap node as a VM
externalBridge: bridge0
externalMACAddress: '52:54:00:00:fa:0f'
bootstrapProvisioningIP: 10.1.1.9
provisioningBridge: provbr0
provisioningNetworkInterface: enp0s3
provisioningMACAddress: '52:54:00:00:fb:0f'
# This needs to be done to avoid nested virtualisation if provisioner is a VM.
libvirtURI: qemu+ssh://root@hypervisor.example.com/system
hosts:
- name: controlplane1
role: master
bmc:
address: ipmi://hypervisor.example.com:6211
disableCertificateVerification: true
username: admin
password: password
# This is the provisioning network interface.
bootMACAddress: 52:54:00:00:fb:11
bootMode: legacy
# We need this for proper targetting of root device (vda).
hardwareProfile: libvirt
- name: controlplane2
role: master
bmc:
address: ipmi://hypervisor.example.com:6212
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:12
bootMode: legacy
hardwareProfile: libvirt
- name: controlplane3
role: master
bmc:
address: ipmi://hypervisor.example.com:6213
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:13
bootMode: legacy
hardwareProfile: libvirt
- name: worker1
role: master
bmc:
address: ipmi://hypervisor.example.com:6221
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:21
bootMode: legacy
hardwareProfile: libvirt
- name: worker2
role: master
bmc:
address: ipmi://hypervisor.example.com:6222
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:22
bootMode: legacy
hardwareProfile: libvirt
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"...","email":"..."},"quay.io":{"auth":"...","email":"..."},...}}'
sshKey: "ssh-rsa ..."
</pre>

== Installation ==

== Post-Install Smoke Tests ==

= Conclusion =

OCP4-IPI-libvirt

2024-01-26T12:07:23Z

Gregab: /* Creating Configuration Manually */ switch from deprecated fields

= Introduction =

== What I Assume ==

* You know how OpenShift installation works and what the difference between IPI and UPI is.
* You know about the OpenShift Machine API and various underlying mechanisms.
* You understand the different types of network interfaces on Linux and different <code>libvirt</code> networks.
* You are familiar and comfortable with NetworkManager and the <code>nmcli</code> tool.
* You are familiar and comfortable with <code>libvirt</code> CLI and XML.
* You are familiar and comfortable with <code>qemu-img</code> tool.
* We are not talking about any firewall restrictions here - it is your responsibility to ensure traffic is not blocked.

== Outcomes ==

The installation described is for a fully managed IPI running OpenShift Container Platform v4.14, initially with three master and two worker nodes.

''At a later point, I will add a couple of steps needed to grow the cluster by one extra worker node.''

= OpenShift Container Platform IPI Installation Using Libvirt =

== Prerequisites ==

Hardware requirements for the cluster:

* 136 GiB RAM (32 GiB per control plane, 20 GiB per compute node), max overcommit ratio of 1.5 (make sure enough swap is available)
* 52 vCPUs (12 per control plane, 8 per compute node), max overcommit ratio of 1.3 (higher might work, but will slow down the installation horribly and may ultimately fail)
* one physical network interface that will be used for the public bridged network
* a physical or virtual network interface that will be used for the provisioning network bridge

Hardware requirements for the installation client (provisioner) machine:

* a minimum of 8 GiB RAM and 4 CPUs
* a network connection to both the public bridged network and the provisioning network

Due to the fact provisioner needs access to both networks, and the provisioning network in this guide is a virtual one, it might be best if you define the provisioner as a VM, with the same network interface settings as the control/compute nodes.

In the case you want to run the workloads spread across several hypervisor hosts, there are some extra steps, but nothing big. More on that in [[#Installation Spanning Multiple Hypervisors]] below.

Software artifacts needed on the provisioner host:

* <code>oc</code>, the command line client, of the corresponding version - download from https://mirror.openshift.com/pub/openshift-v4/clients/ocp/
* <code>libvirt-client</code> package is required for <code>openshift-baremetal-install</code> to be able to communicate to hypervisor(s)
* <code>ipmitool</code> or some other IPMI client
* a <code>pull-secret</code> file containing authentication credentials for OpenShift Container Platform registries - download from https://console.redhat.com/openshift/
* an SSH keypair that can be used for accessing OpenShift nodes

'''IMPORTANT''': The external IP addresses of cluster nodes must be assigned by your infrastructure DHCP server.

== Host Configuration ==

Beyond the logical requirement of having <code>libvirt</code> installed and started, here are the other configuration details for the hypervisor.

=== Network Settings ===

First thing you definitely need to make sure of, is that IP forwarding is enabled.

<pre>
$ sysctl net.ipv4.ip_forward
1
</pre>

Linux network settings need to be configured to have two '''Linux''' bridges, a public and a private provisioning one.

* public bridge, call it <code>bridge0</code>, needs to have the public network interface enslaved to it
* private bridge, call it <code>provbr0</code>, can be a virtual bridge since it is only needed for the provisioning network, which is supposed to be isolated and without any infrastructure services (such as DHCP, DNS, etc.)

It would be wonderful if the bridges could be OpenVSwitch ones, but unfortunately the Terraform bundled with <code>openshift-baremetal-install</code> currently does not include an OpenVSwitch provider, so there's goodbye to that.

As an example, here is my host configuration.

Public bridge:
<pre>
$ ip addr show bridge0
6: bridge0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff
inet 172.25.35.2/24 brd 172.25.35.255 scope global noprefixroute bridge0
valid_lft forever preferred_lft forever
inet6 fe80::4a21:bff:fe57:e06/64 scope link
valid_lft forever preferred_lft forever

$ ip addr show enp86s0
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master bridge0 state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff

$ bridge link | grep "master bridge0"
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master bridge0 state forwarding priority 32 cost 100
</pre>

Provisioning bridge:
<pre>
$ ip addr show provbr0
5: provbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ce:70:26:9c:88:a4 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.2/24 brd 10.1.1.255 scope global noprefixroute provbr0
valid_lft forever preferred_lft forever
inet6 fe80::cc70:26ff:fe9c:88a4/64 scope link
valid_lft forever preferred_lft forever

$ bridge link | grep "master provbr0"
</pre>

=== Installation Spanning Multiple Hypervisors ===

If you want to have your cluster spanning multiple hypervisors, make sure there is also a VXLAN connection between all the provisioning bridges.

You can do that by creating a <code>vxlan</code> type interface, which is a slave connection of type <code>bridge</code>, and the master is set to <code>provbr0</code>. Choose any unique VXLAN ID, and make sure it is the same on all interconnected hosts.

As an example, here is one VXLAN interface connecting hypervisor A to B.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1703164860
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.2
vxlan.remote: 172.25.35.3
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

And this is the corresponding VXLAN interface definition connecting host B to A.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1697549049
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.3
vxlan.remote: 172.25.35.2
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

In this case, <code>bridge link</code> will of course initially also show the <code>provbr0-vxlan10</code> interface as a slave, and will not show empty as above.

<pre>
$ bridge link | grep "master provbr0"
7: provbr0-vxlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master provbr0 state forwarding priority 32 cost 100
</pre>

=== <code>libvirt</code> Settings ===

Your <code>libvirt</code> will of course need to know about those network bridges in order to be able to attach VMs to them.

For that, you will need two network definitions, looking a bit like the following XML. Make sure they are autostart for least headache.

<pre>
$ sudo virsh net-dumpxml external
<network>
<name>external</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='bridge0'/>
</network>

$ sudo virsh net-dumpxml provisioning
<network>
<name>provisioning</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='provbr0'/>
</network>
</pre>

Additionally, you want to ensure that the storage pool is big enough, but that is not directly related to the subject at hand.

<pre>
$ sudo virsh pool-info default
Name: default
UUID: whatever
State: running
Persistent: yes
Autostart: yes
Capacity: 250.92 GiB
Allocation: 0 GiB
Available: 250.92 GiB
</pre>

=== VirtualBMC ===

The most important part of IPI facilitation is to be able to simulate a baseboard management controller for your VMs. <code>libvirt</code> obviously doesn't do this, but luckily there's a small bit of Python code that does, and it's called <code>virtualbmc</code>.

In most Python environments you can install it using <code>pip3</code>, just make sure <code>pip3</code> is up-to-date first.

<pre>
$ pip3 install --upgrade pip
...

$ pip3 install virtualbmc
...
</pre>

This gives you <code>/usr/local/bin/vbmcd</code> which you can control using the following systemd unit:

<pre>
[Unit]
Description=vbmcd
[Service]
Type=forking
ExecStart=/usr/local/bin/vbmcd
[Install]
WantedBy=multi-user.target
</pre>

Put the above content into <code>/etc/systemd/system/vbmcd.service</code>, reload systemd, and enable/start the service.

<pre>
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now vbmcd
</pre>

You now have the ability to associate a TCP port with a virtual machine defined on the hypervisor host, and have it simulate an IPMI BMC for that VM!

== Virtual Machine Configuration ==

=== VM Definitions ===

The virtual machines need to be configured with sufficient amount of compute resources, as per [[#Prerequisites]] above.

This section ties into the [[#Network Settings]] section above. You need two bridges on your hypervisor(s), <code>bridge0</code> and <code>provbr0</code>.

An example control plane node definition in <code>libvirt XML</code> would look like this:

<pre>
<domain type='kvm'>
<name>controlplane1</name>
<memory unit='GiB'>32</memory>
<currentMemory unit='GiB'>32</currentMemory>
<vcpu placement='static'>12</vcpu>
<os>
<type arch='x86_64' machine='q35'>hvm</type>
<boot dev='hd'/>
<boot dev='network'/>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model' check='partial'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/controlplane1-vda.qcow2'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'/>
<controller type='pci' index='0' model='pcie-root'/>
<interface type='bridge'>
<mac address='52:54:00:00:fb:11'/>
<source bridge='provbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:00:fa:11'/>
<source bridge='bridge0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<console type='pty'/>
<channel type='unix'>
<source mode='bind'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' autoport='yes' listen='0.0.0.0'/>
<video>
<model type='virtio'/>
</video>
<memballoon model='virtio'/>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
</rng>
</devices>
</domain>
</pre>

A couple of things to note:

* the first network interface is attached to <code>provbr0</code>, and its PCI address is ''lower'' (0x03), causing it to be the PXE default device
* the second network interface is attached to <code>bridge0</code>, and its PCI address is ''higher'' (0x09), making it the second interface (for external connections)
* boot order is set to hard disk first, network second, which means the host will only PXE boot if the disk image is unbootable
* the disk image needs to be 64GiB in size at the minimum, but you can make it larger and/or add more disk images if you intend to use the local storage operator

When configuring other nodes, simply remember to change the node name, disk image name, and MAC addresses to be unique. Adjust hardware resources accordingly for compute nodes.

=== IPMI BMC ===

What is important for OCP IPI to be able to perform the installation properly is to register each virtual machine with <code>vbmcd</code> and assign it with a port.

<pre>
$ vbmc add --port=6211 controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | down | :: | 6211 |
+-------------------+---------+---------+------+
$ vbmc start controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | running | :: | 6211 |
+-------------------+---------+---------+------+
$ sudo ss -aunp | grep 6211
UNCONN 0 0 *:6211 *:* users:(("vbmcd",pid=766290,fd=21))
</pre>

There is no need to run the <code>vbmc</code> client as root as it is the daemon that is running as root and can see all the VMs accessible through the <code>qemu:///system</code> URL.

Of course there are options. For any VM you add, you can specify a custom set of IPMI admin credentials (options <code>--username</code> and <code>--password</code>, they default to <code>admin</code> and <code>password</code>), a custom libvirt URL and credentials if necessary (options <code>--libvirt-uri</code>, <code>--libvirt-sasl-username</code>, and <code>--libvirt-sasl-password</code>), and a custom IP address to listen on (defaults to all addresses, use option <code>--address</code> to restrict it).

<pre>
$ vbmc show controlplane1
+-----------------------+-------------------+
| Property | Value |
+-----------------------+-------------------+
| active | True |
| address | :: |
| domain_name | controlplane1 |
| libvirt_sasl_password | *** |
| libvirt_sasl_username | None |
| libvirt_uri | qemu:///system |
| password | *** |
| port | 6211 |
| status | running |
| username | admin |
+-----------------------+-------------------+
</pre>

Once started (which just opens the port) you can test the BMC connection using <code>ipmitool</code> or similar.

<pre>
$ ipmitool -I lanplus -H localhost -p 6211 -U admin -P password chassis status
System Power : off
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : always-off
Last Power Event :
Chassis Intrusion : inactive
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false
</pre>

That's it! We're ready to install OCP!

== Installer Configuration ==

This is where it all comes together. You need to execute the steps in this section (and next) on the ''provisioner'' host, that is, a system that has access to both external and provisioning networks of the OpenShift cluster-to-be. The access need not be direct, it can be routed, but if you, as in our example, configured the provisioning network to be an isolated virtual bridge, you will be best off by creating an additional VM that is directly connected to both bridges.

=== Gathering the Bits Together ===

First step is to make sure the following artifacts are available:

* <code>oc</code>
* <code>pull-secret</code>
* an SSH public key for compute node access

The global cluster network settings that we will need to configure are:

* the parent DNS domain of the cluster (such as <code>example.com</code>)
* the name of the cluster (concatenated with DNS domain, for example <code>mycluster</code> will become <code>mycluster.example.com</code>)
* the provisioning network CIDR (in our case, it can be any IP address block not overlapping with the external network as the provisioning network is isolated)
* from the external network address space, a designated:
** API server VIP (<code>api.mycluster.example.com</code> should point to it)
** ingress load balancer VIP (any host within <code>apps.mycluster.example.com</code> should resolve to it, usually via a wildcard record)

Additionally, you will need, for each node its:

* node name
* provisioning interface MAC address
* IPMI BMC address and port
* IPMI BMC credentials
* the name of the provisioning interface as seen from within the VM (re [[#Virtual Machine Configuration]] above - since the PCI address of the interface is ''bus 0x0 slot 0x3'' it will be named <code>enp0s3</code>)

As already said, DHCP address assignment to external interfaces is not managed by the installer. It must be handled by your infrastructure.

=== Preparing the Software ===

First, make sure your <code>pull-secret</code> file is on the provisioner.

<pre>
$ ls -l pull-secret
-rw-r-----. 1 provisioner provisioner 2734 Oct 27 12:21 pull-secret
</pre>

Then after downloading <code>oc</code> (if you didn't do that already)...

<pre>
$ curl -OL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/openshift-client-linux.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 60.9M 100 60.9M 0 0 4287k 0 0:00:14 0:00:14 --:--:-- 5642k

$ tar xf openshift-client-linux.tar.gz oc

$ ./oc version
Client Version: 4.14.9
Kustomize Version: v5.0.1

$ sudo cp ./oc /usr/local/bin/oc
$ rm -f ./oc
</pre>

...and optionally generating a bash completion file...

<pre>
$ oc completion bash > oc.completion
$ sudo cp oc.completion /etc/bash_completion.d/oc
$ rm -f oc.completion
$ source /etc/bash_completion.d/oc
</pre>

...you can use it to extract <code>openshift-baremetal-install</code> from the release image you intend to use.

<pre>
$ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/release.txt | grep "Pull From:"
Pull From: quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ oc adm release extract --registry-config=pull-secret --command=openshift-baremetal-install --to=. \
quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ ./openshift-baremetal-install version
./openshift-baremetal-install 4.14.9
built from commit dfafb5ca972a6ed4677257aebfe4f284ac020830
release image quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44
release architecture amd64

$ sudo cp ./openshift-baremetal-install /usr/local/bin/

$ openshift-baremetal-install completion bash > oinst.completion
$ sudo cp oinst.completion /etc/bash_completion.d/oinst
$ rm -f oinst.completion
</pre>

'''NOTE''': The extraction process can take up to 5 minutes, depending on your network speed and Quay.io responsiveness.

=== Creating Installer Configuration ===

Installer configuration file can be created interactively, using <code>openshift-baremetal-install</code>, or you can simply write it out with an editor.

==== Using Interactive Mode ====

'''IMPORTANT''': In interactive mode, machine network (the network segment the nodes have their IPs allocated from) is expected to be <code>10.0.0.0/16</code>. If your VIPs are not on that network, the installer will fail. If your machine network is not <code>10.0.0.0/16</code>, the only way to install is to create <code>install-config.yaml</code> manually.

'''IMPORTANT''': Interactive installer expects three control plane nodes and three workers. It will fail if you add less than three of each type of nodes to the cluster.

The interactive mode will ask you a series of questions and generate <code>install-config.yaml</code> at the end, but there are very few customisation options using this method.

Initially, you must select the SSH public key to be published on cluster nodes, and specify some general settings:

<pre>
$ openshift-baremetal-install --dir=cluster create install-config
? SSH Public Key /home/provisioner/.ssh/id_rsa.pub
? Platform baremetal
? Provisioning Network Managed
? Provisioning Network CIDR 172.22.0.0/24
? Provisioning bridge provbr0
? Provisioning Network Interface enp0s3
? External bridge bridge0
? Add a Host: [Use arrows to move, type to filter]
> control plane
worker
</pre>

As you can see above, after those have been answered, you may add any number of nodes, either control plane or worker.

For both type of nodes, the questions are the same:

<pre>
? Add a Host: control plane
? Name controlplane1
? BMC Address ipmi://192.168.1.1:6211
? BMC Username admin
? BMC Password ********
? Boot MAC Address 52:54:00:00:fb:11
? Add another host? (y/N) y
</pre>

After all the hosts have been added, there is a final series of cluster-related questions:

<pre>
? Add another host? No
? Base Domain example.com
? Cluster Name mycluster
? Pull Secret [? for help] ****************************...
</pre>

At this point, some checks are performed against DNS server to see if <code>api.mycluster.example.com</code> resolves, and if the IP is a part of the machine network. The same check is performed for the ingress VIP.

'''NOTE''': If your VIPs are not on the default machine network, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [platform.baremetal.apiVIPs: Invalid value: "172.25.3.10": IP expected to be in one of the machine networks: 10.0.0.0/16, platform.baremetal.ingressVIPs: Invalid value: "172.25.3.11": IP expected to be in one of the machine networks: 10.0.0.0/16]
</pre>

'''NOTE''': If you added fewer than three control plane nodes, or fewer than three workers, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [...]
</pre>

==== Creating Configuration Manually ====

The installer configuration file consists of a number of mandatory sections:

* cluster domain and name
* network settings (such as machine and cluster IP ranges)
* control plane and compute node settings
* infrastructure platform settings
* pull secret and ssh key

There are also some optional sections which are useful in special cases such as disconnected installation or special install modes.

'''NOTE''': There is an <code>explain</code> subcommand in the installer binary that explains the structure of the <code>installconfig</code> resource.

<pre>
$ openshift-baremetal-install explain installconfig
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
InstallConfig is the configuration for an OpenShift install.

FIELDS:
additionalTrustBundle <string>
AdditionalTrustBundle is a PEM-encoded X.509 certificate bundle that will be added to the nodes' trusted certificate store.
...

$ openshift-baremetal-install explain installconfig.platform.baremetal
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
BareMetal is the configuration used when installing on bare metal.

FIELDS:
apiVIP <string>
Format: ip
DeprecatedAPIVIP is the VIP to use for internal API communication Deprecated: Use APIVIPs

apiVIPs <[]string>
Format: ip
APIVIPs contains the VIP(s) to use for internal API communication. In dual stack clusters it contains an IPv4 and IPv6 address, otherwise only one VIP
...
</pre>

An example <code>install-config.yaml</code> for baremetal IPI looks like this:

<pre>
apiVersion: v1
baseDomain: example.com
metadata:
name: mycluster
networking:
networkType: OVNKubernetes
# These are the networks external IPs will be allocated from.
machineNetwork:
- 172.25.3.0/24
# This is the pod network.
clusterNetworks:
- cidr: 10.200.0.0/14
hostPrefix: 23
# Only one entry is supported.
serviceNetwork:
- 172.30.0.0/16
compute:
- name: worker
replicas: 2
controlPlane:
name: master
replicas: 3
platform:
baremetal: {}
platform:
baremetal:
apiVIPs:
- 172.25.3.10
ingressVIPs:
- 172.25.3.11
provisioningNetwork: Managed
provisioningNetworkCIDR: 10.1.1.0/24
provisioningDHCPRange: 10.1.1.200,10.1.1.210
# These settings are to configure the temporary bootstrap node as a VM
externalBridge: bridge0
externalMACAddress: '52:54:00:00:fa:0f'
bootstrapProvisioningIP: 10.1.1.9
provisioningBridge: provbr0
provisioningNetworkInterface: enp0s3
provisioningMACAddress: '52:54:00:00:fb:0f'
# This needs to be done to avoid nested virtualisation if provisioner is a VM.
libvirtURI: qemu+ssh://root@hypervisor.example.com/system
hosts:
- name: controlplane1
role: master
bmc:
address: ipmi://hypervisor.example.com:6211
disableCertificateVerification: true
username: admin
password: password
# This is the provisioning network interface.
bootMACAddress: 52:54:00:00:fb:11
bootMode: legacy
# We need this for proper targetting of root device (vda).
hardwareProfile: libvirt
- name: controlplane2
role: master
bmc:
address: ipmi://hypervisor.example.com:6212
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:12
bootMode: legacy
hardwareProfile: libvirt
- name: controlplane3
role: master
bmc:
address: ipmi://hypervisor.example.com:6213
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:13
bootMode: legacy
hardwareProfile: libvirt
- name: worker1
role: master
bmc:
address: ipmi://hypervisor.example.com:6221
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:21
bootMode: legacy
hardwareProfile: libvirt
- name: worker2
role: master
bmc:
address: ipmi://hypervisor.example.com:6222
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:22
bootMode: legacy
hardwareProfile: libvirt
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"...","email":"..."},"quay.io":{"auth":"...","email":"..."},...}}'
sshKey: "ssh-rsa ..."
</pre>

== Installation ==

== Post-Install Smoke Tests ==

= Conclusion =

OCP4-IPI-libvirt

2024-01-26T11:01:20Z

Gregab: /* Creating Configuration Manually */ added install-config.yaml

= Introduction =

== What I Assume ==

* You know how OpenShift installation works and what the difference between IPI and UPI is.
* You know about the OpenShift Machine API and various underlying mechanisms.
* You understand the different types of network interfaces on Linux and different <code>libvirt</code> networks.
* You are familiar and comfortable with NetworkManager and the <code>nmcli</code> tool.
* You are familiar and comfortable with <code>libvirt</code> CLI and XML.
* You are familiar and comfortable with <code>qemu-img</code> tool.
* We are not talking about any firewall restrictions here - it is your responsibility to ensure traffic is not blocked.

== Outcomes ==

The installation described is for a fully managed IPI running OpenShift Container Platform v4.14, initially with three master and two worker nodes.

''At a later point, I will add a couple of steps needed to grow the cluster by one extra worker node.''

= OpenShift Container Platform IPI Installation Using Libvirt =

== Prerequisites ==

Hardware requirements for the cluster:

* 136 GiB RAM (32 GiB per control plane, 20 GiB per compute node), max overcommit ratio of 1.5 (make sure enough swap is available)
* 52 vCPUs (12 per control plane, 8 per compute node), max overcommit ratio of 1.3 (higher might work, but will slow down the installation horribly and may ultimately fail)
* one physical network interface that will be used for the public bridged network
* a physical or virtual network interface that will be used for the provisioning network bridge

Hardware requirements for the installation client (provisioner) machine:

* a minimum of 8 GiB RAM and 4 CPUs
* a network connection to both the public bridged network and the provisioning network

Due to the fact provisioner needs access to both networks, and the provisioning network in this guide is a virtual one, it might be best if you define the provisioner as a VM, with the same network interface settings as the control/compute nodes.

In the case you want to run the workloads spread across several hypervisor hosts, there are some extra steps, but nothing big. More on that in [[#Installation Spanning Multiple Hypervisors]] below.

Software artifacts needed on the provisioner host:

* <code>oc</code>, the command line client, of the corresponding version - download from https://mirror.openshift.com/pub/openshift-v4/clients/ocp/
* <code>libvirt-client</code> package is required for <code>openshift-baremetal-install</code> to be able to communicate to hypervisor(s)
* <code>ipmitool</code> or some other IPMI client
* a <code>pull-secret</code> file containing authentication credentials for OpenShift Container Platform registries - download from https://console.redhat.com/openshift/
* an SSH keypair that can be used for accessing OpenShift nodes

'''IMPORTANT''': The external IP addresses of cluster nodes must be assigned by your infrastructure DHCP server.

== Host Configuration ==

Beyond the logical requirement of having <code>libvirt</code> installed and started, here are the other configuration details for the hypervisor.

=== Network Settings ===

First thing you definitely need to make sure of, is that IP forwarding is enabled.

<pre>
$ sysctl net.ipv4.ip_forward
1
</pre>

Linux network settings need to be configured to have two '''Linux''' bridges, a public and a private provisioning one.

* public bridge, call it <code>bridge0</code>, needs to have the public network interface enslaved to it
* private bridge, call it <code>provbr0</code>, can be a virtual bridge since it is only needed for the provisioning network, which is supposed to be isolated and without any infrastructure services (such as DHCP, DNS, etc.)

It would be wonderful if the bridges could be OpenVSwitch ones, but unfortunately the Terraform bundled with <code>openshift-baremetal-install</code> currently does not include an OpenVSwitch provider, so there's goodbye to that.

As an example, here is my host configuration.

Public bridge:
<pre>
$ ip addr show bridge0
6: bridge0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff
inet 172.25.35.2/24 brd 172.25.35.255 scope global noprefixroute bridge0
valid_lft forever preferred_lft forever
inet6 fe80::4a21:bff:fe57:e06/64 scope link
valid_lft forever preferred_lft forever

$ ip addr show enp86s0
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master bridge0 state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff

$ bridge link | grep "master bridge0"
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master bridge0 state forwarding priority 32 cost 100
</pre>

Provisioning bridge:
<pre>
$ ip addr show provbr0
5: provbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ce:70:26:9c:88:a4 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.2/24 brd 10.1.1.255 scope global noprefixroute provbr0
valid_lft forever preferred_lft forever
inet6 fe80::cc70:26ff:fe9c:88a4/64 scope link
valid_lft forever preferred_lft forever

$ bridge link | grep "master provbr0"
</pre>

=== Installation Spanning Multiple Hypervisors ===

If you want to have your cluster spanning multiple hypervisors, make sure there is also a VXLAN connection between all the provisioning bridges.

You can do that by creating a <code>vxlan</code> type interface, which is a slave connection of type <code>bridge</code>, and the master is set to <code>provbr0</code>. Choose any unique VXLAN ID, and make sure it is the same on all interconnected hosts.

As an example, here is one VXLAN interface connecting hypervisor A to B.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1703164860
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.2
vxlan.remote: 172.25.35.3
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

And this is the corresponding VXLAN interface definition connecting host B to A.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1697549049
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.3
vxlan.remote: 172.25.35.2
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

In this case, <code>bridge link</code> will of course initially also show the <code>provbr0-vxlan10</code> interface as a slave, and will not show empty as above.

<pre>
$ bridge link | grep "master provbr0"
7: provbr0-vxlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master provbr0 state forwarding priority 32 cost 100
</pre>

=== <code>libvirt</code> Settings ===

Your <code>libvirt</code> will of course need to know about those network bridges in order to be able to attach VMs to them.

For that, you will need two network definitions, looking a bit like the following XML. Make sure they are autostart for least headache.

<pre>
$ sudo virsh net-dumpxml external
<network>
<name>external</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='bridge0'/>
</network>

$ sudo virsh net-dumpxml provisioning
<network>
<name>provisioning</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='provbr0'/>
</network>
</pre>

Additionally, you want to ensure that the storage pool is big enough, but that is not directly related to the subject at hand.

<pre>
$ sudo virsh pool-info default
Name: default
UUID: whatever
State: running
Persistent: yes
Autostart: yes
Capacity: 250.92 GiB
Allocation: 0 GiB
Available: 250.92 GiB
</pre>

=== VirtualBMC ===

The most important part of IPI facilitation is to be able to simulate a baseboard management controller for your VMs. <code>libvirt</code> obviously doesn't do this, but luckily there's a small bit of Python code that does, and it's called <code>virtualbmc</code>.

In most Python environments you can install it using <code>pip3</code>, just make sure <code>pip3</code> is up-to-date first.

<pre>
$ pip3 install --upgrade pip
...

$ pip3 install virtualbmc
...
</pre>

This gives you <code>/usr/local/bin/vbmcd</code> which you can control using the following systemd unit:

<pre>
[Unit]
Description=vbmcd
[Service]
Type=forking
ExecStart=/usr/local/bin/vbmcd
[Install]
WantedBy=multi-user.target
</pre>

Put the above content into <code>/etc/systemd/system/vbmcd.service</code>, reload systemd, and enable/start the service.

<pre>
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now vbmcd
</pre>

You now have the ability to associate a TCP port with a virtual machine defined on the hypervisor host, and have it simulate an IPMI BMC for that VM!

== Virtual Machine Configuration ==

=== VM Definitions ===

The virtual machines need to be configured with sufficient amount of compute resources, as per [[#Prerequisites]] above.

This section ties into the [[#Network Settings]] section above. You need two bridges on your hypervisor(s), <code>bridge0</code> and <code>provbr0</code>.

An example control plane node definition in <code>libvirt XML</code> would look like this:

<pre>
<domain type='kvm'>
<name>controlplane1</name>
<memory unit='GiB'>32</memory>
<currentMemory unit='GiB'>32</currentMemory>
<vcpu placement='static'>12</vcpu>
<os>
<type arch='x86_64' machine='q35'>hvm</type>
<boot dev='hd'/>
<boot dev='network'/>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model' check='partial'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/controlplane1-vda.qcow2'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'/>
<controller type='pci' index='0' model='pcie-root'/>
<interface type='bridge'>
<mac address='52:54:00:00:fb:11'/>
<source bridge='provbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:00:fa:11'/>
<source bridge='bridge0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<console type='pty'/>
<channel type='unix'>
<source mode='bind'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' autoport='yes' listen='0.0.0.0'/>
<video>
<model type='virtio'/>
</video>
<memballoon model='virtio'/>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
</rng>
</devices>
</domain>
</pre>

A couple of things to note:

* the first network interface is attached to <code>provbr0</code>, and its PCI address is ''lower'' (0x03), causing it to be the PXE default device
* the second network interface is attached to <code>bridge0</code>, and its PCI address is ''higher'' (0x09), making it the second interface (for external connections)
* boot order is set to hard disk first, network second, which means the host will only PXE boot if the disk image is unbootable
* the disk image needs to be 64GiB in size at the minimum, but you can make it larger and/or add more disk images if you intend to use the local storage operator

When configuring other nodes, simply remember to change the node name, disk image name, and MAC addresses to be unique. Adjust hardware resources accordingly for compute nodes.

=== IPMI BMC ===

What is important for OCP IPI to be able to perform the installation properly is to register each virtual machine with <code>vbmcd</code> and assign it with a port.

<pre>
$ vbmc add --port=6211 controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | down | :: | 6211 |
+-------------------+---------+---------+------+
$ vbmc start controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | running | :: | 6211 |
+-------------------+---------+---------+------+
$ sudo ss -aunp | grep 6211
UNCONN 0 0 *:6211 *:* users:(("vbmcd",pid=766290,fd=21))
</pre>

There is no need to run the <code>vbmc</code> client as root as it is the daemon that is running as root and can see all the VMs accessible through the <code>qemu:///system</code> URL.

Of course there are options. For any VM you add, you can specify a custom set of IPMI admin credentials (options <code>--username</code> and <code>--password</code>, they default to <code>admin</code> and <code>password</code>), a custom libvirt URL and credentials if necessary (options <code>--libvirt-uri</code>, <code>--libvirt-sasl-username</code>, and <code>--libvirt-sasl-password</code>), and a custom IP address to listen on (defaults to all addresses, use option <code>--address</code> to restrict it).

<pre>
$ vbmc show controlplane1
+-----------------------+-------------------+
| Property | Value |
+-----------------------+-------------------+
| active | True |
| address | :: |
| domain_name | controlplane1 |
| libvirt_sasl_password | *** |
| libvirt_sasl_username | None |
| libvirt_uri | qemu:///system |
| password | *** |
| port | 6211 |
| status | running |
| username | admin |
+-----------------------+-------------------+
</pre>

Once started (which just opens the port) you can test the BMC connection using <code>ipmitool</code> or similar.

<pre>
$ ipmitool -I lanplus -H localhost -p 6211 -U admin -P password chassis status
System Power : off
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : always-off
Last Power Event :
Chassis Intrusion : inactive
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false
</pre>

That's it! We're ready to install OCP!

== Installer Configuration ==

This is where it all comes together. You need to execute the steps in this section (and next) on the ''provisioner'' host, that is, a system that has access to both external and provisioning networks of the OpenShift cluster-to-be. The access need not be direct, it can be routed, but if you, as in our example, configured the provisioning network to be an isolated virtual bridge, you will be best off by creating an additional VM that is directly connected to both bridges.

=== Gathering the Bits Together ===

First step is to make sure the following artifacts are available:

* <code>oc</code>
* <code>pull-secret</code>
* an SSH public key for compute node access

The global cluster network settings that we will need to configure are:

* the parent DNS domain of the cluster (such as <code>example.com</code>)
* the name of the cluster (concatenated with DNS domain, for example <code>mycluster</code> will become <code>mycluster.example.com</code>)
* the provisioning network CIDR (in our case, it can be any IP address block not overlapping with the external network as the provisioning network is isolated)
* from the external network address space, a designated:
** API server VIP (<code>api.mycluster.example.com</code> should point to it)
** ingress load balancer VIP (any host within <code>apps.mycluster.example.com</code> should resolve to it, usually via a wildcard record)

Additionally, you will need, for each node its:

* node name
* provisioning interface MAC address
* IPMI BMC address and port
* IPMI BMC credentials
* the name of the provisioning interface as seen from within the VM (re [[#Virtual Machine Configuration]] above - since the PCI address of the interface is ''bus 0x0 slot 0x3'' it will be named <code>enp0s3</code>)

As already said, DHCP address assignment to external interfaces is not managed by the installer. It must be handled by your infrastructure.

=== Preparing the Software ===

First, make sure your <code>pull-secret</code> file is on the provisioner.

<pre>
$ ls -l pull-secret
-rw-r-----. 1 provisioner provisioner 2734 Oct 27 12:21 pull-secret
</pre>

Then after downloading <code>oc</code> (if you didn't do that already)...

<pre>
$ curl -OL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/openshift-client-linux.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 60.9M 100 60.9M 0 0 4287k 0 0:00:14 0:00:14 --:--:-- 5642k

$ tar xf openshift-client-linux.tar.gz oc

$ ./oc version
Client Version: 4.14.9
Kustomize Version: v5.0.1

$ sudo cp ./oc /usr/local/bin/oc
$ rm -f ./oc
</pre>

...and optionally generating a bash completion file...

<pre>
$ oc completion bash > oc.completion
$ sudo cp oc.completion /etc/bash_completion.d/oc
$ rm -f oc.completion
$ source /etc/bash_completion.d/oc
</pre>

...you can use it to extract <code>openshift-baremetal-install</code> from the release image you intend to use.

<pre>
$ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/release.txt | grep "Pull From:"
Pull From: quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ oc adm release extract --registry-config=pull-secret --command=openshift-baremetal-install --to=. \
quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ ./openshift-baremetal-install version
./openshift-baremetal-install 4.14.9
built from commit dfafb5ca972a6ed4677257aebfe4f284ac020830
release image quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44
release architecture amd64

$ sudo cp ./openshift-baremetal-install /usr/local/bin/

$ openshift-baremetal-install completion bash > oinst.completion
$ sudo cp oinst.completion /etc/bash_completion.d/oinst
$ rm -f oinst.completion
</pre>

'''NOTE''': The extraction process can take up to 5 minutes, depending on your network speed and Quay.io responsiveness.

=== Creating Installer Configuration ===

Installer configuration file can be created interactively, using <code>openshift-baremetal-install</code>, or you can simply write it out with an editor.

==== Using Interactive Mode ====

'''IMPORTANT''': In interactive mode, machine network (the network segment the nodes have their IPs allocated from) is expected to be <code>10.0.0.0/16</code>. If your VIPs are not on that network, the installer will fail. If your machine network is not <code>10.0.0.0/16</code>, the only way to install is to create <code>install-config.yaml</code> manually.

'''IMPORTANT''': Interactive installer expects three control plane nodes and three workers. It will fail if you add less than three of each type of nodes to the cluster.

The interactive mode will ask you a series of questions and generate <code>install-config.yaml</code> at the end, but there are very few customisation options using this method.

Initially, you must select the SSH public key to be published on cluster nodes, and specify some general settings:

<pre>
$ openshift-baremetal-install --dir=cluster create install-config
? SSH Public Key /home/provisioner/.ssh/id_rsa.pub
? Platform baremetal
? Provisioning Network Managed
? Provisioning Network CIDR 172.22.0.0/24
? Provisioning bridge provbr0
? Provisioning Network Interface enp0s3
? External bridge bridge0
? Add a Host: [Use arrows to move, type to filter]
> control plane
worker
</pre>

As you can see above, after those have been answered, you may add any number of nodes, either control plane or worker.

For both type of nodes, the questions are the same:

<pre>
? Add a Host: control plane
? Name controlplane1
? BMC Address ipmi://192.168.1.1:6211
? BMC Username admin
? BMC Password ********
? Boot MAC Address 52:54:00:00:fb:11
? Add another host? (y/N) y
</pre>

After all the hosts have been added, there is a final series of cluster-related questions:

<pre>
? Add another host? No
? Base Domain example.com
? Cluster Name mycluster
? Pull Secret [? for help] ****************************...
</pre>

At this point, some checks are performed against DNS server to see if <code>api.mycluster.example.com</code> resolves, and if the IP is a part of the machine network. The same check is performed for the ingress VIP.

'''NOTE''': If your VIPs are not on the default machine network, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [platform.baremetal.apiVIPs: Invalid value: "172.25.3.10": IP expected to be in one of the machine networks: 10.0.0.0/16, platform.baremetal.ingressVIPs: Invalid value: "172.25.3.11": IP expected to be in one of the machine networks: 10.0.0.0/16]
</pre>

'''NOTE''': If you added fewer than three control plane nodes, or fewer than three workers, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [...]
</pre>

==== Creating Configuration Manually ====

The installer configuration file consists of a number of mandatory sections:

* cluster domain and name
* network settings (such as machine and cluster IP ranges)
* control plane and compute node settings
* infrastructure platform settings
* pull secret and ssh key

There are also some optional sections which are useful in special cases such as disconnected installation or special install modes.

'''NOTE''': There is an <code>explain</code> subcommand in the installer binary that explains the structure of the <code>installconfig</code> resource.

<pre>
$ openshift-baremetal-install explain installconfig
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
InstallConfig is the configuration for an OpenShift install.

FIELDS:
additionalTrustBundle <string>
AdditionalTrustBundle is a PEM-encoded X.509 certificate bundle that will be added to the nodes' trusted certificate store.
...

$ openshift-baremetal-install explain installconfig.platform.baremetal
KIND: InstallConfig
VERSION: v1

RESOURCE: <object>
BareMetal is the configuration used when installing on bare metal.

FIELDS:
apiVIP <string>
Format: ip
DeprecatedAPIVIP is the VIP to use for internal API communication Deprecated: Use APIVIPs

apiVIPs <[]string>
Format: ip
APIVIPs contains the VIP(s) to use for internal API communication. In dual stack clusters it contains an IPv4 and IPv6 address, otherwise only one VIP
...
</pre>

An example <code>install-config.yaml</code> for baremetal IPI looks like this:

<pre>
apiVersion: v1
baseDomain: example.com
metadata:
name: mycluster
networking:
machineCIDR: 172.25.3.0/24
networkType: OVNKubernetes
clusterNetwork:
- cidr: 10.200.0.0/14
hostPrefix: 23
compute:
- name: worker
replicas: 2
controlPlane:
name: master
replicas: 3
platform:
baremetal: {}
platform:
baremetal:
apiVIPs:
- 172.25.3.10
ingressVIPs:
- 172.25.3.11
provisioningNetwork: Managed
provisioningNetworkCIDR: 10.1.1.0/24
provisioningDHCPRange: 10.1.1.200,10.1.1.210
# These settings are to configure the temporary bootstrap node as a VM
externalBridge: bridge0
externalMACAddress: '52:54:00:00:fa:0f'
bootstrapProvisioningIP: 10.1.1.9
provisioningBridge: provbr0
provisioningNetworkInterface: enp0s3
provisioningMACAddress: '52:54:00:00:fb:0f'
# This needs to be done to avoid nested virtualisation if provisioner is a VM.
libvirtURI: qemu+ssh://root@hypervisor.example.com/system
hosts:
- name: controlplane1
role: master
bmc:
address: ipmi://hypervisor.example.com:6211
disableCertificateVerification: true
username: admin
password: password
# This is the provisioning network interface.
bootMACAddress: 52:54:00:00:fb:11
bootMode: legacy
# We need this for proper targetting of root device (vda).
hardwareProfile: libvirt
- name: controlplane2
role: master
bmc:
address: ipmi://hypervisor.example.com:6212
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:12
bootMode: legacy
hardwareProfile: libvirt
- name: controlplane3
role: master
bmc:
address: ipmi://hypervisor.example.com:6213
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:13
bootMode: legacy
hardwareProfile: libvirt
- name: worker1
role: master
bmc:
address: ipmi://hypervisor.example.com:6221
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:21
bootMode: legacy
hardwareProfile: libvirt
- name: worker2
role: master
bmc:
address: ipmi://hypervisor.example.com:6222
disableCertificateVerification: true
username: admin
password: password
bootMACAddress: 52:54:00:00:fb:22
bootMode: legacy
hardwareProfile: libvirt
pullSecret: '{"auths":{"cloud.openshift.com":{"auth":"...","email":"..."},"quay.io":{"auth":"...","email":"..."},...}}'
sshKey: "ssh-rsa ..."
</pre>

== Installation ==

== Post-Install Smoke Tests ==

= Conclusion =

OCP4-IPI-libvirt

2024-01-26T10:48:17Z

Gregab: /* Installer Configuration */ fix hostnames, IPs, and macs

= Introduction =

== What I Assume ==

* You know how OpenShift installation works and what the difference between IPI and UPI is.
* You know about the OpenShift Machine API and various underlying mechanisms.
* You understand the different types of network interfaces on Linux and different <code>libvirt</code> networks.
* You are familiar and comfortable with NetworkManager and the <code>nmcli</code> tool.
* You are familiar and comfortable with <code>libvirt</code> CLI and XML.
* You are familiar and comfortable with <code>qemu-img</code> tool.
* We are not talking about any firewall restrictions here - it is your responsibility to ensure traffic is not blocked.

== Outcomes ==

The installation described is for a fully managed IPI running OpenShift Container Platform v4.14, initially with three master and two worker nodes.

''At a later point, I will add a couple of steps needed to grow the cluster by one extra worker node.''

= OpenShift Container Platform IPI Installation Using Libvirt =

== Prerequisites ==

Hardware requirements for the cluster:

* 136 GiB RAM (32 GiB per control plane, 20 GiB per compute node), max overcommit ratio of 1.5 (make sure enough swap is available)
* 52 vCPUs (12 per control plane, 8 per compute node), max overcommit ratio of 1.3 (higher might work, but will slow down the installation horribly and may ultimately fail)
* one physical network interface that will be used for the public bridged network
* a physical or virtual network interface that will be used for the provisioning network bridge

Hardware requirements for the installation client (provisioner) machine:

* a minimum of 8 GiB RAM and 4 CPUs
* a network connection to both the public bridged network and the provisioning network

Due to the fact provisioner needs access to both networks, and the provisioning network in this guide is a virtual one, it might be best if you define the provisioner as a VM, with the same network interface settings as the control/compute nodes.

In the case you want to run the workloads spread across several hypervisor hosts, there are some extra steps, but nothing big. More on that in [[#Installation Spanning Multiple Hypervisors]] below.

Software artifacts needed on the provisioner host:

* <code>oc</code>, the command line client, of the corresponding version - download from https://mirror.openshift.com/pub/openshift-v4/clients/ocp/
* <code>libvirt-client</code> package is required for <code>openshift-baremetal-install</code> to be able to communicate to hypervisor(s)
* <code>ipmitool</code> or some other IPMI client
* a <code>pull-secret</code> file containing authentication credentials for OpenShift Container Platform registries - download from https://console.redhat.com/openshift/
* an SSH keypair that can be used for accessing OpenShift nodes

'''IMPORTANT''': The external IP addresses of cluster nodes must be assigned by your infrastructure DHCP server.

== Host Configuration ==

Beyond the logical requirement of having <code>libvirt</code> installed and started, here are the other configuration details for the hypervisor.

=== Network Settings ===

First thing you definitely need to make sure of, is that IP forwarding is enabled.

<pre>
$ sysctl net.ipv4.ip_forward
1
</pre>

Linux network settings need to be configured to have two '''Linux''' bridges, a public and a private provisioning one.

* public bridge, call it <code>bridge0</code>, needs to have the public network interface enslaved to it
* private bridge, call it <code>provbr0</code>, can be a virtual bridge since it is only needed for the provisioning network, which is supposed to be isolated and without any infrastructure services (such as DHCP, DNS, etc.)

It would be wonderful if the bridges could be OpenVSwitch ones, but unfortunately the Terraform bundled with <code>openshift-baremetal-install</code> currently does not include an OpenVSwitch provider, so there's goodbye to that.

As an example, here is my host configuration.

Public bridge:
<pre>
$ ip addr show bridge0
6: bridge0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff
inet 172.25.35.2/24 brd 172.25.35.255 scope global noprefixroute bridge0
valid_lft forever preferred_lft forever
inet6 fe80::4a21:bff:fe57:e06/64 scope link
valid_lft forever preferred_lft forever

$ ip addr show enp86s0
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master bridge0 state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff

$ bridge link | grep "master bridge0"
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master bridge0 state forwarding priority 32 cost 100
</pre>

Provisioning bridge:
<pre>
$ ip addr show provbr0
5: provbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ce:70:26:9c:88:a4 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.2/24 brd 10.1.1.255 scope global noprefixroute provbr0
valid_lft forever preferred_lft forever
inet6 fe80::cc70:26ff:fe9c:88a4/64 scope link
valid_lft forever preferred_lft forever

$ bridge link | grep "master provbr0"
</pre>

=== Installation Spanning Multiple Hypervisors ===

If you want to have your cluster spanning multiple hypervisors, make sure there is also a VXLAN connection between all the provisioning bridges.

You can do that by creating a <code>vxlan</code> type interface, which is a slave connection of type <code>bridge</code>, and the master is set to <code>provbr0</code>. Choose any unique VXLAN ID, and make sure it is the same on all interconnected hosts.

As an example, here is one VXLAN interface connecting hypervisor A to B.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1703164860
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.2
vxlan.remote: 172.25.35.3
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

And this is the corresponding VXLAN interface definition connecting host B to A.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1697549049
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.3
vxlan.remote: 172.25.35.2
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

In this case, <code>bridge link</code> will of course initially also show the <code>provbr0-vxlan10</code> interface as a slave, and will not show empty as above.

<pre>
$ bridge link | grep "master provbr0"
7: provbr0-vxlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master provbr0 state forwarding priority 32 cost 100
</pre>

=== <code>libvirt</code> Settings ===

Your <code>libvirt</code> will of course need to know about those network bridges in order to be able to attach VMs to them.

For that, you will need two network definitions, looking a bit like the following XML. Make sure they are autostart for least headache.

<pre>
$ sudo virsh net-dumpxml external
<network>
<name>external</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='bridge0'/>
</network>

$ sudo virsh net-dumpxml provisioning
<network>
<name>provisioning</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='provbr0'/>
</network>
</pre>

Additionally, you want to ensure that the storage pool is big enough, but that is not directly related to the subject at hand.

<pre>
$ sudo virsh pool-info default
Name: default
UUID: whatever
State: running
Persistent: yes
Autostart: yes
Capacity: 250.92 GiB
Allocation: 0 GiB
Available: 250.92 GiB
</pre>

=== VirtualBMC ===

The most important part of IPI facilitation is to be able to simulate a baseboard management controller for your VMs. <code>libvirt</code> obviously doesn't do this, but luckily there's a small bit of Python code that does, and it's called <code>virtualbmc</code>.

In most Python environments you can install it using <code>pip3</code>, just make sure <code>pip3</code> is up-to-date first.

<pre>
$ pip3 install --upgrade pip
...

$ pip3 install virtualbmc
...
</pre>

This gives you <code>/usr/local/bin/vbmcd</code> which you can control using the following systemd unit:

<pre>
[Unit]
Description=vbmcd
[Service]
Type=forking
ExecStart=/usr/local/bin/vbmcd
[Install]
WantedBy=multi-user.target
</pre>

Put the above content into <code>/etc/systemd/system/vbmcd.service</code>, reload systemd, and enable/start the service.

<pre>
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now vbmcd
</pre>

You now have the ability to associate a TCP port with a virtual machine defined on the hypervisor host, and have it simulate an IPMI BMC for that VM!

== Virtual Machine Configuration ==

=== VM Definitions ===

The virtual machines need to be configured with sufficient amount of compute resources, as per [[#Prerequisites]] above.

This section ties into the [[#Network Settings]] section above. You need two bridges on your hypervisor(s), <code>bridge0</code> and <code>provbr0</code>.

An example control plane node definition in <code>libvirt XML</code> would look like this:

<pre>
<domain type='kvm'>
<name>controlplane1</name>
<memory unit='GiB'>32</memory>
<currentMemory unit='GiB'>32</currentMemory>
<vcpu placement='static'>12</vcpu>
<os>
<type arch='x86_64' machine='q35'>hvm</type>
<boot dev='hd'/>
<boot dev='network'/>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model' check='partial'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/controlplane1-vda.qcow2'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'/>
<controller type='pci' index='0' model='pcie-root'/>
<interface type='bridge'>
<mac address='52:54:00:00:fb:11'/>
<source bridge='provbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:00:fa:11'/>
<source bridge='bridge0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<console type='pty'/>
<channel type='unix'>
<source mode='bind'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' autoport='yes' listen='0.0.0.0'/>
<video>
<model type='virtio'/>
</video>
<memballoon model='virtio'/>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
</rng>
</devices>
</domain>
</pre>

A couple of things to note:

* the first network interface is attached to <code>provbr0</code>, and its PCI address is ''lower'' (0x03), causing it to be the PXE default device
* the second network interface is attached to <code>bridge0</code>, and its PCI address is ''higher'' (0x09), making it the second interface (for external connections)
* boot order is set to hard disk first, network second, which means the host will only PXE boot if the disk image is unbootable
* the disk image needs to be 64GiB in size at the minimum, but you can make it larger and/or add more disk images if you intend to use the local storage operator

When configuring other nodes, simply remember to change the node name, disk image name, and MAC addresses to be unique. Adjust hardware resources accordingly for compute nodes.

=== IPMI BMC ===

What is important for OCP IPI to be able to perform the installation properly is to register each virtual machine with <code>vbmcd</code> and assign it with a port.

<pre>
$ vbmc add --port=6211 controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | down | :: | 6211 |
+-------------------+---------+---------+------+
$ vbmc start controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | running | :: | 6211 |
+-------------------+---------+---------+------+
$ sudo ss -aunp | grep 6211
UNCONN 0 0 *:6211 *:* users:(("vbmcd",pid=766290,fd=21))
</pre>

There is no need to run the <code>vbmc</code> client as root as it is the daemon that is running as root and can see all the VMs accessible through the <code>qemu:///system</code> URL.

Of course there are options. For any VM you add, you can specify a custom set of IPMI admin credentials (options <code>--username</code> and <code>--password</code>, they default to <code>admin</code> and <code>password</code>), a custom libvirt URL and credentials if necessary (options <code>--libvirt-uri</code>, <code>--libvirt-sasl-username</code>, and <code>--libvirt-sasl-password</code>), and a custom IP address to listen on (defaults to all addresses, use option <code>--address</code> to restrict it).

<pre>
$ vbmc show controlplane1
+-----------------------+-------------------+
| Property | Value |
+-----------------------+-------------------+
| active | True |
| address | :: |
| domain_name | controlplane1 |
| libvirt_sasl_password | *** |
| libvirt_sasl_username | None |
| libvirt_uri | qemu:///system |
| password | *** |
| port | 6211 |
| status | running |
| username | admin |
+-----------------------+-------------------+
</pre>

Once started (which just opens the port) you can test the BMC connection using <code>ipmitool</code> or similar.

<pre>
$ ipmitool -I lanplus -H localhost -p 6211 -U admin -P password chassis status
System Power : off
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : always-off
Last Power Event :
Chassis Intrusion : inactive
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false
</pre>

That's it! We're ready to install OCP!

== Installer Configuration ==

This is where it all comes together. You need to execute the steps in this section (and next) on the ''provisioner'' host, that is, a system that has access to both external and provisioning networks of the OpenShift cluster-to-be. The access need not be direct, it can be routed, but if you, as in our example, configured the provisioning network to be an isolated virtual bridge, you will be best off by creating an additional VM that is directly connected to both bridges.

=== Gathering the Bits Together ===

First step is to make sure the following artifacts are available:

* <code>oc</code>
* <code>pull-secret</code>
* an SSH public key for compute node access

The global cluster network settings that we will need to configure are:

* the parent DNS domain of the cluster (such as <code>example.com</code>)
* the name of the cluster (concatenated with DNS domain, for example <code>mycluster</code> will become <code>mycluster.example.com</code>)
* the provisioning network CIDR (in our case, it can be any IP address block not overlapping with the external network as the provisioning network is isolated)
* from the external network address space, a designated:
** API server VIP (<code>api.mycluster.example.com</code> should point to it)
** ingress load balancer VIP (any host within <code>apps.mycluster.example.com</code> should resolve to it, usually via a wildcard record)

Additionally, you will need, for each node its:

* node name
* provisioning interface MAC address
* IPMI BMC address and port
* IPMI BMC credentials
* the name of the provisioning interface as seen from within the VM (re [[#Virtual Machine Configuration]] above - since the PCI address of the interface is ''bus 0x0 slot 0x3'' it will be named <code>enp0s3</code>)

As already said, DHCP address assignment to external interfaces is not managed by the installer. It must be handled by your infrastructure.

=== Preparing the Software ===

First, make sure your <code>pull-secret</code> file is on the provisioner.

<pre>
$ ls -l pull-secret
-rw-r-----. 1 provisioner provisioner 2734 Oct 27 12:21 pull-secret
</pre>

Then after downloading <code>oc</code> (if you didn't do that already)...

<pre>
$ curl -OL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/openshift-client-linux.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 60.9M 100 60.9M 0 0 4287k 0 0:00:14 0:00:14 --:--:-- 5642k

$ tar xf openshift-client-linux.tar.gz oc

$ ./oc version
Client Version: 4.14.9
Kustomize Version: v5.0.1

$ sudo cp ./oc /usr/local/bin/oc
$ rm -f ./oc
</pre>

...and optionally generating a bash completion file...

<pre>
$ oc completion bash > oc.completion
$ sudo cp oc.completion /etc/bash_completion.d/oc
$ rm -f oc.completion
$ source /etc/bash_completion.d/oc
</pre>

...you can use it to extract <code>openshift-baremetal-install</code> from the release image you intend to use.

<pre>
$ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/release.txt | grep "Pull From:"
Pull From: quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ oc adm release extract --registry-config=pull-secret --command=openshift-baremetal-install --to=. \
quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ ./openshift-baremetal-install version
./openshift-baremetal-install 4.14.9
built from commit dfafb5ca972a6ed4677257aebfe4f284ac020830
release image quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44
release architecture amd64

$ sudo cp ./openshift-baremetal-install /usr/local/bin/

$ openshift-baremetal-install completion bash > oinst.completion
$ sudo cp oinst.completion /etc/bash_completion.d/oinst
$ rm -f oinst.completion
</pre>

'''NOTE''': The extraction process can take up to 5 minutes, depending on your network speed and Quay.io responsiveness.

=== Creating Installer Configuration ===

Installer configuration file can be created interactively, using <code>openshift-baremetal-install</code>, or you can simply write it out with an editor.

==== Using Interactive Mode ====

'''IMPORTANT''': In interactive mode, machine network (the network segment the nodes have their IPs allocated from) is expected to be <code>10.0.0.0/16</code>. If your VIPs are not on that network, the installer will fail. If your machine network is not <code>10.0.0.0/16</code>, the only way to install is to create <code>install-config.yaml</code> manually.

'''IMPORTANT''': Interactive installer expects three control plane nodes and three workers. It will fail if you add less than three of each type of nodes to the cluster.

The interactive mode will ask you a series of questions and generate <code>install-config.yaml</code> at the end, but there are very few customisation options using this method.

Initially, you must select the SSH public key to be published on cluster nodes, and specify some general settings:

<pre>
$ openshift-baremetal-install --dir=cluster create install-config
? SSH Public Key /home/provisioner/.ssh/id_rsa.pub
? Platform baremetal
? Provisioning Network Managed
? Provisioning Network CIDR 172.22.0.0/24
? Provisioning bridge provbr0
? Provisioning Network Interface enp0s3
? External bridge bridge0
? Add a Host: [Use arrows to move, type to filter]
> control plane
worker
</pre>

As you can see above, after those have been answered, you may add any number of nodes, either control plane or worker.

For both type of nodes, the questions are the same:

<pre>
? Add a Host: control plane
? Name controlplane1
? BMC Address ipmi://192.168.1.1:6211
? BMC Username admin
? BMC Password ********
? Boot MAC Address 52:54:00:00:fb:11
? Add another host? (y/N) y
</pre>

After all the hosts have been added, there is a final series of cluster-related questions:

<pre>
? Add another host? No
? Base Domain example.com
? Cluster Name mycluster
? Pull Secret [? for help] ****************************...
</pre>

At this point, some checks are performed against DNS server to see if <code>api.mycluster.example.com</code> resolves, and if the IP is a part of the machine network. The same check is performed for the ingress VIP.

'''NOTE''': If your VIPs are not on the default machine network, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [platform.baremetal.apiVIPs: Invalid value: "172.25.3.10": IP expected to be in one of the machine networks: 10.0.0.0/16, platform.baremetal.ingressVIPs: Invalid value: "172.25.3.11": IP expected to be in one of the machine networks: 10.0.0.0/16]
</pre>

'''NOTE''': If you added fewer than three control plane nodes, or fewer than three workers, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [...]
</pre>

==== Creating Configuration Manually ====

'''NOTE''': There is an <code>explain</code> subcommand in the installer binary that explains the structure of the <code>installconfig</code> resource.

== Installation ==

== Post-Install Smoke Tests ==

= Conclusion =

OCP4-IPI-libvirt

2024-01-26T10:19:25Z

Gregab: /* Creating Installer Configuration */ split into two sections, add reqs for interactive

= Introduction =

== What I Assume ==

* You know how OpenShift installation works and what the difference between IPI and UPI is.
* You know about the OpenShift Machine API and various underlying mechanisms.
* You understand the different types of network interfaces on Linux and different <code>libvirt</code> networks.
* You are familiar and comfortable with NetworkManager and the <code>nmcli</code> tool.
* You are familiar and comfortable with <code>libvirt</code> CLI and XML.
* You are familiar and comfortable with <code>qemu-img</code> tool.
* We are not talking about any firewall restrictions here - it is your responsibility to ensure traffic is not blocked.

== Outcomes ==

The installation described is for a fully managed IPI running OpenShift Container Platform v4.14, initially with three master and two worker nodes.

''At a later point, I will add a couple of steps needed to grow the cluster by one extra worker node.''

= OpenShift Container Platform IPI Installation Using Libvirt =

== Prerequisites ==

Hardware requirements for the cluster:

* 136 GiB RAM (32 GiB per control plane, 20 GiB per compute node), max overcommit ratio of 1.5 (make sure enough swap is available)
* 52 vCPUs (12 per control plane, 8 per compute node), max overcommit ratio of 1.3 (higher might work, but will slow down the installation horribly and may ultimately fail)
* one physical network interface that will be used for the public bridged network
* a physical or virtual network interface that will be used for the provisioning network bridge

Hardware requirements for the installation client (provisioner) machine:

* a minimum of 8 GiB RAM and 4 CPUs
* a network connection to both the public bridged network and the provisioning network

Due to the fact provisioner needs access to both networks, and the provisioning network in this guide is a virtual one, it might be best if you define the provisioner as a VM, with the same network interface settings as the control/compute nodes.

In the case you want to run the workloads spread across several hypervisor hosts, there are some extra steps, but nothing big. More on that in [[#Installation Spanning Multiple Hypervisors]] below.

Software artifacts needed on the provisioner host:

* <code>oc</code>, the command line client, of the corresponding version - download from https://mirror.openshift.com/pub/openshift-v4/clients/ocp/
* <code>libvirt-client</code> package is required for <code>openshift-baremetal-install</code> to be able to communicate to hypervisor(s)
* <code>ipmitool</code> or some other IPMI client
* a <code>pull-secret</code> file containing authentication credentials for OpenShift Container Platform registries - download from https://console.redhat.com/openshift/
* an SSH keypair that can be used for accessing OpenShift nodes

'''IMPORTANT''': The external IP addresses of cluster nodes must be assigned by your infrastructure DHCP server.

== Host Configuration ==

Beyond the logical requirement of having <code>libvirt</code> installed and started, here are the other configuration details for the hypervisor.

=== Network Settings ===

First thing you definitely need to make sure of, is that IP forwarding is enabled.

<pre>
$ sysctl net.ipv4.ip_forward
1
</pre>

Linux network settings need to be configured to have two '''Linux''' bridges, a public and a private provisioning one.

* public bridge, call it <code>bridge0</code>, needs to have the public network interface enslaved to it
* private bridge, call it <code>provbr0</code>, can be a virtual bridge since it is only needed for the provisioning network, which is supposed to be isolated and without any infrastructure services (such as DHCP, DNS, etc.)

It would be wonderful if the bridges could be OpenVSwitch ones, but unfortunately the Terraform bundled with <code>openshift-baremetal-install</code> currently does not include an OpenVSwitch provider, so there's goodbye to that.

As an example, here is my host configuration.

Public bridge:
<pre>
$ ip addr show bridge0
6: bridge0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff
inet 172.25.35.2/24 brd 172.25.35.255 scope global noprefixroute bridge0
valid_lft forever preferred_lft forever
inet6 fe80::4a21:bff:fe57:e06/64 scope link
valid_lft forever preferred_lft forever

$ ip addr show enp86s0
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master bridge0 state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff

$ bridge link | grep "master bridge0"
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master bridge0 state forwarding priority 32 cost 100
</pre>

Provisioning bridge:
<pre>
$ ip addr show provbr0
5: provbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ce:70:26:9c:88:a4 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.2/24 brd 10.1.1.255 scope global noprefixroute provbr0
valid_lft forever preferred_lft forever
inet6 fe80::cc70:26ff:fe9c:88a4/64 scope link
valid_lft forever preferred_lft forever

$ bridge link | grep "master provbr0"
</pre>

=== Installation Spanning Multiple Hypervisors ===

If you want to have your cluster spanning multiple hypervisors, make sure there is also a VXLAN connection between all the provisioning bridges.

You can do that by creating a <code>vxlan</code> type interface, which is a slave connection of type <code>bridge</code>, and the master is set to <code>provbr0</code>. Choose any unique VXLAN ID, and make sure it is the same on all interconnected hosts.

As an example, here is one VXLAN interface connecting hypervisor A to B.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1703164860
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.2
vxlan.remote: 172.25.35.3
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

And this is the corresponding VXLAN interface definition connecting host B to A.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1697549049
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.3
vxlan.remote: 172.25.35.2
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

In this case, <code>bridge link</code> will of course initially also show the <code>provbr0-vxlan10</code> interface as a slave, and will not show empty as above.

<pre>
$ bridge link | grep "master provbr0"
7: provbr0-vxlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master provbr0 state forwarding priority 32 cost 100
</pre>

=== <code>libvirt</code> Settings ===

Your <code>libvirt</code> will of course need to know about those network bridges in order to be able to attach VMs to them.

For that, you will need two network definitions, looking a bit like the following XML. Make sure they are autostart for least headache.

<pre>
$ sudo virsh net-dumpxml external
<network>
<name>external</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='bridge0'/>
</network>

$ sudo virsh net-dumpxml provisioning
<network>
<name>provisioning</name>
<uuid>whatever</uuid>
<forward mode='bridge'/>
<bridge name='provbr0'/>
</network>
</pre>

Additionally, you want to ensure that the storage pool is big enough, but that is not directly related to the subject at hand.

<pre>
$ sudo virsh pool-info default
Name: default
UUID: whatever
State: running
Persistent: yes
Autostart: yes
Capacity: 250.92 GiB
Allocation: 0 GiB
Available: 250.92 GiB
</pre>

=== VirtualBMC ===

The most important part of IPI facilitation is to be able to simulate a baseboard management controller for your VMs. <code>libvirt</code> obviously doesn't do this, but luckily there's a small bit of Python code that does, and it's called <code>virtualbmc</code>.

In most Python environments you can install it using <code>pip3</code>, just make sure <code>pip3</code> is up-to-date first.

<pre>
$ pip3 install --upgrade pip
...

$ pip3 install virtualbmc
...
</pre>

This gives you <code>/usr/local/bin/vbmcd</code> which you can control using the following systemd unit:

<pre>
[Unit]
Description=vbmcd
[Service]
Type=forking
ExecStart=/usr/local/bin/vbmcd
[Install]
WantedBy=multi-user.target
</pre>

Put the above content into <code>/etc/systemd/system/vbmcd.service</code>, reload systemd, and enable/start the service.

<pre>
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now vbmcd
</pre>

You now have the ability to associate a TCP port with a virtual machine defined on the hypervisor host, and have it simulate an IPMI BMC for that VM!

== Virtual Machine Configuration ==

=== VM Definitions ===

The virtual machines need to be configured with sufficient amount of compute resources, as per [[#Prerequisites]] above.

This section ties into the [[#Network Settings]] section above. You need two bridges on your hypervisor(s), <code>bridge0</code> and <code>provbr0</code>.

An example control plane node definition in <code>libvirt XML</code> would look like this:

<pre>
<domain type='kvm'>
<name>controlplane1</name>
<memory unit='GiB'>32</memory>
<currentMemory unit='GiB'>32</currentMemory>
<vcpu placement='static'>12</vcpu>
<os>
<type arch='x86_64' machine='q35'>hvm</type>
<boot dev='hd'/>
<boot dev='network'/>
<bootmenu enable='yes'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model' check='partial'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/controlplane1-vda.qcow2'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'/>
<controller type='pci' index='0' model='pcie-root'/>
<interface type='bridge'>
<mac address='52:54:00:00:fb:11'/>
<source bridge='provbr0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='52:54:00:00:fa:11'/>
<source bridge='bridge0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</interface>
<console type='pty'/>
<channel type='unix'>
<source mode='bind'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
</channel>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' autoport='yes' listen='0.0.0.0'/>
<video>
<model type='virtio'/>
</video>
<memballoon model='virtio'/>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
</rng>
</devices>
</domain>
</pre>

A couple of things to note:

* the first network interface is attached to <code>provbr0</code>, and its PCI address is ''lower'' (0x03), causing it to be the PXE default device
* the second network interface is attached to <code>bridge0</code>, and its PCI address is ''higher'' (0x09), making it the second interface (for external connections)
* boot order is set to hard disk first, network second, which means the host will only PXE boot if the disk image is unbootable
* the disk image needs to be 64GiB in size at the minimum, but you can make it larger and/or add more disk images if you intend to use the local storage operator

When configuring other nodes, simply remember to change the node name, disk image name, and MAC addresses to be unique. Adjust hardware resources accordingly for compute nodes.

=== IPMI BMC ===

What is important for OCP IPI to be able to perform the installation properly is to register each virtual machine with <code>vbmcd</code> and assign it with a port.

<pre>
$ vbmc add --port=6211 controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | down | :: | 6211 |
+-------------------+---------+---------+------+
$ vbmc start controlplane1
$ vbmc list
+-------------------+---------+---------+------+
| Domain name | Status | Address | Port |
+-------------------+---------+---------+------+
| controlplane1 | running | :: | 6211 |
+-------------------+---------+---------+------+
$ sudo ss -aunp | grep 6211
UNCONN 0 0 *:6211 *:* users:(("vbmcd",pid=766290,fd=21))
</pre>

There is no need to run the <code>vbmc</code> client as root as it is the daemon that is running as root and can see all the VMs accessible through the <code>qemu:///system</code> URL.

Of course there are options. For any VM you add, you can specify a custom set of IPMI admin credentials (options <code>--username</code> and <code>--password</code>, they default to <code>admin</code> and <code>password</code>), a custom libvirt URL and credentials if necessary (options <code>--libvirt-uri</code>, <code>--libvirt-sasl-username</code>, and <code>--libvirt-sasl-password</code>), and a custom IP address to listen on (defaults to all addresses, use option <code>--address</code> to restrict it).

<pre>
$ vbmc show controlplane1
+-----------------------+-------------------+
| Property | Value |
+-----------------------+-------------------+
| active | True |
| address | :: |
| domain_name | controlplane1 |
| libvirt_sasl_password | *** |
| libvirt_sasl_username | None |
| libvirt_uri | qemu:///system |
| password | *** |
| port | 6211 |
| status | running |
| username | admin |
+-----------------------+-------------------+
</pre>

Once started (which just opens the port) you can test the BMC connection using <code>ipmitool</code> or similar.

<pre>
$ ipmitool -I lanplus -H localhost -p 6211 -U admin -P password chassis status
System Power : off
Power Overload : false
Power Interlock : inactive
Main Power Fault : false
Power Control Fault : false
Power Restore Policy : always-off
Last Power Event :
Chassis Intrusion : inactive
Front-Panel Lockout : inactive
Drive Fault : false
Cooling/Fan Fault : false
</pre>

That's it! We're ready to install OCP!

== Installer Configuration ==

This is where it all comes together. You need to execute the steps in this section (and next) on the ''provisioner'' host, that is, a system that has access to both external and provisioning networks of the OpenShift cluster-to-be. The access need not be direct, it can be routed, but if you, as in our example, configured the provisioning network to be an isolated virtual bridge, you will be best off by creating an additional VM that is directly connected to both bridges.

=== Gathering the Bits Together ===

First step is to make sure the following artifacts are available:

* <code>oc</code>
* <code>pull-secret</code>
* an SSH public key for compute node access

The global cluster network settings that we will need to configure are:

* the parent DNS domain of the cluster (such as <code>example.com</code>)
* the name of the cluster (concatenated with DNS domain, for example <code>mycluster</code> will become <code>mycluster.example.com</code>)
* the provisioning network CIDR (in our case, it can be any IP address block not overlapping with the external network as the provisioning network is isolated)
* from the external network address space, a designated:
** API server VIP (<code>api.mycluster.example.com</code> should point to it)
** ingress load balancer VIP (any host within <code>apps.mycluster.example.com</code> should resolve to it, usually via a wildcard record)

Additionally, you will need, for each node its:

* node name
* provisioning interface MAC address
* IPMI BMC address and port
* IPMI BMC credentials
* the name of the provisioning interface as seen from within the VM (re [[#Virtual Machine Configuration]] above - since the PCI address of the interface is ''bus 0x0 slot 0x3'' it will be named <code>enp0s3</code>)

As already said, DHCP address assignment to external interfaces is not managed by the installer. It must be handled by your infrastructure.

=== Preparing the Software ===

First, make sure your <code>pull-secret</code> file is on the provisioner.

<pre>
$ ls -l pull-secret
-rw-r-----. 1 provisioner provisioner 2734 Oct 27 12:21 pull-secret
</pre>

Then after downloading <code>oc</code> (if you didn't do that already)...

<pre>
$ curl -OL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/openshift-client-linux.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 60.9M 100 60.9M 0 0 4287k 0 0:00:14 0:00:14 --:--:-- 5642k

$ tar xf openshift-client-linux.tar.gz oc

$ ./oc version
Client Version: 4.14.9
Kustomize Version: v5.0.1

$ sudo cp ./oc /usr/local/bin/oc
$ rm -f ./oc
</pre>

...and optionally generating a bash completion file...

<pre>
$ oc completion bash > oc.completion
$ sudo cp oc.completion /etc/bash_completion.d/oc
$ rm -f oc.completion
$ source /etc/bash_completion.d/oc
</pre>

...you can use it to extract <code>openshift-baremetal-install</code> from the release image you intend to use.

<pre>
$ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable-4.14/release.txt | grep "Pull From:"
Pull From: quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ oc adm release extract --registry-config=pull-secret --command=openshift-baremetal-install --to=. \
quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44

$ ./openshift-baremetal-install version
./openshift-baremetal-install 4.14.9
built from commit dfafb5ca972a6ed4677257aebfe4f284ac020830
release image quay.io/openshift-release-dev/ocp-release@sha256:f5eaf0248779a0478cfd83f055d56dc7d755937800a68ad55f6047c503977c44
release architecture amd64

$ sudo cp ./openshift-baremetal-install /usr/local/bin/

$ openshift-baremetal-install completion bash > oinst.completion
$ sudo cp oinst.completion /etc/bash_completion.d/oinst
$ rm -f oinst.completion
</pre>

'''NOTE''': The extraction process can take up to 5 minutes, depending on your network speed and Quay.io responsiveness.

=== Creating Installer Configuration ===

Installer configuration file can be created interactively, using <code>openshift-baremetal-install</code>, or you can simply write it out with an editor.

==== Using Interactive Mode ====

'''IMPORTANT''': In interactive mode, machine network (the network segment the nodes have their IPs allocated from) is expected to be <code>10.0.0.0/16</code>. If your VIPs are not on that network, the installer will fail. If your machine network is not <code>10.0.0.0/16</code>, the only way to install is to create <code>install-config.yaml</code> manually.

'''IMPORTANT''': Interactive installer expects three control plane nodes and three workers. It will fail if you add less than three of each type of nodes to the cluster.

The interactive mode will ask you a series of questions and generate <code>install-config.yaml</code> at the end, but there are very few customisation options using this method.

Initially, you must select the SSH public key to be published on cluster nodes, and specify some general settings:

<pre>
$ openshift-baremetal-install --dir=cluster create install-config
? SSH Public Key /home/provisioner/.ssh/id_rsa.pub
? Platform baremetal
? Provisioning Network Managed
? Provisioning Network CIDR 172.22.0.0/24
? Provisioning bridge provbr0
? Provisioning Network Interface enp0s3
? External bridge bridge0
? Add a Host: [Use arrows to move, type to filter]
> control plane
worker
</pre>

As you can see, after those have been answered, you may add any number of nodes, either control plane or worker.

For both type of nodes, the questions are the same:

<pre>
? Add a Host: control plane
? Name 3node-node0
? BMC Address ipmi://172.25.35.2:6220
? BMC Username admin
? BMC Password ********
? Boot MAC Address 52:54:00:00:fb:10
? Add another host? (y/N) y
</pre>

After all the hosts have been added, there is a final series of cluster-related questions:

<pre>
? Add another host? No
? Base Domain p0f.local
? Cluster Name 3node
? Pull Secret [? for help] ****************************...
</pre>

At this point, some checks are performed against DNS server to see if <code>api.mycluster.example.com</code> resolves, and if the IP is a part of the machine network. The same check is performed for the ingress VIP.

'''NOTE''': If your VIPs are not on the default machine network, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [platform.baremetal.apiVIPs: Invalid value: "172.25.35.14": IP expected to be in one of the machine networks: 10.0.0.0/16, platform.baremetal.ingressVIPs: Invalid value: "172.25.35.15": IP expected to be in one of the machine networks: 10.0.0.0/16]
</pre>

'''NOTE''': If you added fewer than three control plane nodes, or fewer than three workers, the installer will fail at this point.

<pre>
FATAL failed to fetch Install Config: failed to generate asset "Install Config": invalid install config: [...]
</pre>

==== Creating Configuration Manually ====

'''NOTE''': There is an <code>explain</code> subcommand in the installer binary that explains the structure of the <code>installconfig</code> resource.

== Installation ==

== Post-Install Smoke Tests ==

= Conclusion =

OCP4-IPI-libvirt

2024-01-26T10:12:48Z

Gregab: /* Installer Configuration */ add interactive config

OCP4-IPI-libvirt

2024-01-26T09:11:43Z

Gregab: /* Preparing the Software */ add pull-secret step

OCP4-IPI-libvirt

2024-01-26T09:01:19Z

Gregab: /* Installer Configuration */ added extract tools

OCP4-IPI-libvirt

2024-01-26T07:58:54Z

Gregab: /* Installer Configuration */ add node name

OCP4-IPI-libvirt

2024-01-26T07:25:56Z

Gregab: /* Installer Configuration */ add dns details

OCP4-IPI-libvirt

2024-01-26T07:18:39Z

Gregab: /* Prerequisites */ add DHCP requirement

OCP4-IPI-libvirt

2024-01-26T07:16:16Z

Gregab: /* Virtual Machine Configuration */ reorg

OCP4-IPI-libvirt

2024-01-26T07:13:38Z

Gregab: /* Installer Configuration */ enumerated bits needed for install

OCP4-IPI-libvirt

2024-01-25T18:47:07Z

Gregab: /* Installer Configuration */ intro

OCP4-IPI-libvirt

2024-01-25T13:28:55Z

Gregab: /* Virtual Machine Configuration */ added vbmc notes

OCP4-IPI-libvirt

2024-01-25T13:12:34Z

Gregab: /* Virtual Machine Configuration */ small bits

OCP4-IPI-libvirt

2024-01-25T10:59:58Z

Gregab: /* Virtual Machine Configuration */ libvirt xml and notes added

OCP4-IPI-libvirt

2024-01-25T10:49:50Z

Gregab: /* Prerequisites */ fix link

OCP4-IPI-libvirt

2024-01-23T18:56:50Z

Gregab: finish vbmcd section

OCP4-IPI-libvirt

2024-01-19T22:09:30Z

Gregab: down to vbmc

OCP4-IPI-libvirt

2024-01-19T12:03:26Z

Gregab: added linux network settings

= Introduction =

== What I Assume ==

* You are familiar and comfortable with <code>libvirt</code> CLI and XML.
* You are familiar and comfortable with <code>qemu-img</code> tool.
* You understand the different types of network interfaces on Linux and different <code>libvirt</code> networks.
* You are familiar and comfortable with NetworkManager and the <code>nmcli</code> tool.
* You know how OpenShift installation works and what the difference between IPI and UPI is.
* You know about the OpenShift Machine API and various underlying mechanisms.

== Outcomes ==

The installation described is for a fully managed IPI running OpenShift Container Platform v4.14, initially with three master and two worker nodes.

''At a later point, I will add a couple of steps needed to grow the cluster by one extra worker node.''

= OpenShift Container Platform IPI Installation Using Libvirt =

== Prerequisites ==

Hardware requirements for the cluster:

* 136 GiB RAM (32 GiB per control plane, 20 GiB per compute node), max overcommit ratio of 1.5 (make sure enough swap is available)
* 52 vCPUs (12 per control plane, 8 per compute node), max overcommit ratio of 1.3 (higher might work, but will slow down the installation horribly and may ultimately fail)
* one physical network interface that will be used for the public bridged network
* a physical or virtual network interface that will be used for the provisioning network bridge

Hardware requirements for the installation client (provisioner) machine:

* a minimum of 8 GiB RAM and 4 CPUs
* a network connection to both the public bridged network and the provisioning network

Due to the fact provisioner needs access to both networks, and the provisioning network in this guide is a virtual one, it might be best if you define the provisioner as a VM, with the same network interface settings as the control/compute nodes.

In the case you want to run the workloads spread across several hypervisor hosts, there are some extra steps, but nothing big. More on that in [[#Network Settings]] below.

Software artifacts needed on the provisioner host:

* <code>oc</code>, the command line client, of the corresponding version - download from https://mirror.openshift.com/pub/openshift-v4/clients/ocp/
* <code>libvirt-client</code> package is required for <code>openshift-baremetal-install</code> to be able to communicate to hypervisor(s)
* <code>ipmitool</code> or some other IPMI client
* a <code>pull-secret</code> file containing authentication credentials for OpenShift Container Platform registries - download from https://console.redhat.com/openshift/
* an SSH keypair that can be used for accessing OpenShift nodes

== Host Configuration ==

Beyond the logical requirement of having <code>libvirt</code> installed and started, here are the other configuration details for the hypervisor.

=== Network Settings ===

First thing you definitely need to make sure of, is that IP forwarding is enabled.

<pre>
$ sysctl net.ipv4.ip_forward
1
</pre>

Linux network settings need to be configured to have two '''Linux''' bridges, a public and a private provisioning one.

* public bridge, call it <code>bridge0</code>, needs to have the public network interface enslaved to it
* private bridge, call it <code>provbr0</code>, can be a virtual bridge since it is only needed for the provisioning network, which is supposed to be isolated and without any infrastructure services (such as DHCP, DNS, etc.)

It would be wonderful if the bridges could be OpenVSwitch ones, but unfortunately the Terraform bundled with <code>openshift-baremetal-install</code> currently does not include an OpenVSwitch provider, so there's goodbye to that.

As an example, here is my host configuration.

Public bridge:
<pre>
$ ip addr show bridge0
6: bridge0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff
inet 172.25.35.2/24 brd 172.25.35.255 scope global noprefixroute bridge0
valid_lft forever preferred_lft forever
inet6 fe80::4a21:bff:fe57:e06/64 scope link
valid_lft forever preferred_lft forever

$ ip addr show enp86s0
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master bridge0 state UP group default qlen 1000
link/ether 48:21:0b:57:0e:06 brd ff:ff:ff:ff:ff:ff

$ bridge link | grep "master bridge0"
2: enp86s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master bridge0 state forwarding priority 32 cost 100
</pre>

Provisioning bridge:
<pre>
$ ip addr show provbr0
5: provbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ce:70:26:9c:88:a4 brd ff:ff:ff:ff:ff:ff
inet 10.1.1.2/24 brd 10.1.1.255 scope global noprefixroute provbr0
valid_lft forever preferred_lft forever
inet6 fe80::cc70:26ff:fe9c:88a4/64 scope link
valid_lft forever preferred_lft forever

$ bridge link | grep "master provbr0"
</pre>

=== Installation Spanning Multiple Hypervisors ===

If you want to have your cluster spanning multiple hypervisors, make sure there is also a VXLAN connection between all the provisioning bridges.

You can do that by creating a <code>vxlan</code> type interface, which is a slave connection of type <code>bridge</code>, and the master is set to <code>provbr0</code>. Choose any unique VXLAN ID, and make sure it is the same on all interconnected hosts.

As an example, here is one VXLAN interface connecting hypervisor A to B.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1703164860
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.2
vxlan.remote: 172.25.35.3
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

And this is the corresponding VXLAN interface definition connecting host B to A.

<pre>
$ nmcli con show provbr0-vxlan10 | grep -E '^(connection|vxlan)' | grep -vE '(default|uuid|--|-1|unknown)'
connection.id: provbr0-vxlan10
connection.type: vxlan
connection.interface-name: provbr0-vxlan10
connection.autoconnect: yes
connection.autoconnect-priority: 0
connection.timestamp: 1697549049
connection.read-only: no
connection.master: provbr0
connection.slave-type: bridge
connection.gateway-ping-timeout: 0
vxlan.id: 10
vxlan.local: 172.25.35.3
vxlan.remote: 172.25.35.2
vxlan.source-port-min: 0
vxlan.source-port-max: 0
vxlan.destination-port: 4790
vxlan.tos: 0
vxlan.ttl: 0
vxlan.ageing: 300
vxlan.limit: 0
vxlan.learning: yes
vxlan.proxy: no
vxlan.rsc: no
vxlan.l2-miss: no
vxlan.l3-miss: no
</pre>

In this case, <code>bridge link</code> will of course initially also show the <code>provbr0-vxlan10</code> interface as a slave, and will not show empty as above.

<pre>
$ bridge link | grep "master provbr0"
7: provbr0-vxlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master provbr0 state forwarding priority 32 cost 100
</pre>

== Virtual Machine Configuration ==

== Installer Configuration ==

== Installation ==

== Post-Install Smoke Tests ==

= Conclusion =

OCP4-IPI-libvirt

2024-01-19T11:36:45Z

Gregab: update expectations, add outcomes and prereqs

OCP4-IPI-libvirt

2024-01-19T11:02:51Z

Gregab: add expectations

= Introduction =

== What I Assume ==

* You are familiar and comfortable with <code>libvirt</code> CLI and XML.
* You understand the different types of network interfaces on Linux and different <code>libvirt</code> networks.
* You know how OpenShift installation works and what the difference between IPI and UPI is.
* You know about the OpenShift Machine API and various underlying mechanisms.

= OpenShift Container Platform IPI Installation Using Libvirt =

== Prerequisites ==

== Host Configuration ==

== Virtual Machine Configuration ==

== Installer Configuration ==

== Installation ==

== Post-Install Smoke Tests ==

= Conclusion =

OCP4-IPI-libvirt

2024-01-12T11:14:59Z

Gregab: add structure

= Introduction =

== What I Assume ==

= OpenShift Container Platform IPI Installation Using Libvirt =

== Prerequisites ==

== Host Configuration ==

== Virtual Machine Configuration ==

== Installer Configuration ==

== Installation ==

== Post-Install Smoke Tests ==

= Conclusion =

OpenShift Admin

2024-01-12T11:12:07Z

Gregab: add link to ocp4-ipi-libvirt

= Installation =

* OCP4 IPI vs UPI

* [[OCP4-IPI-libvirt|OCP4 IPI installation using <code>libvirt</code>]]

OpenShift Admin

2024-01-12T10:51:36Z

Gregab: add some installation content

= Installation =

* OCP4 IPI vs UPI

* OCP4 IPI installation using <code>libvirt</code>

Main Page

2024-01-12T10:46:59Z

Gregab: change java to include quarkus, link to ocp admin

List of topics by category:

* Container Technology
* [[OpenShift Admin|OpenShift Administration]]

* Ansible Automation

* [[Java Devel|Java SE/EE/Quarkus]]
* [[OpenShift Devel|OpenShift Application Development]]

* macOS Tips & Tricks

* Linux System Administration
* JBoss EAP Administration

Links will be added as time permits.

Main Page

2024-01-12T10:44:26Z

Gregab: reorg, add ansible

List of topics by category:

* Container Technology
* OpenShift Administration

* Ansible Automation

* [[Java Devel|Java SE/EE Development]]
* [[OpenShift Devel|OpenShift Application Development & SDLC]]

* macOS Tips & Tricks

* Linux System Administration
* JBoss EAP Administration

Links will be added as time permits.

New Features in Java 12 through 15

2021-01-18T12:04:35Z

Gregab: /* What I assume */ change reqs

= Introduction =

JDK versions just keep coming, and since my last article about [[Changes in Java 10 and 11|new features in Java v10 and v11]], there have been four new major Java releases with a number of interesting (and useful) new features.

= What I assume =

I assume that you are:

* fluent in the Java language (according to v11 language specification)
* aware of how Java garbage collectors work and why we need them
* have a clue about what Linux software packages are and what they do

= Language =

== Switch Expressions (v12) ==

* JEP: https://openjdk.java.net/jeps/325

Lifecycle phases:

* v12: preview
* v13: preview
* v14: standard

== Text Blocks (v13) ==

* JEP: https://openjdk.java.net/jeps/355

Lifecycle phases:

* v13: preview
* v14: second preview
* v15: standard

== Pattern Matching for instanceof (v14) ==

* JEP: https://openjdk.java.net/jeps/305

Lifecycle phases:

* v14: preview
* v15: second preview

== Records (v14) ==

* JEP: https://openjdk.java.net/jeps/359

Lifecycle phases:

* v14: preview
* v15: second preview

== Sealed Classes (v15) ==

* JEP: https://openjdk.java.net/jeps/360

Lifecycle phases:

* v15: preview

= Virtual Machine =

== Shenandoah Garbage Collector (v12) ==

Lifecycle phases:

* v12: experimental
* v15: GA

== Abortable Mixed Collections for G1 (v12) ==

https://openjdk.java.net/jeps/344

Make G1 mixed collections abortable if they might exceed the pause target.

== Promptly Return Unused Committed Memory from G1 (v12) ==

https://openjdk.java.net/jeps/346

G1 only returns memory from the Java heap at either a full GC or during a concurrent cycle. Since G1 tries hard to completely avoid full GCs, and only triggers a concurrent cycle based on Java heap occupancy and allocation activity, it will not return Java heap memory in many cases unless forced to do so externally.

This feature enhances the G1 garbage collector to automatically return Java heap memory to the operating system when idle.

== Return Unused Memory to OS from ZGC (v13) ==

* JEP: https://openjdk.java.net/jeps/351

== Bye-bye, CMS (v14) ==

* JEP: https://openjdk.java.net/jeps/363

== Hidden Classes (v15) ==

* JEP: https://openjdk.java.net/jeps/371

= Tools =

== New OS-Native Packaging Tool for Self-Contained Apps (v14) ==

* JEP: https://openjdk.java.net/jeps/343

Lifecycle phases:

* v14: incubator

== Better NullPointerException Messages (v14) ==

* JEP: https://openjdk.java.net/jeps/358

New Features in Java 12 through 15

2021-01-18T11:25:34Z

Gregab: Typo.

= Introduction =

JDK versions just keep coming, and since my last article about [[Changes in Java 10 and 11|new features in Java v10 and v11]], there have been four new major Java releases with a number of interesting (and useful) new features.

= What I assume =

I assume that you are:

* fluent in the Java language (according to v11 language specification)
* aware of how Java garbage collectors work and why we need them
* familiar with what software profiling and auditing are and how they work
* have a clue about Linux scripting

= Language =

== Switch Expressions (v12) ==

* JEP: https://openjdk.java.net/jeps/325

Lifecycle phases:

* v12: preview
* v13: preview
* v14: standard

== Text Blocks (v13) ==

* JEP: https://openjdk.java.net/jeps/355

Lifecycle phases:

* v13: preview
* v14: second preview
* v15: standard

== Pattern Matching for instanceof (v14) ==

* JEP: https://openjdk.java.net/jeps/305

Lifecycle phases:

* v14: preview
* v15: second preview

== Records (v14) ==

* JEP: https://openjdk.java.net/jeps/359

Lifecycle phases:

* v14: preview
* v15: second preview

== Sealed Classes (v15) ==

* JEP: https://openjdk.java.net/jeps/360

Lifecycle phases:

* v15: preview

= Virtual Machine =

== Shenandoah Garbage Collector (v12) ==

Lifecycle phases:

* v12: experimental
* v15: GA

== Abortable Mixed Collections for G1 (v12) ==

https://openjdk.java.net/jeps/344

Make G1 mixed collections abortable if they might exceed the pause target.

== Promptly Return Unused Committed Memory from G1 (v12) ==

https://openjdk.java.net/jeps/346

G1 only returns memory from the Java heap at either a full GC or during a concurrent cycle. Since G1 tries hard to completely avoid full GCs, and only triggers a concurrent cycle based on Java heap occupancy and allocation activity, it will not return Java heap memory in many cases unless forced to do so externally.

This feature enhances the G1 garbage collector to automatically return Java heap memory to the operating system when idle.

== Return Unused Memory to OS from ZGC (v13) ==

* JEP: https://openjdk.java.net/jeps/351

== Bye-bye, CMS (v14) ==

* JEP: https://openjdk.java.net/jeps/363

== Hidden Classes (v15) ==

* JEP: https://openjdk.java.net/jeps/371

= Tools =

== New OS-Native Packaging Tool for Self-Contained Apps (v14) ==

* JEP: https://openjdk.java.net/jeps/343

Lifecycle phases:

* v14: incubator

== Better NullPointerException Messages (v14) ==

* JEP: https://openjdk.java.net/jeps/358

New Features in Java 12 through 15

2021-01-18T11:25:00Z

Gregab: Placeholders with links and metadata.

= Introduction =

JDK versions just keep coming, and since my last article about [[Changes in Java 10 and 11|new features in Java v10 and v11]], there have been three new major Java releases with a number of interesting (and useful) new features.

= What I assume =

I assume that you are:

* fluent in the Java language (according to v11 language specification)
* aware of how Java garbage collectors work and why we need them
* familiar with what software profiling and auditing are and how they work
* have a clue about Linux scripting

= Language =

== Switch Expressions (v12) ==

* JEP: https://openjdk.java.net/jeps/325

Lifecycle phases:

* v12: preview
* v13: preview
* v14: standard

== Text Blocks (v13) ==

* JEP: https://openjdk.java.net/jeps/355

Lifecycle phases:

* v13: preview
* v14: second preview
* v15: standard

== Pattern Matching for instanceof (v14) ==

* JEP: https://openjdk.java.net/jeps/305

Lifecycle phases:

* v14: preview
* v15: second preview

== Records (v14) ==

* JEP: https://openjdk.java.net/jeps/359

Lifecycle phases:

* v14: preview
* v15: second preview

== Sealed Classes (v15) ==

* JEP: https://openjdk.java.net/jeps/360

Lifecycle phases:

* v15: preview

= Virtual Machine =

== Shenandoah Garbage Collector (v12) ==

Lifecycle phases:

* v12: experimental
* v15: GA

== Abortable Mixed Collections for G1 (v12) ==

https://openjdk.java.net/jeps/344

Make G1 mixed collections abortable if they might exceed the pause target.

== Promptly Return Unused Committed Memory from G1 (v12) ==

https://openjdk.java.net/jeps/346

G1 only returns memory from the Java heap at either a full GC or during a concurrent cycle. Since G1 tries hard to completely avoid full GCs, and only triggers a concurrent cycle based on Java heap occupancy and allocation activity, it will not return Java heap memory in many cases unless forced to do so externally.

This feature enhances the G1 garbage collector to automatically return Java heap memory to the operating system when idle.

== Return Unused Memory to OS from ZGC (v13) ==

* JEP: https://openjdk.java.net/jeps/351

== Bye-bye, CMS (v14) ==

* JEP: https://openjdk.java.net/jeps/363

== Hidden Classes (v15) ==

* JEP: https://openjdk.java.net/jeps/371

= Tools =

== New OS-Native Packaging Tool for Self-Contained Apps (v14) ==

* JEP: https://openjdk.java.net/jeps/343

Lifecycle phases:

* v14: incubator

== Better NullPointerException Messages (v14) ==

* JEP: https://openjdk.java.net/jeps/358

Changes in Java 10 and 11

2021-01-18T11:15:53Z

Gregab: /* ZGC (v11) */ added v14 news

= Introduction =

JDK 11 was released almost two months ago (as of this writing), yet it almost seems as if nobody noticed it.

After the roar and thunder of Java 9's JPMS (or Jigsaw, as some may know it) it's gone awfully quiet on the Java front.

But that's far from saying nothing is happening - after [http://openjdk.java.net/jeps/261 Java Module System] (which was about the single biggest feature in [https://docs.oracle.com/javase/9/whatsnew/toc.htm Java 9]), the world moves on.

If version 9 was mostly a clean-up release with the much-hated Java Plugin and the Applet API having been deprecated, CMS GC and all garbage collector combinations involving it, as well, several rather big improvements and additions had been made to both the language and the virtual machine in the two subsequent versions, [https://openjdk.java.net/projects/jdk/10/ Java 10] and [https://openjdk.java.net/projects/jdk/11/ Java 11].

== What I assume ==

I assume that you are:

* fluent in the Java language (according to v8 language specification)
* aware of how Java garbage collectors work and why we need them
* familiar with what software profiling and auditing are and how they work
* have a clue about Linux scripting

= Language =

== Type Inference for Local Variables (v10) ==

* JEP: https://openjdk.java.net/jeps/286

Among the biggest additions to Java 10 is its [https://openjdk.java.net/jeps/286 ability to figure out the type of a variable] that has an initialiser, that is to say, perform LHS (left-hand-side) type inference.

RHS type inference should be quite familiar by now as it was by far the biggest change in Java 8 - [https://openjdk.java.net/projects/lambda/ Lambda Expressions for the Java Language] allow a programmer to save space, time, and improve readability, making heavy use of type inference all the time:

MockEventHandler eh = new MockEventHandler();
eh.setOnSomeEvent('''event -> new EventProcessor().process(event)''');

Actually, the above expression can be simplified even further:

MockEventHandler eh = new MockEventHandler();
eh.setOnSomeEvent('''EventProcessor::process''');

Notice how parameters in lambda expressions lack type declarations. Infact, they completely lack ''any'' declarations! You can [[Java 8 Lambda Expressions|read more about this in a small write-up of mine]].

In Java 10, this inference is extended even further. For cases when the compiler can infer the type of a variable, its declaration can be simplified to just the newly introduced ''reserved type'' <code>var</code>:

'''var''' x = new Integer(5);
'''var''' y = MyClass.Factory.getInstance();

if (x instanceof Integer) {
// matches
}
if (y instanceof MyClass) {
// matches
}

There are obviously some limitations for this new expression, such as:

* declarations without initialisers (which would require some distant action to infer the type, and that could violate the strongly-typed nature of the language because of multiple possible ''different'' initialisers at different execution points),
* <code>null</code> assignments (note that <code>null</code> ''is'' a type, but it is fairly useless in and of its own),
* [http://www.devcodenote.com/2015/04/variable-capture-in-java.html capture variables] (which need to be final, so they can not be inferred),
* [http://iteratrlearning.com/java/generics/2016/05/12/intersection-types-java-generics.html intersection types] (which can not be reliably inferred),

or generally whenever an assignment type is not denotable (such as with inline arrays, method references, and even <code>c.getClass()</code>).

I may write more about those limitations when I've used LHS type inference more, until then feel free to have a look at [https://openjdk.java.net/jeps/286 the JEP defining Java type inference] and its many references.

== Local Variable Syntax for Lambda Parameters (v11) ==

* JEP: https://openjdk.java.net/jeps/323

BLABLA

== New Nest-Based Field/Method Access Control (v11) ==

* JEP: https://openjdk.java.net/jeps/181

BLABLA

= Virtual Machine =

== Application Class Data Sharing (v10) ==

* JEP: https://openjdk.java.net/jeps/310

BLABLA

== No-Op Garbage Collector (v11) ==

* JEP: https://openjdk.java.net/jeps/318

BLABLA

== ZGC (v11) ==

* JEP: https://openjdk.java.net/jeps/333

Lifecycle phases:

* v11: experimental (Linux-only)
* v14: experimental ports for macOS and Windows
* v15: GA

BLABLA

= Tools =

== Flight Recorder (v11) ==

* JEP: https://openjdk.java.net/jeps/328

BLABLA

== Low-Impact Heap Profiling (v11) ==

* JEP: https://openjdk.java.net/jeps/331

BLABLA

== Launching Single-File Programs from Source (v11) ==

* JEP: https://openjdk.java.net/jeps/330

BLABLA

Changes in Java 10 and 11

2021-01-18T11:00:48Z

Gregab: Added JEPs.

= Introduction =

JDK 11 was released almost two months ago (as of this writing), yet it almost seems as if nobody noticed it.

After the roar and thunder of Java 9's JPMS (or Jigsaw, as some may know it) it's gone awfully quiet on the Java front.

But that's far from saying nothing is happening - after [http://openjdk.java.net/jeps/261 Java Module System] (which was about the single biggest feature in [https://docs.oracle.com/javase/9/whatsnew/toc.htm Java 9]), the world moves on.

If version 9 was mostly a clean-up release with the much-hated Java Plugin and the Applet API having been deprecated, CMS GC and all garbage collector combinations involving it, as well, several rather big improvements and additions had been made to both the language and the virtual machine in the two subsequent versions, [https://openjdk.java.net/projects/jdk/10/ Java 10] and [https://openjdk.java.net/projects/jdk/11/ Java 11].

== What I assume ==

I assume that you are:

* fluent in the Java language (according to v8 language specification)
* aware of how Java garbage collectors work and why we need them
* familiar with what software profiling and auditing are and how they work
* have a clue about Linux scripting

= Language =

== Type Inference for Local Variables (v10) ==

* JEP: https://openjdk.java.net/jeps/286

Among the biggest additions to Java 10 is its [https://openjdk.java.net/jeps/286 ability to figure out the type of a variable] that has an initialiser, that is to say, perform LHS (left-hand-side) type inference.

RHS type inference should be quite familiar by now as it was by far the biggest change in Java 8 - [https://openjdk.java.net/projects/lambda/ Lambda Expressions for the Java Language] allow a programmer to save space, time, and improve readability, making heavy use of type inference all the time:

MockEventHandler eh = new MockEventHandler();
eh.setOnSomeEvent('''event -> new EventProcessor().process(event)''');

Actually, the above expression can be simplified even further:

MockEventHandler eh = new MockEventHandler();
eh.setOnSomeEvent('''EventProcessor::process''');

Notice how parameters in lambda expressions lack type declarations. Infact, they completely lack ''any'' declarations! You can [[Java 8 Lambda Expressions|read more about this in a small write-up of mine]].

In Java 10, this inference is extended even further. For cases when the compiler can infer the type of a variable, its declaration can be simplified to just the newly introduced ''reserved type'' <code>var</code>:

'''var''' x = new Integer(5);
'''var''' y = MyClass.Factory.getInstance();

if (x instanceof Integer) {
// matches
}
if (y instanceof MyClass) {
// matches
}

There are obviously some limitations for this new expression, such as:

* declarations without initialisers (which would require some distant action to infer the type, and that could violate the strongly-typed nature of the language because of multiple possible ''different'' initialisers at different execution points),
* <code>null</code> assignments (note that <code>null</code> ''is'' a type, but it is fairly useless in and of its own),
* [http://www.devcodenote.com/2015/04/variable-capture-in-java.html capture variables] (which need to be final, so they can not be inferred),
* [http://iteratrlearning.com/java/generics/2016/05/12/intersection-types-java-generics.html intersection types] (which can not be reliably inferred),

or generally whenever an assignment type is not denotable (such as with inline arrays, method references, and even <code>c.getClass()</code>).

I may write more about those limitations when I've used LHS type inference more, until then feel free to have a look at [https://openjdk.java.net/jeps/286 the JEP defining Java type inference] and its many references.

== Local Variable Syntax for Lambda Parameters (v11) ==

* JEP: https://openjdk.java.net/jeps/323

BLABLA

== New Nest-Based Field/Method Access Control (v11) ==

* JEP: https://openjdk.java.net/jeps/181

BLABLA

= Virtual Machine =

== Application Class Data Sharing (v10) ==

* JEP: https://openjdk.java.net/jeps/310

BLABLA

== No-Op Garbage Collector (v11) ==

* JEP: https://openjdk.java.net/jeps/318

BLABLA

== ZGC (v11) ==

* JEP: https://openjdk.java.net/jeps/333

Lifecycle phases:

* v11: experimental
* v15: GA

BLABLA

= Tools =

== Flight Recorder (v11) ==

* JEP: https://openjdk.java.net/jeps/328

BLABLA

== Low-Impact Heap Profiling (v11) ==

* JEP: https://openjdk.java.net/jeps/331

BLABLA

== Launching Single-File Programs from Source (v11) ==

* JEP: https://openjdk.java.net/jeps/330

BLABLA

Java Devel

2021-01-18T10:45:46Z

Gregab: /* Java SE */ Added v12-v15

= Java SE =

* [[Java 8 Lambda Expressions]]
* [[Changes in Java 10 and 11]]
* [[New Features in Java 12 through 15]]

= Java EE =

Something coming here.

Java 8 Lambda Expressions

2018-12-07T17:54:39Z

Gregab: Added advanced.

= Introduction =

Lambda Expressions for the Java language are meant to reduce bracket noise, thus saving time and improving readability of Java code.

However, they can only be used under certain conditions, and in this article I want to expand on what these conditions are.

== What I assume ==

To best focus on the content of this article, I assume the reader is:

* fluent in Java language (according to v7 language specification)

= The Anatomy of a Lambda Expression =

== Prerequisites ==

Java 8 introduced ''functional interfaces'', a special type of an interface which only has one abstract method.

The key excerpts from the ''[https://www.jcp.org/en/jsr/detail?id=337 Java 8 Language Specification]'' that Lambda Expressions build upon are:

<blockquote>In addition to the usual process of creating an interface instance by declaring and instantiating a class (§15.9), instances of functional interfaces can be created with method reference expressions and lambda expressions (§15.13, §15.27).</blockquote>

and

<blockquote>The function type of a functional interface I is a method type (§8.2) that can be used to override (§8.4.8) the abstract method(s) of I.</blockquote>

The above two excerpts define when lambda expressions can be used, and define the rules for deriving the signature of that expression from the function that the interface proposes.

== Building on Foundation ==

TROLOLO.

== Advanced Shit ==

<blockquote>When the parameter types of a lambda expression are inferred, the same lambda body can be interpreted in different ways, depending on the context in which it appears.</blockquote>

= Examples =

== Parameterless Void Method ==

Consider interface <code>Runnable</code>, which requires you to supply a <code>void run()</code> method in an implementation:

public interface Runnable {
void run();
}

This is a common way of starting <code>Thread</code>s:

public void startThread() {
Runnable r = new Runnable() {
public void run() {
new JobRunner().doJob();
}
};
Thread t = new Thread(r);
t.start();
}

Instead of going the long way, an anonymous inner class can be used in constructor parameter:

public void startThread() {
Thread t = new Thread(new Runnable() {
public void run() {
new JobRunner().doJob();
}
});
t.start();
}

One can now use a lambda expression to reduce the amount of boilerplate:

public class MyCode {
// ...
public void startThread() {
Runnable r = () -> { new JobRunner().doJob(); };
Thread t = new Thread(r);
t.start();
}
// ...
}

Or even?

public class MyCode {
// ...
public void startThread() {
Thread t = new Thread(() -> { new JobRunner().doJob(); });
t.start();
}
// ...
}

Java 8 Lambda Expressions

2018-12-07T13:51:13Z

Gregab: Shuffled bits around.

= Introduction =

Lambda Expressions for the Java language are meant to reduce bracket noise, thus saving time and improving readability of Java code.

However, they can only be used under certain conditions, and in this article I want to expand on what these conditions are.

== What I assume ==

To best focus on the content of this article, I assume the reader is:

* fluent in Java language (according to v7 language specification)

= The Anatomy of a Lambda Expression =

== Prerequisites ==

Java 8 introduced ''functional interfaces'', a special type of an interface which only has one abstract method.

The key excerpts from the ''[https://www.jcp.org/en/jsr/detail?id=337 Java 8 Language Specification]'' that Lambda Expressions build upon are:

<blockquote>In addition to the usual process of creating an interface instance by declaring and instantiating a class (§15.9), instances of functional interfaces can be created with method reference expressions and lambda expressions (§15.13, §15.27).</blockquote>

and

<blockquote>The function type of a functional interface I is a method type (§8.2) that can be used to override (§8.4.8) the abstract method(s) of I.</blockquote>

The above two excerpts define when lambda expressions can be used, and define the rules for deriving the signature of that expression from the function that the interface proposes.

== Building on Foundations ==

TROLOLO.

= Examples =

== Parameterless Void Method ==

Consider interface <code>Runnable</code>, which requires you to supply a <code>void run()</code> method in an implementation:

public interface Runnable {
void run();
}

This is a common way of starting <code>Thread</code>s:

public void startThread() {
Runnable r = new Runnable() {
public void run() {
new JobRunner().doJob();
}
};
Thread t = new Thread(r);
t.start();
}

Instead of going the long way, an anonymous inner class can be used in constructor parameter:

public void startThread() {
Thread t = new Thread(new Runnable() {
public void run() {
new JobRunner().doJob();
}
});
t.start();
}

One can now use a lambda expression to reduce the amount of boilerplate:

public class MyCode {
// ...
public void startThread() {
Runnable r = () -> { new JobRunner().doJob(); };
Thread t = new Thread(r);
t.start();
}
// ...
}

Or even?

public class MyCode {
// ...
public void startThread() {
Thread t = new Thread(() -> { new JobRunner().doJob(); });
t.start();
}
// ...
}

Java 8 Lambda Expressions

2018-12-07T13:13:55Z

Gregab: Added assumptions and intro.

= Introduction =

Java 8 introduced ''functional interfaces'', a special type of an interface which only has one abstract method.

The key excerpt from the ''[https://www.jcp.org/en/jsr/detail?id=337 Java 8 Language Specification]'' related to the subject of this article seems to be:

<blockquote>In addition to the usual process of creating an interface instance by declaring and instantiating a class (§15.9), instances of functional interfaces can be created with method reference expressions and lambda expressions (§15.13, §15.27).</blockquote>

== What I assume ==

To best focus on the content of this article, I assume the reader is:

* fluent in Java language (according to v7 language specification)

= Examples =

== Parameterless Void Method ==

Consider interface <code>Runnable</code>, which requires you to supply a <code>void run()</code> method in an implementation:

public interface Runnable {
void run();
}

This is a common way of starting <code>Thread</code>s:

public void startThread() {
Runnable r = new Runnable() {
public void run() {
new JobRunner().doJob();
}
};
Thread t = new Thread(r);
t.start();
}

Instead of going the long way, an anonymous inner class can be used in constructor parameter:

public void startThread() {
Thread t = new Thread(new Runnable() {
public void run() {
new JobRunner().doJob();
}
});
t.start();
}

One can now use a lambda expression to reduce the amount of boilerplate:

public class MyCode {
// ...
public void startThread() {
Runnable r = () -> { new JobRunner().doJob(); };
Thread t = new Thread(r);
t.start();
}
// ...
}

Or even?

public class MyCode {
// ...
public void startThread() {
Thread t = new Thread(() -> { new JobRunner().doJob(); });
t.start();
}
// ...
}

Changes in Java 10 and 11

2018-11-26T08:05:27Z

Gregab: /* Type Inference for Local Variables (v10) */ Added a method reference example. Fixed some formatting.

= Introduction =

JDK 11 was released almost two months ago (as of this writing), yet it almost seems as if nobody noticed it.

After the roar and thunder of Java 9's JPMS (or Jigsaw, as some may know it) it's gone awfully quiet on the Java front.

But that's far from saying nothing is happening - after [http://openjdk.java.net/jeps/261 Java Module System] (which was about the single biggest feature in [https://docs.oracle.com/javase/9/whatsnew/toc.htm Java 9]), the world moves on.

If version 9 was mostly a clean-up release with the much-hated Java Plugin and the Applet API having been deprecated, CMS GC and all garbage collector combinations involving it, as well, several rather big improvements and additions had been made to both the language and the virtual machine in the two subsequent versions, [https://openjdk.java.net/projects/jdk/10/ Java 10] and [https://openjdk.java.net/projects/jdk/11/ Java 11].

== What I assume ==

I assume that you are:

* fluent in the Java language (according to v8 language specification)
* aware of how Java garbage collectors work and why we need them
* familiar with what software profiling and auditing are and how they work
* have a clue about Linux scripting

= Language =

== Type Inference for Local Variables (v10) ==

Among the biggest additions to Java 10 is its [https://openjdk.java.net/jeps/286 ability to figure out the type of a variable] that has an initialiser, that is to say, perform LHS (left-hand-side) type inference.

RHS type inference should be quite familiar by now as it was by far the biggest change in Java 8 - [https://openjdk.java.net/projects/lambda/ Lambda Expressions for the Java Language] allow a programmer to save space, time, and improve readability, making heavy use of type inference all the time:

MockEventHandler eh = new MockEventHandler();
eh.setOnSomeEvent('''event -> new EventProcessor().process(event)''');

Actually, the above expression can be simplified even further:

MockEventHandler eh = new MockEventHandler();
eh.setOnSomeEvent('''EventProcessor::process''');

Notice how parameters in lambda expressions lack type declarations. Infact, they completely lack ''any'' declarations! You can [[Java 8 Lambda Expressions|read more about this in a small write-up of mine]].

In Java 10, this inference is extended even further. For cases when the compiler can infer the type of a variable, its declaration can be simplified to just the newly introduced ''reserved type'' <code>var</code>:

'''var''' x = new Integer(5);
'''var''' y = MyClass.Factory.getInstance();

if (x instanceof Integer) {
// matches
}
if (y instanceof MyClass) {
// matches
}

There are obviously some limitations for this new expression, such as:

* declarations without initialisers (which would require some distant action to infer the type, and that could violate the strongly-typed nature of the language because of multiple possible ''different'' initialisers at different execution points),
* <code>null</code> assignments (note that <code>null</code> ''is'' a type, but it is fairly useless in and of its own),
* [http://www.devcodenote.com/2015/04/variable-capture-in-java.html capture variables] (which need to be final, so they can not be inferred),
* [http://iteratrlearning.com/java/generics/2016/05/12/intersection-types-java-generics.html intersection types] (which can not be reliably inferred),

or generally whenever an assignment type is not denotable (such as with inline arrays, method references, and even <code>c.getClass()</code>).

I may write more about those limitations when I've used LHS type inference more, until then feel free to have a look at [https://openjdk.java.net/jeps/286 the JEP defining Java type inference] and its many references.

== Local Variable Syntax for Lambda Parameters (v11) ==

BLABLA

== New Nest-Based Field/Method Access Control (v11) ==

BLABLA

= Virtual Machine =

== Application Class Data Sharing (v10) ==

BLABLA

== No-Op Garbage Collector (v11) ==

BLABLA

== ZGC (v11) ==

BLABLA

= Tools =

== Flight Recorder (v11) ==

BLABLA

== Low-Impact Heap Profiling (v11) ==

BLABLA

== Launching Single-File Programs from Source (v11) ==

BLABLA

OpenShift Devel

2018-11-26T07:15:55Z

Gregab:

= General Topics =

TBD.

= Local OpenShift Development =

* [[CDK Tips]]

= Remote Cluster Development =

Something coming here.

CDK Tips

2018-11-26T07:12:34Z

Gregab: Added assumptions.

= Introduction =

CDK and Minishift are a nice, portable way of running a single-node OCP/OKD cluster on your development workstation. I collected some useful tips for an easier life with them.

== What I assume ==

I assume that you are:

* proficient in using the <code>bash</code> shell
* understand the basic of container mechanics such as:
** what is an image
** why do you need a registry
* know how to work with containers using <code>docker</code> or <code>podman</code>
* have used CDK or Minishift before

= Various CDK (Minishift) Tips =

First off: I shall drop the Minishift term here (unless where absolutely necessary) and regard CDK as an alias for Minishift.

== Configuration / Start Up ==

=== Useful Startup Options ===

Sometimes you need to figure out what the hell went wrong and why CDK is acting up on you. Try these:

$ '''cdk start --alsologtostderr --show-libmachine-logs -v 3'''

This will bump the log level up to ''debug'' and be very noisy on the console, but at least you'll get some info about what's cooking.

I personally tend to use these a lot when I'm playing around with some low-level settings and I need to see which components are affected by my changes.

=== Offline Use ===

Because I spend way too much time in airplanes and other transport modalities where internet is more or less absent, I've had a jolly fun time trying to make CDK work offline.

One of these days I'll write a howto on getting CDK to work '''''fully offline''''', with a local registry VM, a Gogs instance, and working DNS resolution.

Until then, there are some simple tricks that should work, provided you have all the images you need in local cache and all the hostnames you need in <code>/etc/hosts</code>.

Some of the interesting options:

; <code>--skip-registration</code>
: '''(CDK only)''' This will not attempt to register your VM in the Red Hat Customer Portal.
; <code>--skip-registry-check</code>
: This will skip the test for online registry availability, but will fail horribly somewhere down the line, unless you're certain you really have all the platform images ''for your version of OCP/OKD''.
; <code>--skip-startup-checks</code>
: This will skip all other start-up checks (such as image versions, oc client availability, etc.) not only the online ones. Make sure your CDK is in good shape by running the startup checks at least once before starting to skip them.

Have a look at <code>cdk config</code> to see the list of individual checks you can skip (look for the <code>skip-check</code> pattern):

$ '''cdk config | grep skip-check'''
* skip-check-deprecation
* skip-check-kvm-driver
* skip-check-xhyve-driver
* skip-check-hyperv-driver
* skip-check-iso-url
* skip-check-vm-driver
* skip-check-vbox-installed
* skip-check-openshift-version
* skip-check-openshift-release
* skip-check-clusterup-flags
* skip-check-instance-ip
* skip-check-network-host
* skip-check-network-ping
* skip-check-network-http
* skip-check-storage-mount
* skip-check-storage-usage
* skip-check-nameservers

'''NOTE:''' Unfortunately, although they are listed as configuration options in <code>cdk config</code> output, these are ignored if you set them using <code>cdk config set</code>.

== Administration ==

=== CDK Administration ===

TBD.

=== Cluster Administration ===

Sometimes you need to do something as <code>system:admin</code>, not just any cluster admin.

For example: in the ''CDK 3.6 / OCP 3.11'' combo, the <code>admin</code> addon fails for some reason, which means you end up with no remote cluster admin capability. What now?

* First, get a shell in the <code>origin</code> container inside the ''boot2docker'' VM:
$ '''cdk ssh'''
Last login: Fri Nov 16 06:25:09 2018 from gateway
[docker@minishift ~]$ '''docker exec -it origin sh'''
sh-4.2#

* Notice that the <code>oc</code> command is not configured to load auth data by default:
sh-4.2# '''oc whoami'''
error: Missing or incomplete configuration info. Please login or point to an existing, complete config file:

1. Via the command-line flag --config
2. Via the KUBECONFIG environment variable
3. In your home directory as ~/.kube/config

To view or setup config directly use the 'config' command.

* Next, run the <code>oc</code> command with the <code>--config</code> option, telling it where to find the <code>kubeconfig</code> file with <code>system:admin</code> credentials:
sh-4.2# '''oc --config=./openshift.local.config/master/admin.kubeconfig whoami'''
system:admin

* Then, depending on how long you intend to spend in the VM, alias the <code>oc</code> command:
sh-4.2# '''alias oc="oc --config=./openshift.local.config/master/admin.kubeconfig"'''
: Alternatively, you could set the <code>KUBECONFIG</code> env variable, of course.

* Then simply do your stuff!
sh-4.2# '''oc describe clusterrolebinding cluster-admin'''
Name: cluster-admin
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate=true
Role:
Kind: ClusterRole
Name: cluster-admin
Subjects:
Kind Name Namespace
---- ---- ---------
Group system:masters
sh-4.2# '''oc get users'''
NAME UID FULL NAME IDENTITIES
admin 6f98e840-e990-11e8-8676-16ea592bfedd anypassword:admin
developer d482a716-e98f-11e8-8676-16ea592bfedd anypassword:developer
sh-4.2# '''oc adm policy add-cluster-role-to-user cluster-admin admin'''
cluster role "cluster-admin" added: "admin"

== Local Shell Environment ==

=== Bash Completion for CDK ===

==== Enabling ====

Running <code>cdk completion bash</code> will output a '''bash completion''' recipe.

You can then place it into <code>/etc/bash_completion.d</code> (or wherever your shell expects it).
: (hint: on macOS, if using MacPorts, this is <code>/opt/local/etc/bash_completion.d</code>)

==== CDK vs Minishift ====

One caveat is that if you're using <code>cdk</code> rather than <code>minishift</code>, you'll have to replace (almost) everything in that file that says <code>minishift</code>, with <code>cdk</code> (because <code>--minishift-home</code> is a valid option for both).

So, a shortcut to getting it to work with <code>cdk</code> in a single pipeline would be:

$ cdk completion bash | sed 's/minishift/cdk/g; s/-cdk-/-minishift-/g' > cdk
$ sudo mv cdk /etc/bash_completion.d/
$ . /etc/bash_completion.d/cdk
$ cdk '''<TAB><TAB>'''
addons config delete docker-env image logs openshift setup-cdk start stop
completion console dns hostfolder ip oc-env profile ssh status version

Voila!

CDK Tips

2018-11-26T07:02:49Z

Gregab: Fixed a typo.

= Various CDK (Minishift) Tips =

First off: I shall drop the Minishift term here (unless where absolutely necessary) and regard CDK as an alias for Minishift.

== Configuration / Start Up ==

=== Useful Startup Options ===

Sometimes you need to figure out what the hell went wrong and why CDK is acting up on you. Try these:

$ '''cdk start --alsologtostderr --show-libmachine-logs -v 3'''

This will bump the log level up to ''debug'' and be very noisy on the console, but at least you'll get some info about what's cooking.

I personally tend to use these a lot when I'm playing around with some low-level settings and I need to see which components are affected by my changes.

=== Offline Use ===

Because I spend way too much time in airplanes and other transport modalities where internet is more or less absent, I've had a jolly fun time trying to make CDK work offline.

One of these days I'll write a howto on getting CDK to work '''''fully offline''''', with a local registry VM, a Gogs instance, and working DNS resolution.

Until then, there are some simple tricks that should work, provided you have all the images you need in local cache and all the hostnames you need in <code>/etc/hosts</code>.

Some of the interesting options:

; <code>--skip-registration</code>
: '''(CDK only)''' This will not attempt to register your VM in the Red Hat Customer Portal.
; <code>--skip-registry-check</code>
: This will skip the test for online registry availability, but will fail horribly somewhere down the line, unless you're certain you really have all the platform images ''for your version of OCP/OKD''.
; <code>--skip-startup-checks</code>
: This will skip all other start-up checks (such as image versions, oc client availability, etc.) not only the online ones. Make sure your CDK is in good shape by running the startup checks at least once before starting to skip them.

Have a look at <code>cdk config</code> to see the list of individual checks you can skip (look for the <code>skip-check</code> pattern):

$ '''cdk config | grep skip-check'''
* skip-check-deprecation
* skip-check-kvm-driver
* skip-check-xhyve-driver
* skip-check-hyperv-driver
* skip-check-iso-url
* skip-check-vm-driver
* skip-check-vbox-installed
* skip-check-openshift-version
* skip-check-openshift-release
* skip-check-clusterup-flags
* skip-check-instance-ip
* skip-check-network-host
* skip-check-network-ping
* skip-check-network-http
* skip-check-storage-mount
* skip-check-storage-usage
* skip-check-nameservers

'''NOTE:''' Unfortunately, although they are listed as configuration options in <code>cdk config</code> output, these are ignored if you set them using <code>cdk config set</code>.

== Administration ==

=== CDK Administration ===

TBD.

=== Cluster Administration ===

Sometimes you need to do something as <code>system:admin</code>, not just any cluster admin.

For example: in the ''CDK 3.6 / OCP 3.11'' combo, the <code>admin</code> addon fails for some reason, which means you end up with no remote cluster admin capability. What now?

* First, get a shell in the <code>origin</code> container inside the ''boot2docker'' VM:
$ '''cdk ssh'''
Last login: Fri Nov 16 06:25:09 2018 from gateway
[docker@minishift ~]$ '''docker exec -it origin sh'''
sh-4.2#

* Notice that the <code>oc</code> command is not configured to load auth data by default:
sh-4.2# '''oc whoami'''
error: Missing or incomplete configuration info. Please login or point to an existing, complete config file:

1. Via the command-line flag --config
2. Via the KUBECONFIG environment variable
3. In your home directory as ~/.kube/config

To view or setup config directly use the 'config' command.

* Next, run the <code>oc</code> command with the <code>--config</code> option, telling it where to find the <code>kubeconfig</code> file with <code>system:admin</code> credentials:
sh-4.2# '''oc --config=./openshift.local.config/master/admin.kubeconfig whoami'''
system:admin

* Then, depending on how long you intend to spend in the VM, alias the <code>oc</code> command:
sh-4.2# '''alias oc="oc --config=./openshift.local.config/master/admin.kubeconfig"'''
: Alternatively, you could set the <code>KUBECONFIG</code> env variable, of course.

* Then simply do your stuff!
sh-4.2# '''oc describe clusterrolebinding cluster-admin'''
Name: cluster-admin
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate=true
Role:
Kind: ClusterRole
Name: cluster-admin
Subjects:
Kind Name Namespace
---- ---- ---------
Group system:masters
sh-4.2# '''oc get users'''
NAME UID FULL NAME IDENTITIES
admin 6f98e840-e990-11e8-8676-16ea592bfedd anypassword:admin
developer d482a716-e98f-11e8-8676-16ea592bfedd anypassword:developer
sh-4.2# '''oc adm policy add-cluster-role-to-user cluster-admin admin'''
cluster role "cluster-admin" added: "admin"

== Local Shell Environment ==

=== Bash Completion for CDK ===

==== Enabling ====

Running <code>cdk completion bash</code> will output a '''bash completion''' recipe.

You can then place it into <code>/etc/bash_completion.d</code> (or wherever your shell expects it).
: (hint: on macOS, if using MacPorts, this is <code>/opt/local/etc/bash_completion.d</code>)

==== CDK vs Minishift ====

One caveat is that if you're using <code>cdk</code> rather than <code>minishift</code>, you'll have to replace (almost) everything in that file that says <code>minishift</code>, with <code>cdk</code> (because <code>--minishift-home</code> is a valid option for both).

So, a shortcut to getting it to work with <code>cdk</code> in a single pipeline would be:

$ cdk completion bash | sed 's/minishift/cdk/g; s/-cdk-/-minishift-/g' > cdk
$ sudo mv cdk /etc/bash_completion.d/
$ . /etc/bash_completion.d/cdk
$ cdk '''<TAB><TAB>'''
addons config delete docker-env image logs openshift setup-cdk start stop
completion console dns hostfolder ip oc-env profile ssh status version

Voila!

Changes in Java 10 and 11

2018-11-25T19:17:27Z

Gregab: Some more assumptions.

= Introduction =

JDK 11 was released almost two months ago (as of this writing), yet it almost seems as if nobody noticed it.

After the roar and thunder of Java 9's JPMS (or Jigsaw, as some may know it) it's gone awfully quiet on the Java front.

But that's far from saying nothing is happening - after [http://openjdk.java.net/jeps/261 Java Module System] (which was about the single biggest feature in [https://docs.oracle.com/javase/9/whatsnew/toc.htm Java 9]), the world moves on.

If version 9 was mostly a clean-up release with the much-hated Java Plugin and the Applet API having been deprecated, CMS GC and all garbage collector combinations involving it, as well, several rather big improvements and additions had been made to both the language and the virtual machine in the two subsequent versions, [https://openjdk.java.net/projects/jdk/10/ Java 10] and [https://openjdk.java.net/projects/jdk/11/ Java 11].

== What I assume ==

I assume that you are:

* fluent in the Java language (according to v8 language specification)
* aware of how Java garbage collectors work and why we need them
* familiar with what software profiling and auditing are and how they work
* have a clue about Linux scripting

= Language =

== Type Inference for Local Variables (v10) ==

Among the biggest additions to Java 10 is its [https://openjdk.java.net/jeps/286 ability to figure out the type of a variable] that has an initialiser, that is to say, perform LHS (left-hand-side) type inference.

RHS type inference should be quite familiar by now as it was by far the biggest change in Java 8 - [https://openjdk.java.net/projects/lambda/ Lambda Expressions for the Java Language] allow a programmer to save space, time, and improve readability, making heavy use of type inference all the time:

MockEventHandler eh = new MockEventHandler();
eh.setOnSomeEvent(event -> new EventProcessor().process(event));

Notice how parameters in lambda expressions lack type declarations. Infact, they completely lack ''any'' declarations! You can [[Java 8 Lambda Expressions|read more about this in a small write-up of mine]].

In Java 10, this inference is extended even further. For cases when the compiler can infer the type of a variable, its declaration can be simplified to just the newly introduced ''reserved type'' <code>var</code>:

var x = new Integer(5);
var y = MyClass.Factory.getInstance();

if (x instanceof Integer) {
// matches
}
if (y instanceof MyClass) {
// matches
}

There are obviously some limitations for this new expression, such as:

* declarations without initialisers (which would require some distant action to infer the type, and that could violate the strongly-typed nature of the language because of multiple possible ''different'' initialisers at different execution points),
* <code>null</code> assignments (note that <code>null</code> ''is'' a type, but it is fairly useless in and of its own),
* [http://www.devcodenote.com/2015/04/variable-capture-in-java.html capture variables] (which need to be final, so they can not be inferred),
* [http://iteratrlearning.com/java/generics/2016/05/12/intersection-types-java-generics.html intersection types] (which can not be reliably inferred),

or generally whenever an assignment type is not denotable (such as with inline arrays, method references, and even <code>c.getClass()</code>).

I may write more about those limitations when I've used LHS type inference more, until then feel free to have a look at [https://openjdk.java.net/jeps/286 the JEP defining Java type inference] and its many references.

== Local Variable Syntax for Lambda Parameters (v11) ==

BLABLA

== New Nest-Based Field/Method Access Control (v11) ==

BLABLA

= Virtual Machine =

== Application Class Data Sharing (v10) ==

BLABLA

== No-Op Garbage Collector (v11) ==

BLABLA

== ZGC (v11) ==

BLABLA

= Tools =

== Flight Recorder (v11) ==

BLABLA

== Low-Impact Heap Profiling (v11) ==

BLABLA

== Launching Single-File Programs from Source (v11) ==

BLABLA

Changes in Java 10 and 11

2018-11-25T19:16:04Z

Gregab: Added assumptions.

= Introduction =

JDK 11 was released almost two months ago (as of this writing), yet it almost seems as if nobody noticed it.

After the roar and thunder of Java 9's JPMS (or Jigsaw, as some may know it) it's gone awfully quiet on the Java front.

But that's far from saying nothing is happening - after [http://openjdk.java.net/jeps/261 Java Module System] (which was about the single biggest feature in [https://docs.oracle.com/javase/9/whatsnew/toc.htm Java 9]), the world moves on.

If version 9 was mostly a clean-up release with the much-hated Java Plugin and the Applet API having been deprecated, CMS GC and all garbage collector combinations involving it, as well, several rather big improvements and additions had been made to both the language and the virtual machine in the two subsequent versions, [https://openjdk.java.net/projects/jdk/10/ Java 10] and [https://openjdk.java.net/projects/jdk/11/ Java 11].

== What I assume ==

I assume that you are:

* fluent in the Java language (according to v8 language specification)
* know how Java garbage collectors work and why we need them
* have a clue about Linux scripting

= Language =

== Type Inference for Local Variables (v10) ==

Among the biggest additions to Java 10 is its [https://openjdk.java.net/jeps/286 ability to figure out the type of a variable] that has an initialiser, that is to say, perform LHS (left-hand-side) type inference.

RHS type inference should be quite familiar by now as it was by far the biggest change in Java 8 - [https://openjdk.java.net/projects/lambda/ Lambda Expressions for the Java Language] allow a programmer to save space, time, and improve readability, making heavy use of type inference all the time:

MockEventHandler eh = new MockEventHandler();
eh.setOnSomeEvent(event -> new EventProcessor().process(event));

Notice how parameters in lambda expressions lack type declarations. Infact, they completely lack ''any'' declarations! You can [[Java 8 Lambda Expressions|read more about this in a small write-up of mine]].

In Java 10, this inference is extended even further. For cases when the compiler can infer the type of a variable, its declaration can be simplified to just the newly introduced ''reserved type'' <code>var</code>:

var x = new Integer(5);
var y = MyClass.Factory.getInstance();

if (x instanceof Integer) {
// matches
}
if (y instanceof MyClass) {
// matches
}

There are obviously some limitations for this new expression, such as:

* declarations without initialisers (which would require some distant action to infer the type, and that could violate the strongly-typed nature of the language because of multiple possible ''different'' initialisers at different execution points),
* <code>null</code> assignments (note that <code>null</code> ''is'' a type, but it is fairly useless in and of its own),
* [http://www.devcodenote.com/2015/04/variable-capture-in-java.html capture variables] (which need to be final, so they can not be inferred),
* [http://iteratrlearning.com/java/generics/2016/05/12/intersection-types-java-generics.html intersection types] (which can not be reliably inferred),

or generally whenever an assignment type is not denotable (such as with inline arrays, method references, and even <code>c.getClass()</code>).

I may write more about those limitations when I've used LHS type inference more, until then feel free to have a look at [https://openjdk.java.net/jeps/286 the JEP defining Java type inference] and its many references.

== Local Variable Syntax for Lambda Parameters (v11) ==

BLABLA

== New Nest-Based Field/Method Access Control (v11) ==

BLABLA

= Virtual Machine =

== Application Class Data Sharing (v10) ==

BLABLA

== No-Op Garbage Collector (v11) ==

BLABLA

== ZGC (v11) ==

BLABLA

= Tools =

== Flight Recorder (v11) ==

BLABLA

== Low-Impact Heap Profiling (v11) ==

BLABLA

== Launching Single-File Programs from Source (v11) ==

BLABLA

Changes in Java 10 and 11

2018-11-21T19:10:51Z

Gregab: Added bits.

= Introduction =

JDK 11 was released almost two months ago (as of this writing), yet it almost seems as if nobody noticed it.

After the roar and thunder of Java 9's JPMS (or Jigsaw, as some may know it) it's gone awfully quiet on the Java front.

But that's far from saying nothing is happening - after [http://openjdk.java.net/jeps/261 Java Module System] (which was about the single biggest feature in [https://docs.oracle.com/javase/9/whatsnew/toc.htm Java 9]), the world moves on.

If version 9 was mostly a clean-up release with the much-hated Java Plugin and the Applet API having been deprecated, CMS GC and all garbage collector combinations involving it, as well, several rather big improvements and additions had been made to both the language and the virtual machine in the two subsequent versions, [https://openjdk.java.net/projects/jdk/10/ Java 10] and [https://openjdk.java.net/projects/jdk/11/ Java 11].

= Language =

== Type Inference for Local Variables (v10) ==

Among the biggest additions to Java 10 is its [https://openjdk.java.net/jeps/286 ability to figure out the type of a variable] that has an initialiser, that is to say, perform LHS (left-hand-side) type inference.

RHS type inference should be quite familiar by now as it was by far the biggest change in Java 8 - [https://openjdk.java.net/projects/lambda/ Lambda Expressions for the Java Language] allow a programmer to save space, time, and improve readability, making heavy use of type inference all the time:

MockEventHandler eh = new MockEventHandler();
eh.setOnSomeEvent(event -> new EventProcessor().process(event));

Notice how parameters in lambda expressions lack type declarations. Infact, they completely lack ''any'' declarations! You can [[Java 8 Lambda Expressions|read more about this in a small write-up of mine]].

In Java 10, this inference is extended even further. For cases when the compiler can infer the type of a variable, its declaration can be simplified to just the newly introduced ''reserved type'' <code>var</code>:

var x = new Integer(5);
var y = MyClass.Factory.getInstance();

if (x instanceof Integer) {
// matches
}
if (y instanceof MyClass) {
// matches
}

There are obviously some limitations for this new expression, such as:

* declarations without initialisers (which would require some distant action to infer the type, and that could violate the strongly-typed nature of the language because of multiple possible ''different'' initialisers at different execution points),
* <code>null</code> assignments (note that <code>null</code> ''is'' a type, but it is fairly useless in and of its own),
* [http://www.devcodenote.com/2015/04/variable-capture-in-java.html capture variables] (which need to be final, so they can not be inferred),
* [http://iteratrlearning.com/java/generics/2016/05/12/intersection-types-java-generics.html intersection types] (which can not be reliably inferred),

or generally whenever an assignment type is not denotable (such as with inline arrays, method references, and even <code>c.getClass()</code>).

I may write more about those limitations when I've used LHS type inference more, until then feel free to have a look at [https://openjdk.java.net/jeps/286 the JEP defining Java type inference] and its many references.

== Local Variable Syntax for Lambda Parameters (v11) ==

BLABLA

== New Nest-Based Field/Method Access Control (v11) ==

BLABLA

= Virtual Machine =

== Application Class Data Sharing (v10) ==

BLABLA

== No-Op Garbage Collector (v11) ==

BLABLA

== ZGC (v11) ==

BLABLA

= Tools =

== Flight Recorder (v11) ==

BLABLA

== Low-Impact Heap Profiling (v11) ==

BLABLA

== Launching Single-File Programs from Source (v11) ==

BLABLA