---
title: Using machine config objects to configure nodes
---

# Using machine config objects to configure nodes {#machine-configs-configure}

You can create `MachineConfig` custom resources (CR) that modify files, systemd unit files, and other operating system features running on OpenShift Container Platform nodes. By using `MachineConfig` objects, you can perform tasks such as disabling chronyd, adding kernel arguments, enabling multipathing, and adding RHCOS extensions.

For more ideas on working with machine configs, see "How to update ssh keys after installation in OpenShift 4?", "Container image signatures", "Enabling SCTP in Openshift Container Platform 4", and "How to provide custom iSCSI initiatornames for nodes".

OpenShift Container Platform supports Ignition specification version 3.5. For more information, see "Configuration Specification v3.5.0 (Ignition documentation)". You should base all new machine configs you create going forward on Ignition specification version 3.5. If you are upgrading your OpenShift Container Platform cluster, any existing machine configs with a previous Ignition specification will be translated automatically to specification version 3.5.

There might be situations where the configuration on a node does not fully match what the currently-applied machine config specifies. This state is called *configuration drift*. The Machine Config Daemon (MCD) regularly checks the nodes for configuration drift. If the MCD detects configuration drift, the MCO marks the node `degraded` until an administrator corrects the node configuration. A degraded node is online and operational, but, it cannot be updated. For more information on configuration drift, see "Understanding configuration drift detection".

> [!TIP]
> Use the following "Configuring chrony time service" procedure as a model for how to go about adding other configuration files to OpenShift Container Platform nodes.

## Configuring chrony time service {#installation-special-config-chrony_machine-configs-configure}

You can set the time server and related settings used by the chrony time service (`chronyd`) by modifying the contents of the `chrony.conf` file and passing those contents to your nodes as a machine config.

For more information on chrony best practices, see the following resources:

- [Configuring chrony (Red Hat Knowledgebase article)](https://access.redhat.com/solutions/3073261)
- [Best practices for NTP (Red Hat Knowledgebase article)](https://access.redhat.com/solutions/778603)
- [Basic chrony NTP troubleshooting (Red Hat Ceph Storage documentation)](https://docs.redhat.com/en/documentation/red_hat_ceph_storage/8/html-single/troubleshooting_guide/basic-chrony-NTP-troubleshooting_diag#basic-chrony-NTP-troubleshooting_diag)

**Procedure**

1. Create a Butane config including the contents of the `chrony.conf` file. For example, to configure chrony on worker nodes, create a `99-worker-chrony.bu` file.

   > [!NOTE]
   > The [Butane version](https://coreos.github.io/butane/specs/) you specify in the config file should match the OpenShift Container Platform version and always ends in `0`. For example, `4.22.0`. See "Creating machine configs with Butane" for information about Butane.

   ```yaml
   variant: openshift
   version: 4.22.0
   metadata:
     name: 99-worker-chrony
     labels:
       machineconfiguration.openshift.io/role: worker
   storage:
     files:
     - path: /etc/chrony.conf
       mode: 0644
       overwrite: true
       contents:
         inline: |
           pool 0.rhel.pool.ntp.org iburst
           driftfile /var/lib/chrony/drift
           makestep 1.0 3
           rtcsync
           logdir /var/log/chrony
   ```

   - `name: 99-worker-chrony` - Specify a name for the machine config file. On control plane nodes, substitute `master` for `worker`.
   - `machineconfiguration.openshift.io/role: worker` - On control plane nodes, substitute `master` for `worker`.
   - `mode: 0644` - Specify an octal value mode for the `mode` field in the machine config file. After creating the file and applying the changes, the `mode` is converted to a decimal value. You can check the YAML file with the command `oc get mc <mc-name> -o yaml`.
   - `pool 0.rhel.pool.ntp.org iburst` - Specify any valid, reachable time source, such as the one provided by your DHCP server.

   > [!NOTE]
   > For all-machine to all-machine communication, the Network Time Protocol (NTP) on UDP is port `123`. If an external NTP time server is configured, you must open UDP port `123`.

   Alternatively, you can specify any of the following NTP servers: `1.rhel.pool.ntp.org`, `2.rhel.pool.ntp.org`, or `3.rhel.pool.ntp.org`. When you use NTP with your DHCP server, you must set the `sourcedir /run/chrony-dhcp` parameter in the `chrony.conf` file.
2. Use Butane to generate a `MachineConfig` object file, `99-worker-chrony.yaml`, containing the configuration to be delivered to the nodes:

   ```terminal
   $ butane 99-worker-chrony.bu -o 99-worker-chrony.yaml
   ```
3. Apply the configurations in one of two ways:

   - If the cluster is not running yet, after you generate manifest files, add the `MachineConfig` object file to the `<installation_directory>/openshift` directory, and then continue to create the cluster.
   - If the cluster is already running, apply the file:

     ```terminal
     $ oc apply -f ./99-worker-chrony.yaml
     ```

## Disabling the chrony time service {#cnf-disable-chronyd_machine-configs-configure}

You can disable the chrony time service (`chronyd`) for nodes with a specific role by using a `MachineConfig` custom resource (CR).

**Prerequisites**

- Install the OpenShift CLI (`oc`).
- Log in as a user with `cluster-admin` privileges.

**Procedure**

1. Create the `MachineConfig` CR that disables `chronyd` for the specified node role.

   1. Save the following YAML in the `disable-chronyd.yaml` file:

      ```yaml
      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        labels:
          machineconfiguration.openshift.io/role: <node_role>
        name: disable-chronyd
      spec:
        config:
          ignition:
            version: 3.5.0
          systemd:
            units:
              - contents: |
                  [Unit]
                  Description=NTP client/server
                  Documentation=man:chronyd(8) man:chrony.conf(5)
                  After=ntpdate.service sntp.service ntpd.service
                  Conflicts=ntpd.service systemd-timesyncd.service
                  ConditionCapability=CAP_SYS_TIME
                  [Service]
                  Type=forking
                  PIDFile=/run/chrony/chronyd.pid
                  EnvironmentFile=-/etc/sysconfig/chronyd
                  ExecStart=/usr/sbin/chronyd $OPTIONS
                  ExecStartPost=/usr/libexec/chrony-helper update-daemon
                  PrivateTmp=yes
                  ProtectHome=yes
                  ProtectSystem=full
                  [Install]
                  WantedBy=multi-user.target
                enabled: false
                name: "chronyd.service"
              - name: "kubelet-dependencies.target"
                contents: |
                  [Unit]
                  Description=Dependencies necessary to run kubelet
                  Documentation=https://github.com/openshift/machine-config-operator/
                  Requires=basic.target network-online.target
                  Wants=NetworkManager-wait-online.service crio-wipe.service
                  Wants=rpc-statd.service
      ```

      where:

      `metadata.labels`
      :   Specifies the node role where you want to disable `chronyd`, for example, `master`.
   2. Create the `MachineConfig` CR by running the following command:

      ```terminal
      $ oc create -f disable-chronyd.yaml
      ```

## Adding kernel arguments to nodes {#nodes-nodes-kernel-arguments_machine-configs-configure}

In some special cases, you can add kernel arguments to a set of nodes in your cluster to customize the kernel behavior to meet specific needs you might have.

You should add kernel arguments with caution and a clear understanding of the implications of the arguments you set.

> [!WARNING]
> Improper use of kernel arguments can result in your systems becoming unbootable.

Examples of kernel arguments you could set include:

- **nosmt**: Disables symmetric multithreading (SMT) in the kernel. Multithreading allows multiple logical threads for each CPU. You could consider `nosmt` in multi-tenant environments to reduce risks from potential cross-thread attacks. By disabling SMT, you essentially choose security over performance.
- **enforcing=0**: Configures Security Enhanced Linux (SELinux) to run in permissive mode. In permissive mode, the system acts as if SELinux is enforcing the loaded security policy, including labeling objects and emitting access denial entries in the logs, but it does not actually deny any operations. While not supported for production systems, permissive mode can be helpful for debugging.

  > [!WARNING]
  > Disabling SELinux on RHCOS in production is not supported. After SELinux has been disabled on a node, it must be re-provisioned before re-inclusion in a production cluster.

See [Kernel.org kernel parameters](https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt) for a list and descriptions of kernel arguments.

In the following procedure, you create a `MachineConfig` object that identifies:

- A set of machines to which you want to add the kernel argument. In this case, machines with a worker role.
- Kernel arguments that are appended to the end of the existing kernel arguments.
- A label that indicates where in the list of machine configs the change is applied.

**Prerequisites**

- You have `cluster-admin` privileges.
- Your cluster is running.

**Procedure**

1. List existing `MachineConfig` objects for your OpenShift Container Platform cluster to determine how to label your machine config:

   ```terminal
   $ oc get MachineConfig
   ```

   ```terminal {title="Example output"}
   NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
   00-master                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   00-worker                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   01-master-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   01-master-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   01-worker-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   01-worker-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   99-master-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   99-master-ssh                                                                                 3.2.0             40m
   99-worker-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   99-worker-ssh                                                                                 3.2.0             40m
   rendered-master-23e785de7587df95a4b517e0647e5ab7   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   rendered-worker-5d596d9293ca3ea80c896a1191735bb1   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   ```
2. Create a `MachineConfig` object file that identifies the kernel argument (for example, `05-worker-kernelarg-selinuxpermissive.yaml`)

   ```yaml
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 05-worker-kernelarg-selinuxpermissive
   spec:
     kernelArguments:
       - enforcing=0
   ```

   where:

   `machineconfiguration.openshift.io/role`
   :   Specifies a label to apply changes to specific nodes.

   `name`
   :   Specifies a name to identify where it fits among the machine configs (05) and what it does (adds a kernel argument to configure SELinux permissive mode).

   `kernelArguments`
   :   Specifies the exact kernel argument as `enforcing=0`.
3. Create the new machine config:

   ```terminal
   $ oc create -f 05-worker-kernelarg-selinuxpermissive.yaml
   ```
4. Check the machine configs to see that the new one was added:

   ```terminal
   $ oc get MachineConfig
   ```

   ```terminal {title="Example output"}
   NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
   00-master                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   00-worker                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   01-master-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   01-master-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   01-worker-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   01-worker-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   05-worker-kernelarg-selinuxpermissive                                                         3.5.0             105s
   99-master-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   99-master-ssh                                                                                 3.2.0             40m
   99-worker-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   99-worker-ssh                                                                                 3.2.0             40m
   rendered-master-23e785de7587df95a4b517e0647e5ab7   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   rendered-worker-5d596d9293ca3ea80c896a1191735bb1   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   ```
5. Check the nodes:

   ```terminal
   $ oc get nodes
   ```

   ```terminal {title="Example output"}
   NAME                           STATUS                     ROLES    AGE   VERSION
   ip-10-0-136-161.ec2.internal   Ready                      worker   28m   v1.35.4
   ip-10-0-136-243.ec2.internal   Ready                      master   34m   v1.35.4
   ip-10-0-141-105.ec2.internal   Ready,SchedulingDisabled   worker   28m   v1.35.4
   ip-10-0-142-249.ec2.internal   Ready                      master   34m   v1.35.4
   ip-10-0-153-11.ec2.internal    Ready                      worker   28m   v1.35.4
   ip-10-0-153-150.ec2.internal   Ready                      master   34m   v1.35.4
   ```

   You can see that scheduling on each worker node is disabled as the change is being applied.
6. Check that the kernel argument worked by going to one of the worker nodes and listing the kernel command-line arguments (in `/proc/cmdline` on the host):

   ```terminal
   $ oc debug node/ip-10-0-141-105.ec2.internal
   ```

   ```terminal {title="Example output"}
   Starting pod/ip-10-0-141-105ec2internal-debug ...
   To use host binaries, run `chroot /host`

   sh-4.2# cat /host/proc/cmdline
   BOOT_IMAGE=/ostree/rhcos-... console=tty0 console=ttyS0,115200n8
   rootflags=defaults,prjquota rw root=UUID=fd0... ostree=/ostree/boot.0/rhcos/16...
   coreos.oem.id=qemu coreos.oem.id=ec2 ignition.platform.id=ec2 enforcing=0

   sh-4.2# exit
   ```

   You should see the `enforcing=0` argument added to the other kernel arguments.

## Enabling multipathing with kernel arguments on RHCOS {#rhcos-enabling-multipath-day-2_machine-configs-configure}

You can achieve higher host availability by enabling multipathing on the primary disk, which allows stronger resilience to hardware failure, by using a `MachineConfig` object.

> [!IMPORTANT]
> Enabling multipathing during installation is supported and recommended for nodes provisioned in OpenShift Container Platform. In setups where any I/O to non-optimized paths results in I/O system errors, you must enable multipathing at installation time. For more information about enabling multipathing during installation time, see "Enabling multipathing post installation" in the *Installing on bare metal* documentation.

> [!IMPORTANT]
> On IBM Z(R) and IBM(R) LinuxONE, you can enable multipathing only if you configured your cluster for it during installation. For more information, see "Installing RHCOS and starting the OpenShift Container Platform bootstrap process" in *Installing a cluster with z/VM on IBM Z(R) and IBM(R) LinuxONE*.

> [!IMPORTANT]
> When an OpenShift Container Platform cluster is installed or configured as a postinstallation activity on a single VIOS host with "vSCSI" storage on IBM Power(R) with multipath configured, the CoreOS nodes with multipath enabled fail to boot. This behavior is expected, as only one path is available to the node.

**Prerequisites**

- You have a running OpenShift Container Platform cluster.
- You are logged in to the cluster as a user with administrative privileges.
- You have confirmed that the disk is enabled for multipathing. Multipathing is only supported on hosts that are connected to a SAN via an HBA adapter.

**Procedure**

1. To enable multipathing postinstallation on control plane nodes:

   - Create a machine config file, such as `99-master-kargs-mpath.yaml`, that instructs the cluster to add the `master` label and that identifies the multipath kernel argument, for example:

     ```yaml
     apiVersion: machineconfiguration.openshift.io/v1
     kind: MachineConfig
     metadata:
       labels:
         machineconfiguration.openshift.io/role: "master"
       name: 99-master-kargs-mpath
     spec:
       kernelArguments:
         - 'rd.multipath=default'
         - 'root=/dev/disk/by-label/dm-mpath-root'
     ```
2. To enable multipathing postinstallation on worker nodes:

   - Create a machine config file, such as `99-worker-kargs-mpath.yaml`, that instructs the cluster to add the `worker` label and that identifies the multipath kernel argument, for example:

     ```yaml
     apiVersion: machineconfiguration.openshift.io/v1
     kind: MachineConfig
     metadata:
       labels:
         machineconfiguration.openshift.io/role: "worker"
       name: 99-worker-kargs-mpath
     spec:
       kernelArguments:
         - 'rd.multipath=default'
         - 'root=/dev/disk/by-label/dm-mpath-root'
     ```
3. Create the new machine config by using either the master or worker YAML file you previously created:

   ```terminal
   $ oc create -f ./99-worker-kargs-mpath.yaml
   ```
4. Check the machine configs to see that the new one was added:

   ```terminal
   $ oc get MachineConfig
   ```

   ```terminal {title="Example output"}
   NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
   00-master                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   00-worker                                          52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   01-master-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   01-master-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   01-worker-container-runtime                        52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   01-worker-kubelet                                  52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   99-master-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   99-master-ssh                                                                                 3.2.0             40m
   99-worker-generated-registries                     52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   99-worker-kargs-mpath                              52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             105s
   99-worker-ssh                                                                                 3.2.0             40m
   rendered-master-23e785de7587df95a4b517e0647e5ab7   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   rendered-worker-5d596d9293ca3ea80c896a1191735bb1   52dd3ba6a9a527fc3ab42afac8d12b693534c8c9   3.5.0             33m
   ```
5. Check the nodes:

   ```terminal
   $ oc get nodes
   ```

   ```terminal {title="Example output"}
   NAME                           STATUS                     ROLES    AGE   VERSION
   ip-10-0-136-161.ec2.internal   Ready                      worker   28m   v1.35.4
   ip-10-0-136-243.ec2.internal   Ready                      master   34m   v1.35.4
   ip-10-0-141-105.ec2.internal   Ready,SchedulingDisabled   worker   28m   v1.35.4
   ip-10-0-142-249.ec2.internal   Ready                      master   34m   v1.35.4
   ip-10-0-153-11.ec2.internal    Ready                      worker   28m   v1.35.4
   ip-10-0-153-150.ec2.internal   Ready                      master   34m   v1.35.4
   ```

   You can see that scheduling on each worker node is disabled as the change is being applied.
6. Check that the kernel argument worked by going to one of the worker nodes and listing the kernel command-line arguments (in `/proc/cmdline` on the host):

   ```terminal
   $ oc debug node/ip-10-0-141-105.ec2.internal
   ```

   ```terminal {title="Example output"}
   Starting pod/ip-10-0-141-105ec2internal-debug ...
   To use host binaries, run `chroot /host`

   sh-4.2# cat /host/proc/cmdline
   ...
   rd.multipath=default root=/dev/disk/by-label/dm-mpath-root
   ...

   sh-4.2# exit
   ```

   You should see the added kernel arguments.

## Adding a real-time kernel to nodes {#nodes-nodes-rtkernel-arguments_machine-configs-configure}

If your OpenShift Container Platform workloads require real-time operating system characteristics, you can switch your machines to the Linux real-time kernel. Switching to the real-time kernel provides a higher degree of determinism for your OpenShift Container Platform workloads.

Even though Linux is not a real-time operating system, the Linux real-time kernel includes a preemptive scheduler that provides the operating system with real-time characteristics. For OpenShift Container Platform, 4.22 you can make the switch to real-time kernel by using a `MachineConfig` object.

Although making the change is as simple as changing a machine config `kernelType` setting to `realtime`, there are a few other considerations before making the change:

- Currently, real-time kernel is supported only on worker nodes, and only for radio access network (RAN) use.
- The following procedure is fully supported with bare metal installations that use systems that are certified for Red Hat Enterprise Linux for Real Time 8.
- Real-time support in OpenShift Container Platform is limited to specific subscriptions.
- The following procedure is also supported for use with Google Cloud.

**Prerequisites**

- Have a running OpenShift Container Platform cluster (version 4.4 or later).
- Log in to the cluster as a user with administrative privileges.

**Procedure**

1. Create a machine config for the real-time kernel: Create a YAML file (for example, `99-worker-realtime.yaml`) that contains a `MachineConfig` object for the `realtime` kernel type. This example tells the cluster to use a real-time kernel for all worker nodes:

   ```terminal
   $ cat << EOF > 99-worker-realtime.yaml
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: "worker"
     name: 99-worker-realtime
   spec:
     kernelType: realtime
   EOF
   ```
2. Add the machine config to the cluster. Type the following to add the machine config to the cluster:

   ```terminal
   $ oc create -f 99-worker-realtime.yaml
   ```
3. Check the real-time kernel: Once each impacted node reboots, log in to the cluster and run the following commands to make sure that the real-time kernel has replaced the regular kernel for the set of nodes you configured:

   ```terminal
   $ oc get nodes
   ```

   ```terminal {title="Example output"}
   NAME                                        STATUS  ROLES    AGE   VERSION
   ip-10-0-143-147.us-east-2.compute.internal  Ready   worker   103m  v1.35.4
   ip-10-0-146-92.us-east-2.compute.internal   Ready   worker   101m  v1.35.4
   ip-10-0-169-2.us-east-2.compute.internal    Ready   worker   102m  v1.35.4
   ```

   ```terminal
   $ oc debug node/ip-10-0-143-147.us-east-2.compute.internal
   ```

   ```terminal {title="Example output"}
   Starting pod/ip-10-0-143-147us-east-2computeinternal-debug ...
   To use host binaries, run `chroot /host`

   sh-4.4# uname -a
   Linux <worker_node> 4.18.0-147.3.1.rt24.96.el8_1.x86_64 #1 SMP PREEMPT RT
           Wed Nov 27 18:29:55 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
   ```

   The kernel name contains `rt` and text “PREEMPT RT” indicates that this is a real-time kernel.
4. To go back to the regular kernel, delete the `MachineConfig` object:

   ```terminal
   $ oc delete -f 99-worker-realtime.yaml
   ```

## Configuring journald settings {#machineconfig-modify-journald_machine-configs-configure}

To configure settings for the `journald` service on OpenShift Container Platform nodes, you can modify the appropriate configuration file and pass the file to the appropriate pool of nodes as a machine config.

This procedure describes how to modify `journald` rate limiting settings in the `/etc/systemd/journald.conf` file and apply them to worker nodes. See the `journald.conf` man page for information on how to use that file.

**Prerequisites**

- Have a running OpenShift Container Platform cluster.
- Log in to the cluster as a user with administrative privileges.

**Procedure**

1. Create a Butane config file, `40-worker-custom-journald.bu`, that includes an `/etc/systemd/journald.conf` file with the required settings.

   > [!NOTE]
   > The [Butane version](https://coreos.github.io/butane/specs/) you specify in the config file should match the OpenShift Container Platform version and always ends in `0`. For example, `4.22.0`. See "Creating machine configs with Butane" for information about Butane.

   ```yaml
   variant: openshift
   version: 4.22.0
   metadata:
     name: 40-worker-custom-journald
     labels:
       machineconfiguration.openshift.io/role: worker
   storage:
     files:
     - path: /etc/systemd/journald.conf
       mode: 0644
       overwrite: true
       contents:
         inline: |
           # Disable rate limiting
           RateLimitInterval=1s
           RateLimitBurst=10000
           Storage=volatile
           Compress=no
           MaxRetentionSec=30s
   ```
2. Use Butane to generate a `MachineConfig` object file, `40-worker-custom-journald.yaml`, containing the configuration to be delivered to the worker nodes:

   ```terminal
   $ butane 40-worker-custom-journald.bu -o 40-worker-custom-journald.yaml
   ```
3. Apply the machine config to the pool:

   ```terminal
   $ oc apply -f 40-worker-custom-journald.yaml
   ```
4. Check that the new machine config is applied and that the nodes are not in a degraded state. It might take a few minutes. The worker pool will show the updates in progress, as each node successfully has the new machine config applied:

   ```terminal
   $ oc get machineconfigpool
   ```

   ```terminal {title="Example output"}
   NAME   CONFIG             UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
   master rendered-master-35 True    False    False    3            3                 3                   0                    34m
   worker rendered-worker-d8 False   True     False    3            1                 1                   0                    34m
   ```
5. To check that the change was applied, you can log in to a worker node:

   ```terminal
   $ oc get node | grep worker
   ```

   ```terminal {title="Example output"}
   ip-10-0-0-1.us-east-2.compute.internal   Ready    worker   39m   v0.0.0-master+$Format:%h$
   ```

   ```terminal
   $ oc debug node/ip-10-0-0-1.us-east-2.compute.internal
   ```

   ```terminal {title="Example output"}
   Starting pod/ip-10-0-141-142us-east-2computeinternal-debug ...
   ...
   sh-4.2# chroot /host
   sh-4.4# cat /etc/systemd/journald.conf
   # Disable rate limiting
   RateLimitInterval=1s
   RateLimitBurst=10000
   Storage=volatile
   Compress=no
   MaxRetentionSec=30s
   sh-4.4# exit
   ```

## Adding extensions to RHCOS {#rhcos-add-extensions_machine-configs-configure}

You can add software packages to Red Hat Enterprise Linux CoreOS (RHCOS) systems by using extension packages to add a minimal set of features to specific nodes.

RHCOS is a minimal container-oriented op-system-base-full operating system, designed to provide a common set of capabilities to OpenShift Container Platform clusters across all platforms. Although adding software packages to RHCOS systems is generally discouraged, you can add any of the following extensions to extend RHCOS:

- **usbguard**: The `usbguard` extension protects RHCOS systems from attacks by intrusive USB devices. For `usbguard`, you must create additional `MachineConfig` objects to start and enable the services as described in the following procedure. For more information, see [USBGuard](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/security_hardening/index#usbguard_protecting-systems-against-intrusive-usb-devices).
- **kerberos**: The `kerberos` extension provides a mechanism that allows both users and machines to identify themselves to the network to receive defined, limited access to the areas and services that an administrator has configured. For more information, see [Using Kerberos](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system-level_authentication_guide/using_kerberos), including how to set up a Kerberos client and mount a Kerberized NFS share.
- **sandboxed-containers**: The `sandboxed-containers` extension contains RPMs for Kata, QEMU, and its dependencies. For more information, see [OpenShift Sandboxed Containers](https://docs.redhat.com/en/documentation/openshift_sandboxed_containers/latest).
- **ipsec**: The `ipsec` extension contains RPMs for libreswan and NetworkManager-libreswan.
- **wasm**: The `wasm` extension enables Developer Preview functionality in OpenShift Container Platform for users who want to use WASM-supported workloads.
- **sysstat**: Adding the `sysstat` extension provides additional performance monitoring for OpenShift Container Platform nodes, including the system activity reporter (`sar`) command for collecting and reporting information.
- **kernel-devel**: The `kernel-devel` extension provides kernel headers and makefiles sufficient to build modules against the kernel package.

The following procedure describes how to use a machine config to add one or more extensions to your RHCOS nodes.

**Prerequisites**

- Have a running OpenShift Container Platform cluster (version 4.6 or later).
- Log in to the cluster as a user with administrative privileges.

**Procedure**

1. Create a Butane configuration file named `80-worker-usbguard.bu` to add the extension, manage the configuration files, and enable the service.

   ```yaml
   variant: openshift
   version: 4.22.0
   metadata:
     name: 80-worker-usbguard
     labels:
       machineconfiguration.openshift.io/role: worker
   extensions:
     - usbguard
   storage:
     files:
       - path: /etc/usbguard/usbguard-daemon.conf
         mode: 0600
         overwrite: true
         contents:
           inline: |
             RuleFile=/etc/usbguard/rules.conf
             PresentDevicePolicy=apply-policy
             IPCAllowedGroups=wheel
       - path: /etc/usbguard/rules.conf
         mode: 0600
         overwrite: false
         contents:
           inline: |
             allow id 1d6b:0002 serial "0000:00:14.0" name "EHCI Host Controller"
   systemd:
     units:
       - name: usbguard.service
         enabled: true
   ```
2. Convert the Butane configuration into a `MachineConfig` manifest by entering the following command:

   ```terminal
   $ butane 80-worker-usbguard.bu -o 80-worker-usbguard.yaml
   ```
3. Apply the `MachineConfig` to the cluster by entering the following command:

   ```terminal
   $ oc apply -f 80-worker-usbguard.yaml
   ```

   This sets all compute nodes to install the `usbguard` RPM package, write the required configuration files, and enable the systemd daemon within a single node rollout cycle.

**Verification**

1. Check that the new machine config was successfully created by entering the following command:

   ```terminal
   $ oc get machineconfig 80-worker-usbguard
   ```

   ```terminal {title="Example output"}
   NAME                GENERATEDBYCONTROLLER IGNITIONVERSION AGE
   80-worker-usbguard                        3.5.0           57s
   ```
2. Check that the machine config is now applied and that the nodes are not in a degraded state. This operation might take a few minutes. The worker pool will show the updates in progress, as each machine successfully has the new machine config applied:

   ```terminal
   $ oc get machineconfigpool
   ```

   ```terminal {title="Example output"}
   NAME   CONFIG             UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
   master rendered-master-35 True    False    False    3            3                 3                   0                    34m
   worker rendered-worker-d8 False   True     False    3            1                 1                   0                    34m
   ```
3. After the pool reports `UPDATED` as `True`, verify that the extension package was installed and that the service is running normally by debugging a compute node. You can complete these tasks by running the following commands:

   ```terminal
   $ oc get node | grep worker
   ```

   ```terminal {title="Example output"}
   NAME                                        STATUS  ROLES    AGE   VERSION
   ip-10-0-169-2.us-east-2.compute.internal    Ready   worker   102m  v1.35.4
   ```

   ```terminal
   $ oc debug node/ip-10-0-169-2.us-east-2.compute.internal
   ```

   ```terminal {title="Example output"}
   ...
   To use host binaries, run `chroot /host`
   sh-4.4# chroot /host
   sh-4.4# rpm -q usbguard
   usbguard-0.7.4-4.el8.x86_64.rpm
   ...
   sh-4.4# systemctl status usbguard.service
   usbguard.service - USBGuard daemon
           Loaded: loaded (/usr/lib/systemd/system/usbguard.service; enabled; preset: disabled)
           Active: active (running)
   ```

## Loading custom firmware blobs in the machine config manifest {#rhcos-load-firmware-blobs_machine-configs-configure}

You can load local firmware blobs that are not managed by RHCOS into the machine config manifest by updating the search path with a machine config.

By default, the location for firmware blobs in `/usr/lib` is read-only.

**Procedure**

1. Create a Butane config file, `98-worker-firmware-blob.bu`, that updates the search path so that it is root-owned and writable to local storage. The following example places the custom blob file from your local workstation onto nodes under `/var/lib/firmware`.

   > [!NOTE]
   > The [Butane version](https://coreos.github.io/butane/specs/) you specify in the config file should match the OpenShift Container Platform version and always ends in `0`. For example, `4.22.0`. See "Creating machine configs with Butane" for information about Butane.

   ```yaml {title="Butane config file for custom firmware blob"}
   variant: openshift
   version: 4.22.0
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: 98-worker-firmware-blob
   storage:
     files:
     - path: /var/lib/firmware/<package_name>
       contents:
         local: <package_name>
       mode: 0644
   openshift:
     kernel_arguments:
       - 'firmware_class.path=/var/lib/firmware'
   ```

   where:

   `storage.files.path`
   :   Specifies the path on the node where the firmware package is copied to.

   `storage.files.contents.local`
   :   Specifies a file with contents that are read from a local file directory on the system running Butane. The path of the local file is relative to a `files-dir` directory, which must be specified by using the `--files-dir` option with Butane in a subsequent step.

   `storage.files.mode`
   :   Specifies the permissions for the file on the RHCOS node. Red Hat recommends setting `0644` permissions.

   `openshift.kernel_arguments`
   :   Specifies the kernel search path of where to look for the custom firmware blob that was copied from your local workstation onto the root file system of the node. This example uses `/var/lib/firmware` as the customized path.
2. Run Butane to generate a `MachineConfig` object file that uses a copy of the firmware blob on your local workstation named `98-worker-firmware-blob.yaml`. The firmware blob contains the configuration to be delivered to the nodes. The following example uses the `--files-dir` option to specify the directory on your workstation where the local file or files are located:

   ```terminal
   $ butane 98-worker-firmware-blob.bu -o 98-worker-firmware-blob.yaml --files-dir <directory_including_package_name>
   ```
3. Apply the configurations to the nodes in one of two ways:

   - If the cluster is not running yet, after you generate manifest files, add the `MachineConfig` object file to the `<installation_directory>/openshift` directory, and then continue to create the cluster.
   - If the cluster is already running, apply the file:

     ```terminal
     $ oc apply -f 98-worker-firmware-blob.yaml
     ```

     A `MachineConfig` object YAML file is created for you to finish configuring your machines.
4. Save the Butane config in case you need to update the `MachineConfig` object in the future.

## Changing the core user password for node access {#core-user-password_machine-configs-configure}

You can use the default `core` user to access a node through a cloud provider serial console or a bare metal baseboard controller manager (BMC) if a node is down and you cannot access that node by using SSH or the `oc debug node` command.

By default, Red Hat Enterprise Linux CoreOS (RHCOS) creates a user named `core` on the nodes in your cluster. However, there is no password for this user. As such, you cannot log in with this user without creating a password by using a machine config. The Machine Config Operator (MCO) assigns the password and injects the password into the `/etc/shadow` file, allowing you to log in with the `core` user. The MCO does not examine the password hash. As such, the MCO cannot report if there is a problem with the password.

> [!NOTE]
> - The password works only through a cloud provider serial console or a BMC. It does not work with SSH.
> - If you have a machine config that includes an `/etc/shadow` file or a systemd unit that sets a password, it takes precedence over the password hash.

You can change the password, if needed, by editing the machine config you used to create the password. Also, you can remove the password by deleting the machine config. Deleting the machine config does not remove the user account.

**Procedure**

1. Using a tool that is supported by your operating system, create a hashed password. For example, create a hashed password using `mkpasswd` by running the following command:

   ```terminal
   $ mkpasswd -m SHA-512 testpass
   ```

   ```terminal {title="Example output"}
   $ $6$CBZwA6s6AVFOtiZe$aUKDWpthhJEyR3nnhM02NM1sKCpHn9XN.NPrJNQ3HYewioaorpwL3mKGLxvW0AOb4pJxqoqP4nFX77y0p00.8.
   ```
2. Create a machine config file that contains the `core` username and the hashed password:

   ```terminal
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: set-core-user-password
   spec:
     config:
       ignition:
         version: 3.5.0
       passwd:
         users:
         - name: core
           passwordHash: <password>
   ```

   where:

   `spec.config.passwd.users.name`
   :   Specifies the user name. This must be `core`.

   `spec.config.passwd.users.passwordHash`
   :   Specifies the hashed password to use with the `core` account.
3. Create the machine config by running the following command:

   ```terminal
   $ oc create -f <file-name>.yaml
   ```

   The nodes do not reboot and should become available in a few moments. You can use the `oc get mcp` to watch for the machine config pools to be updated, as shown in the following example:

   ```
   NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master   rendered-master-d686a3ffc8fdec47280afec446fce8dd   True      False      False      3              3                   3                     0                      64m
   worker   rendered-worker-4605605a5b1f9de1d061e9d350f251e5   False     True       False      3              0                   0                     0                      64m
   ```

**Verification**

1. After the nodes return to the `UPDATED=True` state, start a debug session for a node by running the following command:

   ```terminal
   $ oc debug node/<node_name>
   ```
2. Set `/host` as the root directory within the debug shell by running the following command:

   ```terminal
   sh-4.4# chroot /host
   ```
3. Check the contents of the `/etc/shadow` file:

   ```terminal {title="Example output"}
   ...
   core:$6$2sE/010goDuRSxxv$o18K52wor.wIwZp:19418:0:99999:7:::
   ...
   ```

   The hashed password is assigned to the `core` user.

## Overriding storage or partition setup {#machine-config-install-time-configs_machine-configs-configure}

You can use a `MachineConfig` object to change the disk partition schema, file systems, and RAID configurations that were established during the cluster installation. This allows you to make specific configuration changes that are different from the initial cluster state.

If you specified storage and partition configuration upon cluster installation by using a Butane config, Ignition config, or machine config, those configurations become defaults within your cluster. If you create new nodes, those nodes automatically use those default configurations.

You cannot change these components directly. By default, the Machine Config Operator (MCO) reviews changes in `MachineConfig` objects for specific fields and blocks some changes for security reasons. However, you can override this restriction for disk partition schema, file systems, and RAID configurations by adding the `irreconcilableValidationOverrides` parameter to the `MachineConfiguration` object. Then, you can create a new machine config to make the necessary changes for new nodes.

> [!NOTE]
> Configuration changes made through this process apply to new nodes only.

For example, you might want to override your default storage configuration to add new hardware that uses a different storage partitioning schema or storage file system to your cluster. In this case, you can modify the storage configuration for any new nodes in your cluster.

Or, if you used Ignition to modify the storage configuration as a post-installation task, your cluster might be reporting an `irreconcilableChanges` status in the `MachineConfigNode` object status fields. This messaging can alert you to these differences, so that you can determine if you want new hardware with the new configurations.

> [!IMPORTANT]
> Overriding irreconcilable fields is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
>
> For more information about the support scope of Red Hat Technology Preview features, see [Technology Preview Features Support Scope](https://access.redhat.com/support/offerings/techpreview/).

**Prerequisites**

- You enabled the required Technology Preview features for your cluster by adding the `TechPreviewNoUpgrade` feature set to the `FeatureGate` CR named `cluster`. For information about enabling Feature Gates, see *Enabling features using feature gates*.

  > [!WARNING]
  > Enabling the `TechPreviewNoUpgrade` feature set on your cluster cannot be undone and prevents minor version updates. This feature set allows you to enable these Technology Preview features on test clusters, where you can fully test them. Do not enable this feature set on production clusters.

**Procedure**

1. Edit the `MachineConfiguration` object by using the following command:

   ```terminal
   $ oc edit machineconfiguration
   ```
2. Add the `irreconcilableValidationOverrides` stanza to the `MachineConfiguration` object.

   ```yaml
   apiVersion: operator.openshift.io/v1
   kind: MachineConfiguration
   # ...
   spec:
     irreconcilableValidationOverrides:
       storage:
       - Disks
       - Raid
       - FileSystems
   # ...
   ```

   where:

   `spec.irreconcilableValidationOverrides.storage.Disks`
   :   Allows you to modify the installed storage disk configuration to be used with new nodes. This field is optional.

   `spec.irreconcilableValidationOverrides.storage.Raid`
   :   Allows you to modify the installed RAID configuration to be used with new nodes. This field is optional.

   `spec.irreconcilableValidationOverrides.storage.FileSystems`
   :   Allows you to modify the installed file system configuration to be used with new nodes. This field is optional.
3. Create a YAML file for a `MachineConfig` object with the changes that you need, similar to the following:

   ```yaml
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: worker
     name: extra-disks
   spec:
     config:
       ignition:
         version: "3.5.0"
       storage:
         disks:
         - device: "/dev/sdb"
           wipeTable: true
           partitions:
           - label: raid.1.1
             number: 1
             sizeMiB: 1024
             startMiB: 0
         - device: "/dev/sdc"
           wipeTable: true
           partitions:
           - label: raid.1.2
             number: 1
             sizeMiB: 1024
             startMiB: 0
         raid:
         - devices:
           - "/dev/disk/by-partlabel/raid.1.1"
           - "/dev/disk/by-partlabel/raid.1.2"
           level: stripe
           name: data
         filesystems:
         - device: "/dev/md/data"
           path: "/var/lib/data"
           format: ext4
           label: DATA
   ```

   where:

   `spec.config.storage.disks`
   :   Specifies changes to the installed storage disk configuration in Ignition format. This field is optional.

`spec.config.storage.raid`
:   Specifies changes to the installed RAID configuration in Ignition format. This field is optional.

`spec.config.storage.filesystems`
:   Specifies changes to the installed file system configuration in Ignition format. This field is optional.

1. Create the `MachineConfig` object by using a command similar to the following:

   ```terminal
   $ oc create -f <file_name>.yaml
   ```

   When you create a new node from a machine set with the associated label, the new configurations are applied to the node.

**Additional resources**
{._additional-resources}

- [How to update ssh keys after installation in OpenShift 4? (Red Hat Knowledgebase article)](https://access.redhat.com/solutions/3868301)
- [Container image signatures](/openshift-docs-markdown/security/container_security/security-container-signature#security-container-signature)
- [Enabling SCTP in OpenShift Container Platform 4 (Red Hat Knowledgebase article)](https://access.redhat.com/solutions/4727321)
- [How to provide custom iSCSI initiatornames for nodes in OpenShift Container Platform 4.x (Red Hat Knowledgebase article)](https://access.redhat.com/solutions/5170251)
- [Configuration Specification v3.5.0 (Ignition documentation)](https://coreos.github.io/ignition/configuration-v3_5/)
- [Understanding configuration drift detection](/openshift-docs-markdown/machine_configuration/index#machine-config-drift-detection_machine-config-overview)
- [Creating machine configs with Butane](/openshift-docs-markdown/installing/install_config/installing-customizing#installation-special-config-butane_installing-customizing)
- [Enabling multipathing with kernel arguments on RHCOS](/openshift-docs-markdown/installing/installing_bare_metal/upi/installing-bare-metal#rhcos-enabling-multipath_installing-bare-metal)
- [Creating machine configs with Butane](/openshift-docs-markdown/installing/install_config/installing-customizing#installation-special-config-butane_installing-customizing)
- [Enabling features using feature gates](/openshift-docs-markdown/nodes/clusters/nodes-cluster-enabling-features#nodes-cluster-enabling-features)