--- title: Image mode for OpenShift --- # Image mode for OpenShift {#mco-coreos-layering} You can extend the functionality of your base RHCOS image by layering additional images onto the base image without modifying the base RHCOS image. This layering creates a *custom layered image* that includes all RHCOS functionality and adds additional functionality to specific nodes in the cluster. Image mode is a cloud-native approach to operating system management that treats your OS like a container image. You define your operating system configuration as code, build it as a unified image, and deploy it consistently across your entire fleet. ## About image mode for OpenShift {#coreos-layering-about_mco-coreos-layering} Image mode for OpenShift allows you to customize the underlying node operating system on any of your cluster nodes. This helps keep everything up-to-date, including the node operating system and any added customizations such as specialized software. You create a custom layered image by using a Containerfile and applying it to nodes by using a custom object. At any time, you can remove the custom layered image by deleting that custom object. With image mode for OpenShift, you can install RPMs into your base image, and your custom content will be booted alongside RHCOS. The Machine Config Operator (MCO) can roll out these custom layered images and monitor these custom containers in the same way it does for the default RHCOS image. Image mode for OpenShift gives you greater flexibility in how you manage your RHCOS nodes. > [!IMPORTANT] > Installing realtime kernel and extensions RPMs as custom layered content is not recommended. This is because these RPMs can conflict with RPMs installed by using a machine config. If there is a conflict, the MCO enters a `degraded` state when it tries to install the machine config RPM. You need to remove the conflicting extension from your machine config before proceeding. When you apply the custom layered image to your cluster, you assume the responsibility for the package you applied with the custom layered image and any issues that might arise with the package. There are three methods for deploying a custom layered image onto your nodes: On-cluster image mode : With on-cluster image mode, you create a `MachineOSConfig` object where you include the Containerfile and other parameters. The build is performed on your cluster and the resulting custom layered image is automatically pushed to your repository and applied to the machine config pool that you specified in the `MachineOSConfig` object. The entire process is performed completely within your cluster. Out-of-cluster image mode : With out-of-cluster image mode, you create a Containerfile that references an OpenShift Container Platform image and the RPM that you want to apply, build the layered image in your own environment, and push the image to your repository. Then, in your cluster, create a `MachineConfig` object for the targeted node pool that points to the new image. The Machine Config Operator overrides the base RHCOS image, as specified by the `osImageURL` value in the associated machine config, and boots the new image. During OpenShift Container Platform installation : You can apply a pre-built custom layered image to specific nodes during OpenShift Container Platform installation. > [!IMPORTANT] > For these methods, use the same base RHCOS image installed on the rest of your cluster. Use the `oc adm release info --image-for rhel-coreos` command to obtain the base image used in your cluster. ## Example Containerfiles {#coreos-layering-examples_mco-coreos-layering} Image mode for OpenShift allows you to use the following types of images to create custom layered images: - **OpenShift Container Platform Hotfixes**. You can work with Customer Experience and Engagement (CEE) to obtain and apply Hotfix packages on top of your RHCOS image. In some instances, you might want a bug fix or enhancement before it is included in an official OpenShift Container Platform release. Image mode for OpenShift allows you to easily add the Hotfix before it is officially released and remove the Hotfix when the underlying RHCOS image incorporates the fix. > [!IMPORTANT] > Some Hotfixes require a Red Hat Support Exception and are outside of the normal scope of OpenShift Container Platform support coverage or life cycle policies. Hotfixes are provided to you based on Red Hat Hotfix policy. Apply it on top of the base image and test that new custom layered image in a non-production environment. When you are satisfied that the custom layered image is safe to use in production, you can roll it out on your own schedule to specific node pools. For any reason, you can easily roll back the custom layered image and return to using the default RHCOS. ```yaml {title="Example on-cluster Containerfile to apply a Hotfix"} containerfileArch: noarch content: |- FROM configs AS final #Install hotfix package RUN dnf update -y https://example.com/files/systemd-252-46.el9_4.x86_64.rpm \ https://example.com/files/systemd-journal-remote-252-46.el9_4.x86_64.rpm \ https://example.com/files/systemd-libs-252-46.el9_4.x86_64.rpm \ https://example.com/files/systemd-pam-252-46.el9_4.x86_64.rpm \ https://example.com/files/systemd-udev-252-46.el9_4.x86_64.rpm \ https://example.com/files/systemd-rpm-macros-252-46.el9_4.noarch.rpm && \ dnf clean all && \ bootc container lint ``` ```yaml {title="Example out-of-cluster Containerfile to apply a Hotfix"} FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256... #Install hotfix package RUN dnf update -y https://example.com/files/systemd-252-46.el9_4.x86_64.rpm \ https://example.com/files/systemd-journal-remote-252-46.el9_4.x86_64.rpm \ https://example.com/files/systemd-libs-252-46.el9_4.x86_64.rpm \ https://example.com/files/systemd-pam-252-46.el9_4.x86_64.rpm \ https://example.com/files/systemd-udev-252-46.el9_4.x86_64.rpm \ https://example.com/files/systemd-rpm-macros-252-46.el9_4.noarch.rpm && \ dnf clean all && \ bootc container lint ``` - **RHEL packages**. You can download Red Hat Enterprise Linux (RHEL) packages from the Red Hat Customer Portal, such as chrony, firewalld, and iputils. ```yaml {title="Example out-of-cluster Containerfile to apply the rsyslog utility"} # Using a 4.18.0 image FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256... # Install rsyslog package RUN dnf install -y rsyslog && \ bootc container lint # Copy your custom configuration in ADD remote.conf /etc/rsyslog.d/remote.conf ``` - **Third-party packages**. You can download and install RPMs from third-party organizations, such as the following types of packages: - Bleeding edge drivers and kernel enhancements to improve performance or add capabilities. - Forensic client tools to investigate possible and actual break-ins. - Security agents. - Inventory agents that provide a coherent view of the entire cluster. - SSH Key management packages. ```yaml {title="Example on-cluster Containerfile to apply a third-party package from EPEL"} FROM configs AS final #Enable EPEL (more info at https://docs.fedoraproject.org/en-US/epel/ ) and install htop RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && \ dnf install -y htop && \ dnf clean all && \ bootc container lint ``` ```yaml {title="Example out-of-cluster Containerfile to apply a third-party package from EPEL"} # Get RHCOS base image of target cluster `oc adm release info --image-for rhel-coreos` FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256... #Enable EPEL (more info at https://docs.fedoraproject.org/en-US/epel/ ) and install htop RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm && \ dnf install -y htop && \ dnf clean all && \ bootc container lint ``` This Containerfile installs the RHEL fish program. Because fish requires additional RHEL packages, the image must be built on an entitled RHEL host. For RHEL entitlements to work, you must copy the `etc-pki-entitlement` secret into the `openshift-machine-config-operator` namespace. ```yaml {title="Example on-cluster Containerfile to apply a third-party package that has RHEL dependencies"} FROM configs AS final # RHEL entitled host is needed here to access RHEL packages # Install fish as third party package from EPEL RUN dnf install -y https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/f/fish-3.3.1-3.el9.x86_64.rpm && \ dnf clean all && \ bootc container lint ``` ```yaml {title="Example out-of-cluster Containerfile to apply a third-party package that has RHEL dependencies"} # Get RHCOS base image of target cluster `oc adm release info --image-for rhel-coreos` FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256... # RHEL entitled host is needed here to access RHEL packages # Install fish as third party package from EPEL RUN dnf install -y https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/f/fish-3.3.1-3.el9.x86_64.rpm && \ dnf clean all && \ bootc container lint ``` After you create the machine config, the Machine Config Operator (MCO) performs the following steps: 1. Renders a new machine config for the specified pool or pools. 2. Performs cordon and drain operations on the nodes in the pool or pools. 3. Writes the rest of the machine config parameters onto the nodes. 4. Applies the custom layered image to the node. 5. Reboots the node using the new image. > [!IMPORTANT] > It is strongly recommended that you test your images outside of your production environment before rolling out to your cluster. ## About on-cluster image mode {#coreos-layering-configuring-on_mco-coreos-layering} You can use the image mode for OpenShift on-cluster build process to apply a custom layered image to your nodes by creating a `MachineOSConfig` custom resource (CR). When you create the object, the Machine Config Operator (MCO) creates a `MachineOSBuild` object and a builder pod. The process also creates transient objects, such as config maps, which are cleaned up after the build is complete. The `MachineOSBuild` object and the associated `builder-*` pod use the same naming scheme, `-`, for example: ```terminal {title="Example MachineOSBuild object"} NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED layered-image-c8765e26ebc87e1e17a7d6e0a78e8bae False False True False False ``` ```terminal {title="Example builder pod"} NAME READY STATUS RESTARTS AGE build-layered-image-c8765e26ebc87e1e17a7d6e0a78e8bae 2/2 Running 0 11m ``` You should not need to interact with these new objects or the `machine-os-builder` pod. However, you can use all of these resources for troubleshooting, if necessary. When the build is complete, the MCO pushes the new custom layered image to your repository and rolls the image out to the nodes in the associated machine config pool. You can see the digested image pull spec for the new custom layered image in the `MachineOSConfig` object. This is now the active image pull spec for this `MachineOSConfig`. ```terminal {title="Example digested image pull spec"} apiVersion: machineconfiguration.openshift.io/v1 kind: MachineOSConfig metadata: annotations: machineconfiguration.openshift.io/current-machine-os-build: layered-9a8f89455246fa0c42ecee6ff1fa1a45 labels: machineconfiguration.openshift.io/createdByOnClusterBuildsHelper: "" name: layered-image # ... status: currentImagePullSpec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-image@sha256:3c8fc667adcb432ce0c83581f16086afec08a961dd28fed69bb6bad6db0a0754 ``` > [!TIP] > You can test a `MachineOSBuild` object to make sure it builds correctly without rolling out the custom layered image to active nodes by using a custom machine config pool that contains non-production nodes. Alternatively, you can use a custom machine config pool that has no nodes. The `MachineOSBuild` object builds even if there are no nodes for the MCO to deploy the custom layered image onto. You can apply a custom layered image to any machine config pool in your cluster, including the control plane, worker, or custom pools. > [!NOTE] > For single-node OpenShift clusters, you can apply a custom layered image to the control plane node only. Making certain changes to a `MachineOSConfig` object triggers an automatic rebuild of the associated custom layered image. You can mitigate the effects of the rebuild by pausing the machine config pool where the custom layered image is applied as described in "Pausing the machine config pools". While the pools are paused, the MCO does not roll out the newly built image to the nodes after the build is complete. However, the build runs regardless of whether the pool is paused or not. For example, if you want to remove and replace a `MachineOSCOnfig` object, pausing the machine config pools before making the change prevents the MCO from reverting the associated nodes to the base image, reducing the number of reboots needed. When a machine config pool is paused, the `oc get machineconfigpools` reports the following status: ```terminal {title="Example output"} NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-a0b404d061a6183cc36d302363422aba True False False 3 3 3 0 4h14m worker rendered-worker-221507009cbcdec0eec8ab3ccd789d18 False False False 2 2 2 0 4h14m ``` The `worker` machine config pool is paused, as indicated by the three `False` statuses and the `READYMACHINECOUNT` at `0`. After the changes have been rolled out, you can unpause the machine config pool. In the case of a build failure, for example due to network issues or an invalid secret, the MCO retries the build three additional times before the job fails. The MCO creates a different build pod for each build attempt. Note that the MCO automatically removes these build pods after a short period of time. Also, the affected machine config pool reports a build failure through the `ImageBuildDegraded` status condition. You can use the build pod logs to troubleshoot any build failures. ```terminal {title="Example failed MachineOSBuild object"} NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED AGE layered-image-c8765e26ebc87e1e17a7d6e0a78e8bae False False False False True 12m ``` You can manually rebuild your custom layered image by either modifying your `MachineOSConfig` object or applying an annotation to the `MachineOSConfig` object. For more information, see "Rebuilding an on-cluster custom layered image". If you used a custom machine config pool to apply an on-cluster layered image to a node, you can remove the custom layered image from the node and revert to the base image. For more information, see "Reverting an on-cluster layered node". You can modify an on-custom layered image as needed, to install additional packages, remove existing packages, change repositories, update secrets, or other similar changes, by editing the MachineOSConfig object. For more information, see "Modifying a custom layered image". ### On-cluster image mode known limitations {#coreos-layering-configuring-on-limitations_mco-coreos-layering} Note the following limitations when working with the on-cluster layering feature: - On-cluster image mode is not supported on multi-architecture compute machines. - Using multiple `MachineOSConfig` objects on the same machine config pool is not supported. You need a separate `MachineOSConfig` CR for each machine config pool where you want to use a distinct custom layered image. - If you scale up a machine set that uses a custom layered image, the nodes reboot two times. The first, when the node is initially created with the base image and a second time when the custom layered image is applied. - Node disruption policies are not supported on nodes with a custom layered image. However, the following machine configuration changes do not cause a new image build or the reboot of a node with an on-cluster custom layered image: - Modifying the configuration files in the `/var` or `/etc` directory - Adding or modifying a systemd service - Changing SSH keys - Removing mirroring rules from `ICSP`, `ITMS`, and `IDMS` objects - Changing the trusted CA, by updating the `user-ca-bundle` configmap in the `openshift-config` namespace - The following machine configuration changes do cause a new image build and a node reboot: - Changing the kernel arguments - Changing the `OSImageURL` parameter - Adding or removing extensions - The images used in creating custom layered images take up space in your push registry. Always be aware of the free space in your registry and prune the images as needed. You can automatically remove an on-cluster custom layered image from the repository by deleting the `MachineOSBuild` object that created the image. Note that the credentials provided by the registry push secret must also grant permission to delete an image from the registry. For more information, see "Removing an on-cluster custom layered image". ### Using the on-cluster image mode to apply a custom layered image {#coreos-layering-configuring-on-proc_mco-coreos-layering} You can use the on-cluster build process to apply a custom layered image to your cluster by creating a `MachineOSConfig` custom resource (CR). The `MachineOSConfig` CR specifies the following parameters: - the Containerfile to build - the machine config pool to associate the build - where the final image should be pushed and pulled from - the push and pull secrets to use You can create only one `MachineOSConfig` CR for each machine config pool. **Prerequisites** - You have the pull secret in the `openshift-machine-config-operator` namespace that the Machine Config Operator (MCO) needs in order to pull the base operating system image from your repository. By default, the MCO uses the cluster global pull secret, which it synchronizes into the `openshift-machine-config-operator` namespace. You can add your pull secret to the OpenShift Container Platform global pull secret or you can use a different pull secret. For information on modifying the global pull secret, see "Updating the global cluster pull secret". - You have the push secret of the registry that the MCO needs to push the new custom layered image to. The credentials provided by the secret must also grant permission to delete an image from the registry. > [!NOTE] > In a disconnected environment, ensure that the disconnected cluster can access the registry where you want to push the image. Image mirroring applies only to pulling images. - You have the pull secret that your nodes need to pull the new custom layered image from your registry. This should be a different secret than the one used to push the image to the repository. - You are familiar with how to configure a Containerfile. Instructions on how to create a Containerfile are beyond the scope of this documentation. - Optional: You have a separate machine config pool for the nodes where you want to apply the custom layered image. One benefit to having a custom machine config pool for the nodes it that you can easily revert to the base image, if needed. For more information, see "Reverting an on-cluster layered node". **Procedure** 1. Create a `MachineOSconfig` object: 1. Create a YAML file similar to the following: ```yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineOSConfig metadata: name: layered-image spec: machineConfigPool: name: layered-image containerFile: - containerfileArch: NoArch content: |- FROM configs AS final RUN dnf install -y cowsay && \ dnf clean all && \ bootc container lint imageBuilder: imageBuilderType: Job baseImagePullSecret: name: global-pull-secret-copy renderedImagePushSpec: image-registry.openshift-image-registry.svc:5000/openshift/os-image:latest renderedImagePushSecret: name: builder-dockercfg-mtcl23 ``` where: `apiVersion` : Specifies the `machineconfiguration.openshift.io/v1` API that is required for `MachineConfig` CRs. `metadata.name` : Specifies a name for the `MachineOSConfig` object. The name must match the name of the associated machine config pool. This name is used with other on-cluster image mode resources. The examples in this documentation use the name `layered-image`. `spec.machineConfigPool.name` : Specifies the name of the machine config pool associated with the nodes where you want to deploy the custom layered image. The examples in this documentation use the `layered-image` machine config pool. `spec.containerFile` : Specifies the Containerfile to configure the custom layered image. `spec.containerFile.containerfileArch` : Specifies the architecture this containerfile is to be built for: `ARM64`, `AMD64`, `PPC64LE`, `S390X`, or `NoArch`. The default is `NoArch`, which defines a Containerfile that can be applied to any architecture. `spec.imageBuilder` : Specifies the name of the image builder to use. This must be `Job`, which is a reference to the `job` object that is managing the image build. `spec.baseImagePullSecret` : Specifies the name of the pull secret that the MCO needs to pull the base operating system image from the registry. By default, the global pull secret is used. This parameter is optional. `spec.renderedImagePushSpec` : Specifies the image registry to push the newly-built custom layered image to. This can be any registry that your cluster has access to in the `host[:port][/namespace]/name` or `svc_name.namespace.svc[:port]/repository/name:` format. This example uses the internal OpenShift Container Platform registry. You can specify a mirror registry if you cluster is properly configured to use a mirror registry. `spec.renderedImagePushSecret` : Specifies the name of the push secret that the MCO needs to push the newly-built custom layered image to that registry. 2. Create the `MachineOSConfig` object: ```terminal $ oc create -f .yaml ``` 2. If necessary, when the `MachineOSBuild` object has been created and is in the `READY` state, modify the node spec for the nodes where you want to use the new custom layered image: 1. Check that the `MachineOSBuild` object is ready, by running the following command: ```terminal $ oc get machineosbuild ``` When the `SUCCEEDED` value is `True`, the build is complete: ```terminal {title="Example output showing that the MachineOSBuild object is ready"} NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED AGE layered-image-ad5a3cad36303c363cf458ab0524e7c0-builder False False True False False 43s ``` 2. Edit the nodes where you want to deploy the custom layered image by adding a label for the machine config pool you specified in the `MachineOSConfig` object: ```terminal $ oc label node 'node-role.kubernetes.io/=' ``` where: node-role.kubernetes.io/= : Specifies a node selector that identifies the nodes to deploy the custom layered image. When you save the changes, the MCO drains, cordons, and reboots the nodes. After the reboot, the node uses the new custom layered image. **Verification** 1. Verify that the new pods are ready by running the following command: ```terminal $ oc get pods -n openshift-machine-config-operator ``` ```terminal {title="Example output"} NAME READY STATUS RESTARTS AGE build-layered-image-ad5a3cad36303c363cf458ab0524e7c0-hxrws 2/2 Running 0 2m40s # ... machine-os-builder-6fb66cfb99-zcpvq 1/1 Running 0 2m42s ``` The build pod where the custom layered image is building is named in the `build--` format. The `machine-os-builder-*` pod can be used for troubleshooting. 2. Verify the custom layered image build by running a command similar to the following: ```terminal $ oc get machineconfigpool -o yaml ``` ```terminal {title="Example output"} apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfigPool metadata: labels: machineconfiguration.openshift.io/mco-built-in: "" pools.operator.machineconfiguration.openshift.io/layered: "" name: layered # ... status: # ... conditions # ... - lastTransitionTime: "2025-09-09T13:43:35Z" message: 'Failed to build OS image for pool worker (MachineOSBuild: worker-2d03dc921ff0c242c5892a3ef1ed1608): Failed: Build Failed' reason: BuildFailed status: "True" type: ImageBuildDegraded ``` The `reason: BuildFailed` status indicates whether the custom layered image build failed. If `False`, the build succeeded. If `True`, the build failed. You can use the build pod logs to troubleshoot any build failures. 3. Verify the current stage of your layered build by running the following command: ```terminal $ oc get machineosbuilds ``` ```terminal {title="Example output"} NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED AGE layered-image-ad5a3cad36303c363cf458ab0524e7c0 False True False False False 12m ``` The `MachineOSBuild` object is named in the `-` format. The build is complete when `BUILDING` is `False` and `SUCCEEDED` is `True`. 4. When the build is complete, verify that the image has been applied to the nodes in the affected pool by running a command similar to the following: ```terminal $ oc describe machineconfignode/ ``` ```terminal {title="Example machine config node output"} Name: ip-10-0-14-86.us-west-1.compute.internal API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfigNode # ... Spec: Config Image: Desired Image: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/ocb-image@sha256:b485378fd8f7963ed74f14ce64f4f1e511e1601d49302b3046b1b78a83f539e3 Config Version: Desired: rendered-worker-d63c7736923b60b8b82492ae9a1eef40 Node: Name: ip-10-0-14-86.us-west-1.compute.internal Pool: Name: worker # ... Status: Conditions: # ... Message: Action during update to image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/ocb-image@sha256:b485378fd8f7963ed74f14ce64f4f1e511e1601d49302b3046b1b78a83f539e3: Successfully pulled OS image image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/ocb-image@sha256:b485378fd8f7963ed74f14ce64f4f1e511e1601d49302b3046b1b78a83f539e3 from registry Reason: ImagePulledFromRegistry Status: False Type: ImagePulledFromRegistry # ... Config Image: Current Image: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/ocb-image@sha256:b485378fd8f7963ed74f14ce64f4f1e511e1601d49302b3046b1b78a83f539e3 Desired Image: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/ocb-image@sha256:b485378fd8f7963ed74f14ce64f4f1e511e1601d49302b3046b1b78a83f539e3 # ... ``` The digested image pull spec for the new custom layered image is in the `Spec.Config Image.Desired Image` parameter. > [!IMPORTANT] > The `ImagePulledFromRegistry` condition is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. > > For more information about the support scope of Red Hat Technology Preview features, see [Technology Preview Features Support Scope](https://access.redhat.com/support/offerings/techpreview/). 5. Verify that the `MachineOSConfig` object contains a reference to the new custom layered image by running the following command: ```terminal $ oc describe machineosconfig ``` ```terminal {title="Example digested image pull spec"} apiVersion: machineconfiguration.openshift.io/v1 kind: MachineOSConfig metadata: annotations: machineconfiguration.openshift.io/current-machine-os-build: layered-9a8f89455246fa0c42ecee6ff1fa1a45 labels: machineconfiguration.openshift.io/createdByOnClusterBuildsHelper: "" name: layered-image # ... status: currentImagePullSpec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-image@sha256:3c8fc667adcb432ce0c83581f16086afec08a961dd28fed69bb6bad6db0a0754 ``` where: `status.currentImagePullSpec` : Specifies the digested image pull spec for the new custom layered image. 6. Verify that the appropriate nodes are using the new custom layered image: 1. Start a debug session as root for a control plane node by running the following command: ```terminal $ oc debug node/ ``` 2. Set `/host` as the root directory within the debug shell: ```terminal sh-4.4# chroot /host ``` 3. Run the `rpm-ostree status` command to view that the custom layered image is in use: ```terminal sh-5.1# rpm-ostree status ``` ```terminal {title="Example output"} # ... Deployments: * ostree-unverified-registry:image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images@sha256:3c8fc667adcb432ce0c83581f16086afec08a961dd28fed69bb6bad6db0a0754 Digest: sha256:3c8fc667adcb432ce0c83581f16086afec08a961dd28fed69bb6bad6db0a0754 Version: 419.94.202502100215-0 (2025-02-12T19:20:44Z) ``` The `Deployments` stanza includes the digested image pull spec for the new custom layered image. ### Modifying an on-cluster custom layered image {#coreos-layering-configuring-on-modifying_mco-coreos-layering} You can modify an on-cluster custom layered image, as needed, to install additional packages, remove existing packages, change the pull or push repositories, update secrets, or other similar changes. You can edit the `MachineOSConfig` object, apply changes to the YAML file that created the `MachineOSConfig` object, or create a new YAML file for that purpose. If you modify and apply the `MachineOSConfig` object YAML or create a new YAML file, the YAML overwrites any changes you made directly to the `MachineOSConfig` object itself. Making certain changes to a `MachineOSConfig` object triggers an automatic rebuild of the associated custom layered image. You can mitigate the effects of the rebuild by pausing the machine config pool where the custom layered image is applied as described in "Pausing the machine config pools". While the pools are paused, the MCO does not roll out the newly built image to the nodes after the build is complete. However, the build runs regardless of whether the pool is paused or not. For example, if you want to remove and replace a `MachineOSCOnfig` object, pausing the machine config pools before making the change prevents the MCO from reverting the associated nodes to the base image, reducing the number of reboots needed. When a machine config pool is paused, the `oc get machineconfigpools` reports the following status: ```terminal {title="Example output"} NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-a0b404d061a6183cc36d302363422aba True False False 3 3 3 0 4h14m worker rendered-worker-221507009cbcdec0eec8ab3ccd789d18 False False False 2 2 2 0 4h14m ``` The `worker` machine config pool is paused, as indicated by the three `False` statuses and the `READYMACHINECOUNT` at `0`. After the changes have been rolled out, you can unpause the machine config pool. **Prerequisites** - You have opted in to on-cluster image mode by creating a `MachineOSConfig` object. **Procedure** - Modify an object to update the associated custom layered image: 1. Edit the `MachineOSConfig` object to modify the custom layered image. The following example adds the `rngd` daemon to nodes that already have the tree package that was installed using a custom layered image. ```yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineOSConfig metadata: name: layered-image spec: machineConfigPool: name: layered-image containerFile: - containerfileArch: noarch content: |- FROM configs AS final RUN rpm-ostree install rng-tools && \ systemctl enable rngd && \ rpm-ostree cleanup -m && \ bootc container lint RUN rpm-ostree install tree && \ bootc container lint imageBuilder: imageBuilderType: PodImageBuilder baseImagePullSecret: name: global-pull-secret-copy renderedImagePushspec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/os-images:latest renderedImagePushSecret: name: new-secret-name ``` where: `spec:containerFile.content` : Modify the Containerfile, for example to add or remove packages. This parameter is optional. `spec.baseImagePullSecret` : Update the secret needed to pull the base operating system image from the registry. This parameter is optional. `spec.renderedImagePushspec` : Modify the image registry to push the newly built custom layered image to. This parameter is optional. `spec.renderedImagePushSecret` : Update the secret needed to push the newly built custom layered image to the registry. This parameter is optional. When you save the changes, the MCO drains, cordons, and reboots the nodes. After the reboot, the node uses the cluster base Red Hat Enterprise Linux CoreOS (RHCOS) image. If your changes modify a secret only, no new build is triggered and no reboot is performed. **Verification** 1. Verify that the new `MachineOSBuild` object was created by using the following command: ```terminal $ oc get machineosbuild ``` ```terminal {title="Example output"} NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED AGE layered-image-a5457b883f5239cdcb71b57e1a30b6ef False False True False False 4d17h layered-image-f91f0f5593dd337d89bf4d38c877590b False True False False False 2m41s ``` The value `True` in the `BUILDING` column indicates that the `MachineOSBuild` object is building. When the `SUCCEEDED` column reports `True`, the build is complete. 2. You can watch as the new machine config is rolled out to the nodes by using the following command: ```terminal $ oc get machineconfigpools ``` ```terminal {title="Example output"} NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-a0b404d061a6183cc36d302363422aba True False False 3 3 3 0 3h38m worker rendered-worker-221507009cbcdec0eec8ab3ccd789d18 False True False 2 2 2 0 3h38m ``` The value `FALSE` in the `UPDATED` column indicates that the `MachineOSBuild` object is building. When the `UPDATED` column reports `FALSE`, the new custom layered image has rolled out to the nodes. 3. When the node is back in the `Ready` state, check that the changes were applied: 1. Open an `oc debug` session to the node by running the following command: ```terminal $ oc debug node/ ``` 2. Set `/host` as the root directory within the debug shell by running the following command: ```terminal sh-5.1# chroot /host ``` 3. Use an appropriate command to verify that change was applied. The following examples shows that the `rngd` daemon was installed: ```terminal sh-5.1# rpm -qa |grep rng-tools ``` ```terminal {title="Example output"} rng-tools-6.17-3.fc41.x86_64 ``` ```terminal sh-5.1# rngd -v ``` ```terminal {title="Example output"} rngd 6.16 ``` ### Rebuilding an on-cluster custom layered image {#coreos-layering-configuring-on-rebuild_mco-coreos-layering} You can rebuild an on-cluster custom layered image by either modifying your `MachineOSConfig` object or adding an annotation to the `MachineOSConfig` object. Both of these actions trigger an automatic rebuild of the object. For example, you could perform a rebuild if the you change the Containerfile or need to update the `osimageurl` location in a machine config. After you add the annotation, the Machine Config Operator (MCO) deletes the current `MachineOSBuild` object and creates a new one in its place. When the build process is complete, the MCO automatically removes the annotation. **Prerequisites** - You have opted-in to on-cluster image mode by creating a `MachineOSConfig` object. **Procedure** - Edit the `MachineOSConfig` object to add the `machineconfiguration.openshift.io/rebuild` annotation by using the following command: ```terminal $ oc edit MachineOSConfig ``` ```yaml {title="Example MachineOSConfig object"} apiVersion: machineconfiguration.openshift.io/v1 kind: MachineOSConfig metadata: annotations: machineconfiguration.openshift.io/current-machine-os-build: layering-c26d4a003432df70ee66c83981144cfa machineconfiguration.openshift.io/rebuild: "" # ... name: layered-image # ... ``` Add the `machineconfiguration.openshift.io/rebuild: ""` annotation to trigger a rebuild of the custom layered image. **Verification** - Check that the `MachineOSBuild` object is building by using the following command: ```terminal $ oc get machineosbuild ``` ```terminal {title="Example output"} NAME PREPARED BUILDING SUCCEEDED INTERRUPTED FAILED AGE layered-image-d6b929a29c6dbfa8e4007c8069a2fd08 False True False False False 2m41s ``` The value `True` in the `BUILDING` column indicates that the `MachineOSBuild` object is building. - Edit the `MachineOSConfig` object to verify that the MCO removed the `machineconfiguration.openshift.io/rebuild` annotation by using the following command: ```terminal $ oc edit MachineOSConfig ``` ```yaml {title="Example MachineOSConfig object"} apiVersion: machineconfiguration.openshift.io/v1 kind: MachineOSConfig metadata: annotations: machineconfiguration.openshift.io/current-machine-os-build: layering-c26d4a003432df70ee66c83981144cfa # ... name: layered-image # ... ``` ### Reverting an on-cluster custom layered image {#coreos-layering-configuring-on-revert_mco-coreos-layering} If you applied an on-cluster layered image to a node in a custom machine config pool (MCP), you can remove the custom layered image from the node and revert to the base image. To revert the node, remove the node from the custom MCP by removing the custom machine config pool label from the node. After you remove the label, the Machine Config Operator (MCO) reboots the node with the cluster base Red Hat Enterprise Linux CoreOS (RHCOS) image, overriding the custom layered image. > [!IMPORTANT] > Before you remove the label, make sure the node is associated with another MCP. **Prerequisites** - You have opted-in to On-cluster image mode by creating a `MachineOSConfig` object. - You have applied a `MachineOSConfig` object to a node in a custom machine config pool. **Procedure** - Remove the label from the node by using the following command: ```terminal $ oc label node/ node-role.kubernetes.io/- ``` When you save the changes, the MCO drains, cordons, and reboots the nodes. After the reboot, the node uses the cluster base Red Hat Enterprise Linux CoreOS (RHCOS) image. **Verification** - Verify that the custom layered image is removed by performing any of the following checks: - Check that the worker machine config pool is updating with the previous machine config: ```terminal $ oc get mcp ``` ```terminal {title="Sample output"} NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE layered rendered-layered-e8c8bc1de69777325003e80bc0c04b82 True False False 0 0 0 0 4h20m master rendered-master-50d7bc27ee8b9ca2250383f0647ade7f True False False 3 3 3 0 5h39m worker rendered-worker-e8c8bc1de69777325003e80bc0c04b82 True False False 3 3 3 0 5h39m ``` The custom machine config pool, named `layered`, no longer has any nodes. When the `UPDATING` field is `True`, the machine config pool is updating with the previous machine config. When the field becomes `False`, the worker machine config pool has rolled out to the previous machine config. - Check the nodes to see that scheduling on the nodes is disabled. This indicates that the change is being applied: ```terminal $ oc get nodes ``` ```terminal {title="Example output"} NAME STATUS ROLES AGE VERSION ip-10-0-148-79.us-west-1.compute.internal Ready worker 32m v1.35.4 ip-10-0-155-125.us-west-1.compute.internal Ready,SchedulingDisabled worker 35m v1.35.4 ip-10-0-170-47.us-west-1.compute.internal Ready control-plane,master 42m v1.35.4 ip-10-0-174-77.us-west-1.compute.internal Ready control-plane,master 42m v1.35.4 ip-10-0-211-49.us-west-1.compute.internal Ready control-plane,master 42m v1.35.4 ip-10-0-218-151.us-west-1.compute.internal Ready worker 31m v1.35.4 ``` - When the node is back in the `Ready` state, check that the node is using the base image: 1. Open an `oc debug` session to the node. For example: ```terminal $ oc debug node/ip-10-0-155-125.us-west-1.compute.internal ``` 2. Set `/host` as the root directory within the debug shell: ```terminal sh-4.4# chroot /host ``` 3. Run the `rpm-ostree status` command to view that the base image is in use: ```terminal sh-4.4# rpm-ostree status ``` ```text {title="Example output"} State: idle Deployments: * ostree-unverified-registry:registry.build05.ci.openshift.org/ci-ln-qd0hmqk/stable@sha256:a8bd32573f787f6d1c23e1d669abbefd1e31339826d06e750c0ca632ad6c414f Digest: sha256:a8bd32573f787f6d1c23e1d669abbefd1e31339826d06e750c0ca632ad6c414f Version: 419.96.202501202201-0 (2025-01-20T22:06:13Z) ``` ### Removal of an on-cluster custom layered image {#coreos-layering-configuring-on-remove_mco-coreos-layering} You can remove an on-cluster custom layered image from the repository by deleting the `MachineOSBuild` object that created the image. Removing unneeded custom layered images prevents the images from taking up excessive space in your registry. The credentials provided by the registry push secret that you added to the `MachineOSBuild` object must grant the permission for deleting an image from the registry. If the delete permission is not provided, the image is not removed when you delete the `MachineOSBuild` object. The custom layered image is not deleted if the image is either currently in use on a node or is desired by the nodes, as indicated by the `machineconfiguration.openshift.io/currentImage` or `machineconfiguration.openshift.io/desiredImage` annotations on the node, which are added to the node when you create the `MachineOSConfig` object. ## Using Out-of-cluster image mode to apply a custom layered image {#coreos-layering-configuring_mco-coreos-layering} You can use the image mode for OpenShift out-of-cluster build process to apply a custom layered image to your nodes by creating a `MachineOSConfig` custom resource (CR). When you create the object, the Machine Config Operator (MCO) reboots those nodes with the new custom layered image, overriding the base Red Hat Enterprise Linux CoreOS (RHCOS) image. To apply a custom layered image to your cluster, you must have the custom layered image in a repository that your cluster can access. Then, create a `MachineConfig` object that points to the custom layered image. You need a separate `MachineConfig` object for each machine config pool that you want to configure. > [!IMPORTANT] > As soon as you apply an out-of-cluster custom image to your cluster, you effectively *take ownership* of your custom layered images and those nodes. OpenShift Container Platform no longer automatically updates any node that uses the custom layered image. You become responsible for maintaining and updating your nodes as appropriate. If you roll back the custom layer, OpenShift Container Platform resumes automatically updating the node. See the "Updating with a RHCOS custom layered image" for important information about updating nodes that use a custom layered image. **Prerequisites** - You must create a custom layered image that is based on an OpenShift Container Platform image digest, not a tag. > [!NOTE] > You should use the same base RHCOS image that is installed on the rest of your cluster. Use the `oc adm release info --image-for rhel-coreos` command to obtain the base image being used in your cluster. For example, the following Containerfile creates a custom layered image from an OpenShift Container Platform 4.22 image and overrides the kernel package with one from CentOS 9 Stream: ```yaml {title="Example Containerfile for a custom layer image"} # Using a 4.22.0 image FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256... #Install hotfix rpm RUN rpm-ostree override replace http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/kernel-{,core-,modules-,modules-core-,modules-extra-}5.14.0-295.el9.x86_64.rpm && \ rpm-ostree cleanup -m && \ bootc container lint ``` where: `FROM` : Specifies the RHCOS base image of your cluster. `RUN` : Replaces the kernel packages. > [!NOTE] > Instructions on how to create a Containerfile are beyond the scope of this documentation. - Because the process for building a custom layered image is performed outside of the cluster, you must use the `--authfile /path/to/pull-secret` option with Podman or Buildah. Alternatively, to have the pull secret read by these tools automatically, you can add it to one of the default file locations: `~/.docker/config.json`, `$XDG_RUNTIME_DIR/containers/auth.json`, `~/.docker/config.json`, or `~/.dockercfg`. Refer to the `containers-auth.json` man page for more information. - You must push the custom layered image to a repository that your cluster can access. **Procedure** 1. Create a machine config file. 1. Create a YAML file similar to the following: ```yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: worker name: os-layer-custom spec: osImageURL: quay.io/my-registry/custom-image@sha256... ``` where: `metadata.labels` : Specifies the machine config pool to deploy the custom layered image. `spec.osImageURL` : Specifies the path to the custom layered image in the repository. 2. Create the `MachineConfig` object: ```terminal $ oc create -f .yaml ``` > [!IMPORTANT] > It is strongly recommended that you test your images outside of your production environment before rolling out to your cluster. **Verification** You can verify that the custom layered image is applied by performing any of the following checks: 1. Check that the worker machine config pool has rolled out with the new machine config: 1. Check that the new machine config is created: ```terminal $ oc get mc ``` ```terminal {title="Sample output"} NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE 00-master 5bdb57489b720096ef912f738b46330a8f577803 3.5.0 95m 00-worker 5bdb57489b720096ef912f738b46330a8f577803 3.5.0 95m 01-master-container-runtime 5bdb57489b720096ef912f738b46330a8f577803 3.5.0 95m 01-master-kubelet 5bdb57489b720096ef912f738b46330a8f577803 3.5.0 95m 01-worker-container-runtime 5bdb57489b720096ef912f738b46330a8f577803 3.5.0 95m 01-worker-kubelet 5bdb57489b720096ef912f738b46330a8f577803 3.5.0 95m 99-master-generated-registries 5bdb57489b720096ef912f738b46330a8f577803 3.5.0 95m 99-master-ssh 3.2.0 98m 99-worker-generated-registries 5bdb57489b720096ef912f738b46330a8f577803 3.5.0 95m 99-worker-ssh 3.2.0 98m os-layer-custom 10s rendered-master-15961f1da260f7be141006404d17d39b 5bdb57489b720096ef912f738b46330a8f577803 3.5.0 95m rendered-worker-5aff604cb1381a4fe07feaf1595a797e 5bdb57489b720096ef912f738b46330a8f577803 3.5.0 95m rendered-worker-5de4837625b1cbc237de6b22bc0bc873 5bdb57489b720096ef912f738b46330a8f577803 3.5.0 4s ``` The `os-layer-custom` object is the newly created machine config. The `rendered-worker-5de4837625b1cbc237de6b22bc0bc873` object is the newly created rendered machine config. 2. Check that the `osImageURL` value in the new machine config points to the expected image: ```terminal $ oc describe mc rendered-worker-5de4837625b1cbc237de6b22bc0bc873 ``` ```terminal {title="Example output"} Name: rendered-worker-5de4837625b1cbc237de6b22bc0bc873 Namespace: Labels: Annotations: machineconfiguration.openshift.io/generated-by-controller-version: 5bdb57489b720096ef912f738b46330a8f577803 machineconfiguration.openshift.io/release-image-version: 4.22.0-ec.3 API Version: machineconfiguration.openshift.io/v1 Kind: MachineConfig ... Os Image URL: quay.io/my-registry/custom-image@sha256... ``` 3. Check that the associated machine config pool is updated with the new machine config: ```terminal $ oc get mcp ``` ```terminal {title="Sample output"} NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-15961f1da260f7be141006404d17d39b True False False 3 3 3 0 39m worker rendered-worker-5de4837625b1cbc237de6b22bc0bc873 True False False 3 0 0 0 39m ``` When the `UPDATING` field is `True`, the machine config pool is updating with the new machine config. In this case, you will not see the new machine config listed in the output. When the field becomes `False`, the worker machine config pool has rolled out to the new machine config. 4. Check the nodes to see that scheduling on the nodes is disabled. This indicates that the change is being applied: ```terminal $ oc get nodes ``` ```terminal {title="Example output"} NAME STATUS ROLES AGE VERSION ip-10-0-148-79.us-west-1.compute.internal Ready worker 32m v1.35.4 ip-10-0-155-125.us-west-1.compute.internal Ready,SchedulingDisabled worker 35m v1.35.4 ip-10-0-170-47.us-west-1.compute.internal Ready control-plane,master 42m v1.35.4 ip-10-0-174-77.us-west-1.compute.internal Ready control-plane,master 42m v1.35.4 ip-10-0-211-49.us-west-1.compute.internal Ready control-plane,master 42m v1.35.4 ip-10-0-218-151.us-west-1.compute.internal Ready worker 31m v1.35.4 ``` 2. When the node is back in the `Ready` state, check that the node is using the custom layered image: 1. Open an `oc debug` session to the node. For example: ```terminal $ oc debug node/ip-10-0-155-125.us-west-1.compute.internal ``` 2. Set `/host` as the root directory within the debug shell: ```terminal sh-4.4# chroot /host ``` 3. Run the `rpm-ostree status` command to view that the custom layered image is in use: ```terminal sh-4.4# sudo rpm-ostree status ``` ```text {title="Example output"} State: idle Deployments: * ostree-unverified-registry:quay.io/my-registry/... Digest: sha256:... ``` ### Reverting an out-of-cluster node {#coreos-layering-removing_mco-coreos-layering} You can revert an out-of-cluster custom layered image from the nodes in specific machine config pools. The Machine Config Operator (MCO) reboots those nodes with the cluster base Red Hat Enterprise Linux CoreOS (RHCOS) image, overriding the custom layered image. To remove a Red Hat Enterprise Linux CoreOS (RHCOS) custom layered image from your cluster, you need to delete the machine config that applied the image. **Procedure** - Delete the machine config that applied the custom layered image. ```terminal $ oc delete mc os-layer-custom ``` After deleting the machine config, the nodes reboot. **Verification** You can verify that the custom layered image is removed by performing any of the following checks: 1. Check that the worker machine config pool is updating with the previous machine config: ```terminal $ oc get mcp ``` ```terminal {title="Sample output"} NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-6faecdfa1b25c114a58cf178fbaa45e2 True False False 3 3 3 0 39m worker rendered-worker-6b000dbc31aaee63c6a2d56d04cd4c1b False True False 3 0 0 0 39m ``` When the `UPDATING` field is `True`, the machine config pool is updating with the previous machine config. When the field becomes `False`, the worker machine config pool has rolled out to the previous machine config. 2. Check the nodes to see that scheduling on the nodes is disabled. This indicates that the change is being applied: ```terminal $ oc get nodes ``` ```terminal {title="Example output"} NAME STATUS ROLES AGE VERSION ip-10-0-148-79.us-west-1.compute.internal Ready worker 32m v1.35.4 ip-10-0-155-125.us-west-1.compute.internal Ready,SchedulingDisabled worker 35m v1.35.4 ip-10-0-170-47.us-west-1.compute.internal Ready control-plane,master 42m v1.35.4 ip-10-0-174-77.us-west-1.compute.internal Ready control-plane,master 42m v1.35.4 ip-10-0-211-49.us-west-1.compute.internal Ready control-plane,master 42m v1.35.4 ip-10-0-218-151.us-west-1.compute.internal Ready worker 31m v1.35.4 ``` 3. When the node is back in the `Ready` state, check that the node is using the base image: 1. Open an `oc debug` session to the node by running the following command: ```terminal $ oc debug node/ ``` 2. Set `/host` as the root directory within the debug shell by running the following command: ```terminal sh-5.1# chroot /host ``` 3. Run the `rpm-ostree status` command to view that the custom layered image is in use: ```terminal sh-5.1# sudo rpm-ostree status ``` ```text {title="Example output"} State: idle Deployments: * ostree-unverified-registry:podman pull quay.io/openshift-release-dev/ocp-release@sha256:e2044c3cfebe0ff3a99fc207ac5efe6e07878ad59fd4ad5e41f88cb016dacd73 Digest: sha256:e2044c3cfebe0ff3a99fc207ac5efe6e07878ad59fd4ad5e41f88cb016dacd73 ``` ## Applying a custom layered image during OpenShift Container Platform installation {#coreos-layering-install-time_mco-coreos-layering} You can use the standard OpenShift Container Platform installation process to apply a custom layered image to your nodes by adding a `MachineOSConfig` custom resource (CR) YAML and a push secret YAML to the `/manifests/` directory. This allows you to use image mode for OpenShift to apply additional functionality to specific nodes upon cluster installation. After the installation, if you modify a machine config pool or update the OpenShift Container Platform version, the Machine Config Operator (MCO) builds and applies a new custom layered image, and pushes the updated image to your repository. **Prerequisites** - You have a custom layered image in a repository that your cluster can access. ```yaml {title="Example containerFile for a custom layered image"} FROM quay.io/centos/centos:stream9 AS centos RUN dnf install -y epel-release FROM [rhel-coreos image] AS configs COPY --from=centos /etc/yum.repos.d /etc/yum.repos.d COPY --from=centos /etc/pki/rpm-gpg/RPM-GPG-KEY-* /etc/pki/rpm-gpg/ RUN sed -i 's/\$stream/9-stream/g' /etc/yum.repos.d/centos*.repo && \ rpm-ostree install cowsay && \ ostree container commit ``` - You have a repository and any needed secret where the MCO can push any updated custom layered images. **Procedure** 1. Create a YAML file for the `MachineOSConfig` object similar to the following: ```yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineOSConfig metadata: name: worker annotations: machineconfiguration.openshift.io/pre-built-image: "quay.io/myorg/custom-rhcos@sha256:abc123..." spec: machineConfigPool: name: worker imageBuilder: imageBuilderType: Job renderedImagePushSpec: quay.io/your-registry/layered-rhcos:latest renderedImagePushSecret: name: push-secret containerFile: - containerfileArch: NoArch content: | FROM configs AS final RUN rpm-ostree install cowsay && \ ostree container commit ``` where: `metadata.name` : Specifies a name for the `MachineOSConfig` object. The name must match the name of the associated machine config pool. `metadata.annotations.machineconfiguration.openshift.io/pre-built-image` : Specifies the digested image pull spec of your custom layered image. `spec.machineConfigPool` : Specifies the name of the machine config pool associated with the nodes where you want to deploy the custom layered image. `spec.imageBuilder.imageBuilderType` : Specifies the name of the image builder to use. This must be `Job`, which is a reference to the `job` object that is managing the image build. `spec.renderedImagePushSpec` : Specifies an image registry to push any updated custom layered images to, if needed, after the installation process is complete. This can be any registry that your cluster has access to in the `host[:port][/namespace]/name` or `svc_name.namespace.svc[:port]/repository/name:` format. You can specify a mirror registry if you cluster is properly configured to use a mirror registry. `spec.renderedImagePushSecret.name` : Specifies the name of the push secret needed by the MCO to push the updated custom layered image to that registry. `spec.containerFile` : Specifies the containerFile that you used to create the custom layered image. 1. Create a YAML file for the push secret similar to the following: ```yaml apiVersion: v1 kind: Secret metadata: name: push-secret namespace: openshift-machine-config-operator data: .dockerconfigjson: secret type: kubernetes.io/dockerconfigjson ``` 2. When the `manifests` directory is available, add the `MachineOSConfig` YAML to the directory by using a command similar to the following: ```terminal $ cp .yaml manifests/ ``` where: `file-name` : Specifies the YAML file for the `MachineOSConfig` object. 3. Add the push secret YAML to the `manifests` directory by using a command similar to the following: ```terminal $ cp .yaml manifests/ ``` where: `file-name` : Specifies the YAML file for the push secret. 4. Continue with the installation process as usual. **Verification** - After the installation is complete, check that the `MachineOSConfig` object displays the `PreBuiltImageSeeded` status as `True` and contains a reference to the custom layered image by using the following command: ```terminal $ oc get machineosconfigs.machineconfiguration.openshift.io -o yaml ``` ```yaml {title="Example output"} apiVersion: v1 items: - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineOSConfig metadata: annotations: machineconfiguration.openshift.io/current-machine-os-build: worker-4cedbc10da849ae7019288febc3a2d17 # ... status: conditions: - lastTransitionTime: "2025-11-19T13:32:17Z" message: MachineOSConfig seeded with pre-built image "quay.io/myorg/custom-rhcos@sha256:abc123..." reason: PreBuiltImageSeeded status: "True" type: Seeded currentImagePullSpec: image-registry.openshift-image-registry.svc:5000/openshift-machine-config-operator/layered-rhcos@sha256:3c8fc667adcb432ce0c83581f16086afec08a961dd28fed69bb6bad6db0a0754 ``` where: `items.status.conditions.reason.PreBuiltImageSeeded.True` : Specifies that the associated nodes were created using your custom layered image. `items.status.currentImagePullSpec` : Specifies the digested image pull spec for the new custom layered image. ## Updating with a RHCOS custom layered image {#coreos-layering-updating_mco-coreos-layering} When you configure image mode for OpenShift, OpenShift Container Platform no longer automatically updates the node pool that uses the custom layered image. You become responsible to manually update your nodes as appropriate. Updating a node with a custom layered image is not required. However, if that node gets too far behind the current OpenShift Container Platform version, you could experience unexpected results. To update a node that uses a custom layered image, follow these general steps: 1. The cluster automatically upgrades to version x.y.z+1, except for the nodes that use the custom layered image. 2. You could then create a new Containerfile that references the updated OpenShift Container Platform image and the RPM that you had previously applied. 3. Create a new machine config that points to the updated custom layered image. **Additional resources** {._additional-resources} - [Using the on-cluster image mode to apply a custom layered image](/openshift-docs-markdown/machine_configuration/mco-coreos-layering#coreos-layering-configuring-on-proc_mco-coreos-layering) - [Removing an on-cluster custom layered image](/openshift-docs-markdown/machine_configuration/mco-coreos-layering#coreos-layering-configuring-on-remove_mco-coreos-layering) - [Pausing the machine config pools](/openshift-docs-markdown/updating/updating_a_cluster/update-using-custom-machine-config-pools#update-using-custom-machine-config-pools-pause_update-using-custom-machine-config-pools) - [Rebuilding an on-cluster custom layered image](/openshift-docs-markdown/machine_configuration/mco-coreos-layering#coreos-layering-configuring-on-rebuild_mco-coreos-layering) - [Reverting an on-cluster custom layered image](/openshift-docs-markdown/machine_configuration/mco-coreos-layering#coreos-layering-configuring-on-revert_mco-coreos-layering) - [Modifying a custom layered image](/openshift-docs-markdown/machine_configuration/mco-coreos-layering#coreos-layering-configuring-on-modifying_mco-coreos-layering) - [About checking machine config node status](/openshift-docs-markdown/machine_configuration/index#checking-mco-node-status_machine-config-overview) - [Updating the global cluster pull secret](/openshift-docs-markdown/openshift_images/managing_images/using-image-pull-secrets#images-update-global-pull-secret_using-image-pull-secrets) - [Enabling features using feature gates](/openshift-docs-markdown/nodes/clusters/nodes-cluster-enabling-features#nodes-cluster-enabling-features) - [Updating with a RHCOS custom layered image](/openshift-docs-markdown/machine_configuration/mco-coreos-layering#coreos-layering-updating_mco-coreos-layering) - [What is a Hotfix package? (Red Hat Knowledgebase article)](https://access.redhat.com/solutions/2996001) - [Download Red Hat Enterprise Linux Packages](https://access.redhat.com/downloads/content/479/ver=/rhel---9/9.1/x86_64/packages)