--- title: Installing a cluster on user-provisioned infrastructure in AWS by using CloudFormation templates --- # Installing a cluster on user-provisioned infrastructure in AWS by using CloudFormation templates {#installing-aws-user-infra} To deploy OpenShift Container Platform version 4.22 on Amazon Web Services (AWS) with your own infrastructure, use the `CloudFormation` templates or create resources according to your company’s policies. One way to create this infrastructure is to use the `CloudFormation` templates. You can change the templates to customize your infrastructure or use the information that they contain to create AWS objects according to your company’s policies. > [!IMPORTANT] > The steps for performing a user-provisioned infrastructure installation are an example only. Installing a cluster with your own infrastructure requires knowledge of the cloud provider and the installation process of OpenShift Container Platform. Several `CloudFormation` templates are available to assist in completing these steps or to help model your own. You are also free to create the required resources through other methods; the templates are just an example. ## Prerequisites {#_prerequisites} - You reviewed details about the OpenShift Container Platform installation and update processes. - You read the documentation on selecting a cluster installation method and preparing it for users. - You configured an AWS account to host the cluster. > [!IMPORTANT] > If you have an AWS profile stored on your computer, it must not use a temporary session token that you generated while using a multifactor authentication device. The cluster continues to use your current AWS credentials to create AWS resources for the entire life of the cluster, so you must use key-based, long-term credentials. To generate appropriate keys, see Managing Access Keys for IAM Users in the AWS documentation. You can supply the keys when you run the installation program. - You prepared the user-provisioned infrastructure. - You downloaded the AWS CLI and installed it on your computer. - If you use a firewall, you configured it to allow the sites that your cluster requires access to. > [!NOTE] > Be sure to also review this site list if you are configuring a proxy. - If the cloud identity and access management (IAM) APIs are not accessible in your environment, or if you do not want to store an administrator-level credential secret in the `kube-system` namespace, you can manually create and keep long-term credentials. **Additional resources** {._additional-resources} - [OpenShift Container Platform installation and update](/openshift-docs-markdown/architecture/architecture-installation#architecture-installation) - [Selecting a cluster installation method and preparing it for users](/openshift-docs-markdown/installing/overview/installing-preparing#installing-preparing) - [Configuring an AWS account](/openshift-docs-markdown/installing/installing_aws/installing-aws-account#installing-aws-account) - [Preparing user-provisioned infrastructure](/openshift-docs-markdown/installing/installing_aws/upi/upi-aws-installation-reqs#upi-aws-installation-reqs) - [Configuring your firewall](/openshift-docs-markdown/installing/install_config/configuring-firewall#configuring-firewall-module_configuring-firewall) - [Manually creating long-term credentials](/openshift-docs-markdown/installing/installing_aws/ipi/installing-aws-customizations#manually-create-iam_installing-aws-customizations) - [Managing access keys for IAM Users (AWS documentation)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) - [Install the AWS CLI using the bundled installer (AWS documentation)](https://docs.aws.amazon.com/cli/latest/userguide/install-bundle.html) ## Creating the installation files for AWS {#installation-user-infra-generate_installing-aws-user-infra} To install OpenShift Container Platform on Amazon Web Services by using user-provisioned infrastructure, you must generate the files that the installation program needs to deploy your cluster and modify them so that the cluster creates only the machines that it will use. You generate and customize the `install-config.yaml` file, Kubernetes manifests, and Ignition config files. You also have the option to first set up a separate `var` partition during the preparation phases of installation. ### Creating a separate `/var` partition {#installation-disk-partitioning-upi-templates_installing-aws-user-infra} To isolate growing storage for containers, etcd, or logs, you can optionally create a separate `/var` partition on worker nodes before you generate Ignition configs. It is recommended that disk partitioning for OpenShift Container Platform be left to the installation program. However, there are cases where you might want to create separate partitions in a part of the filesystem that you expect to grow. OpenShift Container Platform supports the addition of a single partition to attach storage to either the `/var` partition or a subdirectory of `/var`. For example: - `/var/lib/containers`: Holds container-related content that can grow as more images and containers are added to a system. - `/var/lib/etcd`: Holds data that you might want to keep separate for purposes such as performance optimization of etcd storage. - `/var`: Holds data that you might want to keep separate for purposes such as auditing. Storing the contents of a `/var` directory separately makes it easier to grow storage for those areas as needed and reinstall OpenShift Container Platform at a later date and keep that data intact. With this method, you will not have to pull all your containers again, nor will you have to copy massive log files when you update systems. Because `/var` must be in place before a fresh installation of Red Hat Enterprise Linux CoreOS (RHCOS), the following procedure sets up the separate `/var` partition by creating a machine config manifest that is inserted during the `openshift-install` preparation phases of an OpenShift Container Platform installation. > [!IMPORTANT] > If you follow the steps to create a separate `/var` partition in this procedure, it is not necessary to create the Kubernetes manifest and Ignition config files again as described later in this section. **Procedure** 1. Create a directory to hold the OpenShift Container Platform installation files: ```terminal $ mkdir $HOME/clusterconfig ``` 2. Run `openshift-install` to create a set of files in the `manifest` and `openshift` subdirectories. Answer the system questions as you are prompted: ```terminal $ openshift-install create manifests --dir $HOME/clusterconfig ``` ```terminal {title="Example output"} ? SSH Public Key ... INFO Credentials loaded from the "myprofile" profile in file "/home/myuser/.aws/credentials" INFO Consuming Install Config from target directory INFO Manifests created in: $HOME/clusterconfig/manifests and $HOME/clusterconfig/openshift ``` 3. Optional: Confirm that the installation program created manifests in the `clusterconfig/openshift` directory: ```terminal $ ls $HOME/clusterconfig/openshift/ ``` ```terminal {title="Example output"} 99_kubeadmin-password-secret.yaml 99_openshift-cluster-api_master-machines-0.yaml 99_openshift-cluster-api_master-machines-1.yaml 99_openshift-cluster-api_master-machines-2.yaml ... ``` 4. Create a Butane config that configures the additional partition. For example, name the file `$HOME/clusterconfig/98-var-partition.bu`, change the disk device name to the name of the storage device on the `worker` systems, and set the storage size as appropriate. This example places the `/var` directory on a separate partition: ```yaml variant: openshift version: 4.22.0 metadata: labels: machineconfiguration.openshift.io/role: worker name: 98-var-partition storage: disks: - device: /dev/disk/by-id/ partitions: - label: var start_mib: size_mib: number: 5 filesystems: - device: /dev/disk/by-partlabel/var path: /var format: xfs mount_options: [defaults, prjquota] with_mount_unit: true ``` where: `` : Specifies the storage device name of the disk that you want to partition. `` : Specifies the `start_mib` parameter. When adding a data partition to the boot disk, a minimum value of 25000 MiB (Mebibytes) is recommended. The root file system is automatically resized to fill all available space up to the specified offset. If no value is specified, or if the specified value is smaller than the recommended minimum, the resulting root file system will be too small, and future reinstalls of RHCOS might overwrite the beginning of the data partition. `` : Specifies the size of the data partition in mebibytes. `storage.filesystems.mount_options` : The `prjquota` mount option must be enabled for filesystems used for container storage. > [!NOTE] > When creating a separate `/var` partition, you cannot use different instance types for worker nodes, if the different instance types do not have the same device name. 5. Create a manifest from the Butane config and save it to the `clusterconfig/openshift` directory. For example, run the following command: ```terminal $ butane $HOME/clusterconfig/98-var-partition.bu -o $HOME/clusterconfig/openshift/98-var-partition.yaml ``` 6. Run `openshift-install` again to create Ignition configs from a set of files in the `manifest` and `openshift` subdirectories: ```terminal $ openshift-install create ignition-configs --dir $HOME/clusterconfig ``` ```terminal $ ls $HOME/clusterconfig/ auth bootstrap.ign master.ign metadata.json worker.ign ``` You can now use the Ignition config files as input to the installation procedures to install Red Hat Enterprise Linux CoreOS (RHCOS) systems. ### Creating the installation configuration file {#installation-generate-aws-user-infra-install-config_installing-aws-user-infra} Generate and customize the installation configuration file that the installation program needs to deploy your cluster. **Prerequisites** - You obtained the OpenShift Container Platform installation program for user-provisioned infrastructure and the pull secret for your cluster. - You checked that you are deploying your cluster to an Amazon Web Services (AWS) Region with an accompanying Red Hat Enterprise Linux CoreOS (RHCOS) AMI published by Red Hat. If you are deploying to an AWS Region that requires a custom AMI, such as an AWS GovCloud Region, you must create the `install-config.yaml` file manually. **Procedure** 1. Create the `install-config.yaml` file. 1. Change to the directory that contains the installation program and run the following command: ```terminal $ ./openshift-install create install-config --dir ``` For ``, specify the directory name to store the files that the installation program creates. > [!IMPORTANT] > Specify an empty directory. Some installation assets, such as bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift Container Platform version. 2. At the prompts, provide the configuration details for your cloud: 1. Optional: Select an SSH key to use to access your cluster machines. > [!NOTE] > For production OpenShift Container Platform clusters on which you want to perform installation debugging or disaster recovery, specify an SSH key that your `ssh-agent` process uses. 2. Select **aws** as the platform to target. 3. If you do not have an AWS profile stored on your computer, enter the AWS access key ID and secret access key for the user that you configured to run the installation program. > [!NOTE] > The AWS access key ID and secret access key are stored in `~/.aws/credentials` in the home directory of the current user on the installation host. You are prompted for the credentials by the installation program if the credentials for the exported profile are not present in the file. Any credentials that you provide to the installation program are stored in the file. 4. Select the AWS Region to deploy the cluster to. 5. Select the base domain for the Route 53 service that you configured for your cluster. 6. Enter a descriptive name for your cluster. 7. Paste the [pull secret from Red Hat OpenShift Cluster Manager](https://console.redhat.com/openshift/install/pull-secret). 2. If you are installing a three-node cluster, modify the `install-config.yaml` file by setting the `compute.replicas` parameter to `0`. This ensures that the cluster’s control planes are schedulable. For more information, see "Installing a three-node cluster on AWS". 3. Optional: Back up the `install-config.yaml` file. > [!IMPORTANT] > The `install-config.yaml` file is consumed during the installation process. If you want to reuse the file, you must back it up now. **Additional resources** {._additional-resources} - [Configuration and credential file settings (AWS documentation)](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) ### Configuring the cluster-wide proxy during installation {#installation-configure-proxy_installing-aws-user-infra} Production environments can deny direct access to the internet and instead have an HTTP or HTTPS proxy available. You can configure a new OpenShift Container Platform cluster to use a proxy by configuring the proxy settings in the `install-config.yaml` file. **Prerequisites** - You have an existing `install-config.yaml` file. - You have reviewed the sites that your cluster requires access to and determined whether any of them need to bypass the proxy. By default, the proxy handles all cluster egress traffic, including calls to hosting cloud provider APIs. You added sites to the `Proxy` object’s `spec.noProxy` field to bypass the proxy if necessary. > [!NOTE] > The `Proxy` object `status.noProxy` field includes the values of the `networking.machineNetwork[].cidr`, `networking.clusterNetwork[].cidr`, and `networking.serviceNetwork[]` fields from your installation configuration. > > For installations on Amazon Web Services (AWS), Google Cloud, Microsoft Azure, and Red Hat OpenStack Platform (RHOSP), the `Proxy` object `status.noProxy` field also includes the instance metadata endpoint (`169.254.169.254`). **Procedure** 1. Edit your `install-config.yaml` file and add the proxy settings. For example: ```yaml apiVersion: v1 baseDomain: my.domain.com proxy: httpProxy: http://:@: httpsProxy: https://:@: noProxy: ec2..amazonaws.com,elasticloadbalancing..amazonaws.com,s3..amazonaws.com additionalTrustBundle: | -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- additionalTrustBundlePolicy: # ... ``` where: `proxy.httpProxy` : Specifies a proxy URL to use for creating HTTP connections outside the cluster. The URL scheme must be `http`. `proxy.httpsProxy` : Specifies a proxy URL to use for creating HTTPS connections outside the cluster. `proxy.noProxy` : Specifies a comma-separated list of destination domain names, IP addresses, or other network CIDRs to exclude from proxying. Preface a domain with `.` to match subdomains only. For example, `.y.com` matches `x.y.com`, but not `y.com`. Use `*` to bypass the proxy for all destinations. If you have added the Amazon `EC2`, `Elastic Load Balancing`, and `S3` VPC endpoints to your VPC, you must add these endpoints to the `noProxy` field. `additionalTrustBundle` : If you specify this value, the installation program generates a config map named `user-ca-bundle` in the `openshift-config` namespace to hold the additional CA certificates. If you specify `additionalTrustBundle` and at least one proxy setting, the `Proxy` object references the `user-ca-bundle` config map in the `trustedCA` field. The Cluster Network Operator then creates a `trusted-ca-bundle` config map that merges the contents specified for the `trustedCA` parameter with the RHCOS trust bundle. You must set the `additionalTrustBundle` field unless an authority from the RHCOS trust bundle signs the proxy’s identity certificate. `additionalTrustBundlePolicy` : Specifies the policy that determines the configuration of the `Proxy` object to reference the `user-ca-bundle` config map in the `trustedCA` field. The allowed values are `Proxyonly` and `Always`. Use `Proxyonly` to reference the `user-ca-bundle` config map only when you configure an `http/https` proxy. Use `Always` to always reference the `user-ca-bundle` config map. The default value is `Proxyonly`. Optional parameter. > [!NOTE] > The installation program does not support the proxy `readinessEndpoints` field. > [!NOTE] > If the installation program times out, restart and then complete the deployment by using the `wait-for` command of the installation program. For example: > > ```terminal > $ ./openshift-install wait-for install-complete --log-level debug > ``` 2. Save the file and reference it when installing OpenShift Container Platform. The installation program creates a cluster-wide proxy named `cluster` that uses the proxy settings in the `install-config.yaml` file. If you do not give proxy settings, the installation program still creates a `cluster` `Proxy` object, but it has a nil `spec`. > [!NOTE] > Only the `Proxy` object named `cluster` is supported, and you cannot create additional proxies. ### Creating the Kubernetes manifest and Ignition config files {#installation-user-infra-generate-k8s-manifest-ignition_installing-aws-user-infra} Because you manually provision infrastructure, you must generate the Kubernetes manifest and Ignition config files that the cluster requires. The installation program converts the installation configuration into Kubernetes manifests and then wraps them into Ignition configuration files. You use these Ignition files to configure the cluster machines. > [!IMPORTANT] > - The Ignition config files that the OpenShift Container Platform installation program generates contain certificates that expire after 24 hours, which the system then renews. If you shut down the cluster before the system renews the certificates and you later restart the cluster after the 24 hours have elapsed, the cluster automatically recovers the expired certificates. The exception is that you must manually approve the pending `node-bootstrapper` certificate signing requests (CSRs) to recover kubelet certificates. See the documentation for *Recovering from expired control plane certificates* for more information. > - Use Ignition config files within 12 hours after you generate them, because the 24-hour certificate rotates from 16 to 22 hours after you install the cluster. By using the Ignition config files within 12 hours, you can avoid installation failure if the certificate update runs during installation. **Prerequisites** - You obtained the OpenShift Container Platform installation program. - You created the `install-config.yaml` installation configuration file. **Procedure** 1. Change to the directory that contains the OpenShift Container Platform installation program and generate the Kubernetes manifests for the cluster: ```terminal $ ./openshift-install create manifests --dir ``` where: `` : Specifies the installation directory that contains the `install-config.yaml` file you created. 2. Remove the Kubernetes manifest files that define the control plane machines: ```terminal $ rm -f /openshift/99_openshift-cluster-api_master-machines-*.yaml ``` By removing these files, you prevent the cluster from automatically generating control plane machines. 3. Remove the Kubernetes manifest files that define the control plane machine set: ```terminal $ rm -f /openshift/99_openshift-machine-api_master-control-plane-machine-set.yaml ``` 4. Remove the Kubernetes manifest files that define the worker machines: ```terminal $ rm -f /openshift/99_openshift-cluster-api_worker-machineset-*.yaml ``` > [!IMPORTANT] > If you disabled the `MachineAPI` capability when installing a cluster on user-provisioned infrastructure, you must remove the Kubernetes manifest files that define the worker machines. Otherwise, your cluster fails to install. Because you create and manage the worker machines yourself, you do not need to initialize these machines. > [!WARNING] > If you are installing a three-node cluster, skip the following step to allow the control plane nodes to be schedulable. > [!IMPORTANT] > When you configure control plane nodes from the default unschedulable to schedulable, you require additional subscriptions because control plane nodes then become compute nodes. 5. Verify that the `mastersSchedulable` parameter in the `/manifests/cluster-scheduler-02-config.yml` Kubernetes manifest file is set to `false`. This setting prevents pods from being scheduled on the control plane machines: 1. Open the `/manifests/cluster-scheduler-02-config.yml` file. 2. Locate the `mastersSchedulable` parameter and verify that it is set to `false`. 3. Save and exit the file. 6. Optional: If you do not want [the Ingress Operator](https://github.com/openshift/cluster-ingress-operator) to create DNS records on your behalf, remove the `privateZone` and `publicZone` sections from the `/manifests/cluster-dns-02-config.yml` DNS configuration file: ```yaml apiVersion: config.openshift.io/v1 kind: DNS metadata: creationTimestamp: null name: cluster spec: baseDomain: example.openshift.com privateZone: id: mycluster-100419-private-zone publicZone: id: example.openshift.com status: {} ``` `spec.privateZone`: Remove this section completely. If you do so, you must add ingress DNS records manually in a later step. 7. To create the Ignition configuration files, run the following command from the directory that contains the installation program: ```terminal $ ./openshift-install create ignition-configs --dir ``` where: `` : Specifies the same installation directory. The installation program creates Ignition config files for the bootstrap, control plane, and compute nodes in the installation directory. The program also creates the `kubeadmin-password` and `kubeconfig` files in the `.//auth` directory: ``` . ├── auth │ ├── kubeadmin-password │ └── kubeconfig ├── bootstrap.ign ├── master.ign ├── metadata.json └── worker.ign ``` ## Extracting the infrastructure name {#installation-extracting-infraid_installing-aws-user-infra} To identify your cluster resources in Amazon Web Services, extract the unique infrastructure name from the Ignition config files. The Ignition config files contain a unique cluster identifier that you can use to uniquely identify your cluster in Amazon Web Services. The infrastructure name is also used to locate the appropriate AWS resources during an OpenShift Container Platform installation. The provided CloudFormation templates contain references to this infrastructure name, so you must extract it. > [!WARNING] > Do not run the `openshift-install create manifests` command again after creating any Google Cloud resources. Running the command again generates a new cluster identifier, which will cause errors in existing resources. If you need to regenerate the manifests because you modified the `install-config.yaml` file, delete any Google Cloud resources you created and re-create them with the new cluster identifier. **Prerequisites** - You obtained the OpenShift Container Platform installation program and the pull secret for your cluster. - You generated the Ignition config files for your cluster. - You installed the `jq` package. **Procedure** - To extract and view the infrastructure name from the Ignition config file metadata, run the following command: ```terminal $ jq -r .infraID /metadata.json ``` where `` is the path to the directory that you stored the installation files in. ```terminal {title="Example output"} openshift-vw9j6 ``` The output of this command is your cluster name and a random string. ## Creating a VPC in AWS {#installation-creating-aws-vpc_installing-aws-user-infra} To provide the network foundation for your OpenShift Container Platform cluster, create a Virtual Private Cloud (VPC) in Amazon Web Services (AWS) by using the provided CloudFormation template. You can customize the VPC to meet your requirements, including VPN and route tables. You can use the provided CloudFormation template and a custom parameter file to create a stack of AWS resources that represent the VPC. > [!NOTE] > If you do not use the provided CloudFormation template to create your AWS infrastructure, you must review the provided information and manually create the infrastructure. If your cluster does not initialize correctly, you might have to contact Red Hat support with your installation logs. **Prerequisites** - You added your AWS keys and region to your local AWS profile by running `aws configure`. **Procedure** 1. Create a JSON file that contains the parameter values that the template requires: ```json [ { "ParameterKey": "VpcCidr", "ParameterValue": "10.0.0.0/16" }, { "ParameterKey": "AvailabilityZoneCount", "ParameterValue": "1" }, { "ParameterKey": "SubnetBits", "ParameterValue": "12" } ] ``` where: `VpcCidr` : Specifies the CIDR block for the VPC in the format `x.x.x.x/16-24`. `AvailabilityZoneCount` : Specifies the number of availability zones to deploy the VPC in. Set the value to an integer between `1` and `3`. `SubnetBits` : Specifies the size of each subnet in each availability zone. Set the value to an integer between `5` and `13`, where `5` is `/27` and `13` is `/19`. 2. Copy the template from the **CloudFormation template for the VPC** section of this topic and save it as a YAML file on your computer. This template describes the VPC that your cluster requires. 3. Launch the CloudFormation template to create a stack of AWS resources that represent the VPC: > [!IMPORTANT] > You must enter the command on a single line. ```terminal $ aws cloudformation create-stack --stack-name \ --template-body file://