--- title: Configuring a private cluster --- # Configuring a private cluster {#configuring-private-cluster} After installing OpenShift Container Platform, you can restrict access to cluster DNS, ingress, API server, and Azure registry storage endpoints to make core cluster services private. ## About private clusters {#private-clusters-about_configuring-private-cluster} You can make a deployed cluster private by restricting DNS, Ingress Controller, and API server access to internal networks. By default, OpenShift Container Platform is provisioned using publicly-accessible DNS and endpoints. You can set the DNS, Ingress Controller, and API server to private after you deploy your private cluster. > [!IMPORTANT] > If the cluster has any public subnets, load balancer services created by administrators might be publicly accessible. To ensure cluster security, verify that these services are explicitly annotated as private. ### DNS {#private-clusters-about-dns_configuring-private-cluster} If you install OpenShift Container Platform on installer-provisioned infrastructure, the installation program creates records in a pre-existing public zone and, where possible, creates a private zone for the cluster’s own DNS resolution. In both the public zone and the private zone, the installation program or cluster creates DNS entries for `*.apps`, for the `Ingress` object, and `api`, for the API server. The `*.apps` records in the public and private zone are identical, so when you delete the public zone, the private zone seamlessly provides all DNS resolution for the cluster. ### Ingress Controller {#private-clusters-about-ingress-controller_configuring-private-cluster} Because the default `Ingress` object is created as public, the load balancer is internet-facing and in the public subnets. The Ingress Operator generates a default certificate for an Ingress Controller to serve as a placeholder until you configure a custom default certificate. Do not use Operator-generated default certificates in production clusters. The Ingress Operator does not rotate its own signing certificate or the default certificates that it generates. Operator-generated default certificates are intended as placeholders for custom default certificates that you configure. ### API server {#private-clusters-about-api-server_configuring-private-cluster} By default, the installation program creates appropriate network load balancers for the API server to use for both internal and external traffic. On Amazon Web Services (AWS), separate public and private load balancers are created. The load balancers are identical except that an additional port is available on the internal one for use within the cluster. Although the installation program automatically creates or destroys the load balancer based on API server requirements, the cluster does not manage or maintain them. As long as you preserve the cluster’s access to the API server, you can manually modify or move the load balancers. For the public load balancer, port 6443 is open and the health check is configured for HTTPS against the `/readyz` path. On Google Cloud, a single load balancer is created to manage both internal and external API traffic, so you do not need to modify the load balancer. On Microsoft Azure, both public and private load balancers are created. However, because of limitations in current implementation, you just retain both load balancers in a private cluster. ## Configuring DNS records to be published in a private zone {#private-clusters-setting-dns-private_configuring-private-cluster} You can remove the public zone from the cluster DNS configuration so that new DNS records are published only to the private zone and remain available to internal clients. For all OpenShift Container Platform clusters, whether public or private, DNS records are published in a public zone by default. You can remove the public zone from the cluster DNS configuration to avoid exposing DNS records to the public. You might want to avoid exposing sensitive information, such as internal domain names, internal IP addresses, or the number of clusters at an organization, or you might simply have no need to publish records publicly. If all the clients that should be able to connect to services within the cluster use a private DNS service that has the DNS records from the private zone, then there is no need to have a public DNS record for the cluster. After you deploy a cluster, you can modify its DNS to use only a private zone by modifying the `DNS` custom resource (CR). Modifying the `DNS` CR in this way means that any DNS records that are subsequently created are not published to public DNS servers, which keeps knowledge of the DNS records isolated to internal users. This can be done when you configure the cluster to be private, or if you never want DNS records to be publicly resolvable. Alternatively, even in a private cluster, you might keep the public zone for DNS records because it allows clients to resolve DNS names for applications running on that cluster. For example, an organization can have machines that connect to the public internet and then establish VPN connections for certain private IP ranges in order to connect to private IP addresses. The DNS lookups from these machines use the public DNS to determine the private addresses of those services, and then connect to the private addresses over the VPN. **Procedure** 1. Review the `DNS` CR for your cluster by running the following command and observing the output: ```terminal $ oc get dnses.config.openshift.io/cluster -o yaml ``` ```yaml {title="Example output"} apiVersion: config.openshift.io/v1 kind: DNS metadata: creationTimestamp: "2019-10-25T18:27:09Z" generation: 2 name: cluster resourceVersion: "37966" selfLink: /apis/config.openshift.io/v1/dnses/cluster uid: 0e714746-f755-11f9-9cb1-02ff55d8f976 spec: baseDomain: privateZone: tags: Name: -int kubernetes.io/cluster/: owned publicZone: id: Z2XXXXXXXXXXA4 status: {} ``` Note that the `spec` section contains both a private and a public zone. 2. Patch the `DNS` CR to remove the public zone by running the following command: ```terminal $ oc patch dnses.config.openshift.io/cluster --type=merge --patch='{"spec": {"publicZone": null}}' ``` ```yaml {title="Example output"} dns.config.openshift.io/cluster patched ``` The Ingress Operator consults the `DNS` CR definition when it creates DNS records for `IngressController` objects. If only private zones are specified, only private records are created. > [!IMPORTANT] > Existing DNS records are not modified when you remove the public zone. You must manually delete previously published public DNS records if you no longer want them to be published publicly. **Verification** - Review the `DNS` CR for your cluster and confirm that the public zone was removed, by running the following command and observing the output: ```terminal $ oc get dnses.config.openshift.io/cluster -o yaml ``` ```yaml {title="Example output"} apiVersion: config.openshift.io/v1 kind: DNS metadata: creationTimestamp: "2019-10-25T18:27:09Z" generation: 2 name: cluster resourceVersion: "37966" selfLink: /apis/config.openshift.io/v1/dnses/cluster uid: 0e714746-f755-11f9-9cb1-02ff55d8f976 spec: baseDomain: privateZone: tags: Name: -int kubernetes.io/cluster/-wfpg4: owned status: {} ``` ## Setting the Ingress Controller to private {#private-clusters-setting-ingress-private_configuring-private-cluster} You can configure the default Ingress Controller to use an internal endpoint so that application routes are published only in the private DNS zone. After you deploy a cluster, you can modify its Ingress Controller to use only a private zone. **Procedure** 1. Modify the default Ingress Controller to use only an internal endpoint: ```terminal $ oc replace --force --wait --filename - < [!IMPORTANT] > The OpenShift CLI (`oc`) steps that remove the external load balancers require the Machine API. For clusters that cannot use the Machine API, you must manually remove the external load balancers. > > Clusters with the infrastructure platform type `none` cannot use the Machine API. To view the platform type for your cluster, run the following command: > > ```terminal > $ oc get infrastructure cluster -o jsonpath='{.status.platform}' > ``` **Prerequisites** - You have installed an OpenShift Container Platform cluster on AWS. - You have access to the AWS console as a user with administrator privileges. - You have access to the OpenShift CLI (`oc`) as a user with administrator privileges. **Procedure** 1. Log in to the AWS console as a user with administrator privileges. 2. Delete the external load balancer. > [!NOTE] > The API DNS entry in the private zone already points to the internal load balancer, which uses an identical configuration, so you do not need to modify the internal load balancer. 3. Delete the `api..` DNS entry in the public zone. where `` is the name of the cluster and `` is the base domain for the cluster. 4. To remove the external load balancers, log in to the OpenShift CLI (`oc`) as a user with administrator privileges. - If your cluster uses a control plane machine set, remove the external load balancers by editing the `ControlPlaneMachineSet` custom resource (CR). 1. Edit the `ControlPlaneMachineSet` CR by running the following command: ```terminal $ oc edit controlplanemachineset.machine.openshift.io cluster \ -n openshift-machine-api ``` 2. Remove the external load balancers by deleting the corresponding lines in the control plane machine set custom resource (CR). In the `spec.template.spec.providerSpec.value.loadBalancers` section of the CR, the `name` value for the external load balancer ends in `-ext`. Delete the line with the external load balancer `name` value and the line with the external load balancer `type` value that accompanies it. ```yaml apiVersion: machine.openshift.io/v1 kind: ControlPlaneMachineSet metadata: name: cluster namespace: openshift-machine-api spec: # ... template: # ... spec: providerSpec: value: loadBalancers: - name: -ext type: network - name: -int type: network # ... ``` 3. Save your changes and exit the object specification. When you save an update to the control plane machine set, the Control Plane Machine Set Operator updates the control plane machines according to your configured update strategy. For more information, see "Updating the control plane configuration". - If your cluster does not use a control plane machine set, you must delete the external load balancers from each control plane machine. 1. List the cluster machines by running the following command: ```terminal $ oc get machine -n openshift-machine-api ``` ```text {title="Example output"} NAME STATE TYPE REGION ZONE AGE -master-0 running m4.xlarge us-east-1 us-east-1a 17m -master-1 running m4.xlarge us-east-1 us-east-1b 17m -master-2 running m4.xlarge us-east-1 us-east-1a 17m -worker-us-east-1a- running m4.xlarge us-east-1 us-east-1a 15m -worker-us-east-1a- running m4.xlarge us-east-1 us-east-1a 15m -worker-us-east-1b- running m4.xlarge us-east-1 us-east-1b 15m ``` The control plane machines contain the `master` string in their names. 2. Remove the external load balancer from each control plane machine: 1. Edit a control plane machine object to by running the following command: ```terminal $ oc edit machines -n openshift-machine-api ``` where `` is the name of the control plane machine object to modify. 2. Remove the lines that describe the external load balancer. In the `spec.providerSpec.value.loadBalancers` section of the CR, the `name` value for the external load balancer ends in `-ext`. Delete the line with the external load balancer `name` value and the the line with the external load balancer `type` value that accompanies it. ```yaml apiVersion: machine.openshift.io/v1beta1 kind: Machine metadata: name: namespace: openshift-machine-api spec: providerSpec: value: loadBalancers: - name: -ext type: network - name: -int type: network # ... ``` 3. Save your changes and exit the object specification. 4. Repeat this process for each control plane machine. **Additional resources** {._additional-resources} - [Updating the control plane configuration](/openshift-docs-markdown/machine_management/control_plane_machine_management/cpmso-managing-machines#cpmso-feat-config-update_cpmso-managing-machines) ## Restricting the API server to private for an Microsoft Azure cluster {#private-clusters-setting-api-private-azure_configuring-private-cluster} If the security posture of your organization does not allow clusters to use an open API endpoint, you can restrict the API server to use only internal load balancers. To implement this API server restriction, use the Microsoft Azure console to delete the external load balancer component. **Prerequisites** - You have installed an OpenShift Container Platform cluster on Azure. - You have access to the Azure console as a user with administrator privileges. **Procedure** 1. Log in to the Azure console as a user with administrator privileges. 2. Delete the following resources: - The `api-v4` rule for the public load balancer. - The `frontendIPConfiguration` parameter that is associated with the `api-v4` rule for the public load balancer. - The public IP address that is specified in the `frontendIPConfiguration` parameter. 3. Configure the Ingress Controller endpoint publishing scope to `Internal`. For more information, see "Configuring the Ingress Controller endpoint publishing scope to Internal". 4. Delete the `api.` DNS entry in the public zone. where `` is the name of the cluster. **Additional resources** {._additional-resources} - [Configuring the Ingress Controller endpoint publishing scope to Internal](/openshift-docs-markdown/networking/ingress_load_balancing/configuring_ingress_cluster_traffic/nw-configuring-ingress-controller-endpoint-publishing-strategy#nw-ingresscontroller-change-internal_nw-configuring-ingress-controller-endpoint-publishing-strategy) ## Configuring a private storage endpoint on Azure {#registry-configuring-private-storage-endpoint-azure_configuring-private-cluster} You can configure the Image Registry Operator to use a private Azure storage endpoint so that registry storage is not exposed through a public-facing endpoint. You can leverage the Image Registry Operator to use private endpoints on Azure, which enables seamless configuration of private storage accounts when OpenShift Container Platform is deployed on private Azure clusters. This allows you to deploy the image registry without exposing public-facing storage endpoints. > [!IMPORTANT] > Do not configure a private storage endpoint on Microsoft Azure Red Hat OpenShift (ARO), because the endpoint can put your Microsoft Azure Red Hat OpenShift cluster in an unrecoverable state. You can configure the Image Registry Operator to use private storage endpoints on Azure in one of two ways: - By configuring the Image Registry Operator to discover the VNet and subnet names - With user-provided Azure Virtual Network (VNet) and subnet names ### Limitations for configuring a private storage endpoint on Azure {#limitations-configuring-private-storage-endpoint-azure} The following limitations apply when configuring a private storage endpoint on Azure: - When configuring the Image Registry Operator to use a private storage endpoint, public network access to the storage account is disabled. Consequently, pulling images from the registry outside of OpenShift Container Platform only works by setting `disableRedirect: true` in the registry Operator configuration. With redirect enabled, the registry redirects the client to pull images directly from the storage account, which will no longer work due to disabled public network access. For more information, see "Disabling redirect when using a private storage endpoint on Azure". - This operation cannot be undone by the Image Registry Operator. ### Configuring a private storage endpoint on Azure by enabling the Image Registry Operator to discover VNet and subnet names {#configuring-private-storage-endpoint-azure-vnet-subnet-iro-discovery_configuring-private-cluster} You can configure a private Azure storage endpoint by enabling the Image Registry Operator to discover the VNet and subnet, allowing registry storage without public network access. The following procedure shows you how to set up a private storage endpoint on Azure by configuring the Image Registry Operator to discover VNet and subnet names. **Prerequisites** - You have configured the image registry to run on Azure. - Your network has been set up using the Installer Provisioned Infrastructure installation method. For users with a custom network setup, see "Configuring a private storage endpoint on Azure with user-provided VNet and subnet names". **Procedure** 1. Edit the Image Registry Operator `config` object and set `networkAccess.type` to `Internal`: ```terminal $ oc edit configs.imageregistry/cluster ``` ```terminal # ... spec: # ... storage: azure: # ... networkAccess: type: Internal # ... ``` 2. Optional: Enter the following command to confirm that the Operator has completed provisioning. This might take a few minutes. ```terminal $ oc get configs.imageregistry/cluster -o=jsonpath="{.spec.storage.azure.privateEndpointName}" -w ``` 3. Optional: If the registry is exposed by a route, and you are configuring your storage account to be private, you must disable redirect if you want pulls external to the cluster to continue to work. Enter the following command to disable redirect on the Image Operator configuration: ```terminal $ oc patch configs.imageregistry cluster --type=merge -p '{"spec":{"disableRedirect": true}}' ``` > [!NOTE] > When redirect is enabled, pulling images from outside of the cluster will not work. **Verification** 1. Fetch the registry service name by running the following command: ```terminal $ oc get imagestream -n openshift ``` ```terminal {title="Example output"} NAME IMAGE REPOSITORY TAGS UPDATED cli image-registry.openshift-image-registry.svc:5000/openshift/cli latest 8 hours ago ... ``` 2. Enter debug mode by running the following command: ```terminal $ oc debug node/ ``` 3. Run the suggested `chroot` command. For example: ```terminal $ chroot /host ``` 4. Enter the following command to log in to your container registry: ```terminal $ podman login --tls-verify=false -u unused -p $(oc whoami -t) image-registry.openshift-image-registry.svc:5000 ``` ```terminal {title="Example output"} Login Succeeded! ``` 5. Enter the following command to verify that you can pull an image from the registry: ```terminal $ podman pull --tls-verify=false image-registry.openshift-image-registry.svc:5000/openshift/tools ``` ```terminal {title="Example output"} Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/tools/openshift/tools... Getting image source signatures Copying blob 6b245f040973 done Copying config 22667f5368 done Writing manifest to image destination Storing signatures 22667f53682a2920948d19c7133ab1c9c3f745805c14125859d20cede07f11f9 ``` ### Configuring a private storage endpoint on Azure with user-provided VNet and subnet names {#configuring-private-storage-endpoint-azure-user-provided-vnet-subnet_configuring-private-cluster} You can configure a private Azure storage endpoint for the image registry by specifying user-provided VNet and subnet names, enabling registry storage without public network access. Use the following procedure to configure a storage account that has public network access disabled and is exposed behind a private storage endpoint on Azure. **Prerequisites** - You have configured the image registry to run on Azure. - You must know the VNet and subnet names used for your Azure environment. - If your network was configured in a separate resource group in Azure, you must also know its name. **Procedure** 1. Edit the Image Registry Operator `config` object and configure the private endpoint using your VNet and subnet names: ```terminal $ oc edit configs.imageregistry/cluster ``` ```terminal # ... spec: # ... storage: azure: # ... networkAccess: type: Internal internal: subnetName: vnetName: networkResourceGroupName: # ... ``` 2. Optional: Enter the following command to confirm that the Operator has completed provisioning. This might take a few minutes. ```terminal $ oc get configs.imageregistry/cluster -o=jsonpath="{.spec.storage.azure.privateEndpointName}" -w ``` > [!NOTE] > When redirect is enabled, pulling images from outside of the cluster will not work. **Verification** 1. Fetch the registry service name by running the following command: ```terminal $ oc get imagestream -n openshift ``` ```terminal {title="Example output"} NAME IMAGE REPOSITORY TAGS UPDATED cli image-registry.openshift-image-registry.svc:5000/openshift/cli latest 8 hours ago ... ``` 2. Enter debug mode by running the following command: ```terminal $ oc debug node/ ``` 3. Run the suggested `chroot` command. For example: ```terminal $ chroot /host ``` 4. Enter the following command to log in to your container registry: ```terminal $ podman login --tls-verify=false -u unused -p $(oc whoami -t) image-registry.openshift-image-registry.svc:5000 ``` ```terminal {title="Example output"} Login Succeeded! ``` 5. Enter the following command to verify that you can pull an image from the registry: ```terminal $ podman pull --tls-verify=false image-registry.openshift-image-registry.svc:5000/openshift/tools ``` ```terminal {title="Example output"} Trying to pull image-registry.openshift-image-registry.svc:5000/openshift/tools/openshift/tools... Getting image source signatures Copying blob 6b245f040973 done Copying config 22667f5368 done Writing manifest to image destination Storing signatures 22667f53682a2920948d19c7133ab1c9c3f745805c14125859d20cede07f11f9 ``` ### Optional: Disabling redirect when using a private storage endpoint on Azure {#disabling-redirect-private-storage-endpoint-azure_configuring-private-cluster} You can disable redirect when using a private Azure storage endpoint so that users outside the cluster can pull images through the image registry route. By default, redirect is enabled when using the image registry. Redirect allows off-loading of traffic from the registry pods into the object storage, which makes pull faster. When redirect is enabled and the storage account is private, users from outside of the cluster are unable to pull images from the registry. In some cases, users might want to disable redirect so that users from outside of the cluster can pull images from the registry. **Prerequisites** - You have configured the image registry to run on Azure. - You have configured a route. **Procedure** - Enter the following command to disable redirect on the image registry configuration: ```terminal $ oc patch configs.imageregistry cluster --type=merge -p '{"spec":{"disableRedirect": true}}' ``` **Verification** 1. Fetch the registry service name by running the following command: ```terminal $ oc get imagestream -n openshift ``` ```terminal {title="Example output"} NAME IMAGE REPOSITORY TAGS UPDATED cli default-route-openshift-image-registry./cli latest 8 hours ago ... ``` 2. Enter the following command to log in to your container registry: ```terminal $ podman login --tls-verify=false -u unused -p $(oc whoami -t) default-route-openshift-image-registry. ``` ```terminal {title="Example output"} Login Succeeded! ``` 3. Enter the following command to verify that you can pull an image from the registry: ```terminal $ podman pull --tls-verify=false default-route-openshift-image-registry. /openshift/tools ``` ```terminal {title="Example output"} Trying to pull default-route-openshift-image-registry./openshift/tools... Getting image source signatures Copying blob 6b245f040973 done Copying config 22667f5368 done Writing manifest to image destination Storing signatures 22667f53682a2920948d19c7133ab1c9c3f745805c14125859d20cede07f11f9 ```