Integrating SPIRE federation with multi-cluster Red Hat OpenShift Service Mesh¶

Configure SPIFFE Runtime Environment (SPIRE) federation across multiple OpenShift Container Platform clusters to enable cross-cluster mutual TLS (mTLS) authentication and zero trust workload identity in a multi-cluster service mesh deployment.

Multi-cluster SPIFFE Runtime Environment integration with Red Hat OpenShift Service Mesh¶

Understand how SPIFFE Runtime Environment (SPIRE) federation integrates with multi-cluster Red Hat OpenShift Service Mesh. Cross-cluster mutual TLS (mTLS) lets workloads on separate clusters authenticate each other under a unified zero-trust identity framework.

Multi-cluster SPIRE integration extends single-cluster SPIRE capabilities to enable workloads in different clusters to authenticate each other using Secure Production Identity Framework for Everyone (SPIFFE) identities. This eliminates the need for separate certificate authorities per cluster and enables true cross-cluster zero-trust architecture.

What gets federated¶

Federation happens at two layers, and both are required:

Layer	What	How
SPIRE Federation	Trust bundles	SPIRE Servers exchange bundles via `https_spiffe` profile
Istio Federation	Service discovery and routing	`Istiod` discovers remote endpoints via remote secrets, routes traffic through east-west gateways

Preparing the environment for multi-cluster SPIFFE Runtime Environment federation¶

Export kubeconfig paths, trust domains, federation endpoints, and JWT issuer URLs for Cluster A and Cluster B before you deploy federated SPIFFE Runtime Environment (SPIRE) operands on both clusters.

Prerequisites

You have two OpenShift Container Platform clusters (4.x) with network connectivity between them.
You have installed Zero Trust Workload Identity Manager on both clusters.
The OpenShift CLI (oc) is configured with access to both clusters.
You have installed the istioctl CLI tool.
You have installed helm. This is used for gateway deployment.
You have Istio version 1.29.2 or later.

Procedure

Export the namespace variables by running the following commands:

$ export ZTWIM_NS=zero-trust-workload-identity-manager

$ export OSSM_NS=istio-system

$ export OSSM_CNI=istio-cni

Export the kubeconfig file paths for Cluster A and Cluster B by running the following commands:

$ export CLUSTER_A_KUBECONFIG="/path/to/cluster-a/kubeconfig"

$ export CLUSTER_B_KUBECONFIG="/path/to/cluster-b/kubeconfig"

Set the base domain environment variables for Cluster A and Cluster B by running the following commands:

$ export CLUSTER_A_BASE_DOMAIN=$(oc get ingresses.config/cluster \
  -o jsonpath='{.spec.domain}' --kubeconfig "${CLUSTER_A_KUBECONFIG}")

$ export CLUSTER_B_BASE_DOMAIN=$(oc get ingresses.config/cluster \
  -o jsonpath='{.spec.domain}' --kubeconfig "${CLUSTER_B_KUBECONFIG}")

Export the trust domain environment variables from the base domain of each cluster by running the following commands:

$ export CLUSTER_A_TRUST_DOMAIN="${CLUSTER_A_BASE_DOMAIN}"

$ export CLUSTER_B_TRUST_DOMAIN="${CLUSTER_B_BASE_DOMAIN}"

Define the cluster and network environment variables by running the following commands:

$ export CLUSTER_A=cluster-a

$ export CLUSTER_B=cluster-b

$ export NETWORK_A=network-a

$ export NETWORK_B=network-b

Export the federation endpoint URLs for Cluster A and Cluster B by running the following commands:

$ export FEDERATION_ENDPOINT_A="https://federation.${CLUSTER_A_BASE_DOMAIN}"

$ export FEDERATION_ENDPOINT_B="https://federation.${CLUSTER_B_BASE_DOMAIN}"

Set the JWT issuer environment variables for Cluster A and Cluster B by running the following commands:

$ export JWT_ISSUER_A="https://oidc-discovery.${CLUSTER_A_BASE_DOMAIN}"

$ export JWT_ISSUER_B="https://oidc-discovery.${CLUSTER_B_BASE_DOMAIN}"

Deploying SPIFFE Runtime Environment with federation on both clusters¶

Deploy SPIFFE Runtime Environment (SPIRE) operand custom resources (CRs) with federation enabled on Cluster A and Cluster B, wait for operands to become ready, and verify SDS configuration.

Prerequisites

You have completed preparing the environment for multi-cluster SPIRE federation. For more information, see "Preparing the environment for multi-cluster SPIFFE Runtime Environment federation".
The environment variables from the "Preparing the environment for multi-cluster SPIFFE Runtime Environment federation" procedure are set.

Procedure

Deploy SPIRE with federation enabled on Cluster A:

Create a YAML file that defines the ZeroTrustWorkloadIdentityManager CR on Cluster A:

apiVersion: operator.openshift.io/v1alpha1
kind: ZeroTrustWorkloadIdentityManager
metadata:
  name: cluster
spec:
  trustDomain: ${CLUSTER_A_TRUST_DOMAIN}
  clusterName: ${CLUSTER_A}
  bundleConfigMap: "spire-bundle"

Apply the YAML file on Cluster A by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_A_KUBECONFIG}" -f <filename>
```

Create a YAML file that defines the SpireServer CR on Cluster A:

apiVersion: operator.openshift.io/v1alpha1
kind: SpireServer
metadata:
  name: cluster
spec:
  logLevel: "info"
  logFormat: "text"
  jwtIssuer: $JWT_ISSUER_A
  caValidity: "24h"
  defaultX509Validity: "1h"
  defaultJWTValidity: "5m"
  caKeytype: "rsa-2048"
  jwtKeyType: "rsa-2048"
  keyManager: ""
  caSubject:
    country: "US"
    organization: "RH"
    commonName: "SPIRE Server CA"
  persistence:
    size: "5Gi"
    accessMode: "ReadWriteOnce"
  datastore:
    databaseType: "sqlite3"
    connectionString: "/run/spire/data/datastore.sqlite3"
    tlsSecretName: ""
    maxOpenConns: 100
    maxIdleConns: 10
    connMaxLifetime: 0
    disableMigration: "false"
  federation:
    bundleEndpoint:
      profile: "https_spiffe"

Apply the YAML file on Cluster A by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_A_KUBECONFIG}" -f <filename>
```

Create a YAML file that defines the SpireAgent CR on Cluster A:

apiVersion: operator.openshift.io/v1alpha1
kind: SpireAgent
metadata:
  name: cluster
spec:
  socketPath: "/run/spire/agent-sockets"
  logLevel: "info"
  logFormat: "text"
  nodeAttestor:
    k8sPSATEnabled: "true"
  workloadAttestors:
    k8sEnabled: "true"
    workloadAttestorsVerification:
      type: "auto"
      hostCertBasePath: "/etc/kubernetes"
      hostCertFileName: "kubelet-ca.crt"
    useNewContainerLocator: "true"

Apply the YAML file on Cluster A by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_A_KUBECONFIG}" -f <filename>
```

Create a YAML file that defines the SpiffeCSIDriver CR on Cluster A:

apiVersion: operator.openshift.io/v1alpha1
kind: SpiffeCSIDriver
metadata:
  name: cluster
spec:
  agentSocketPath: "/run/spire/agent-sockets"
  pluginName: csi.spiffe.io

Apply the YAML file on Cluster A by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_A_KUBECONFIG}" -f <filename>
```

Create a YAML file that defines the SpireOIDCDiscoveryProvider CR on Cluster A:

apiVersion: operator.openshift.io/v1alpha1
kind: SpireOIDCDiscoveryProvider
metadata:
  name: cluster
spec:
  logLevel: "info"
  logFormat: "text"
  csiDriverName: "csi.spiffe.io"
  jwtIssuer: $JWT_ISSUER_A
  replicaCount: 1
  managedRoute: "true"

Apply the YAML file on Cluster A by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_A_KUBECONFIG}" -f <filename>
```

Deploy SPIRE with federation enabled on Cluster B:

Create a YAML file that defines the ZeroTrustWorkloadIdentityManager CR on Cluster B:

apiVersion: operator.openshift.io/v1alpha1
kind: ZeroTrustWorkloadIdentityManager
metadata:
  name: cluster
spec:
  trustDomain: ${CLUSTER_B_TRUST_DOMAIN}
  clusterName: ${CLUSTER_B}
  bundleConfigMap: "spire-bundle"

Apply the YAML file on Cluster B by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_B_KUBECONFIG}" -f <filename>
```

Create a YAML file that defines the SpireServer CR on Cluster B:

apiVersion: operator.openshift.io/v1alpha1
kind: SpireServer
metadata:
  name: cluster
spec:
  logLevel: "info"
  logFormat: "text"
  jwtIssuer: $JWT_ISSUER_B
  caValidity: "24h"
  defaultX509Validity: "1h"
  defaultJWTValidity: "5m"
  caKeytype: "rsa-2048"
  jwtKeyType: "rsa-2048"
  keyManager: ""
  caSubject:
    country: "US"
    organization: "RH"
    commonName: "SPIRE Server CA"
  persistence:
    size: "5Gi"
    accessMode: "ReadWriteOnce"
  datastore:
    databaseType: "sqlite3"
    connectionString: "/run/spire/data/datastore.sqlite3"
    tlsSecretName: ""
    maxOpenConns: 100
    maxIdleConns: 10
    connMaxLifetime: 0
    disableMigration: "false"
  federation:
    bundleEndpoint:
      profile: "https_spiffe"

Apply the YAML file on Cluster B by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_B_KUBECONFIG}" -f <filename>
```

Create a YAML file that defines the SpireAgent CR on Cluster B:

apiVersion: operator.openshift.io/v1alpha1
kind: SpireAgent
metadata:
  name: cluster
spec:
  socketPath: "/run/spire/agent-sockets"
  logLevel: "info"
  logFormat: "text"
  nodeAttestor:
    k8sPSATEnabled: "true"
  workloadAttestors:
    k8sEnabled: "true"
    workloadAttestorsVerification:
      type: "auto"
      hostCertBasePath: "/etc/kubernetes"
      hostCertFileName: "kubelet-ca.crt"
    useNewContainerLocator: "true"

Apply the YAML file on Cluster B by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_B_KUBECONFIG}" -f <filename>
```

Create a YAML file that defines the SpiffeCSIDriver CR on Cluster B:

apiVersion: operator.openshift.io/v1alpha1
kind: SpiffeCSIDriver
metadata:
  name: cluster
spec:
  agentSocketPath: "/run/spire/agent-sockets"
  pluginName: csi.spiffe.io

Apply the YAML file on Cluster B by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_B_KUBECONFIG}" -f <filename>
```

Create a YAML file that defines the SpireOIDCDiscoveryProvider CR on Cluster B:

apiVersion: operator.openshift.io/v1alpha1
kind: SpireOIDCDiscoveryProvider
metadata:
  name: cluster
spec:
  logLevel: "info"
  logFormat: "text"
  csiDriverName: "csi.spiffe.io"
  jwtIssuer: $JWT_ISSUER_B
  replicaCount: 1
  managedRoute: "true"

Apply the YAML file on Cluster B by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_B_KUBECONFIG}" -f <filename>
```

Wait for the spire-server StatefulSet to become ready on Cluster A by running the following command:

$ oc rollout status statefulset/spire-server --kubeconfig="${CLUSTER_A_KUBECONFIG}" -n ${ZTWIM_NS} --timeout=300s

Wait for the spire-agent DaemonSet to become ready on Cluster A by running the following command:

$ oc rollout status daemonset/spire-agent --kubeconfig="${CLUSTER_A_KUBECONFIG}" -n ${ZTWIM_NS} --timeout=300s

Wait for the spire-spiffe-csi-driver DaemonSet to become ready on Cluster A by running the following command:

$ oc rollout status daemonset/spire-spiffe-csi-driver --kubeconfig="${CLUSTER_A_KUBECONFIG}" -n ${ZTWIM_NS} --timeout=300s

Wait for the spire-spiffe-oidc-discovery-provider deployment to become available on Cluster A by running the following command:

$ oc wait --for=condition=Available deployment/spire-spiffe-oidc-discovery-provider \
  --kubeconfig="${CLUSTER_A_KUBECONFIG}" -n ${ZTWIM_NS} --timeout=300s

Wait for the spire-server StatefulSet to become ready on Cluster B by running the following command:

$ oc rollout status statefulset/spire-server --kubeconfig="${CLUSTER_B_KUBECONFIG}" -n ${ZTWIM_NS} --timeout=300s

Wait for the spire-agent DaemonSet to become ready on Cluster B by running the following command:

$ oc rollout status daemonset/spire-agent --kubeconfig="${CLUSTER_B_KUBECONFIG}" -n ${ZTWIM_NS} --timeout=300s

Wait for the spire-spiffe-csi-driver DaemonSet to become ready on Cluster B by running the following command:

$ oc rollout status daemonset/spire-spiffe-csi-driver --kubeconfig="${CLUSTER_B_KUBECONFIG}" -n ${ZTWIM_NS} --timeout=300s

Wait for the spire-spiffe-oidc-discovery-provider deployment to become available on Cluster B by running the following command:

$ oc wait --for=condition=Available deployment/spire-spiffe-oidc-discovery-provider \
  --kubeconfig="${CLUSTER_B_KUBECONFIG}" -n ${ZTWIM_NS} --timeout=300s

Verification

Verify that the SDS configuration is available on Cluster A by running the following command:

$ oc get cm spire-agent --kubeconfig="${CLUSTER_A_KUBECONFIG}" -n "${ZTWIM_NS}" \
  -o jsonpath='{.data.agent\.conf}' | grep -A5 '"sds"'

Verify that the SDS configuration is available on Cluster B by running the following command:

$ oc get cm spire-agent --kubeconfig="${CLUSTER_B_KUBECONFIG}" -n "${ZTWIM_NS}" \
  -o jsonpath='{.data.agent\.conf}' | grep -A5 '"sds"'

Example output

"sds": {
  "default_all_bundles_name": "ROOTCA",
  "default_bundle_name": "null"
},

Additional resources¶

Configuring Red Hat OpenShift Service Mesh for multi-cluster SPIFFE Runtime Environment integration¶

Configure Red Hat OpenShift Service Mesh on each cluster with federation settings, east-west gateways, and remote secrets to enable cross-cluster service communication by using SPIRE-issued certificates.

Prerequisites

You deployed SPIFFE Runtime Environment (SPIRE) with federation for multi-cluster integration. For more information, see "Deploying SPIFFE Runtime Environment with federation on both clusters".

Procedure

Verify that the federation routes are created on Cluster A by running the following command:
```
$ oc get route -n ${ZTWIM_NS} --kubeconfig="${CLUSTER_A_KUBECONFIG}" | grep federation
```
Verify that the federation routes are created on Cluster B by running the following command:
```
$ oc get route -n ${ZTWIM_NS} --kubeconfig="${CLUSTER_B_KUBECONFIG}" | grep federation
```

On Cluster A, create a ClusterFederatedTrustDomain object pointing to Cluster B by running the following command:

Create a YAML file that defines the ClusterFederatedTrustDomain CR on Cluster A:

apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterFederatedTrustDomain
metadata:
  name: federation-to-cluster-b
spec:
  trustDomain: ${CLUSTER_B_TRUST_DOMAIN}
  bundleEndpointURL: ${FEDERATION_ENDPOINT_B}
  bundleEndpointProfile:
    type: https_spiffe
    endpointSPIFFEID: spiffe://${CLUSTER_B_TRUST_DOMAIN}/spire/server

Apply the YAML file on Cluster A by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_A_KUBECONFIG}" -f <filename>
```

On Cluster B, create a ClusterFederatedTrustDomain object pointing to Cluster A by running the following command:

Create a YAML file that defines the ClusterFederatedTrustDomain CR on Cluster B:

apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterFederatedTrustDomain
metadata:
  name: federation-to-cluster-a
spec:
  trustDomain: ${CLUSTER_A_TRUST_DOMAIN}
  bundleEndpointURL: ${FEDERATION_ENDPOINT_A}
  bundleEndpointProfile:
    type: https_spiffe
    endpointSPIFFEID: spiffe://${CLUSTER_A_TRUST_DOMAIN}/spire/server

Apply the YAML file on Cluster B by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_B_KUBECONFIG}" -f <filename>
```

Verification

Verify that the SPIRE Server on Cluster A has the trust bundle from Cluster B by running the following command:

$ oc exec --kubeconfig="${CLUSTER_A_KUBECONFIG}" -n ${ZTWIM_NS} spire-server-0 -c spire-server -- \
  spire-server bundle list -socketPath /tmp/spire-server/private/api.sock -format spiffe 2>&1 | head -5

The output must show public keys for ${CLUSTER_B_TRUST_DOMAIN}.

Example output

{
  "trust_domains": {
    "${CLUSTER_B_TRUST_DOMAIN}": {
      "keys": [

Verify that the SPIRE Server on Cluster B has the trust bundle from Cluster A by running the following command:

$ oc exec --kubeconfig="${CLUSTER_B_KUBECONFIG}" -n ${ZTWIM_NS} spire-server-0 -c spire-server -- \
  spire-server bundle list -socketPath /tmp/spire-server/private/api.sock -format spiffe 2>&1 | head -5

The output must show public keys for ${CLUSTER_A_TRUST_DOMAIN}.

Example output

{
  "trust_domains": {
    "${CLUSTER_A_TRUST_DOMAIN}": {
      "keys": [

Verify that the federation endpoint on Cluster A is reachable by running the following command:
```
$ curl -sk "${FEDERATION_ENDPOINT_A}" | python3 -c "import sys,json; print(f'Keys: {len(json.load(sys.stdin).get(\"keys\",[]))}')"
```
The output must show at least one x509-svid key and one jwt-svid key.
Example output
```
Keys: 2
```
Verify that the federation endpoint on Cluster B is reachable by running the following command:
```
$ curl -sk "${FEDERATION_ENDPOINT_B}" | python3 -c "import sys,json; print(f'Keys: {len(json.load(sys.stdin).get(\"keys\",[]))}')"
```
The output must show at least one x509-svid key and one jwt-svid key.
Example output
```
Keys: 2
```

Deploying the Red Hat OpenShift Service Mesh CNI on both clusters¶

Deploy the IstioCNI CR and federated ClusterSPIFFEID resources on Cluster A and Cluster B. This configures Red Hat OpenShift Service Mesh CNI networking and federated SPIFFE trust for cross-cluster mesh workloads.

Prerequisites

You have configured Red Hat OpenShift Service Mesh for multi-cluster integration. For more information, see "Configuring Red Hat OpenShift Service Mesh for multi-cluster SPIFFE Runtime Environment integration".
The environment variables from the "Preparing the environment for multi-cluster SPIFFE Runtime Environment federation" and "Deploying SPIFFE Runtime Environment with federation on both clusters" procedures are set.
You have installed Red Hat OpenShift Service Mesh 2.6.11 on both clusters.

Procedure

Create the Red Hat OpenShift Service Mesh CNI namespace on Cluster A by running the following command:
```
$ oc new-project "${OSSM_CNI}" --kubeconfig="${CLUSTER_A_KUBECONFIG}" 2>/dev/null || true
```
Deploy the IstioCNI CR on Cluster A by running the following command:
1. Create a YAML file that defines the IstioCNI CR on Cluster A:
```
apiVersion: sailoperator.io/v1
kind: IstioCNI
metadata:
  name: default
spec:
  namespace: ${OSSM_CNI}
```
2. Apply the YAML file on Cluster A by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_A_KUBECONFIG}" -f <filename>
```

Wait for the istio-cni-node DaemonSet to be created on Cluster A by running the following command:

$ until oc get daemonset/istio-cni-node --kubeconfig="${CLUSTER_A_KUBECONFIG}" -n "${OSSM_CNI}" &> /dev/null; do
  sleep 3
done

Wait for the IstioCNI DaemonSet to become ready on Cluster A by running the following command:

$ oc rollout status daemonset/istio-cni-node --kubeconfig="${CLUSTER_A_KUBECONFIG}" -n "${OSSM_CNI}" --timeout=300s

Create the Red Hat OpenShift Service Mesh CNI namespace on Cluster B by running the following command:
```
$ oc new-project "${OSSM_CNI}" --kubeconfig="${CLUSTER_B_KUBECONFIG}" 2>/dev/null || true
```
Deploy the IstioCNI CR on Cluster B by running the following command:
1. Create a YAML file that defines the IstioCNI CR on Cluster B:
```
apiVersion: sailoperator.io/v1
kind: IstioCNI
metadata:
  name: default
spec:
  namespace: ${OSSM_CNI}
```
2. Apply the YAML file on Cluster B by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_B_KUBECONFIG}" -f <filename>
```

Wait for the istio-cni-node DaemonSet to be created on Cluster B by running the following command:

$ until oc get daemonset/istio-cni-node --kubeconfig="${CLUSTER_B_KUBECONFIG}" -n "${OSSM_CNI}" &> /dev/null; do
  sleep 3
done

Wait for the IstioCNI DaemonSet to become ready on Cluster B by running the following command:

$ oc rollout status daemonset/istio-cni-node --kubeconfig="${CLUSTER_B_KUBECONFIG}" -n "${OSSM_CNI}" --timeout=300s

Create the federated ClusterSPIFFEID resources on Cluster A by running the following command:

Create a YAML file that defines the federated ClusterSPIFFEID resources on Cluster A:

apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
  name: sample-federation
spec:
  className: zero-trust-workload-identity-manager-spire
  spiffeIDTemplate: "spiffe://{{ .TrustDomain }}/ns/{{ .PodMeta.Namespace }}/sa/{{ .PodSpec.ServiceAccountName }}"
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: sample
  federatesWith:
    - "${CLUSTER_B_TRUST_DOMAIN}"
---
apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
  name: istio-system-federation
spec:
  className: zero-trust-workload-identity-manager-spire
  spiffeIDTemplate: "spiffe://{{ .TrustDomain }}/ns/{{ .PodMeta.Namespace }}/sa/{{ .PodSpec.ServiceAccountName }}"
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: istio-system
  federatesWith:
    - "${CLUSTER_B_TRUST_DOMAIN}"

Apply the YAML file on Cluster A by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_A_KUBECONFIG}" -f <filename>
```

Create federated ClusterSPIFFEID resources on Cluster B by running the following command:

Create a YAML file that defines the federated ClusterSPIFFEID resources on Cluster B:

apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
  name: sample-federation
spec:
  className: zero-trust-workload-identity-manager-spire
  spiffeIDTemplate: "spiffe://{{ .TrustDomain }}/ns/{{ .PodMeta.Namespace }}/sa/{{ .PodSpec.ServiceAccountName }}"
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: sample
  federatesWith:
    - "${CLUSTER_A_TRUST_DOMAIN}"
---
apiVersion: spire.spiffe.io/v1alpha1
kind: ClusterSPIFFEID
metadata:
  name: istio-system-federation
spec:
  className: zero-trust-workload-identity-manager-spire
  spiffeIDTemplate: "spiffe://{{ .TrustDomain }}/ns/{{ .PodMeta.Namespace }}/sa/{{ .PodSpec.ServiceAccountName }}"
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: istio-system
  federatesWith:
    - "${CLUSTER_A_TRUST_DOMAIN}"

Apply the YAML file on Cluster B by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_B_KUBECONFIG}" -f <filename>
```
Warning

Do not patch the default ClusterSPIFFEID (zero-trust-workload-identity-manager-spire-default). The Zero Trust Workload Identity Manager reconciles and reverts manual changes. Instead, create custom ClusterSPIFFEID resources for the specific namespaces.

Deploying the Istio custom resource with the federation configuration¶

Deploy the Istio custom resource on Cluster A and Cluster B with SPIFFE Runtime Environment (SPIRE) federation and multi-cluster Red Hat OpenShift Service Mesh settings. This configures Istiod to obtain workload certificates from SPIRE and to trust SPIFFE bundles from both clusters for cross-cluster mTLS.

The Istio CR must include the following fields and values:

A meshConfig.trustDomain value that matches the SPIRE trust domain.
A meshConfig.caCertificates value with bundle URLs for both clusters. This handles cross-trust-domain validation.
A WORKLOAD_IDENTITY_SOCKET_FILE value for SPIRE SDS integration.
A jwksResolverExtraRootCA value for OIDC validation.
A multi-cluster configuration that includes meshID, clusterName, and network.
A SPIRE injection template configuration.

Note

Do not use meshConfig.trustDomainAliases. Use meshConfig.caCertificates with spiffeBundleUrl instead.

Prerequisites

You have completed deploying the Istio Container Network Interface (CNI) on both clusters. For more information, see "Deploying Red Hat OpenShift Service Mesh CNI on both clusters".
The environment variables from the "Preparing the environment for multi-cluster SPIFFE Runtime Environment federation" and "Deploying SPIFFE Runtime Environment with federation on both clusters" procedures are set.

Procedure

Extract the OpenID Connect (OIDC) certificate on Cluster A by running the following command:

$ export EXTRA_ROOT_CA_A="$(oc get secret oidc-serving-cert \
  --kubeconfig="${CLUSTER_A_KUBECONFIG}" -n ${ZTWIM_NS} -o json | \
  jq -r '.data."tls.crt"' | base64 -d | sed 's/^/        /')"

Extract the OpenID Connect (OIDC) certificate on Cluster B by running the following command:

$ export EXTRA_ROOT_CA_B="$(oc get secret oidc-serving-cert \
  --kubeconfig="${CLUSTER_B_KUBECONFIG}" -n ${ZTWIM_NS} -o json | \
  jq -r '.data."tls.crt"' | base64 -d | sed 's/^/        /')"

Get the bundle endpoint URL for Cluster A by running the following command:
```
$ export BUNDLE_URL_A="${FEDERATION_ENDPOINT_A}"
```
Get the bundle endpoint URL for Cluster B by running the following command:
```
$ export BUNDLE_URL_B="${FEDERATION_ENDPOINT_B}"
```

Create the Istio custom resource (CR) on Cluster A by running the following command:

$ oc new-project "${OSSM_NS}" --kubeconfig="${CLUSTER_A_KUBECONFIG}" 2>/dev/null || true

Apply the Istio CR on Cluster A by running the following command:

Create a YAML file that defines the Istio CR on Cluster A:

apiVersion: sailoperator.io/v1
kind: Istio
metadata:
  name: default
spec:
  namespace: istio-system
  updateStrategy:
    type: InPlace
  values:
    meshConfig:
      trustDomain: ${CLUSTER_A_TRUST_DOMAIN}
      defaultConfig:
        proxyMetadata:
          WORKLOAD_IDENTITY_SOCKET_FILE: "spire-agent.sock"
      caCertificates:
        - spiffeBundleUrl: ${BUNDLE_URL_A}
          trustDomains:
            - ${CLUSTER_A_TRUST_DOMAIN}
        - spiffeBundleUrl: ${BUNDLE_URL_B}
          trustDomains:
            - ${CLUSTER_B_TRUST_DOMAIN}
    global:
      meshID: mesh1
      multiCluster:
        clusterName: ${CLUSTER_A}
      network: ${NETWORK_A}
    pilot:
      jwksResolverExtraRootCA: |
        ${EXTRA_ROOT_CA_A}
      env:
        ENABLE_CA_SERVER: "true"
    sidecarInjectorWebhook:
      templates:
        spire: |
          spec:
            initContainers:
            - name: istio-proxy
              volumeMounts:
              - name: workload-socket
                mountPath: /run/secrets/workload-spiffe-uds
                readOnly: true
            volumes:
              - name: workload-socket
                csi:
                  driver: "csi.spiffe.io"
                  readOnly: true
        spireGw: |
          spec:
            containers:
            - name: istio-proxy
              volumeMounts:
              - name: workload-socket
                mountPath: /run/secrets/workload-spiffe-uds
                readOnly: true
            volumes:
              - name: workload-socket
                csi:
                  driver: "csi.spiffe.io"
                  readOnly: true

Apply the YAML file on Cluster A by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_A_KUBECONFIG}" -f <filename>
```

Create the Istio CR on Cluster B by running the following command:

$ oc new-project "${OSSM_NS}" --kubeconfig="${CLUSTER_B_KUBECONFIG}" 2>/dev/null || true

Apply the Istio CR on Cluster B by running the following command:

Create a YAML file that defines the Istio CR on Cluster B:

apiVersion: sailoperator.io/v1
kind: Istio
metadata:
  name: default
spec:
  namespace: istio-system
  updateStrategy:
    type: InPlace
  values:
    meshConfig:
      trustDomain: ${CLUSTER_B_TRUST_DOMAIN}
      defaultConfig:
        proxyMetadata:
          WORKLOAD_IDENTITY_SOCKET_FILE: "spire-agent.sock"
      caCertificates:
        - spiffeBundleUrl: ${BUNDLE_URL_B}
          trustDomains:
            - ${CLUSTER_B_TRUST_DOMAIN}
        - spiffeBundleUrl: ${BUNDLE_URL_A}
          trustDomains:
            - ${CLUSTER_A_TRUST_DOMAIN}
    global:
      meshID: mesh1
      multiCluster:
        clusterName: ${CLUSTER_B}
      network: ${NETWORK_B}
    pilot:
      jwksResolverExtraRootCA: |
        ${EXTRA_ROOT_CA_B}
      env:
        ENABLE_CA_SERVER: "true"
    sidecarInjectorWebhook:
      templates:
        spire: |
          spec:
            initContainers:
            - name: istio-proxy
              volumeMounts:
              - name: workload-socket
                mountPath: /run/secrets/workload-spiffe-uds
                readOnly: true
            volumes:
              - name: workload-socket
                csi:
                  driver: "csi.spiffe.io"
                  readOnly: true
        spireGw: |
          spec:
            containers:
            - name: istio-proxy
              volumeMounts:
              - name: workload-socket
                mountPath: /run/secrets/workload-spiffe-uds
                readOnly: true
            volumes:
              - name: workload-socket
                csi:
                  driver: "csi.spiffe.io"
                  readOnly: true

Apply the YAML file on Cluster B by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_B_KUBECONFIG}" -f <filename>
```

Wait for the istiod deployment to be created on Cluster A by running the following command:

$ until oc get deployment istiod --kubeconfig="${CLUSTER_A_KUBECONFIG}" -n "${OSSM_NS}" &> /dev/null; do
  sleep 3
done

Wait for Istiod to become ready on Cluster A by running the following command:

$ oc wait --for=condition=Available deployment/istiod \
  --kubeconfig="${CLUSTER_A_KUBECONFIG}" -n "${OSSM_NS}" --timeout=300s

Wait for the istiod deployment to be created on Cluster B by running the following command:

$ until oc get deployment istiod --kubeconfig="${CLUSTER_B_KUBECONFIG}" -n "${OSSM_NS}" &> /dev/null; do
  sleep 3
done

Wait for Istiod to become ready on Cluster B by running the following command:

$ oc wait --for=condition=Available deployment/istiod \
  --kubeconfig="${CLUSTER_B_KUBECONFIG}" -n "${OSSM_NS}" --timeout=300s

Verifying SPIRE integration with Istio on each cluster¶

Verify that Red Hat OpenShift Service Mesh on Cluster A and Cluster B obtains workload certificates from SPIFFE Runtime Environment (SPIRE). This confirms Istio sidecars use SPIRE-issued identities rather than the built-in Istio certificate authority (CA) before you proceed with cross-cluster mesh verification.

Prerequisites

You have deployed the Istio custom resource (CR) with the federation configuration. For more information, see "Deploying the Istio custom resource with the federation configuration".
The environment variables from the "Preparing the environment for multi-cluster SPIFFE Runtime Environment federation" and "Deploying SPIFFE Runtime Environment with federation on both clusters" procedures are set.
Istiod is running and ready on Cluster A and Cluster B.

Procedure

Set the verification namespace variable by running the following command:
```
$ export VERIFY_NS=verify-ossm-ztwim
```

Prepare the verification namespace on both clusters by running the following commands:

Create the verification namespace on Cluster A:

$ oc create namespace ${VERIFY_NS} --kubeconfig="${CLUSTER_A_KUBECONFIG}" 2>/dev/null || true

Enable Istio injection for the verification namespace on Cluster A:

$ oc label namespace ${VERIFY_NS} istio-injection=enabled \
  --kubeconfig="${CLUSTER_A_KUBECONFIG}" --overwrite

Create the verification namespace on Cluster B:

$ oc create namespace ${VERIFY_NS} --kubeconfig="${CLUSTER_B_KUBECONFIG}" 2>/dev/null || true

Enable Istio injection for the verification namespace on Cluster B:

$ oc label namespace ${VERIFY_NS} istio-injection=enabled \
  --kubeconfig="${CLUSTER_B_KUBECONFIG}" --overwrite

Deploy the httpbin workload on Cluster A by running the following command:

Create a YAML file that defines the httpbin Deployment on Cluster A:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  namespace: ${VERIFY_NS}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      annotations:
        inject.istio.io/templates: "sidecar,spire"
        spiffe.io/audience: "test-audience"
      labels:
        app: httpbin
        version: v1
    spec:
      containers:
      - image: docker.io/mccutchen/go-httpbin:v2.15.0
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 8080

Apply the YAML file on Cluster A by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_A_KUBECONFIG}" -f <filename>
```

Deploy the httpbin workload on Cluster B by running the following command:

Create a YAML file that defines the httpbin Deployment on Cluster B:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  namespace: ${VERIFY_NS}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      annotations:
        inject.istio.io/templates: "sidecar,spire"
        spiffe.io/audience: "test-audience"
      labels:
        app: httpbin
        version: v1
    spec:
      containers:
      - image: docker.io/mccutchen/go-httpbin:v2.15.0
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 8080

Apply the YAML file on Cluster B by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_B_KUBECONFIG}" -f <filename>
```

Wait for the httpbin deployment to become available on Cluster A by running the following command:

$ oc rollout status deployment/httpbin \
  -n "${VERIFY_NS}" --kubeconfig="${CLUSTER_A_KUBECONFIG}" --timeout=300s

Wait for the httpbin deployment to become available on Cluster B by running the following command:

$ oc rollout status deployment/httpbin \
  -n "${VERIFY_NS}" --kubeconfig="${CLUSTER_B_KUBECONFIG}" --timeout=300s

Verify the Envoy sidecar certificate on Cluster A by running the following commands:

Get the httpbin pod name on Cluster A:

$ HTTPBIN_POD=$(oc get pod -l app=httpbin -n "${VERIFY_NS}" \
  --kubeconfig="${CLUSTER_A_KUBECONFIG}" -o jsonpath="{.items[0].metadata.name}")

Export the Envoy sidecar certificate chain for the httpbin pod on Cluster A:

$ istioctl --kubeconfig="${CLUSTER_A_KUBECONFIG}" proxy-config secret "${HTTPBIN_POD}" \
  -n "${VERIFY_NS}" -o json \
  | jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' \
  | base64 --decode > chain-a.pem

Confirm the certificate was issued by SPIRE on Cluster A:
```
$ openssl x509 -in chain-a.pem -text | grep SPIRE
```

Verify the Envoy sidecar certificate on Cluster B by running the following commands:

Get the httpbin pod name on Cluster B:

$ HTTPBIN_POD=$(oc get pod -l app=httpbin -n "${VERIFY_NS}" \
  --kubeconfig="${CLUSTER_B_KUBECONFIG}" -o jsonpath="{.items[0].metadata.name}")

Export the Envoy sidecar certificate chain for the httpbin pod on Cluster B:

$ istioctl --kubeconfig="${CLUSTER_B_KUBECONFIG}" proxy-config secret "${HTTPBIN_POD}" \
  -n "${VERIFY_NS}" -o json \
  | jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' \
  | base64 --decode > chain-b.pem

Confirm the certificate was issued by SPIRE on Cluster B:
```
$ openssl x509 -in chain-b.pem -text | grep SPIRE
```
Example output
```
 Issuer: C=US, O=RH, CN=<APP_DOMAIN>/serialNumber=...
        Subject: C=US, O=SPIRE
```
If you see SPIRE in both Issuer and Subject on each cluster, Red Hat OpenShift Service Mesh is obtaining workload certificates from SPIRE rather than the Istio built-in CA.

Remove the verification namespace from both clusters by running the following commands:

Remove the verification namespace from Cluster A:

$ oc delete namespace ${VERIFY_NS} --kubeconfig="${CLUSTER_A_KUBECONFIG}" --ignore-not-found

Remove the verification namespace from Cluster B:

$ oc delete namespace ${VERIFY_NS} --kubeconfig="${CLUSTER_B_KUBECONFIG}" --ignore-not-found

Verifying workload mTLS with SPIRE-issued identities on each cluster¶

Deploy httpbin and curl test workloads with SPIFFE Runtime Environment (SPIRE) sidecar injection on both clusters, enable STRICT mTLS with ISTIO_MUTUAL, and verify HTTP connectivity on each cluster. This confirms workloads use SPIRE-issued certificates under STRICT mTLS.

Prerequisites

You have verified that SPIRE is integrated with Istio on each cluster. For more information, see "Verifying SPIRE integration with Istio on each cluster".
The environment variables from the "Preparing the environment for multi-cluster SPIFFE Runtime Environment federation" and "Deploying SPIFFE Runtime Environment with federation on both clusters" procedures are set.
Istiod is running and ready on both clusters.

Procedure

Set the test environment variables by running the following commands:
1. Set the test namespace environment variable:
```
$ export TPJ=test-ossm-with-ztwim
```
2. Set the SPIFFE audience environment variable:
```
$ export SPIFFE_AUDIENCE="sky-computing-demo"
```

Prepare the test namespace on both clusters by running the following commands:

Create the test namespace on Cluster A:

$ oc create namespace ${TPJ} --kubeconfig="${CLUSTER_A_KUBECONFIG}" 2>/dev/null || true

Enable Istio injection for the test namespace on Cluster A:

$ oc label namespace ${TPJ} istio-injection=enabled \
  --kubeconfig="${CLUSTER_A_KUBECONFIG}" --overwrite

Create the test namespace on Cluster B:

$ oc create namespace ${TPJ} --kubeconfig="${CLUSTER_B_KUBECONFIG}" 2>/dev/null || true

Enable Istio injection for the test namespace on Cluster B:

$ oc label namespace ${TPJ} istio-injection=enabled \
  --kubeconfig="${CLUSTER_B_KUBECONFIG}" --overwrite

Create the httpbin server on Cluster A by running the following command:

Create a YAML file that defines the httpbin ServiceAccount, Service, and Deployment on Cluster A:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
  namespace: ${TPJ}
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  namespace: ${TPJ}
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http-ex-spiffe
    port: 443
    targetPort: 8080
  - name: http
    port: 80
    targetPort: 8080
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  namespace: ${TPJ}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      annotations:
        inject.istio.io/templates: "sidecar,spire"
        spiffe.io/audience: "${SPIFFE_AUDIENCE}"
      labels:
        app: httpbin
        version: v1
    spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/mccutchen/go-httpbin:v2.15.0
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 8080

Apply the YAML file on Cluster A by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_A_KUBECONFIG}" -f <filename>
```

Create the httpbin server on Cluster B by running the following command:

Create a YAML file that defines the httpbin ServiceAccount, Service, and Deployment on Cluster B:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
  namespace: ${TPJ}
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  namespace: ${TPJ}
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http-ex-spiffe
    port: 443
    targetPort: 8080
  - name: http
    port: 80
    targetPort: 8080
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  namespace: ${TPJ}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      annotations:
        inject.istio.io/templates: "sidecar,spire"
        spiffe.io/audience: "${SPIFFE_AUDIENCE}"
      labels:
        app: httpbin
        version: v1
    spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/mccutchen/go-httpbin:v2.15.0
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 8080

Apply the YAML file on Cluster B by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_B_KUBECONFIG}" -f <filename>
```

Wait for the httpbin deployment to become available on both clusters by running the following commands:

Wait for the httpbin deployment on Cluster A:

$ oc rollout status deployment/httpbin \
  -n "${TPJ}" --kubeconfig="${CLUSTER_A_KUBECONFIG}" --timeout=300s

Wait for the httpbin deployment on Cluster B:

$ oc rollout status deployment/httpbin \
  -n "${TPJ}" --kubeconfig="${CLUSTER_B_KUBECONFIG}" --timeout=300s

Create the curl client on Cluster A by running the following command:

Create a YAML file that defines the curl ServiceAccount, Service, and Deployment on Cluster A:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: curl
  namespace: ${TPJ}
---
apiVersion: v1
kind: Service
metadata:
  name: curl
  namespace: ${TPJ}
  labels:
    app: curl
    service: curl
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: curl
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: curl
  namespace: ${TPJ}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: curl
  template:
    metadata:
      annotations:
        inject.istio.io/templates: "sidecar,spire"
        spiffe.io/audience: "${SPIFFE_AUDIENCE}"
      labels:
        app: curl
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: curl
      containers:
      - name: curl
        image: curlimages/curl:8.16.0
        command:
        - /bin/sh
        - -c
        - sleep inf
        imagePullPolicy: IfNotPresent

Apply the YAML file on Cluster A by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_A_KUBECONFIG}" -f <filename>
```

Create the curl client on Cluster B by running the following command:

Create a YAML file that defines the curl ServiceAccount, Service, and Deployment on Cluster B:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: curl
  namespace: ${TPJ}
---
apiVersion: v1
kind: Service
metadata:
  name: curl
  namespace: ${TPJ}
  labels:
    app: curl
    service: curl
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: curl
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: curl
  namespace: ${TPJ}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: curl
  template:
    metadata:
      annotations:
        inject.istio.io/templates: "sidecar,spire"
        spiffe.io/audience: "${SPIFFE_AUDIENCE}"
      labels:
        app: curl
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: curl
      containers:
      - name: curl
        image: curlimages/curl:8.16.0
        command:
        - /bin/sh
        - -c
        - sleep inf
        imagePullPolicy: IfNotPresent

Apply the YAML file on Cluster B by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_B_KUBECONFIG}" -f <filename>
```

Wait for the curl deployment to become available on both clusters by running the following commands:

Wait for the curl deployment on Cluster A:

$ oc rollout status deployment/curl \
  -n "${TPJ}" --kubeconfig="${CLUSTER_A_KUBECONFIG}" --timeout=300s

Wait for the curl deployment on Cluster B:

$ oc rollout status deployment/curl \
  -n "${TPJ}" --kubeconfig="${CLUSTER_B_KUBECONFIG}" --timeout=300s

Verify that the curl client can reach httpbin on both clusters before enabling STRICT mTLS by running the following commands:

Verify connectivity on Cluster A:

$ oc exec deploy/curl -n "${TPJ}" --kubeconfig="${CLUSTER_A_KUBECONFIG}" -it -- \
  curl -s -o /dev/null -w "%{http_code}" http://httpbin

Verify connectivity on Cluster B:

$ oc exec deploy/curl -n "${TPJ}" --kubeconfig="${CLUSTER_B_KUBECONFIG}" -it -- \
  curl -s -o /dev/null -w "%{http_code}" http://httpbin

Example output

You must receive an HTTP 200 status code on each cluster.

Enable STRICT mTLS between the services on Cluster A by running the following command:

Create a YAML file that defines the PeerAuthentication and DestinationRule resources on Cluster A:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: ${TPJ}
spec:
  mtls:
    mode: STRICT
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: curl
  namespace: ${TPJ}
spec:
  host: curl
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: httpbin
  namespace: ${TPJ}
spec:
  host: httpbin
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

Apply the YAML file on Cluster A by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_A_KUBECONFIG}" -f <filename>
```

Enable STRICT mTLS between the services on Cluster B by running the following command:

Create a YAML file that defines the PeerAuthentication and DestinationRule resources on Cluster B:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: ${TPJ}
spec:
  mtls:
    mode: STRICT
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: curl
  namespace: ${TPJ}
spec:
  host: curl
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: httpbin
  namespace: ${TPJ}
spec:
  host: httpbin
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

Apply the YAML file on Cluster B by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_B_KUBECONFIG}" -f <filename>
```

Verify that the curl client can reach httpbin on both clusters with STRICT mTLS enabled by running the following commands:
1. Verify connectivity on Cluster A:
```
$ oc exec deploy/curl -n "${TPJ}" --kubeconfig="${CLUSTER_A_KUBECONFIG}" -it -- \
  curl -s -o /dev/null -w "%{http_code}" http://httpbin
```
2. Verify connectivity on Cluster B:
```
$ oc exec deploy/curl -n "${TPJ}" --kubeconfig="${CLUSTER_B_KUBECONFIG}" -it -- \
  curl -s -o /dev/null -w "%{http_code}" http://httpbin
```
  Example output
```
200
```
  If you receive an HTTP 200 status code on each cluster, Red Hat OpenShift Service Mesh workloads are communicating under STRICT mTLS using SPIRE-issued identities.

Remove the test namespace from both clusters by running the following commands:

Remove the test namespace from Cluster A:

$ oc delete namespace ${TPJ} --kubeconfig="${CLUSTER_A_KUBECONFIG}" --ignore-not-found

Remove the test namespace from Cluster B:

$ oc delete namespace ${TPJ} --kubeconfig="${CLUSTER_B_KUBECONFIG}" --ignore-not-found

Deploying east-west gateways¶

Deploy SPIRE-enabled east-west gateways on both clusters using Helm. Red Hat OpenShift Service Mesh uses east-west gateways to connect cluster networks and enable secure cross-cluster communication in a multi-cluster mesh.

Prerequisites

You deployed the Istio custom resource with the federation configuration. For more information, see "Deploying the Istio custom resource with the federation configuration".
The environment variables from the "Preparing the environment for multi-cluster SPIFFE Runtime Environment federation" and "Deploying SPIFFE Runtime Environment with federation on both clusters" procedures are set.
Federated ClusterSPIFFEID resources exist on both clusters.

Procedure

Add the Istio Helm repository by running the following command:

$ helm repo add istio https://istio-release.storage.googleapis.com/charts

Update the Istio Helm repository by running the following command:
```
$ helm repo update
```

Grant security context constraints (SCC) permissions on Cluster A by running the following command:

$ oc adm policy add-scc-to-user anyuid \
  -z istio-eastwestgateway -n istio-system --kubeconfig="${CLUSTER_A_KUBECONFIG}"

Grant security context constraints (SCC) permissions on Cluster B by running the following command:

$ oc adm policy add-scc-to-user anyuid \
  -z istio-eastwestgateway -n istio-system --kubeconfig="${CLUSTER_B_KUBECONFIG}"

Install the Istio gateway on Cluster A by running the following command:

$ helm upgrade --install istio-eastwestgateway istio/gateway \
  -n istio-system \
  --set-json 'podAnnotations={"inject.istio.io/templates":"gateway,spireGw"}' \
  --set name=istio-eastwestgateway \
  --set networkGateway="${NETWORK_A}" \
  --kubeconfig="${CLUSTER_A_KUBECONFIG}"

Install the Istio gateway on Cluster B by running the following command:

$ helm upgrade --install istio-eastwestgateway istio/gateway \
  -n istio-system \
  --set-json 'podAnnotations={"inject.istio.io/templates":"gateway,spireGw"}' \
  --set name=istio-eastwestgateway \
  --set networkGateway="${NETWORK_B}" \
  --kubeconfig="${CLUSTER_B_KUBECONFIG}"

Wait for the east-west gateway to become available on Cluster A by running the following command:

$ oc wait --for=condition=Available deployment/istio-eastwestgateway \
  --kubeconfig="${CLUSTER_A_KUBECONFIG}" -n istio-system --timeout=300s

Wait for the east-west gateway to become available on Cluster B by running the following command:

$ oc wait --for=condition=Available deployment/istio-eastwestgateway \
  --kubeconfig="${CLUSTER_B_KUBECONFIG}" -n istio-system --timeout=300s

Create the cross-network Gateway custom resource (CR) on Cluster A by running the following command:

Create a YAML file that defines the Gateway CR on Cluster A:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: cross-network-gateway
  namespace: istio-system
spec:
  selector:
    istio: eastwestgateway
  servers:
    - port:
        number: 15443
        name: tls
        protocol: TLS
      tls:
        mode: AUTO_PASSTHROUGH
      hosts:
        - "*.local"

Apply the YAML file on Cluster A by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_A_KUBECONFIG}" -f <filename>
```

Create the cross-network Gateway CR on Cluster B by running the following command:

Create a YAML file that defines the Gateway CR on Cluster B:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: cross-network-gateway
  namespace: istio-system
spec:
  selector:
    istio: eastwestgateway
  servers:
    - port:
        number: 15443
        name: tls
        protocol: TLS
      tls:
        mode: AUTO_PASSTHROUGH
      hosts:
        - "*.local"

Apply the YAML file on Cluster B by running the following command:
```
$ oc apply --kubeconfig="${CLUSTER_B_KUBECONFIG}" -f <filename>
```
The Gateway CRs configure the east-west gateway deployment to accept cross-cluster TLS traffic on port 15443 using AUTO_PASSTHROUGH mode. This preserves SPIRE-issued certificates for end-to-end mTLS.

Verification

Verify that the cross-network Gateway exists on Cluster A by running the following command:

$ oc get gateway cross-network-gateway -n istio-system \
  --kubeconfig="${CLUSTER_A_KUBECONFIG}" \
  -o jsonpath='{.spec.servers[0].tls.mode}{"\n"}'

Example output

AUTO_PASSTHROUGH

Verify that the cross-network Gateway exists on Cluster B by running the following command:

$ oc get gateway cross-network-gateway -n istio-system \
  --kubeconfig="${CLUSTER_B_KUBECONFIG}" \
  -o jsonpath='{.spec.servers[0].tls.mode}{"\n"}'

Example output

AUTO_PASSTHROUGH

Exchanging remote secrets¶

Create remote secrets on both clusters so Istiod can discover services in the peer cluster and route cross-cluster traffic through the east-west gateways.

Prerequisites

You have deployed the east-west gateway, including the cross-network Gateway CR on both clusters. For more information, see "Deploying east-west gateways".
The environment variables from the "Preparing the environment for multi-cluster SPIFFE Runtime Environment federation" and "Deploying SPIFFE Runtime Environment with federation on both clusters" procedures are set.
The istioctl CLI is available and configured for both clusters.

Procedure

Create an Istio remote secret on Cluster A by running the following command:

$ istioctl create-remote-secret \
  --kubeconfig="${CLUSTER_A_KUBECONFIG}" \
  --name="${CLUSTER_A}" \
  --istioNamespace=istio-system | \
  oc apply --kubeconfig="${CLUSTER_B_KUBECONFIG}" -f -

Create an Istio remote secret on Cluster B by running the following command:

$ istioctl create-remote-secret \
  --kubeconfig="${CLUSTER_B_KUBECONFIG}" \
  --name="${CLUSTER_B}" \
  --istioNamespace=istio-system | \
  oc apply --kubeconfig="${CLUSTER_A_KUBECONFIG}" -f -

Verify that the remote cluster is synced on Cluster A by running the following command:
```
$ istioctl remote-clusters --kubeconfig="${CLUSTER_A_KUBECONFIG}"
```
The output must show ${CLUSTER_B} with status synced.
Example output
```
NAME         STATUS   SECRET
cluster-b    synced   istio-remote-secret-cluster-b
```
Verify that the remote cluster is synced on Cluster B by running the following command:
```
$ istioctl remote-clusters --kubeconfig="${CLUSTER_B_KUBECONFIG}"
```
The output must show ${CLUSTER_A} with status synced.
Example output
```
NAME         STATUS   SECRET
cluster-a    synced   istio-remote-secret-cluster-a
```

Verifying cross-cluster service communication¶

Verify cross-cluster service communication between Red Hat OpenShift Service Mesh clusters using sample workloads. This confirms SPIRE-issued identities and federated mesh routing enable end-to-end cross-cluster communication.

Prerequisites

You have deployed east-west gateways and created the cross-network Gateway CR on both clusters.
You have exchanged remote secrets between clusters.

Procedure

Set the sample namespace environment variable by running the following command:
```
$ export SAMPLE_NS=sample
```

Create the sample namespace on Cluster A by running the following command:

$ oc create namespace ${SAMPLE_NS} --kubeconfig="${CLUSTER_A_KUBECONFIG}" 2>/dev/null || true

Enable Istio injection for the sample namespace on Cluster A by running the following command:

$ oc label namespace ${SAMPLE_NS} istio-injection=enabled \
  --kubeconfig="${CLUSTER_A_KUBECONFIG}" --overwrite

Create the sample namespace on Cluster B by running the following command:

$ oc create namespace ${SAMPLE_NS} --kubeconfig="${CLUSTER_B_KUBECONFIG}" 2>/dev/null || true

Enable Istio injection for the sample namespace on Cluster B by running the following command:

$ oc label namespace ${SAMPLE_NS} istio-injection=enabled \
  --kubeconfig="${CLUSTER_B_KUBECONFIG}" --overwrite

Install the Istio HelloWorld Service in Cluster B by running the following command:

Create a YAML file that defines the HelloWorld Service in Cluster B:

apiVersion: v1
kind: Service
metadata:
  name: helloworld
  labels:
    app: helloworld
    service: helloworld
spec:
  ports:
  - port: 5000
    name: http
  selector:
    app: helloworld

Apply the YAML file in Cluster B by running the following command:

$ oc apply --kubeconfig="${CLUSTER_B_KUBECONFIG}" -n ${SAMPLE_NS} -f <filename>

Install the helloworld-v1 Deployment in Cluster B by running the following command:

Create a YAML file that defines the helloworld-v1 Deployment in Cluster B:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-v1
  labels:
    app: helloworld
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloworld
      version: v1
  template:
    metadata:
      labels:
        app: helloworld
        version: v1
    spec:
      containers:
      - name: helloworld
        image: registry.istio.io/release/examples-helloworld-v1:1.0
        resources:
          requests:
            cpu: "100m"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 5000

Apply the YAML file in Cluster B by running the following command:

$ oc apply --kubeconfig="${CLUSTER_B_KUBECONFIG}" -n ${SAMPLE_NS} -f <filename>

Install the Istio HelloWorld Service in Cluster A by running the following command:

Create a YAML file that defines the HelloWorld Service in Cluster A:

apiVersion: v1
kind: Service
metadata:
  name: helloworld
  labels:
    app: helloworld
    service: helloworld
spec:
  ports:
  - port: 5000
    name: http
  selector:
    app: helloworld

Apply the YAML file in Cluster A by running the following command:

$ oc apply --kubeconfig="${CLUSTER_A_KUBECONFIG}" -n ${SAMPLE_NS} -f <filename>

Install the sleep client in Cluster A by running the following command:

Create a YAML file that defines the sleep ServiceAccount, Service, and Deployment in Cluster A:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image: docker.io/curlimages/curl:8.16.0
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true

Apply the YAML file in Cluster A by running the following command:

$ oc apply --kubeconfig="${CLUSTER_A_KUBECONFIG}" -n ${SAMPLE_NS} -f <filename>

Add the SPIRE injection template to the sleep application in Cluster A by running the following command:

$ oc patch deploy sleep \
    -n ${SAMPLE_NS} \
    --type='merge' \
    --kubeconfig="${CLUSTER_A_KUBECONFIG}" \
    -p '{"spec": {"template": {"metadata": {"annotations": {"inject.istio.io/templates": "sidecar,spire"}}}}}'

Add the SPIRE injection template to the HelloWorld application in Cluster B by running the following command:

$ oc patch deploy helloworld-v1 \
   -n ${SAMPLE_NS} \
   --type='merge' \
   --kubeconfig="${CLUSTER_B_KUBECONFIG}" \
    -p '{"spec": {"template": {"metadata": {"annotations": {"inject.istio.io/templates": "sidecar,spire"}}}}}'

Wait for the sleep deployment to become available on Cluster A by running the following command:

$ oc rollout status deploy/sleep --kubeconfig "${CLUSTER_A_KUBECONFIG}" -n ${SAMPLE_NS} --timeout=300s

Wait for the helloworld-v1 deployment to become available on Cluster B by running the following command:

$ oc rollout status deploy/helloworld-v1 --kubeconfig "${CLUSTER_B_KUBECONFIG}" -n ${SAMPLE_NS} --timeout=300s

Verify that the sleep pod uses a SPIRE-issued identity by running the following command:

$ oc exec deploy/sleep -n ${SAMPLE_NS} --kubeconfig="${CLUSTER_A_KUBECONFIG}" -c istio-proxy -- \
  curl -s localhost:15000/certs | jq -r '.certificates[0].cert_chain[0].subject_alt_names[0].uri'

Example output

spiffe://${CLUSTER_A_TRUST_DOMAIN}/ns/sample/sa/sleep

Verify that the sleep pod on Cluster A can reach the helloworld.sample service by running the following command:

$ oc exec deploy/sleep \
  -n ${SAMPLE_NS} \
  --kubeconfig="${CLUSTER_A_KUBECONFIG}" \
  -- curl -sS helloworld.sample:5000/hello

Example output

Hello version: v1, instance: helloworld-v1-5859666d7-pcb8v

Additional resources

Multi-cluster configuration overview