In-Cluster Workload Identity with Okustera & Terraform

Pod IAM is Optional

Pod IAM Workload Identity is an optional security feature. By default, the Okustera cluster runs in Standard Mode, routing API traffic directly to the backend with 0ms subrequest latency and minimal resource footprint.

• Standard Mode (Default): In-cluster GitOps runners and pods authenticate using scoped OpenStack Keystone Application Credentials or API tokens mounted via Kubernetes Secrets.
• Enforced Mode (Optional): In-cluster pods project cryptographic ServiceAccount tokens verified against the cluster's JWKS and evaluated by omc-iam-authorizer.

This guide explains both approaches: authenticating in Standard Mode (recommended for most workloads) and authenticating in Enforced Mode (for strict zero-trust environments).

---

Architecture: Standard Mode vs. Enforced Mode

Dimension	Standard Mode (Default)	Enforced Mode (Pod IAM)
Ingress Routing	Direct APISIX proxying to backend (0ms overhead)	APISIX forward-auth plugin intercepts all /api/*
Cluster Services	Authorizer deployment scaled to 0 replicas	2 authorizer replicas running in omc-iam-system
Authentication Primitive	Keystone Application Credentials / API Token	Projected ServiceAccount JWT (okustera-platform audience)
Policy Definition	Standard Keystone RBAC roles (admin, member, reader)	Declarative OMCIAMRole path & method policies

---

Approach A: Standard Mode Authentication (Recommended)

In Standard Mode, you don't need projected tokens or authorizer pods. You provision an unprivileged, scoped Keystone Application Credential via Terraform and mount it into your runner pod:

1. Provision Application Credential in Terraform

resource "openstack_identity_application_credential_v3" "ci_runner" {
  name        = "gitlab-ci-runner-credential"
  description = "Scoped credential for CI/CD Terraform runner"
  roles       = ["member"]
}

resource "kubernetes_secret" "ci_runner_auth" {
  metadata {
    name      = "okustera-ci-credentials"
    namespace = "gitlab-runners"
  }
  data = {
    OS_AUTH_TYPE                   = "v3applicationcredential"
    OS_AUTH_URL                    = "https://keystone.okustera.com/v3"
    OS_APPLICATION_CREDENTIAL_ID     = openstack_identity_application_credential_v3.ci_runner.id
    OS_APPLICATION_CREDENTIAL_SECRET = openstack_identity_application_credential_v3.ci_runner.secret
  }
}

2. CI/CD Runner Pod Configuration (Standard Mode)

apiVersion: batch/v1
kind: Job
metadata:
  name: terraform-apply-production
  namespace: gitlab-runners
spec:
  template:
    spec:
      serviceAccountName: default
      restartPolicy: Never
      containers:
        - name: terraform
          image: hashicorp/terraform:1.6.0
          command: ["terraform", "apply", "-auto-approve"]
          envFrom:
            - secretRef:
                name: okustera-ci-credentials

---

Approach B: Enforced Mode Authentication (Zero-Trust Workload Identity)

If your platform administrator has enabled Enforced Mode, follow these steps to configure zero-trust pod token verification.

The Problem with Static Credentials in GitOps

Traditionally, running Terraform in CI/CD requires injecting a static secret:

• OKUSTERA_API_TOKEN stored in CI/CD pipeline environment variables.
• Or static OpenStack admin passwords stored in Kubernetes Secrets.

Risks:

• Static secrets can be leaked through CI job logs or compromised build artifacts.
• Secrets require manual rotation policies.
• A leaked token grants permanent, unconstrained access across the entire tenant project.

---

The Workload Identity Solution

With Pod-Level IAM Role Isolation, Kubernetes itself acts as the identity provider:

1. Every Kubernetes Pod running under a ServiceAccount can project a cryptographically signed JSON Web Token (JWT) issued by the cluster API server.
2. The JWT includes the pod's identity (sub: system:serviceaccount:<namespace>:<sa>), audience (okustera-platform), and expiry (1 hour).
3. The Terraform Provider client (okustera-client-go) automatically reads this token file (/var/run/secrets/okustera/token).
4. When Terraform calls Apache APISIX (http://apisix-gateway.apisix.svc/api/v1/*), APISIX validates the JWT signature against the cluster's JWKS endpoint (/openid/v1/jwks).
5. APISIX resolves the pod's assigned OkusteraIAMRole and verifies that the role allows the requested API operations (e.g., creating a database, updating a route).

---

Step-by-Step Setup

Step 1: Declare the IAM Role for Terraform in Okustera

Create an IAM Role defining what this Terraform runner is allowed to manage:

resource "okustera_iam_role" "terraform_deployer" {
  name        = "terraform-deployer"
  description = "Permissions for GitOps Terraform runner to deploy app stack"

  endpoint_policies = [
    {
      effect  = "Allow"
      paths   = [
        "/api/v1/dbaas/*",
        "/api/v1/storage/*",
        "/api/v1/functions/*",
        "/api/v1/apisix/*"
      ]
      methods = ["GET", "POST", "PUT", "DELETE"]
    },
    {
      effect  = "Deny"
      paths   = [
        "/api/v1/billing/payment-methods*",
        "/api/v1/iam/roles/admin*"
      ]
      methods = ["*"]
    }
  ]
}

Step 2: Bind the ServiceAccount to the IAM Role

Bind the Kubernetes ServiceAccount used by your CI/CD runner pod:

resource "okustera_pod_identity_binding" "ci_runner_binding" {
  namespace       = "gitlab-runners"
  service_account = "terraform-worker-sa"
  role_name       = okustera_iam_role.terraform_deployer.name
}

Step 3: Configure the Runner Pod Manifest

In your CI runner or Kubernetes Job manifest, project the ServiceAccount token:

apiVersion: batch/v1
kind: Job
metadata:
  name: terraform-apply-production
  namespace: gitlab-runners
spec:
  template:
    metadata:
      annotations:
        okustera.io/iam-role: "terraform-deployer"
    spec:
      serviceAccountName: terraform-worker-sa
      restartPolicy: Never
      containers:
        - name: terraform
          image: hashicorp/terraform:1.6.0
          command: ["terraform", "apply", "-auto-approve"]
          volumeMounts:
            - name: okustera-token
              mountPath: /var/run/secrets/okustera
              readOnly: true
          env:
            - name: OKUSTERA_ENDPOINT
              value: "http://apisix-gateway.apisix.svc/api/v1"
            - name: OKUSTERA_TOKEN_FILE
              value: "/var/run/secrets/okustera/token"
      volumes:
        - name: okustera-token
          projected:
            sources:
              - serviceAccountToken:
                  path: token
                  audience: "okustera-platform"
                  expirationSeconds: 3600

Step 4: Configure the Provider Block

In your application's main.tf, simply enable workload_identity:

provider "okustera" {
  endpoint = "http://apisix-gateway.apisix.svc/api/v1"

  workload_identity = {
    enabled = true
  }
}

The provider will automatically read the token from /var/run/secrets/okustera/token (or OKUSTERA_TOKEN_FILE / OMC_TOKEN_FILE), reload it dynamically as it rotates, and present it to APISIX.

---

Alternative: Declaring IAM Roles via kubernetes_manifest

If you are using the standard HashiCorp Kubernetes provider or managing manifests directly, you can declare the exact same role and binding using native Custom Resource Definitions:

resource "kubernetes_manifest" "terraform_deployer_role" {
  manifest = {
    apiVersion = "iam.opencloud.local/v1alpha1"
    kind       = "OMCIAMRole"
    metadata = {
      name      = "terraform-deployer"
      namespace = "gitlab-runners"
    }
    spec = {
      description = "Permissions for GitOps Terraform runner to deploy app stack"
      endpointPolicies = [
        {
          path    = "/api/v1/dbaas/*"
          methods = ["GET", "POST", "PUT", "DELETE"]
          effect  = "Allow"
        },
        {
          path    = "/api/v1/storage/*"
          methods = ["GET", "POST", "PUT", "DELETE"]
          effect  = "Allow"
        },
        {
          path    = "/api/v1/billing/*"
          methods = ["*"]
          effect  = "Deny"
        }
      ]
    }
  }
}

resource "kubernetes_manifest" "ci_runner_binding" {
  manifest = {
    apiVersion = "iam.opencloud.local/v1alpha1"
    kind       = "OMCPodIdentityBinding"
    metadata = {
      name      = "ci-runner-binding"
      namespace = "gitlab-runners"
    }
    spec = {
      serviceAccountName = "terraform-worker-sa"
      roleRef = {
        name = "terraform-deployer"
      }
    }
  }
}

---

Platform Administrator: Toggling Pod IAM via Terraform

Platform engineers managing Okustera cluster infrastructure can switch between Standard Mode and Enforced Mode declaratively via Terraform:

variable "enable_pod_iam" {
  type        = bool
  default     = false
  description = "Set to true to activate APISIX forward-auth and authorizer pods"
}

# 1. Apply APISIX Route manifest based on mode
resource "kubernetes_manifest" "omc_portal_route" {
  manifest = yamldecode(file(
    var.enable_pod_iam 
      ? "${path.module}/infrastructure/k8s/iam/apisix-forward-auth-route.yaml"
      : "${path.module}/infrastructure/k8s/omc-portal/apisix-route.yaml"
  ))
}

# 2. Scale omc-iam-authorizer deployment
resource "kubernetes_manifest" "authorizer_scale" {
  manifest = {
    apiVersion = "apps/v1"
    kind       = "Deployment"
    metadata = {
      name      = "omc-iam-authorizer"
      namespace = "omc-iam-system"
    }
    spec = {
      replicas = var.enable_pod_iam ? 2 : 0
    }
  }
}