Skip to main content

Managing Terraform State & Remote Backends

This guide explains how Terraform manages application state on the Okustera / OpenCloud platform, comparing local development state with production-grade remote state on Ceph RADOS Gateway (RGW) S3 object storage.

1. Overview & Architecture​

When deploying applications with Terraform, the state file acts as the single source of truth mapping your declarative HCL configurations to actual physical and virtual resources in the Okustera cloud (Keystone tenants, APISIX routes, Ceph S3 buckets, PostgreSQL clusters, and serverless functions).

Depending on your environment, state is managed in two ways:

  1. Local State (Default for Development & Quickstarts): Stored directly on disk in the directory where terraform apply is executed (terraform.tfstate).
  2. Remote S3 State (Mandatory for Production & CI/CD GitOps): Stored centrally, encrypted, and locked in a dedicated tenant S3 bucket on Ceph RADOS Gateway (RGW).

2. Local State Workflow (Development & Testing)​

How Local State Works​

When no backend block is specified in your Terraform configuration (such as in local prototyping or test sandboxes):

  • Terraform writes the state to a local file named terraform.tfstate in the current working directory.
  • When resources are modified, Terraform creates a snapshot of the prior state in terraform.tfstate.backup.

Critical Security Requirement: .gitignore​

Because Terraform state contains unmasked resource attributes—including database credentials, API keys, S3 secrets, and Keystone tokens—you must never commit .tfstate files to version control.

Always include the following patterns in your project's .gitignore:

# Terraform local state files
*.tfstate
*.tfstate.*
*.tfstate.backup
crash.log
crash.*.log

# Terraform local directory and lock file
.terraform/
.terraform.lock.hcl

# Variable definitions containing sensitive secrets
terraform.tfvars
terraform.tfvars.json
*.auto.tfvars
*.auto.tfvars.json

[!CAUTION] Committing terraform.tfstate to Git exposes sensitive tenant credentials and violates ISO/IEC 27001 and Okustera Zero-Trust security invariants.

3. Production Remote State: Ceph RGW S3 Backend​

In production environments, collaborative teams, and CI/CD pipelines, state must be stored in a shared, highly available remote backend. Because Okustera provides an AWS S3-compatible Ceph RADOS Gateway (RGW), you use Terraform's standard s3 backend.

Create a file named backend.tf in your Terraform root module:

terraform {
  backend "s3" {
    bucket                      = "okustera-tfstate-tenant-prod"
    key                         = "apps/production-app/terraform.tfstate"
    endpoint                    = "http://s3.omc-poc.local:7480" # Or https://s3.okustera.com
    region                      = "us-east-1"                   # Dummy region required by AWS S3 client
    skip_credentials_validation = true                          # Ceph RGW bypasses AWS STS
    skip_region_validation      = true                          # Ceph RGW manages internal failure domains
    skip_metadata_api_check     = true                          # Bypasses AWS EC2 metadata lookup
    force_path_style            = true                          # Mandatory: Ceph RGW uses path-style routing
  }
}

Parameter Reference​

ParameterRequiredDescription
bucketYesDedicated tenant S3 bucket created for Terraform state storage.
keyYesPath within the bucket where the state file is stored (e.g. apps/<app-name>/terraform.tfstate).
endpointYesURL of the Okustera Ceph RADOS Gateway (e.g. http://s3.omc-poc.local:7480 or https://s3.okustera.com).
regionYesSet to "us-east-1". The AWS SDK client requires a region string even when connecting to Ceph.
force_path_styleYesMust be set to true. Instructs the client to use endpoint/bucket/key rather than subdomain addressing.
skip_credentials_validationYesMust be true to disable AWS STS credentials validation.
skip_region_validationYesMust be true to avoid querying AWS region endpoints.
skip_metadata_api_checkYesMust be true to prevent timeouts attempting to reach AWS IMDS (169.254.169.254).

4. Provisioning the State Bucket​

Tenants can provision their state storage bucket and S3 credentials declaratively using the Okustera Terraform Provider or through the Okustera Web Portal.

Declarative Setup with Okustera Provider​

# 1. Provision a dedicated S3 bucket for Terraform state
resource "okustera_s3_bucket" "tf_state_bucket" {
  name        = "okustera-tfstate-tenant-prod"
  versioning  = true
  description = "Dedicated storage bucket for Terraform state files with object versioning"
}

# 2. Provision scoped S3 credentials for automation
resource "okustera_s3_credentials" "tf_state_creds" {
  tenant_id   = "tenant-prod"
  description = "Scoped S3 access keys for CI/CD Terraform state backend"
}

output "tfstate_s3_access_key" {
  value       = okustera_s3_credentials.tf_state_creds.access_key
  description = "S3 Access Key for Terraform backend"
}

Passing S3 Credentials Securely​

Do not hardcode S3 credentials in your backend.tf. Provide them to the Terraform CLI via environment variables:

export AWS_ACCESS_KEY_ID="<YOUR_S3_ACCESS_KEY>"
export AWS_SECRET_ACCESS_KEY="<YOUR_S3_SECRET_KEY>"

terraform init

5. State Locking & Concurrency Protection​

In team workflows and automated pipelines, two operators or CI runs executing terraform apply simultaneously can corrupt the state file.

Locking Mechanisms Supported on Okustera​

  1. Ceph S3 Object Lock: Enabled automatically when versioning = true is configured on the state bucket.
  2. DynamoDB-Compatible Locking Table: In high-velocity GitOps pipelines, you can specify an external lock table or a distributed key-value store (e.g. Valkey / Redis lock coordinator):
terraform {
  backend "s3" {
    bucket         = "okustera-tfstate-tenant-prod"
    key            = "apps/production-app/terraform.tfstate"
    endpoint       = "http://s3.omc-poc.local:7480"
    region         = "us-east-1"
    dynamodb_table = "okustera-tfstate-locks"
    # ... other parameters ...
  }
}

6. Migrating from Local State to Remote Ceph S3​

If you started your project with local state and want to transition to Ceph S3 without losing existing resource tracking:

Step 1: Add the backend "s3" block to backend.tf​

Add the configuration as shown in Section 3.

Step 2: Re-initialize with -migrate-state​

Run the initialization command. Terraform detects the change from local disk to remote S3:

terraform init -migrate-state

Terraform prompts for confirmation:

Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to confirm and "no" to cancel.

  Enter a value: yes

Step 3: Verify Remote State​

List the tracked resources from the newly configured remote backend:

terraform state list

Once migration is complete, you can safely remove the local terraform.tfstate and terraform.tfstate.backup files.

7. Disaster Recovery & Enterprise SLA Guarantees​

Under the Okustera Disaster Recovery Architecture (documented in platform Tier 1 standards):

  • Classification: Tier 1 Critical Infrastructure.
  • Target RPO (Recovery Point Objective): Real-time. Every state modification committed by terraform apply is written synchronously to Ceph object storage.
  • Target RTO (Recovery Time Objective): < 5 minutes.
  • Encryption at Rest: State objects stored in Ceph RGW are protected using AES-256 server-side encryption backed by OpenStack Barbican key management.
  • Multi-Site Replication: For multi-zone deployments, Ceph RGW multi-site replication automatically mirrors the okustera-tfstate bucket to secondary disaster recovery clusters asynchronously.