CA Rotation

AlmaForge issues every cluster credential through its built-in certificate authorities. Staged CA rotation keeps agents, users, and active sessions connected while replacing the signing key. Both the old and new chains remain trusted during the transition. Verify that services and external trust stores have adopted the new CA before completing rotation. Components that miss the transition can lose access when the old CA is removed.

When to rotate

Rotate on a regular schedule or in response to an event:

• Annual maintenance: rotate at least once per year as part of routine cluster hygiene.
• Post-incident: rotate immediately after suspected or confirmed credential exposure, key-store compromise, or unexpected privileged access.
• Hardware change: after adding or removing an HSM or KMS from the Auth Service, rotate all CAs so every active Auth replica holds the new key material.

Certificate authorities at a glance

AlmaForge maintains the following built-in CAs:

Type	Flag value	Purpose	Blast radius if rotation stalls
Host CA	host	Signs host identity certs for servers, proxies, and Auth replicas.	Services can fail to prove identity to each other.
User CA	user	Signs user SSH and TLS certs for clients and agents.	New logins and refreshed user certs can fail.
Database CA	db	Signs short-lived client certs for database access.	New database sessions can fail certificate validation.
OpenSSH CA	openssh	Signs certs for agentless OpenSSH-native nodes.	Agentless OpenSSH-native nodes can reject new sessions.
JWT signer	jwt	Signs JWTs for application-access and workload-identity tokens.	Tokens signed by the JWT signer can fail validation.
OIDC IdP CA	oidc_idp	Signs tokens when the built-in OIDC provider is active.	Downstream OIDC clients can reject newly issued tokens.

Use the same phase sequence for every type. The auth rotate command requires --type, so run one command per CA family.

Production rotation scope

Rotate one CA family at a time in production. Complete Host CA rotation through standby before starting User CA rotation, then rotate Database, OpenSSH, JWT, or OIDC separately unless the maintenance plan explicitly accepts the combined blast radius.

Prerequisites

Before starting:

1. Maintenance window scheduled. Operator on-call and available for the full grace period (default 30 hours).
2. State backend backed up. Verify the most recent snapshot is current before initiating. See Backups.
3. Break-glass path tested. Confirm you can reach Auth via --auth-server from the Auth host. See Break-Glass Access.
4. All Auth replicas reachable. Rotation propagates to every Auth replica. A replica that is offline during update_servers can miss the new chain and fail cert validation after rotation completes.
5. Login profile active. You need a valid alma session with an admin role to run auth rotate.
6. External signer access ready. If CAs are backed by an HSM or KMS, coordinate with the signer operator before phase 1. New key material is created during init.

Pre-rotation baseline

Capture the starting point before the first phase change. Use the same terminal, ticket, or runbook page for all snapshots so the post-rotation comparison is explicit.

Terminal

alma statusalma get ca > ca-rotation-ca-before.txtalma get ca -o yaml > ca-rotation-ca-before.yamlalma get auth -o yaml > ca-rotation-auth-before.yamlalma get proxy -o yaml > ca-rotation-proxy-before.yamlalma get node -o yaml > ca-rotation-nodes-before.yamlalma get db -o yaml > ca-rotation-db-before.yamlalma get app -o yaml > ca-rotation-apps-before.yamlalma get kube -o yaml > ca-rotation-kube-before.yaml

Save the Cluster: value from alma status. The examples below use <cluster-name> for that value. Save the CA fingerprints from ca-rotation-ca-before.txt, the fleet counts from the get snapshots, and the active session count from your normal monitoring source if you track live sessions outside the CLI.

Rotation phases

Rotation is a state machine. The supported phases in sequence are:

text

standby --> init --> update_clients --> update_servers --> standby
                         |                    |
                         +--------rollback----+

The cluster starts and ends in standby. The rollback phase is available from init, update_clients, or update_servers if you need to abort.

The CA resource is the source of truth for the cluster phase. Some service inventory resources also expose local spec.rotation from their heartbeat. There is no separate per-agent CA-rotation ready label in this release, so use the CA phase, service inventory, and fresh connection tests together as the gate.

Manual rotation procedure

Manual mode lets you advance each phase explicitly. This is the recommended approach for production clusters because you can verify each phase before proceeding. The commands below rotate the Host CA as the worked example. Finish that CA through standby before repeating the full sequence for user, db, openssh, jwt, or oidc_idp.

Phase 1: init

What happens: Auth generates a new CA key pair. Both the old and new CAs are trusted for verification. All new certificates are still signed by the old CA. Offline components can discover the new CA before they are asked to trust it.

Action:

Terminal

alma auth rotate --type=host --manual --phase=init

Verify: Check that the CA resources entered init.

Terminal

alma get ca/host/<cluster-name> -o yaml

The spec.rotation.phase field should read init for the CA type you are rotating.

Readiness gate: Do not advance based on elapsed time alone. Re-run the relevant inventory snapshots and compare them with the baseline. For each service class in scope, the resource count should match the baseline and the resource should not be stale or missing.

Terminal

alma get auth -o yaml > ca-rotation-auth-init.yamlalma get proxy -o yaml > ca-rotation-proxy-init.yamlalma get node -o yaml > ca-rotation-nodes-init.yamlalma get db -o yaml > ca-rotation-db-init.yamlalma get app -o yaml > ca-rotation-apps-init.yamlalma get kube -o yaml > ca-rotation-kube-init.yaml

Rollback if: any Auth replica is unreachable, a required service class is missing from inventory, or external CA-bundle distribution cannot finish before update_clients.

Phase 1.5: distribute exported CA bundles

After init and before update_clients, update any external system that pins exported CA bundles. Common examples are monitoring collectors, bastion hosts, SIEM parsers, and embedded clients.

Terminal

alma auth export --type=tls-host > cluster-host-ca.pemalma auth export --type=tls-user > cluster-user-ca.pemalma auth export --type=db > cluster-db-ca.pemalma auth export --type=openssh > cluster-openssh-ca.pub

Export only the types you use externally. Store the new files wherever the external system loads its trusted CA bundle.

Phase 2: update_clients

What happens: The selected CA starts signing new credentials with its new key. During Host CA rotation, services renew their client identities while server identities still use the old key. User CA rotation requires users to obtain new credentials through sign-in. Both CA chains remain trusted, and rollback is still available.

Action:

Terminal

alma auth rotate --type=host --manual --phase=update_clients

Verify:

Terminal

alma loginalma statusalma get ca/host/<cluster-name> -o yaml

The CA spec.rotation.phase field should read update_clients. After re-login, the session info block should still show the expected Cluster: value and a non-expired profile.

Readiness gate: Query the service inventory again. For each resource type in scope, spec.rotation.phase should read update_clients where the resource exposes rotation state. The resource count should still match the baseline.

Terminal

alma get auth -o yaml > ca-rotation-auth-update-clients.yamlalma get proxy -o yaml > ca-rotation-proxy-update-clients.yamlalma get node -o yaml > ca-rotation-nodes-update-clients.yamlalma get db -o yaml > ca-rotation-db-update-clients.yamlalma get app -o yaml > ca-rotation-apps-update-clients.yamlalma get kube -o yaml > ca-rotation-kube-update-clients.yaml

Rollback if: new logins fail, any service logs a rotation-state out-of-sync error, or the inventory loses a required Auth, Proxy, agent, database, app, or Kubernetes service entry.

Phase 3: update_servers

What happens: Auth, Proxy, and agent servers reload and serve TLS under the new CA. Old-CA certs are still trusted for verification so that components that have not yet reconnected can finish gracefully. You can still roll back from this phase.

Action:

Terminal

alma auth rotate --type=host --manual --phase=update_servers

Verify: Attempt a new login and a test connection to at least one server to confirm the new chain is being served and accepted.

Terminal

alma loginalma get nodealma get ca/host/<cluster-name> -o yaml

The CA spec.rotation.phase field should read update_servers.

Readiness gate: Open at least one fresh connection for every access surface in scope: SSH, database, application, and Kubernetes where those services are deployed. Re-run the inventory snapshots and confirm counts still match the baseline.

Rollback if: any fresh connection path fails, an Auth or Proxy replica is missing from inventory, or agents report rotation-state out-of-sync errors.

Phase 4: standby (complete)

What happens: The old CA is removed from the trust store. All components reload one final time and no longer accept old-CA certs. Rotation is complete.

Action:

Terminal

alma auth rotate --type=host --manual --phase=standby

Verify:

Terminal

alma get ca/host/<cluster-name> -o yamlalma get ca > ca-rotation-ca-after.txt

The CA spec.rotation.phase field should read standby. Compare ca-rotation-ca-after.txt with the pre-rotation CA fingerprint snapshot and confirm the expected CA types changed.

Automatic rotation

Auto mode rotates through all phases on a schedule derived from the --grace-period flag (default: 30 hours). It is convenient for low-traffic or single-node test clusters. For production, prefer manual mode so you can verify each phase before proceeding.

Terminal

# Start auto rotation with the default 30-hour grace periodalma auth rotate --type=host

If an automatic phase transition fails, Auth switches back to manual mode. Check alma status and advance manually from whatever phase the rotation stopped at.

JWT-only rotation

Use the same phase sequence with --type=jwt when only the JWT signer needs to move.

Terminal

alma auth rotate --type=jwt --manual --phase=initalma auth rotate --type=jwt --manual --phase=update_clientsalma auth rotate --type=jwt --manual --phase=update_serversalma auth rotate --type=jwt --manual --phase=standby

Keep JWT-only rotations separate from Host and User rotations so application-token validation problems do not coincide with server or user credential changes. In automatic mode, choose a --grace-period that matches the token lifetime and the maintenance window.

Rollback

Roll back if you discover a problem during any phase before standby. Rollback is available from init, update_clients, or update_servers.

What happens: The new CA key is demoted and flagged as trusted-only (not signing). Components reload and return to old-CA certs. When you then advance from rollback to standby, the new CA is discarded and the cluster returns to the original state.

Abort from any in-progress phase:

Terminal

alma auth rotate --type=host --manual --phase=rollback

Complete the rollback (return to standby):

Terminal

alma auth rotate --type=host --manual --phase=standby

What is recoverable: All credentials issued by the old CA remain valid. The cluster returns to its pre-rotation state. Components that received new-CA certs during update_clients will re-receive old-CA certs after rollback.

What is not recoverable: If you advance past standby (rotation completes) and then discover a problem, rollback is no longer available. Start a new rotation to replace the CA again.

After rotation

1. Verify client connectivity. Log in from an operator workstation, list nodes, and open a test session to a managed server.
2. Compare the baseline. The same Auth, Proxy, agent, database, app, and Kubernetes services should appear in the post-rotation snapshots unless the maintenance window intentionally changed them.
3. Preserve the evidence. Keep the auth rotate command output, CA snapshots, inventory snapshots, and Auth logs with the maintenance record. Auth logs record phase changes. Audit logs record newly issued certificates as CertificateCreate (TC000I).
4. Schedule the next rotation. Set a reminder for one year out, or sooner if the cluster handles high-value workloads.

Frequency guidance

Rotate at least once per year. Clusters that handle privileged access to production infrastructure or regulated data benefit from a six-month cadence. After any incident involving suspected credential exposure, rotate immediately regardless of when the last rotation occurred.

Rotate Host and User CAs sequentially in production unless the incident response plan requires a faster combined rotation. Rotate Database, OpenSSH, JWT, and OIDC separately when the trigger is limited to one credential family.

Troubleshooting

Rotation stuck in update_clients

Agents that were offline when update_clients was issued have not received new-CA certs. Bring offline agents back online and wait for them to reconnect. Once all expected agents reappear in get node and fresh connections succeed, advance to update_servers.

If an agent cannot reconnect because it only trusts the old CA, check whether it re-joined the cluster during init. If the agent missed init, roll back to standby and restart the rotation from init.

Agent fails cert verification after update_servers

Both CA chains remain trusted during update_servers, so an old certificate alone does not explain a validation failure. Check the agent's trust bundle, certificate validity, and reported rotation phase. Do not advance to standby while a required connection fails. If the failure cannot be resolved within the maintenance window, move the CA to rollback, complete rollback to standby, and retry after the agent reconnects cleanly.

Auth replica does not acknowledge the new phase

Check that the replica is running and reachable. A replica that is down during update_servers will not have the new CA in its active key set. Bring the replica back online. It will sync from the state backend and reload. Do not advance to standby until all Auth replicas are healthy.

auth rotate returns rotation is already in progress

A rotation is already running. Check the current phase:

Terminal

alma get ca/host/<cluster-name> -o yaml

Advance or roll back from the current phase rather than starting a new rotation.

Rollback does not return to standby

Query the CA resource and confirm the current phase before retrying:

Terminal

alma get ca/host/<cluster-name> -o yaml

If the CA is still in rollback, re-run the same standby command for the same --type. If Auth replicas disagree or the command cannot reach a quorum, stop advancing phases. Use the tested break-glass path, capture the current CA and inventory snapshots, and restore the state backend from the pre-rotation backup if Auth cannot serve the cluster consistently.

Further reading

• Break-Glass Access: emergency access if the normal login path is unavailable during or after rotation.
• Backups: verify a current backup before starting any rotation.
• High Availability Guide: how certificate lifecycle fits into a multi-replica cluster.
• Cluster on Linux: step-by-step Auth and Proxy setup where rotation applies.