High-Availability Architecture

Run Auth and Proxy replicas across independent failure domains, with a shared state backend, durable audit and recording storage, and a highly available entry point. This page explains the topology and what users experience when a component fails.

An HA AlmaForge deployment runs several copies of the cluster services across hosts to keep serving through a replica failure. The cluster's durable data (configuration, audit events, and session recordings) lives in external storage. Configure storage and load balancing for redundancy too, so neither becomes a single point of failure.

For step-by-step installation, see Cluster on Linux or Cluster on Kubernetes. Both ship the colocated topology described below.

Services

Every AlmaForge daemon is the same almad binary. The daemon's environment decides which services it runs, and a single process can run any non-empty subset of them:

Service	Enable with	What it does
Auth	`ALMA_AUTH_SERVICE=true`	Stores cluster configuration, decides who can access what, and signs the short-lived certificates clients use to log in. Talks to the state backend (etcd, PostgreSQL, DynamoDB, or Firestore), where the cluster stores all of its durable configuration.
Proxy	`ALMA_PROXY_SERVICE=true`	The public entry point on TCP/443. Accepts client connections, asks Auth to authorize them, and forwards each session to the right agent.
SSH	`ALMA_SSH_SERVICE=true`	Registers SSH hosts with the cluster and keeps a long-lived outbound connection to the Proxy (a reverse tunnel) so the Proxy can route inbound SSH sessions back through it.
Kubernetes	`ALMA_KUBERNETES_SERVICE=true`	Same pattern for Kubernetes clusters.
Application	`ALMA_APP_SERVICE=true`	Same pattern for HTTP applications.
Database	`ALMA_DATABASE_SERVICE=true`	Same pattern for databases.

The first two (Auth and Proxy) are control-plane services: together they form the cluster that everything else talks to. The last four are agent services that expose individual resources to the cluster.

Each variable is a tri-state boolean (true, false, or unset), and unset leaves the service to auto-detection. On the command line the same choice is --auth-service and --no-auth-service, and so on. See Environment Configuration for the complete ALMA_* catalog and Infrastructure as Code for the broader configuration model: environment, drop-in manifests, and API-managed resources.

How a request flows

To make the parts concrete, take an example: an engineer runs kubectl get pods against a cluster registered in AlmaForge.

The user holds a short-lived X.509 certificate from a prior alma login. Its lifetime is bounded by the user's roles.
DNS resolves the public hostname to an L4 load balancer (a TCP-level forwarder that does not decrypt TLS). The LB picks one of the healthy Proxy replicas and forwards the connection.
kubectl negotiates TLS through that connection with the AlmaForge Proxy.
The Proxy terminates TLS, sees the user certificate, and asks Auth to verify the certificate and resolve the user's roles.
The Proxy looks up which agent owns the target Kubernetes cluster and finds the reverse tunnel that agent already holds open to the Proxy.
The Proxy forwards the request through the tunnel. The agent calls the Kubernetes API as the user the role allows, hits the cluster's API server, and streams the response back.

Audit events are emitted throughout this path (login, session start, request completion) and written to the external audit store.

Every component in this flow has multiple replicas, so the path does not depend on which specific replica handles each step. With that picture in mind, the rest of this page covers how to compose those components, what state they share, and how each part fails.

Composing a deployment

Operators have three points of flexibility when shaping a AlmaForge deployment. Every shape in this document is a particular choice along these three axes.

Services per process. One almad process runs any non-empty subset of the six services above. A cluster host typically runs Auth and Proxy together. An SSH agent host typically runs only the SSH service. All six in one process is fine for evaluation.
Replicas per role. Run as many processes with the same role set as the workload needs. A group of replicas running the same service combination is called a fleet. Three colocated Auth+Proxy replicas. Two SSH agents per pool. Five Proxy replicas behind the LB when the Auth fleet runs separately.
Network segmentation. Put Auth on a more restricted segment (a separate VLAN, subnet, or firewall zone) than Proxy. Put agent hosts in the same network as the resources they expose, not next to the control plane.

The connectivity rules that constrain these choices:

Auth needs the state backend, the audit store (external database for security events), and the recording store (external object storage for session recordings).
Proxy needs to reach Auth on the multiplex listener (TCP/443). The connection uses mutual TLS: each side presents a certificate signed by the cluster CA, the cluster's own internal certificate authority that issues every host and user credential. A Proxy in the same process as Auth reaches it without touching the network at all. A Proxy on its own host makes a private network call.
Agents (SSH, Kubernetes, Application, Database) dial out to Proxy on TCP/443. Reverse tunnels need no inbound connection from the Proxy to the agent.
Clients (operators, agents, automation) reach Proxy on TCP/443.
Every cluster host needs working NTP or an equivalent clock source. Cluster identities are time-bound certificates and clock skew breaks them.

Every other choice is up to the operator.

Cluster topology

Two shapes show up most often for the Auth + Proxy pair. They are points on the spectrum the rules above describe, not exclusive options.

Colocated (default)

Every cluster host runs one almad process with both Auth and Proxy enabled. The Proxy service reaches its own colocated Auth inside the process, without touching the network at all.

Colocated topology

Public path. Clients reach any host on TCP/443. An L4 load balancer (HAProxy, AWS NLB, a cloud LoadBalancer Service, or equivalent) picks a healthy host. DNS round-robin works as a fallback when an L4 LB is not available. See the Networking Guide for setup detail.
Internal path. None across the wire. Auth-to-Proxy traffic is loopback inside each process.
Recommended replica count. Three almad replicas. During a rolling upgrade one restarts at a time while the other two keep serving. Check their health before restarting the next replica.
Sizing trade-off. Auth and Proxy load grow together. You cannot scale one without the other. For small to medium clusters the operational simplicity of one fleet outweighs the lost flexibility.

Both shipped deployment guides build this topology by default. See Cluster on Linux and Cluster on Kubernetes.

Split Auth and Proxy

The Auth fleet and the Proxy fleet run on separate hosts. Auth hosts enable only the Auth service and bind the gRPC listener on a private interface. Proxy hosts enable only the Proxy service and dial the Auth gRPC listener over mTLS.

Split-fleet topology

Public path. Same as colocated. TCP/443 from clients hits the L4 LB, which forwards to the Proxy fleet.
Internal path. Proxy fleet reaches Auth fleet over mTLS gRPC on the Auth hosts' multiplex listener, TCP/443. The connection is mutually authenticated against the cluster CA. Only the Proxy fleet needs to reach it. Put the Auth fleet on a more restricted network segment if your compliance posture asks for it.
Recommended replica count. Two Auth replicas (one can restart while the other keeps serving) and three Proxy replicas (the LB keeps at least two healthy targets during rolling upgrades).
Sizing trade-off. Each fleet scales on its own signal. Proxy scales with concurrent sessions and bandwidth. Auth scales with API call rate and cert issuance. Add capacity to whichever fleet saturates first.

The split shape is not in the shipped example bundles. Build it from the colocated overlay by separating the Auth env from the Proxy env and pointing each Proxy at the Auth fleet's gRPC listener.

When to choose

You want	Pick
Proxy and Auth at very different scales	Split
Auth on a more restricted network segment than Proxy	Split
Different upgrade cadence per fleet	Split

Default to colocated. Move to split when one of the split-side reasons applies. The migration is mechanical: stand up the new fleet, drain hosts off the old fleet, decommission the old hosts. The cluster state, agents, and clients see no protocol change.

Recommended production floor

Component	Colocated	Split
`almad` replicas	3	-
Auth fleet	-	2
Proxy fleet	-	3
State backend	3 etcd nodes (or equivalent HA shape)	3 etcd nodes (or equivalent HA shape)
Audit and recordings	External, replicated storage	External, replicated storage
Agent pool	2 agents per pool	2 agents per pool
Public ingress	HA L4 load balancer	HA L4 load balancer

DynamoDB tops out at two Auth Service instances per cluster, a limit of the DynamoDB Streams reader. Use etcd, PostgreSQL, or Firestore for the recommended three-replica colocated topology. A two-replica colocated deployment has less maintenance headroom.

Place each cluster replica and each etcd node in a distinct failure domain (AZ or rack). The cluster tolerates one failure-domain loss. A AlmaForge cluster is single-region by design. For cross-region disaster recovery, run a second cluster with separate state, CA, and audit. There is no built-in shared-state replication.

Agent topology

Agent services (SSH, Kubernetes, Application, Database) run as their own almad processes on hosts close to the targets they expose. Both cluster topologies above use the same agent shape:

One or more agent processes per resource pool.
Each process runs only the agent services it needs. An SSH agent process typically does not also run the Database service.
Two agents per Database, Application, or Kubernetes target pool for HA. Both agents must be able to reach the same targets. An SSH Service exposes its own host, so another SSH host does not replace that host's filesystem or running processes.
Agents authenticate with a join token: either a pre-shared secret stored on the agent or a platform-issued identity (AWS, GCP, Azure, GitHub, Kubernetes, and a few more). After joining, the agent persists its host identity locally and dials out to Proxy on TCP/443, holding the connection open.

See the access guides for per-service setup.

How replicas coordinate

Different components share state in different ways.

Auth replicas share state through the backend. No Auth replica coordinates the others. Each replica reads and writes the state backend (etcd, PostgreSQL, DynamoDB, or Firestore) directly. The backend is the source of truth. There is no "primary" Auth and no Auth-to-Auth failover path.

Proxy replicas do not share sessions. Each Proxy owns its own active sessions and the tunnels they run through. A Proxy can be started, restarted, or replaced without telling any other Proxy. When it dies, the sessions on it die with it.

Agents own their own tunnels. Each agent holds its own host identity (issued at first join, persisted locally) and its own reverse tunnel. Multiple agents can serve the same pool of targets. Operators run two per pool for HA. Agents do not coordinate explicitly. The cluster tracks which agents are alive and routes new sessions to whichever one answers.

Durable cluster state is synchronized through the storage backend. The backend owns replication, write coordination, and quorum, which is the number of nodes that must agree before a write commits.

Reads are served from local caches. Auth replicas, Proxies, and agents cache the resources they need and receive updates through resource event streams. During a brief Auth outage, Proxies and agents keep serving work they can satisfy from the cache. New logins, resource changes, target registrations, and cache misses wait for Auth to return.

How persistence works

Three kinds of durable data live in three different stores.

Cluster configuration (Role, OIDCConnector, Token, target manifests, the cluster CA) lives in the state backend. Every Auth replica reads and writes it. Process settings, storage configuration, bootstrap manifests, and their secrets also exist outside that backend. Preserve them with the cluster backup so replacement hosts can connect to the existing cluster.

Audit events live in a separate append-only store. The audit-capable drivers are PostgreSQL, DynamoDB, Firestore, and Athena (queries against S3-backed Parquet files). Every meaningful action emits an event, including logins, session starts, and role changes. Auth writes events as they happen. Consumers read them later for compliance and forensics. The audit store is independent of the state backend on purpose. Even if you rebuild the cluster from scratch, the audit history survives.

Session recordings live in object storage: S3, an S3-compatible service like MinIO, GCS, or Azure Blob. Each recorded session is a multi-MB blob. Like audit, recordings outlive the cluster they were recorded from.

Active session tunnels are not durable. Losing the Proxy that owns a tunnel forces a reconnect or a new login. Policy and data already written to external stores survive the loss. Recording data still buffered only on the failed host may be lost, depending on the recording mode.

That separation is the recovery story. Replace Auth and Proxy nodes, point them at the existing state backend, and the cluster comes back with the same policies, connectors, tokens, and targets. Agents auto-reconnect after a restore. Reissue tokens only for agents whose data dir was also wiped.

Failure modes

A Proxy dies. The LB pulls it out of rotation once its health check fails. Active sessions on that Proxy terminate. The client reconnects through another Proxy. Users see a failed request or broken session, then retry.
An Auth replica dies. Remaining replicas keep serving because no Auth replica is primary. In-flight API calls retry on a healthy replica.
A Proxy cannot reach Auth. Active sessions and requests that can be served from the Proxy's local cache may continue. New logins, fresh policy checks, and cache misses fail or wait until the Proxy can reach Auth again. Locks created during the partition cannot reach an isolated enforcement point until connectivity returns. The role's locking mode determines whether a stale lock view blocks access. See lock propagation. In a colocated deployment this scenario implies the Auth service in the same process is failing, so the LB will already be pulling the host out of rotation.
An etcd node dies. Quorum survives losing one node out of three, or two out of five. Writes may pause while etcd elects a leader. Reads on the surviving leader keep flowing.
The state backend loses quorum. Writes pause. Existing sessions that do not need a new Auth decision can keep flowing because the Proxy already holds the tunnel and user certificate, and Proxies and agents have local read caches. New logins, role edits, target registrations, and cache misses fail until quorum returns. For restore planning, see Backups and the backend-specific Storage Guide.
Sustained Auth outage. New logins fail immediately. Existing certificates may support access that can be authorized from cached state, subject to certificate expiry, locking mode, and the protocol's live Auth dependencies. Certificate lifetime and active-session disconnection are separate role options. If every Auth replica is down, break-glass needs at least one Auth host running, so the recovery path is to restore an Auth process first. See Break-glass.
The LB fails. The cluster becomes unreachable. The fix is to run the LB in HA itself, such as active/standby HAProxy, a cloud L4 load balancer across availability zones, or whatever your environment provides. A single-instance LB is a single point of failure regardless of how many cluster replicas you run.
An agent dies. Targets served only by that agent become unreachable. Run two agents per pool with the same join method and the same ALMA_MATCH_LABELS selector.
The SSO IdP goes down. New SSO logins fail. Existing credentials do not require the IdP on each use and remain subject to their expiry and normal access policy. Unlike an Auth outage, cluster policy changes and lock propagation can continue.

Services​

How a request flows​

Composing a deployment​

Cluster topology​

Colocated (default)​

Split Auth and Proxy​

When to choose​

Recommended production floor​

Agent topology​

How replicas coordinate​

How persistence works​

Failure modes​

Further reading​