Skip to main content

Scaling

Every component in AlmaForge scales on a different signal. There is no single "scale the cluster" knob. Read the signal for the component that is under pressure, then act on that component alone.

Signals at a glance:

  • Auth: API latency, audit emit failures, CPU on Auth nodes.
  • Proxy: concurrent sessions, per-Proxy NIC throughput, CPU on Proxy nodes.
  • etcd: request latency. Replica count is not the performance lever.
  • PostgreSQL / DynamoDB / Firestore: write latency and backend-side throttling or queue depth.
  • Agents: file-descriptor exhaustion, CPU pressure, or target reachability when one agent restarts.
  • Audit and recording stores: ingestion lag, per-bucket throughput limits.
  • Load balancer: connection throughput against the LB's own envelope.

For the HA architecture that underpins all of this, see High Availability. In the default colocated deployment, adding a control-plane replica scales Auth and Proxy together. The separate actions below apply to split deployments.

Metrics to wire first​

Use the metrics catalog for exact names before you build alerts. The table below lists the scaling signals that have a cataloged metric in this release:

SignalMetric or check
Auth backend read/write latencybackend_read_seconds, backend_write_seconds
Auth cert/key generation pressureauth_generate_seconds, auth_generate_requests, auth_generate_requests_throttled_total
Audit event emit failuresaudit_failed_emit_events
Audit event ratealma_audit_emit_events
Proxy active SSH sessionsproxy_ssh_sessions_total
Process CPUrate(process_cpu_seconds_total[5m])
File-descriptor pressureprocess_open_fds / process_max_fds
Process memoryprocess_resident_memory_bytes
S3 recording-store latencys3_requests_seconds
GCS recording-store upload latencygcs_event_storage_uploads_seconds
DynamoDB backend latencydynamo_requests_seconds
etcd transaction latencyetcd_backend_tx_seconds

Examples:

PromQL
# Storage backend write p99 by instance
histogram_quantile(0.99,
  sum by (le, instance) (rate(backend_write_seconds_bucket[5m])))

# File-descriptor usage ratio by instance
process_open_fds / process_max_fds

/readyz latency, load-balancer target state, cloud object-store queue depth, and NIC saturation are external checks. Measure those with your black-box probe, load balancer metrics, cloud provider metrics, and host network telemetry.

Auth fleet​

Signal: rising /readyz or gRPC latency, backend read/write P99, audit emit failures, or sustained CPU above 70% on Auth nodes.

Action: add Auth replicas. On Linux hosts, install the package and point the new node at the same state backend and env file as the existing fleet (see Cluster on Linux). On Kubernetes, increment the StatefulSet replica count.

No Auth replica is primary. Every replica reads and writes the same state backend. Give new replicas the same cluster identity, backend, and bootstrap resources as the existing fleet. Add one replica at a time. Wait for the new replica to pass /readyz and for latency to remain stable for at least 60 seconds before adding another. For DynamoDB state, keep the Auth pool within the two-instance limit.

Two Auth replicas is the HA minimum. Three replicas means rolling upgrades and node failures both happen with two healthy replicas remaining instead of one. Many production clusters start at three when the extra instance cost is acceptable.

When replicas are not enough: if tail latency stays high after adding replicas, the bottleneck is usually the state backend or the audit store, not Auth itself. Profile before adding more replicas. Bigger Auth nodes can help when the in-memory cache cannot hold the cluster working set, or during cert-issuance storms after CA rotation or a large fleet restart. Profile before resizing.

Proxy fleet​

Signal: concurrent session count approaching the per-process file-descriptor limit, sustained high CPU, or network bandwidth saturating a Proxy node's NIC.

Action: add Proxy replicas. On Linux hosts, start a new Proxy node from the same almad.env template and join it to the Auth fleet with its token (see Cluster on Linux). On Kubernetes, increment the StatefulSet replica count. The L4 LB picks up the new node as soon as its /readyz health check passes.

Each additional Proxy consumes compute, memory, file descriptors, and connections to Auth and agents. It holds active sessions and a local read cache, but no durable cluster configuration. Removing a Proxy drops its active sessions. Users reconnect through another Proxy.

Bigger Proxy nodes help when the bottleneck is per-session throughput, recording upload speed, or per-connection bandwidth rather than connection count. Connection count is a parallelism problem. Throughput per connection is a per-node capacity problem. Alert when a Proxy is above roughly 80% of its NIC line rate during the business peak.

State backend​

The state backend is the source of truth for cluster configuration. It does not scale the same way the stateless fleets do.

etcd​

etcd scales for failure tolerance, not for performance. Going from 3 to 5 nodes tolerates two node failures instead of one. It does not make reads or writes faster: quorum-write latency stays the same or gets slightly worse as cluster size grows.

Follower reads in supported etcd versions can reduce read pressure on the leader. They do not change the quorum-write path that dominates Auth write latency.

If etcd is the bottleneck:

  • Give etcd nodes faster disks. etcd writes are sequential and latency is dominated by fsync time. Dedicated low-latency SSD volumes or local NVMe are the most effective etcd performance lever.
  • Give etcd nodes more RAM. etcd keeps its working set in memory.
  • Reduce write amplification on the AlmaForge side by auditing StorageConfig options in the Storage Guide.

Do not add etcd nodes to solve performance problems. Use 3 nodes for most production clusters. Use 5 nodes when you need to tolerate two simultaneous node failures.

See the etcd Storage Guide for cluster layout, peer mTLS, and disk sizing guidance.

PostgreSQL​

PostgreSQL scales vertically for write throughput: a bigger primary instance with more CPU and RAM handles more concurrent write connections. Auth writes state changes to the primary, while many reads use local caches. Read replicas do not increase write capacity.

If PostgreSQL write latency is the bottleneck:

  • Move to a larger instance class.
  • Budget max_connections across Auth replicas and set pool_max_conns in each PostgreSQL connection string. The state driver has its own pool and needs a direct connection for logical decoding. See connection pooling constraints.
  • Separate the state database from the audit database if both share the same instance. Audit is append-only and high-volume. State is read/write and latency-sensitive. They have different access patterns.

See the PostgreSQL Storage Guide for connection pooling, sizing, and TLS setup.

DynamoDB​

DynamoDB scales by provisioned capacity or on-demand mode. In on-demand mode, AWS scales capacity automatically. In provisioned mode, increase read and write capacity units when CloudWatch shows consumed capacity approaching the provisioned limit.

No AlmaForge config change is needed. The scaling knob is the DynamoDB table configuration in your AWS account.

See the DynamoDB Storage Guide for table layout, IAM, and capacity planning.

Firestore​

Firestore autoscales storage and request capacity in Native mode. No AlmaForge config change is needed. Watch GCP-side quotas and the per-collection write rate as the cluster grows. Cost scales with operations and stored bytes rather than provisioned capacity.

See the Firestore Storage Guide for IAM and collection layout.

Agent pools​

Signal: file-descriptor exhaustion on an agent, sustained CPU pressure, or targets in a pool becoming unreachable when one agent restarts.

Action: run two or more agents per pool. Set the same ALMA_MATCH_LABELS on each agent so both agents claim the same targets:

This target-pool model applies to Database, Application, and Kubernetes services. An SSH Service exposes the host where it runs. A second SSH host does not replace access to the first host's filesystem or processes.

env
ALMA_MATCH_LABELS=env=prod,region=us-east-1

The cluster tracks which agents are alive and routes new sessions to whichever one answers. Existing sessions stay on the agent that opened them until they finish or the agent restarts. See ALMA_MATCH_LABELS for the full label selector syntax.

Two agents per pool is the HA minimum. It means a single agent restart does not make the pool's targets unreachable. Three or more agents is appropriate when the pool is large, or when the agent role is also under CPU or FD pressure from many concurrent sessions.

For pools with heterogeneous targets, you can shard by label. One agent with ALMA_MATCH_LABELS=env=prod,tier=web handles web targets. Another with ALMA_MATCH_LABELS=env=prod,tier=db handles database targets. Both share the same cluster. Each agent only claims the targets whose labels match its selector.

Audit and recording stores​

Signal: ingestion lag visible in your store's write queue or metrics, dropped recording upload chunks in Proxy logs, or per-bucket throughput limits reported by your cloud provider.

Audit events​

Audit events are append-only. High ingestion volume is the usual pressure, not read latency.

Audit storage supports fan-out through multiple StorageConfig instances named audit or audit-*. Each instance receives the same emitted events. The audit-capable drivers are DynamoDB, PostgreSQL, Firestore, Athena, and local file. Use fan-out when you need an operational sink and a compliance sink at the same time.

For PostgreSQL audit, isolate high-volume ingestion from the state database when the two compete for resources. Monitor autovacuum and query latency, and configure audit retention to bound stored history.

Session recordings​

Recording size depends on session duration and terminal output. In an HA deployment, recordings live in object storage (S3, GCS, Azure Blob). Check the capture service's path to Auth and Auth's path to the store. The capture service depends on the recording mode.

If recording uploads are dropping:

  • Ensure the recording services and Auth hosts have enough bandwidth along the upload path. Asynchronous modes buffer recordings locally, while synchronous modes put recording delivery in the session path. Monitor local disk capacity as well as network throughput.
  • Move recordings to a bucket, container, or account with a higher write-throughput envelope if the store-side limit is the ceiling. A cluster has one recordings StorageConfig instance, so do not plan on per-cluster fan-out for session recordings.

There is no public per-Proxy recording upload-buffer knob in this release. Scale Proxy bandwidth, the object store, or the cluster shape instead.

Load balancer​

The L4 load balancer forwards TCP without decrypting it. Its scaling envelope depends on which LB you run.

  • Cloud L4 load balancers (AWS NLB, GCP TCP LB, Azure LB): scale automatically with connection throughput. No operator action required beyond provisioning cross-zone or multi-AZ.
  • HAProxy: scales vertically (more CPU, more memory, more FDs per process) or by adding HAProxy instances behind a DNS-based or anycast layer. Size each HAProxy VM with your own TLS, connection churn, and recording traffic mix.
  • nginx stream: similar to HAProxy. Single-process per-worker-count scaling.
  • Kubernetes Service (LoadBalancer): delegates to the cloud provider's LB. Scaling behavior matches the cloud L4 case above.

For LB configuration, PROXY protocol setup, and health check wiring, see the Networking Guide.

Geographic distribution​

Keep Auth, Proxy, and the state backend in the same low-latency region. Do not stretch etcd across a high-latency WAN to make one cluster cover every geography. Put agents near the targets they serve and let them dial back to the regional control plane.

For autonomous regional operation, run separate clusters with separate state backends, CA material, and break-glass procedures. This docs set does not describe a shared-state multi-region control plane.

Cost considerations​

Capacity is also a budget decision. Watch these cost drivers before scaling blindly:

  • Recordings: long retention of session recordings is usually the largest long-term storage cost.
  • Proxy bandwidth: more sessions mean more egress and more upload bandwidth for recordings.
  • State backend: faster etcd disks or a larger PostgreSQL primary cost more than adding stateless replicas.
  • DynamoDB: on-demand mode is simple for bursty clusters. Move to provisioned capacity when the workload is steady and predictable.
  • Auth and Proxy compute: three Auth replicas cost more than two, but they preserve two healthy Auth replicas during maintenance.

Capacity planning rules of thumb​

These are qualitative starting points, not benchmarks. Validate with your own load before committing to a shape. There is no published users-per-node or sessions-per-vCPU multiplier for this release.

ComponentStarting shapeWhen to add capacity
Auth2 replicas minimum, 3 preferred for production, 2 vCPU / 4 GB RAM eachStorage backend write P99 over 100 ms, auth_generate_seconds rising, or audit emit failures growing
Proxy3 replicas, 2 vCPU / 4 GB RAM eachproxy_ssh_sessions_total rising with FD usage above 70%, or NIC throughput above 80% of line rate
etcd3 nodes, dedicated low-latency SSD or local NVMe, 8 GB RAM eachRarely. Resize disks or RAM before adding nodes
PostgreSQLOne primary, 4 vCPU / 16 GB RAMWrite latency rising, or connection pool saturation
DynamoDBOn-demand modeTable metrics show consistent throttling
FirestoreNative mode (autoscales)Quota warnings or per-collection write-rate ceiling
Agent pool2 per poolSingle-agent pool, or FD exhaustion approaching

Use the measured bottleneck to choose which component to scale. Session-heavy workloads put pressure on Proxies and agents, while frequent logins, resource changes, and audit events can put pressure on Auth and storage independently of the Proxy count.

Further reading​