etcd

The etcd driver stores shared cluster state for multiple Auth instances. Use it when your team operates an etcd cluster on VMs or Kubernetes. Choose separate drivers for audit events and session recordings.

Storage concern	Supported
Cluster state (name: state)	yes
Audit events (name: audit-*)	no
Session recordings (name: recordings)	no

TL;DR

1. Use etcd only for name: state. You still need separate drivers for audit events and session recordings.
2. Pick a required key prefix, then write a StorageConfig with spec.etcd that points at your etcd client endpoints.
3. Restart the Auth Service and verify with alma get storageconfig/state.

Prerequisites

Before you start, make sure you have:

• etcd 3.5 or newer.
• A working etcd cluster with peer TLS already configured.
• Client credentials for AlmaForge: either mTLS files or username plus password file.
• A planned key prefix for AlmaForge data. prefix is required.

The etcd security guide covers the TLS setup procedure, and the etcd-io/etcd/hack/tls-setup script is a quick way to bootstrap a CA, peer certificates, and client certificates if you do not already have a TLS hierarchy in place.

Configuration

Step 1. Choose the key prefix and endpoints

Use etcd when you already trust etcd as part of your control-plane stack and want AlmaForge on the same operational footing. That is common in Kubernetes-heavy environments because etcd already sits behind the Kubernetes API server.

Plan the prefix before the first start. There is no default, and AlmaForge refuses to start without one.

Step 2. Write the state backend resource

For mTLS authentication:

mtls.yaml:

examples/storage/etcd/mtls.yaml

# Shared etcd state backend authenticated with a client certificate.
apiVersion: almaforge.com/v1
kind: StorageConfig
metadata:
  name: state
spec:
  etcd:
    peers:
      - https://etcd-1.example.com:2379
      - https://etcd-2.example.com:2379
      - https://etcd-3.example.com:2379
    prefix: /almaforge/
    tlsCAFile: /etc/almaforge/etcd/ca.pem
    tlsCertFile: /etc/almaforge/etcd/client.pem
    tlsKeyFile: /etc/almaforge/etcd/client-key.pem

For password authentication instead of mTLS:

password.yaml:

examples/storage/etcd/password.yaml

# Shared etcd state backend authenticated with a password file.
apiVersion: almaforge.com/v1
kind: StorageConfig
metadata:
  name: state
spec:
  etcd:
    peers:
      - https://etcd-1.example.com:2379
    prefix: /almaforge/
    tlsCAFile: /etc/almaforge/etcd/ca.pem
    username: almaforge
    passwordFile: /etc/almaforge/etcd/password

Field	Required	Default	Notes
peers	yes	none	One or more etcd client endpoints.
prefix	yes	none	Key prefix for AlmaForge data. No default. AlmaForge refuses to start without it.
tlsCAFile	yes unless insecure: true	none	CA bundle used to verify the etcd server.
tlsCertFile	no	none	Client certificate for mTLS auth.
tlsKeyFile	no	none	Client key for mTLS auth.
username	no	none	Username for password auth when not using mTLS.
passwordFile	no	none	Path to a file containing the etcd password. Newlines are trimmed on read.
insecure	no	false	Disables normal TLS validation requirements. Do not use this in production.
dialTimeout	no	30s	Connection timeout.
bufferSize	no	1024	Event buffer capacity.
maxClientMsgSizeBytes	no	etcd client default	Raise only when message size errors force it. Keep it in sync with the server limit.
clientPoolSize	no	3	Number of concurrent etcd clients AlmaForge uses.

Step 3. Restart and verify

Restart the Auth Service after updating the bootstrap file, then verify the active resource:

Terminal

sudo systemctl restart almadalma get storageconfig/state -o yaml

The backend is active when the command shows the intended endpoints and prefix and all Auth instances are ready. See Monitoring for readiness checks.

URI form

Equivalent URI string, suitable for an environment variable or CLI flag. Peer endpoints are comma-separated in the URI host, or in a peers= query parameter for additional peers. Other fields go in the query string:

text

etcd://etcd-1.example.com:2379,etcd-2.example.com:2379,etcd-3.example.com:2379?prefix=/almaforge&tls_ca_file=/etc/almaforge/etcd-ca.pem&tls_key_file=/etc/almaforge/etcd-client-key.pem&tls_cert_file=/etc/almaforge/etcd-client-cert.pem&dial_timeout=30s

Canonical query keys are snake_case (tls_ca_file, tls_key_file, tls_cert_file, dial_timeout, password_file). The typed JSON-tag form (tlsCAFile, etc.) is accepted as an alias.

Troubleshooting

AlmaForge refuses to start without prefix

That is expected. prefix is required and has no default.

etcd reports gRPC message size errors

Raise maxClientMsgSizeBytes on AlmaForge and --max-request-bytes on the etcd servers together. Keep the numbers aligned.

You want to store audit events or recordings in etcd

Do not. etcd is state only. Plan separate drivers for audit events and session recordings.

Operations

Cluster sizing

Use three etcd members for a quorum that tolerates one member loss. Five members tolerate two member losses. Two members require both to remain available and provide no member-failure tolerance. See High availability.

Hardware

Follow the etcd operational guide. SSD or NVMe storage is the safe default.

Driver scope

etcd only covers cluster state. You still need separate drivers for audit events and session recordings.

References

• Storage backends overview.
• High availability.
• Break-glass access.
• etcd security guide.
• etcd operational guide.