Backups and restore

AlmaForge holds no durable state in the almad process, so backing up the cluster means backing up the three stores that hold its data. Production clusters use an external service for all three. Single-host clusters keep the stores and the host identity under /var/lib/almaforge, where backup is a filesystem job. See Local disk.

Store	What it holds	Backup is the responsibility of
State backend	Roles, OIDC connectors, tokens, target manifests, cluster CAs	The backend itself (etcd snapshot, PostgreSQL dump, DynamoDB on-demand backup, Firestore export)
Audit events	Structured authentication, access, and protocol records captured by the audit policy	The audit backend. Retention controls when records expire, not how they are recovered
Session recordings	Multi-MB playback blobs	The object store. Bucket lifecycle rules and cross-region replication

almad itself holds only ephemeral process state plus the host identity persisted under /var/lib/almaforge. Rebuilding an Auth or Proxy replica also requires its environment, resource files, TLS material, and access to the configured stores. Include those files and their secret sources in the recovery plan.

Snapshot the state backend

Run the backend-native snapshot tool on a schedule and verify the snapshot before retiring the previous one.

Terminal

# etcdetcdctl snapshot save almaforge-<timestamp>.dbetcdutl snapshot status almaforge-<timestamp>.db
# PostgreSQLpg_dump -Fc -f almaforge-<timestamp>.dump <database>pg_restore -l almaforge-<timestamp>.dump >/dev/null
# DynamoDBaws dynamodb create-backup --table-name <table> --backup-name almaforge-<timestamp>aws dynamodb describe-backup --backup-arn <arn>
# Firestoregcloud firestore export gs://<bucket>/almaforge-<timestamp>

Snapshot at a cadence that matches your tolerable recovery point. Hourly is reasonable for most production deployments. Use a faster cadence for clusters that issue many access requests per minute.

For backend-specific options (continuous WAL archiving on PostgreSQL, point-in-time recovery on DynamoDB, and off-cluster copy for etcd snapshots), see the per-driver page in the Storage Guide.

Local disk

A cluster built from the Standalone guide configures no storage backend, so all three stores sit under /var/lib/almaforge: cluster state in backend/, audit events in log/, and session recordings in records/. The host identity lives there too. One copy of that directory preserves those stores and the host identity. Also preserve the environment configuration and any files it references outside that directory.

SQLite is a live database, so take the copy with the daemon stopped:

Terminal

sudo mkdir -p /var/backupssudo systemctl stop almadsudo cp -a /var/lib/almaforge /var/backups/almaforge-<timestamp>sudo systemctl start almad

cp -a carries the mode and ownership across, which matters because the daemon creates /var/lib/almaforge as root:root at mode 0700 and expects to find it that way. An atomic filesystem or volume snapshot works without the stop, because it captures the database and its journal at one instant.

Restore with the daemon stopped. Preserve the current directory separately, restore the backup as /var/lib/almaforge with its ownership and modes intact, then start the daemon. Do not merge a backup into an existing database directory. Changes made after the backup will not be present in the restored cluster. See the SQLite driver page for the tuning fields and the single-writer constraint.

Move to an external backend before this cluster carries production traffic. Local-disk storage has no host-level redundancy.

Audit and recordings

Audit events are appended during normal operation, but backend durability alone does not protect against accidental deletion or compromised storage credentials. Configure backend-native backups or protected copies and test recovery. Set retention separately to match your organization's evidence requirements.

Session recordings are object-storage blobs. Configure bucket-level versioning, lifecycle transitions to cold storage, and cross-region replication where regulatory rules require it.

Restore drill

A restore path you have not exercised is not a restore path. Run this drill on a staging cluster at least once per quarter:

1. Snapshot the staging state backend, or copy /var/lib/almaforge on a local-disk cluster.
2. Stop staging almad.
3. Restore the snapshot into a fresh backend instance, or put the copied directory back in place.
4. Point almad at the new instance and restart.
5. Verify with alma get role, alma get oidcconnector, alma get node, and an end-to-end login.

Record the wall-clock time the restore takes and the size of the snapshot. Repeat the drill as data volume and deployment size change.

What is not restored

• Active sessions. Existing user certificates remain valid until they expire, but the Proxy tunnels and TCP connections that carried sessions through the old cluster are gone.
• Agent host identities lost with the host. If an agent's /var/lib/almaforge survived (for example, on a separate volume), it reconnects automatically. Otherwise re-issue a Token and rejoin.

For full HA failure-mode analysis, see High Availability.

References

• Storage Guide - per-backend snapshot and encryption guidance.
• CA Rotation - rotate the cluster CA after a restore from a snapshot whose key material might be exposed.
• Break-glass - recover access when the cluster is up but normal sign-in is blocked.