Skip to main content

AlmaForge Hardening Guide

Hardening is a set of choices an SRE makes before a cluster goes live and reviews after production changes. The defaults are safe for evaluation, not for production. This page walks through the choices that matter once the cluster holds real credentials, real audit data, and real production access.

Read the High Availability Guide first if you need a refresher on the cluster moving parts (Auth, Proxy, agents, state backend, cluster CA, fleet, mTLS).

Out of scope here:

  • Role design and access policy - see RBAC.
  • CA rotation procedure - see CA Rotation.
  • On-the-wire TLS, listeners, and PROXY protocol - see Networking Guide.
  • Per-backend storage hardening (encryption, IAM, network policy) - see Storage Guide.

Pre-go-live checklist​

Work the list top to bottom before flipping production traffic at the cluster. Each item links to its section below.

  • State, audit, and recording backends are encrypted at rest (Encryption at rest).
  • Internal mTLS works end to end and ALMA_CA_PIN is captured (mTLS inside the cluster).
  • Bootstrap join tokens are rotated or deleted after first agent join (Secrets and tokens).
  • NTP or equivalent clock sync is configured on every Auth, Proxy, and agent host. Drift is monitored (Clock sync).
  • Auth and Proxy data directories are chmod 0700 and owned by root (Filesystem hardening).
  • Role maxSessionTTL, disconnectExpiredCert, and maxConnections are set on production roles (Session and connection limits).
  • MFA is enforced at your IdP. AlmaForge inherits the IdP's MFA decision and does not enforce it independently (Identity provider).
  • Daemon outbound traffic is restricted to the state, audit, recording, OIDC, ACME, and package-mirror endpoints (Outbound network).
  • L4 load balancer is the only source the daemon trusts for PROXY protocol headers (Web UI hardening).
  • Web UI CSP, CORS, and HSTS behavior is verified on your hostname (Web UI hardening).

Encryption at rest​

AlmaForge stores cluster state, audit events, and session recordings in three separate places. Each one carries different sensitivity and each backend has its own at-rest encryption story. Pick the option that matches your compliance posture before any production data lands. See the per-backend pages under Storage Guide for setup detail.

Disk and OS layer​

Start below the application. For self-hosted etcd, PostgreSQL, and Auth hosts, encrypt the host disks with LUKS, dm-crypt, or the equivalent volume encryption in your platform. Managed databases and cloud buckets still need their own at-rest settings, but disk encryption closes the lowest layer first.

Verification is platform-specific. Capture one command per storage layer in your runbook, for example:

Terminal
findmnt -no SOURCE,FSTYPE,OPTIONS /var/lib/almaforgecryptsetup status <volume-name>aws dynamodb describe-table --table-name cluster-audit --query 'Table.SSEDescription.Status'gcloud storage buckets describe gs://almaforge-recordings --format='value(encryption.defaultKmsKeyName)'az storage account show --name almaforgerecordings --query encryption.keySource

Cluster state backend holds roles, connectors, tokens, target manifests, and the cluster CA. Loss of the at-rest data means an operator-level breach. Options:

  • etcd: encrypt the underlying disks or volumes, including snapshots. etcd does not encrypt its stored key/value data itself. See etcd's storage encryption guidance.
  • PostgreSQL: use encrypted disks or your managed service's storage encryption. Cover data files, WAL, and backups. pgcrypto is not a transparent encryption setting for the AlmaForge state backend. See PostgreSQL encryption options.
  • DynamoDB: server-side encryption with an AWS-owned, AWS-managed, or customer-managed key. The default is AWS-owned. See DynamoDB encryption.
  • Firestore: encryption at rest with Google-managed or customer-managed (CMEK) keys.

Audit events store the security event stream. Anyone who reads the audit store sees who-did-what timestamps. Options match the audit-capable drivers:

  • PostgreSQL: same options as the state backend above.
  • DynamoDB: server-side encryption with an AWS-managed or customer-managed KMS key.
  • Firestore: encryption at rest with Google-managed or customer-managed (CMEK) keys.
  • Athena: queries S3-backed Parquet files. Encrypt the underlying S3 bucket with SSE-S3 or SSE-KMS.

Session recordings contain terminal output and timing from privileged sessions. Protect them with encryption and restricted access.

  • S3 / GCS / Azure Blob: enable bucket-level SSE with a customer-managed KMS key. On GCS, use CMEK. On Azure Blob, use a customer-managed key on the storage account. Lifecycle rules move old recordings to cold storage on the same key.

If you operate a single KMS, the simplest production shape is one customer-managed key per store, with key rotation on the cadence your compliance program requires. Record the rotation blast radius in the runbook: some managed KMS rotations are transparent, while others require explicit object or table re-encryption.

mTLS inside the cluster​

The cluster CA signs every internal certificate. Every connection between Auth, Proxy, and agents is mutually authenticated against that CA. Internal TLS is on by default. The work here is to make sure no piece of the chain falls back to weaker or unauthenticated paths.

The cluster CA is created when the first Auth Service replica starts. Its keys are stored in the state backend. Any user signed in on a workstation can print the trusted key pins through the Proxy:

Terminal
alma auth pinsha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef

The command prints one sha256: pin for each trusted key. CA rotation can produce multiple pins while the cluster trusts both keys. Include every printed pin. A direct join requires a matching pin for every CA certificate returned by the Auth Service. A partial set fails the join.

Distribute the pins to your configuration management system. Set ALMA_CA_PIN to the absolute path of a file containing one pin per line. The pins are checked only when ALMA_AUTH_SERVER selects a direct Auth Service join. Without a pin, that path trusts the certificate presented on its first connection, which opens a window for on-path attacks during bootstrap.

A join through ALMA_PROXY validates the Proxy's TLS certificate against the host's system trust store. Use a publicly trusted Proxy certificate or install its signing CA in the host's trust store. This path ignores ALMA_CA_PIN.

Auth and Proxy mTLS: both ends present a certificate and both verify it against the cluster CA. The Proxy presents the host certificate issued to it when it joined, stored under /var/lib/almaforge and rotated without operator action. Auth presents a host certificate from the same CA, so a Proxy that reaches the wrong server fails the handshake instead of sending it a request.

A Proxy in the same process as Auth reaches it without using the network. A Proxy on another host reaches it over the multiplex listener on TCP/443. Confirm that the listener is present:

Terminal
ss -tlnp | grep almad

Then confirm the auth route on it answers with a certificate from the cluster CA. Name only that route, so a host not serving it closes the connection instead of answering with the web certificate:

Terminal
openssl s_client -connect <auth-host>:443 -alpn alma-auth </dev/null 2>&1 \    | grep -E 'subject=|issuer='

The issuer is your cluster's host CA. Anything else on that route, or a connection that closes without a certificate, means the listener is not serving Auth.

That check proves the route is served by Auth under the cluster CA rather than by plaintext or the web certificate. It does not test client authentication: Auth accepts the handshake and rejects the request afterward, so a certless client gets a TLS session and no access. Only an end-to-end Proxy or agent join tests that path.

Agent join credentials: agents authenticate with credentials scoped to a Token resource. The static join method uses a client_id and client_secret. Cloud and CI/CD join methods use a verified platform identity instead of a secret. After a successful join, the cluster issues the agent a host certificate. The agent rotates that certificate without operator action.

Before an agent joins through ALMA_AUTH_SERVER, retrieve the CA pins with:

Terminal
alma auth pin

Set ALMA_CA_PIN in the agent's env file, including for cloud-identity joins. Agents that join through ALMA_PROXY rely on the Proxy certificate instead, as described above.

Secrets and tokens​

Scope and expire static join secrets, and rotate other credentials according to their provider's lifecycle. Treat filesystem access to the Auth host as equivalent to admin access.

Join token rotation. A static Token carries a client_id and client_secret. Anyone with both can join an agent of the granted service role until the token expires. Defaults:

  • Issue tokens with a short --expires window when bootstrapping agents (minutes to hours, not days).
  • Scope tokens with --services=<list> to the smallest role set the agent needs.
  • Rotate the proxy-fleet token after the initial Proxy fleet has joined and stored its long-lived identity locally.

After the fleet has joined, delete the bootstrap token and verify it is gone:

Terminal
alma delete token/proxy-fleetalma get token/proxy-fleet

The second command should fail with a not-found error. When a new Proxy or agent needs to join later, issue a fresh time-boxed token for that change window and delete it after the host stores its identity.

Bootstrap connector drop-in. The first administrator signs in through a drop-in OIDCConnector (see Linux deployment guide). Keep the connector file-owned in almad.d, manage it through your configuration pipeline, and deploy mapping changes to the watched directory on each Auth host. Connector files reload without a restart. Changes to secrets supplied through the process environment require a restart.

Cloud and CI/CD join methods (aws, ec2, azure, gcp, github, gitlab, circleci, spacelift, kubernetes) avoid a shared secret entirely. Use them where the platform offers a verifiable workload identity. The static join method is the fallback for environments without one.

Secrets in resource manifests. Never commit OAuth client secrets, database passwords, or storage credentials to the manifest files in /etc/almaforge/almad.d. Reference them with ${env:NAME} or ${file:/path} and resolve from the daemon environment or a mounted file. See Resource value expansion.

Identity provider​

AlmaForge delegates authentication entirely to your IdP. Every control that lives at the IdP layer (MFA, conditional access, device posture, geofencing, account lockout) is the IdP's job, not the cluster's. Take that seriously: a cluster with no MFA is a cluster whose IdP has no MFA.

  • Require MFA at the IdP for every account that can sign in to AlmaForge. AlmaForge accepts whatever assertion the IdP produces and has no independent MFA challenge.
  • Audit IdP-side changes. Adding a group, changing claim mapping, or rotating an OIDC client secret at the IdP changes who can reach the cluster. Wire IdP admin events into the same SIEM that watches AlmaForge audit events.
  • Rotate the OIDC client secret on the IdP and update the cluster env in the same change window.
  • Keep OIDC endpoints reachable from every Auth host, including discovery, signing keys, and the token endpoint. Browser access to the IdP alone is not enough. See egress requirements.

Clock sync​

Short-lived certificates depend on time. Clock drift can make valid certificates look expired, distort audit timelines, and break SSO or MFA challenges.

Use NTP, chrony, systemd-timesyncd, or your platform's equivalent on every Auth, Proxy, and agent host. Verify on systemd hosts with:

Terminal
timedatectl show -p NTPSynchronized --valueyes

Filesystem hardening​

Daemon user. The shipped systemd unit runs almad as root. The daemon binds privileged ports, manages on-disk identity material, and reads root-owned config and TLS files, so it stays under the root account rather than a dedicated service user.

The Auth data directory holds the cluster CA bootstrap state, the local admin identity used by --auth-server direct access, and the working files for each running role. Lock the filesystem down.

Treat /var/lib/almaforge access as cluster-admin equivalent

Read access to /var/lib/almaforge is sufficient to take over the cluster through the --auth-server direct-access path. Audit sudo, filesystem mounts, and host authentication events that touch this directory. Require multi-person approval for routine access. A break-glass procedure that any sysadmin can execute silently is not a break-glass procedure.

Required production paths and modes:

  • /var/lib/almaforge (/var/lib/almaforge): owner root:root, mode 0700.
  • /etc/almaforge: owner root:root, mode 0700. The config root, holding the env file and any TLS material.
  • /etc/almaforge/almad.d (/etc/almaforge/almad.d/): owner root:root, mode 0700. Drop-in manifests are read by the root daemon and must not be world- or group-readable.
  • /etc/almaforge/almad.env (/etc/almaforge/almad.env): owner root:root, mode 0600. Holds secrets interpolated into drop-ins.
  • TLS key (/etc/almaforge/tls.key): owner root:root, mode 0600. Same rule for any etcd client key, database password file, or per-Proxy private key.

After install, confirm. On Linux with GNU stat:

Terminal
stat -c '%a %U:%G %n' /var/lib/almaforge /etc/almaforge/almad.env

On macOS or BSD systems inspecting a mounted image or copied tree, use:

Terminal
stat -f '%Mp%Lp %Su:%Sg %N' /var/lib/almaforge /etc/almaforge/almad.env

SELinux and AppArmor. AlmaForge does not ship a custom SELinux policy or AppArmor profile in this release. Run with your distribution's default profile and confine the daemon at the filesystem level via the modes above.

Outbound network​

almad initiates outbound connections to a small, known set of endpoints. A compromised daemon with unrestricted egress is much harder to contain than one with a firewall-enforced allowlist. Permit outbound only to:

  • The state backend endpoints (etcd, PostgreSQL, DynamoDB, Firestore).
  • The audit and recording backend endpoints.
  • The OIDC provider's discovery, JWKS, and token endpoints (typically https://<issuer>/.well-known/openid-configuration and the jwks_uri and token_endpoint it advertises).
  • Resource endpoints reached by database, application, and Kubernetes agents, and SSH endpoints reached by the Proxy for agentless hosts.
  • APIs used by optional integrations and platform join methods you enable. See egress requirements.
  • Internal NTP and DNS resolvers.
  • The internal package mirror and container registry during install and upgrade windows.
  • The ACME directory when ALMA_ACME_ENABLED is set. For Let's Encrypt, allow TCP 443 to acme-v02.api.letsencrypt.org. Keep it allowed for automatic renewal, which begins roughly two months after issuance. A cluster that uses ALMA_TLS_CERT and ALMA_TLS_KEY needs no ACME egress.

Deny everything else at the host firewall or VPC egress layer. Cloud-managed firewalls, Kubernetes NetworkPolicy, or a per-host iptables/nftables ruleset all work. See the Networking Guide for the inbound port matrix.

Session and connection limits​

Limits are applied by role options, not by daemon configuration. Setting them on production roles bounds the blast radius of a compromised credential.

maxSessionTTL caps the lifetime of credentials issued for the role. An unspecified role maximum defaults to 12 hours, and built-in roles can impose shorter limits. Issued credentials cannot exceed the 30-hour hard cap. Set a maximum that matches the role's operational needs. Use disconnectExpiredCert when active sessions must also end at certificate expiry. See role options.

warning

When disconnectExpiredCert is enabled, a shorter certificate lifetime also shortens active sessions. Test long-running maintenance workflows before tightening production roles.

maxConnections caps the number of concurrent connections a single identity may hold. The default is unlimited. Setting it to a small finite number (1 to 5 for human users, higher for bot identities) prevents a stolen certificate from holding open a fleet of sessions.

role-with-limits.yaml:

examples/hardening/role-with-limits.yaml
# Production role with explicit session and connection limits.
# Use this shape for human operator roles. Keep a separate break-glass or
# maintenance role with a longer TTL if your change windows require it.
apiVersion: almaforge.com/v1
kind: Role
metadata:
  name: editor
spec:
  options:
    maxSessionTTL: 4h # Upper bound on a single interactive session.
    maxConnections: 5 # Maximum concurrent connections for one identity.
  allow:
    serverLabels:
      env: prod # Limit this role to production-labeled servers.
    serverLogins:
      - ubuntu # Allow only the OS login names operators need.

Verify the limits after applying:

Terminal
alma get role/editor -o yaml | grep -E 'maxSessionTTL|maxConnections'

Runtime revocation​

Locks block a user, role, login, server, device, or access request cluster-wide at the next decision point, independent of certificate expiry:

Terminal
alma create lock --target-user=alice --message="incident-2026-1234"

This is the runtime complement to the role limits above. See Revoking access for the full lock runbook, including how locks propagate during a partition.

Web UI hardening​

The Proxy serves the AlmaForge Web UI on the public hostname. Content-Security-Policy, secure cookies, and HSTS ship with the Web UI. Verify the headers on the public endpoint before going live, including any changes introduced by infrastructure in front of the Proxy.

Content-Security-Policy. The Proxy emits a CSP that allows inline styles required by the bundled UI and blocks third-party script sources. Its frame-ancestors 'none' directive prevents a parent portal from embedding the Web UI. Open the Web UI directly at the cluster hostname.

warning

CSP can lock out operator workflows. frame-ancestors 'none' blocks every parent portal from framing the Web UI. Preserve this protection when configuring infrastructure in front of the Proxy.

CORS. Access the Web UI and its JSON API through the same cluster origin. Do not add permissive cross-origin headers at an intermediary. Application-access CORS settings are separate from Web UI policy.

HSTS. The Proxy emits a Strict-Transport-Security header with max-age=31536000; includeSubDomains when serving on the public hostname. Confirm the LB does not strip the header:

Terminal
curl -sI https://almaforge.example.com | grep -i strictstrict-transport-security: max-age=31536000; includeSubDomains
warning

HSTS is cached by browsers for the max-age window. With includeSubDomains, every subdomain under the public hostname must keep working over HTTPS. Server-side rollback cannot clear the cache already stored by clients.

For maximum hardening, submit the public hostname to the HSTS preload list only after the header has been live for at least the max-age window. Preload is harder to undo than a normal HSTS header.

Verify the Web UI headers before go-live:

Terminal
curl -sI https://almaforge.example.com \    | grep -Ei 'content-security-policy|strict-transport-security|x-frame-options'curl -sI -H 'Origin: https://dashboard.example.com' https://almaforge.example.com \    | grep -i access-control-allow-origin

Cookies. Session cookies are flagged HttpOnly, Secure, and SameSite=Lax. None of this is configurable. Operators who put a caching CDN in front of the Proxy must exclude Set-Cookie and any auth-bearing path from the cache.

Public exposure. The Proxy is intentionally fronted by a single hostname (ALMA_ADVERTISE). Do not stand up alternate hostnames through reverse proxies that bypass ALMA_ADVERTISE. The cluster issues short-lived certs and validates audience claims against that hostname. Alternate hostnames break SSO redirects and break audit attribution.

PROXY protocol trust. When an L4 load balancer fronts the Proxy, the LB injects the real client IP via PROXY protocol so audit events attribute connections to the originator rather than the LB. Set ALMA_REAL_IP_FROM to the trusted LB addresses and restrict the listener's network exposure to those sources. ALMA_PROXY_PROTOCOL controls header presence, not which sender's address is trusted. Incorrect trust settings can make source-IP attribution spoofable. See the Networking Guide for PROXY protocol setup.

Network segmentation is a hardening choice handled at the infrastructure layer. See Networking Guide for VLAN, security group, and Kubernetes NetworkPolicy patterns.

After hardening​

  • Run through Break-glass once on a staging cluster. A break-glass path you have not exercised is not a path.
  • Schedule a tabletop break-glass exercise within 30 days of go-live.
  • Add the cluster CA fingerprint and break-glass runbook URL to your incident response wiki.
  • Schedule the first CA rotation drill - see CA Rotation.
  • Wire /readyz into your LB health check and Prometheus /metrics into your scrape config. See Cluster on Linux and Monitoring.
  • Alert on high-value audit events such as LockCreate, SessionReject, and repeated login failures. See Monitoring and the Audit Reference.
  • Review this guide after every major upgrade, cluster migration, audit finding, new production role, and scheduled CA rotation. A PCI shop should do that at least quarterly. A SOC2 shop should include it in the annual control review.

References​