Skip to main content

AlmaForge Cluster on Linux

This guide builds a production high-availability deployment of AlmaForge on three Linux hosts. Cluster state, audit events, and session recordings live in external services that you provision separately. Clients reach the cluster through the public hostname. For production, use an L4 load balancer in front of the three hosts to remove unhealthy nodes through active health checks. Use DNS round-robin only when an L4 load balancer is not available. See Load balancer.

Looking for the 10-minute eval?

Try AlmaForge with the Quick Start demo.

Topology​

This topology keeps the production footprint small: three cluster hosts that hold no durable state, plus the entry point that routes clients to healthy hosts.

Linux HA topology with external storage

  • Three peer hosts. Each runs almad with both Auth and Proxy enabled in one process (see the High Availability Guide for the service model). There is no leader, and hosts do not connect directly to each other. A host can fail without taking cluster state, audit history, or recordings with it.
  • Single entry point. Clients reach the cluster through one L4 load balancer that health-checks /readyz. DNS round-robin is an alternative without active health checks. TLS terminates on each host, not at the LB.
  • External durable storage. State (etcd, PostgreSQL, DynamoDB, Firestore), audit (PostgreSQL, DynamoDB, Firestore, Athena), and recordings (S3, GCS, Azure Blob) live outside the hosts. A host failure leaves all of it intact.

Provision prerequisites​

Review System Requirements first. Before you install AlmaForge, prepare the hosts, firewall rules, entry point, DNS records, TLS certificate, storage backends, and OIDC application described below.

External storage​

AlmaForge needs three storage backends before first start:

Provision them before installing AlmaForge. almad does not host any of them.

DynamoDB state supports at most two Auth instances. For the three-host topology in this guide, use etcd, PostgreSQL, or Firestore. See the DynamoDB prerequisites.

See the Storage Guide for the full driver matrix, sizing guidance, and per-driver reference pages.

If you run etcd on the same hosts as almad, plan for extra CPU and RAM and use SSD-backed disk. etcd is latency-sensitive and competes with almad for fsync throughput on shared storage.

Linux hosts​

Prepare three Linux hosts that meet System Requirements. This guide assumes the hosts are called almaforge-1, almaforge-2, and almaforge-3.

Every host reads and writes state, audit events, and recordings through the external storage backends. Place hosts on network segments that can reach those backends over private routes. Diagnostics, metrics, and OS login stay private.

Firewalls​

Open the following ports for this Linux HA layout. Only TCP/443 needs to be reachable by clients.

PortDescription / purpose
TCP/443Client-reachable for users, agents, automation, and alma. With DNS round-robin, clients connect directly to each host. With a load balancer, the load balancer forwards TCP/443 to each host.
TCP/3000Private diagnostics endpoint for /healthz and /readyz when ALMA_DIAG_LISTEN is enabled. Reachable from the load balancer and monitoring systems only.
TCP/3081Private Prometheus metrics endpoint when ALMA_METRICS_LISTEN is enabled. Reachable from monitoring only.
TCP/22Private OS login for installation, upgrades, and break-glass recovery. Allow operator workstations or bastions only.
Outbound connectionsAllow the configured storage backend ports, package mirror HTTPS, and OIDC discovery, signing-key, and token endpoints. See egress requirements.

With DNS round-robin, each host needs an address that clients can reach. With a load balancer, only the load balancer needs a public address. Keep diagnostics, metrics, and SSH on private networks. Outbound connections to your state, audit, and recording backends should also use private network paths where your platform supports it.

Confirm SSH works on all three hosts before continuing.

Load balancer​

End users (both browser and CLI) as well as agents need to reach TCP/443 on one of the nodes.

Recommended: an L4 load balancer in front of the three hosts. It forwards TCP/443 without terminating TLS. Each host terminates TLS. The load balancer runs active health checks against /readyz on the diagnostic port (ALMA_DIAG_LISTEN) and removes unhealthy hosts according to its check interval and failure threshold. TCP/443 alone only proves the socket is open. /readyz proves the process can serve traffic.

See Load balancer configuration for HAProxy, AWS NLB, Kubernetes Service, and nginx examples.

Alternatively, skip the load balancer and use DNS round-robin. Publish one A or AAAA record per host under the same DNS name, and clients connect directly. This keeps the deployment small and avoids an extra moving part, but the trade-offs are real:

  • Slow failover. DNS caches keep clients dialing dead addresses for minutes after a host fails.
  • No health-checking. Every host stays in rotation regardless of state. Sessions can land on nodes that will not serve them.
  • No traffic shaping. Removing a host for upgrades or debugging means editing DNS and waiting for propagation.

DNS round-robin is reasonable for evaluation, single-region lab clusters, or environments where an L4 load balancer is not available. Use a load balancer for any production deployment that needs sub-minute failover or maintenance windows.

DNS​

AlmaForge needs a stable client-facing hostname. Use it in the TLS certificate SAN, ALMA_ADVERTISE, and IdP redirect URI. This guide also uses it as the cluster name for readability, but ClusterName is a separate, immutable identifier. Changing an endpoint later requires coordinating DNS, certificates, IdP configuration, and agent connections.

The minimum DNS records:

RecordPurpose
almaforge.example.com A / AAAAPublic host addresses for DNS round-robin, or the load balancer address if you use one. Inside a private DNS zone for internal-only deployments.
*.almaforge.example.com A / AAAAWildcard subdomain when serving applications.

See Networking Guide for wildcard DNS behavior and advertise-vs-listen details.

TLS certificates​

AlmaForge terminates TLS on every host. Use the same certificate and private key on all three. The cert must include:

  • CN=almaforge.example.com or a subject your CA policy accepts.
  • SAN: almaforge.example.com
  • SAN: *.almaforge.example.com when serving application subdomains.

Source the certificate from your organization's internal CA or a public CA. Request the hostname SAN and any application SANs explicitly. A wildcard SAN covers application subdomains, not the public hostname itself.

Let's Encrypt with multiple hosts

The built-in ACME path keeps challenge state on the host requesting the certificate. In this multi-host topology, a TLS-ALPN-01 challenge can reach a different host and fail. Supply certificates through your central certificate-management process. For Let's Encrypt, use DNS-01 issuance there and distribute the resulting certificate and key to every host. See TLS configuration.

Store the cert and key at /etc/almaforge/tls.crt and /etc/almaforge/tls.key on every host. Step 2 copies them with the required owner and mode before the daemon starts.

After renewal, distribute the new certificate and matching key to every host, then roll the hosts one at a time. To reload certificate files without a restart, configure ALMA_TLS_RELOAD_INTERVAL first. Verify the certificate served by each host. See operator-supplied certificates.

Workstations must trust the CA that signed the certificate before signing in via CLI Setup. The Troubleshooting section has the per-OS commands.

SSO​

AlmaForge authenticates operators through your existing OIDC provider, such as Google Workspace, Okta, Entra ID, Auth0, Keycloak, or any other compliant IdP. Provision the OIDC application on the IdP side now so the credentials are ready when you write almad.env.

You need three values from the IdP:

ValueGoes intoNotes
Issuer URLspec.issuerURL in 10-sso.yamlThe IdP's OIDC discovery base. Examples: https://accounts.google.com, https://<tenant>.okta.com, https://login.microsoftonline.com/<tenant-id>/v2.0.
Client IDALMA_SSO_CLIENT_ID in almad.envPublic identifier for the OIDC app.
Client secretALMA_SSO_CLIENT_SECRET in almad.envTreat as a secret. The connector reads it with ${env:ALMA_SSO_CLIENT_SECRET}.

Register the redirect URI on the IdP side using the public hostname you configured in DNS: https://almaforge.example.com/api/v1/oidc/callback.

AlmaForge derives the same URL from ALMA_ADVERTISE at runtime. A mismatch between what you register on the IdP and what ALMA_ADVERTISE produces surfaces at sign-in as redirect_uri_mismatch.

See the OIDC connector guide for full claim mapping, scopes, PKCE, and provider-specific setup. Per-IdP walk-throughs: Google, Okta, Microsoft.

Provision AlmaForge​

Each host runs one almad process with both Auth and Proxy services enabled. Steps 1-6 install the package, copy configuration, and start systemd on every host. You can run those steps with configuration management or repeat them host by host. Step 7 runs from an administrator workstation.

Step 1. Install the package​

On every host, install AlmaForge from the signed APT or RPM repository for your distribution. The repos let you pin versions, mirror internally, and verify GPG signatures the package manager already trusts. The signed binary tarball is the right choice for minimal images without a package manager. The install.sh tab is a quicker path for demo environments.

These instructions apply to APT-based (Ubuntu and Debian) distributions and their derivatives.

Supported APT-based Linux distributions

The APT package supports these distribution families. Check their upstream maintenance windows when choosing a production host:

DistributionCodenameVersionRelease DateUpstream EOL
UbuntuJammy22.04 LTSApril 2022April 2027
UbuntuNoble24.04 LTSApril 2024April 2029
DebianBookworm12June 2023July 2026 (regular), June 2028 (LTS)
DebianTrixie13August 2025August 2028 (regular), June 2030 (LTS)

See Debian release support for the distinction between regular support and LTS.

Other distributions with glibc 2.28+ may also work. Please contact us if you run into any issues.

The snippet below configures the https://get.almaforge.com repository and installs the almaforge package from the repository:

Terminal
sudo install -m 0755 -d /etc/apt/keyringscurl -sfLS https://get.almaforge.com/almaforge.gpg | \    gpg --dearmor | sudo tee "/etc/apt/keyrings/almaforge.gpg" >/dev/nullecho "deb [signed-by=/etc/apt/keyrings/almaforge.gpg] https://get.almaforge.com generic main" | \    sudo tee "/etc/apt/sources.list.d/almaforge.list"sudo apt-get updatesudo apt-get install -y almaforge
Required tools

curl, gpg, and sudo are often missing in stripped-down images.

Verify that almad is installed:

Terminal
almad versionAlmaForge v2026.1.26+g5ab8dcffcb
Downloadable artifacts

The installer creates the almad daemon binary, the alma admin/client binary, the almad systemd service, and the /var/lib/almaforge directory for runtime state. The daemon runs as root. It does not start the service yet. The remaining steps add configuration. You enable systemd at the end.

Step 2. Write the env file​

/etc/almaforge/almad.env carries process-local settings: which services run, which addresses bind, where state lives, and any secrets the drop-ins interpolate. See Environment Configuration for every supported ALMA_* variable, and Infrastructure as Code for how the env file relates to drop-ins and API-managed resources.

Create the config directories before writing the env file:

Terminal
sudo install -d -o root -g root -m 0700 /etc/almaforge /etc/almaforge/almad.d

almad.env, copied to /etc/almaforge/almad.env on every host:

examples/linux/cluster/etc/almaforge/almad.env
# Environment file for an AlmaForge cluster node.
# Runs both Auth and Proxy services in one process. Same file on
# every host in the 3-node default cluster. Place at
# /etc/almaforge/almad.env. chown root:root, chmod 600.

ALMA_AUTH_SERVICE=true
ALMA_PROXY_SERVICE=true

# The cluster nodes are also reachable as SSH targets.
ALMA_SSH_SERVICE=true

# Public Proxy listener (TCP 443, multiplexes everything end users
# see). Clients reach almad through DNS round-robin, or through an
# optional L4 load balancer. The packaged systemd unit runs as root,
# which can bind 443. If you run unprivileged, grant
# CAP_NET_BIND_SERVICE or bind a high port behind a load balancer.
ALMA_LISTEN=0.0.0.0:443

# Public hostname clients connect to. Must match the TLS cert and DNS.
# Point DNS at the host addresses, or at a load balancer if you add one.
ALMA_ADVERTISE=almaforge.example.com:443

# Public TLS certificate and key. Distribute the same files to every
# host. Roll the cluster one host at a time on renewal.
ALMA_TLS_CERT=/etc/almaforge/tls.crt
ALMA_TLS_KEY=/etc/almaforge/tls.key

# Private diagnostics for health checks and operator checks.
# Firewall this port to monitoring systems and any load balancer you add.
ALMA_DIAG_LISTEN=0.0.0.0:3000

# Private Prometheus metrics endpoint. Firewall to monitoring only.
ALMA_METRICS_LISTEN=0.0.0.0:3081

# Static labels attached to this cluster node in inventory and audit.
ALMA_LABELS=env=prod,team=platform

# Interpolated by the OIDC connector under /etc/almaforge/almad.d/.
ALMA_SSO_CLIENT_ID=<paste-from-your-IdP>
ALMA_SSO_CLIENT_SECRET=<paste-from-your-IdP>

The env file points ALMA_TLS_CERT and ALMA_TLS_KEY at the shared certificate and key from TLS certificates. Copy them to the referenced paths on every host:

Terminal
sudo install -o root -g root -m 0600 tls.crt /etc/almaforge/tls.crtsudo install -o root -g root -m 0600 tls.key /etc/almaforge/tls.key

ALMA_AUTH_SERVICE=true and ALMA_PROXY_SERVICE=true enable both services in the same process. ALMA_LISTEN is the public Proxy multiplex listener on 443. ALMA_ADVERTISE is the public hostname advertised to clients and agents.

Keep the env file readable and writable only by root:

Terminal
sudo chown root:root /etc/almaforge/almad.envsudo chmod 0600 /etc/almaforge/almad.env

Step 3. Add storage drop-ins​

AlmaForge reads bootstrap resources from /etc/almaforge/almad.d at process start. This step writes one StorageConfig per slot for state, audit, and recordings. Each manifest points at a backend you provisioned in External storage. See Infrastructure as Code for how drop-ins, env vars, and API-managed resources fit together.

Before storage, pin the cluster's identity. ClusterName is the stable identifier clients see. It is separate from the public endpoint and does not have to match the public TLS certificate SAN. It is immutable once the first host finishes initializing cluster state, so skipping this step locks the cluster name to the first node's hostname for the life of the deployment. Save almad.d/00-cluster-name.yaml as /etc/almaforge/almad.d/00-cluster-name.yaml on every host, with clusterName set to the public hostname:

examples/linux/cluster/etc/almaforge/almad.d/00-cluster-name.yaml
# Cluster identity. Set this before first start. The values become
# immutable once the first host initializes cluster state. Changing
# them later requires re-bootstrapping the cluster.
# Place in /etc/almaforge/almad.d/ on each host.
apiVersion: almaforge.com/v1
kind: ClusterName
metadata:
  name: cluster-name
spec:
  # Public cluster hostname shown to users and embedded in issued certs.
  clusterName: almaforge.example.com

If your state backend uses mTLS (etcd is the typical case), copy the client CA, certificate, and key to every host at the paths the manifest references. See the etcd storage guide for the certificate profile.

Save almad.d/01-state-storage.yaml as /etc/almaforge/almad.d/01-state-storage.yaml on every host. Numeric prefixes are a convention for readability. almad figures out the actual load order itself. State must be available before any other resource touches the cluster.

examples/linux/cluster/etc/almaforge/almad.d/01-state-storage.yaml
# Cluster state backend: external etcd cluster reached over mTLS.
# Replace endpoints with your provisioned etcd, or swap spec.etcd for
# spec.dynamo / spec.firestore. See storage/index.mdx for the matrix.
# Place in /etc/almaforge/almad.d/ on each host.
apiVersion: almaforge.com/v1
kind: StorageConfig
metadata:
  name: state
spec:
  etcd:
    # Client URLs for every etcd member. These must be reachable from
    # every AlmaForge host over the private network.
    peers:
      - https://etcd-1.example.internal:2379
      - https://etcd-2.example.internal:2379
      - https://etcd-3.example.internal:2379
    # Key prefix that keeps AlmaForge data separate from other etcd users.
    prefix: /almaforge/
    # Client mTLS material for the etcd client API.
    tlsCAFile: /etc/almaforge/etcd-ca.crt
    tlsCertFile: /etc/almaforge/etcd-client.crt
    tlsKeyFile: /etc/almaforge/etcd-client.key

Add the audit and recording StorageConfig manifests before first start. The example bundle uses one default backend for each slot:

Copy each to every host and keep the stable filenames 02-audit-storage.yaml and 03-recording-storage.yaml. Substitute the values for your environment.

For alternative backends, see the per-driver guides: DynamoDB, Firestore, GCS, Azure Blob, and Athena (in the Storage Guide). Swap the default file with the driver-specific YAML from those pages.

After Step 3, the drop-in tree on every host is:

text
/etc/almaforge/almad.d/
|-- 00-cluster-name.yaml
|-- 01-state-storage.yaml
|-- 02-audit-storage.yaml
`-- 03-recording-storage.yaml

Use the StorageConfig reference when you need the full schema.

Step 4. Add the OIDC connector​

AlmaForge authenticates operators through your existing SSO provider. A fresh cluster has no administrator session yet, so you cannot bootstrap the first connector with alma apply -f. Add the connector as a startup drop-in before first start. When the cluster becomes healthy, the first administrator can sign in immediately.

almad.d/10-sso.yaml:

examples/linux/cluster/etc/almaforge/almad.d/10-sso.yaml
# OIDC connector loaded from almad.d at startup. Replace the
# claim/value pair with the SSO claim that should grant the first
# administrator access.
#
# Place in /etc/almaforge/almad.d/ on each host.
# Replace the issuerURL placeholder below with your IdP's OIDC
# discovery base. ${env:ALMA_SSO_CLIENT_ID} and
# ${env:ALMA_SSO_CLIENT_SECRET} are interpolated from
# /etc/almaforge/almad.env.
apiVersion: almaforge.com/v1
kind: OIDCConnector
metadata:
  name: corp
spec:
  issuerURL: <paste-from-your-IdP>
  clientID: ${env:ALMA_SSO_CLIENT_ID} # read from environment (almad.env)
  clientSecret: ${env:ALMA_SSO_CLIENT_SECRET} # read from environment (almad.env)
  scope:
    - openid
    - email
    - profile
    # Uncomment for Microsoft Entra, Okta, and other providers.
    # Google rejects this scope. Use googleAdminEmail and
    # googleServiceAccount instead.
    # https://almaforge.com/sso/google/
    # - groups
  claimsToRoles:
    # Map IdP groups to AlmaForge roles. Every claimsToRoles entry
    # is evaluated on every sign-in, and the user receives the union
    # of all matched roles.
    #
    # `claim` is the OIDC claim name as your IdP emits it. `groups`
    # is the convention for Okta, Entra ID, Keycloak, and Auth0.
    # If your IdP does not emit groups, map `email` or another claim
    # that your IdP includes in the ID token.
    #
    # `value` has to match what the provider actually sends, which is
    # often not the name you see in its admin console:
    #
    #   Okta              group-name
    #   Microsoft Entra   GUID of the group
    #   Google            group-name@example.com
    #
    # `roles` is a list of AlmaForge role names. Built-in roles are
    # `admin` (everything), `auditor` (read-only plus audit),
    # `access` (use resources), `member` (see and request access),
    # and `reviewer` (review access requests). Define custom roles as
    # additional drop-ins under almad.d/.
    - claim: groups
      value: admins
      roles:
        - admin
    - claim: groups
      value: auditors
      roles:
        - auditor

Save the manifest to /etc/almaforge/almad.d/10-sso.yaml on every host. Set ALMA_SSO_CLIENT_ID and ALMA_SSO_CLIENT_SECRET in /etc/almaforge/almad.env on every host. The manifest reads them with ${env:ALMA_SSO_CLIENT_ID} and ${env:ALMA_SSO_CLIENT_SECRET}.

Register https://<ALMA_ADVERTISE>/api/v1/oidc/callback as the redirect URI on the IdP side, with <ALMA_ADVERTISE> set to the public hostname from DNS. AlmaForge generates the same URL from ALMA_ADVERTISE at runtime and does not require it in the connector spec. A mismatch surfaces as an IdP-side redirect_uri_mismatch error at sign-in.

claimsToRoles is the bridge between your IdP and AlmaForge RBAC. The example maps two IdP groups to built-in roles: admins to admin (every resource, including the roles and connectors that decide who else has access) and auditors to auditor (read-only plus audit). The other built-in roles are member, access, and reviewer. See Built-in roles. Replace admins and auditors with the group names your IdP actually emits.

The connector's metadata.name is corp. It shows up as the choice on the browser sign-in page when an operator runs alma login (see CLI Setup).

If you add or change this connector after startup, keep it in almad.d and update the file on every host. Connector manifests reload automatically. Changes to secrets in almad.env require a rolling restart because the process reads its environment at startup.

Step 5. Pre-flight check​

Before starting the daemon, verify the filesystem modes are correct. This is the final check before first start for the package-created paths and the files you copied into place. At minimum (see Hardening Guide for the production-grade checklist):

PathOwnerModePurpose
/var/lib/almaforgeroot:root0700Local daemon state and host identity.
/etc/almaforgeroot:root0700Config root, holding the env file and any TLS material.
/etc/almaforge/almad.d/root:root0700Resource drop-ins.
/etc/almaforge/almad.envroot:root0600Env vars and interpolated secrets.
/etc/almaforge/tls.key, etcd-client.key, audit/recording backend credential filesroot:root0600TLS and storage backend client keys.

Confirm with stat. Every entry should be owned by root:root, with mode 0700 on the three directories and 0600 on every file:

Terminal
sudo stat -c '%a %U:%G %n' \    /etc/almaforge \    /etc/almaforge/almad.d \    /etc/almaforge/almad.env \    /etc/almaforge/tls.crt \    /etc/almaforge/tls.key \    /etc/almaforge/almad.d/*.yaml
Substitute the example hostname everywhere

The bundled almad.env, 00-cluster-name.yaml, OIDC connector, and storage manifests ship with almaforge.example.com and placeholder backend endpoints. Every occurrence must be replaced with your real public hostname from DNS and your real backend endpoints. Missing one of these substitutions surfaces later as TLS hostname mismatches, a wrong immutable cluster name, or a state-backend dial failure.

Terminal
sudo grep -rn 'example\.com\|example\.internal\|<paste-from' /etc/almaforge/

The command should return no matches. Any hit means a placeholder slipped through and the daemon will fail to start or sign-in will fail later.

Step 6. Start the daemon​

Terminal
sudo systemctl enable --now almadsudo systemctl status almad

At this point the env file, storage drop-ins, and OIDC connector are on every host. The first host to start initializes cluster state in the backend: the cluster identity from 00-cluster-name.yaml, the certificate authorities, and the rest of the drop-in resources. No operator session exists yet. That comes next.

Bring up the remaining hosts one at a time and confirm each becomes healthy. Each detects existing cluster state and joins as another peer.

Verify cluster health from any host. Do not use normal alma reads yet. Without a logged-in identity, they fail. Use systemd, the journal, and the local diagnostic endpoint instead:

Terminal
systemctl is-active almadactivesudo journalctl -u almad -n 50 | grep -E "is starting on|registered Proxy"curl -fsS http://localhost:3000/readyz{"status":"ok"}

Repeat on every host. The cluster is healthy when each host reports active, the journal shows the Proxy and Auth listeners starting, and /readyz returns 200. If any host fails one of these checks, jump to Troubleshooting before continuing. End-to-end verification comes next, in Step 7.

Production deployments should also wire the diagnostic and metrics endpoints into your observability stack. See Monitoring for Prometheus scrape targets and alert recipes.

Step 7. Verify the cluster​

Verify from an administrator workstation so the check covers DNS, TLS, the load balancer or DNS round-robin, SSO, and RBAC. If you used an internal CA, make the workstation trust the CA that signed the public TLS certificate before signing in. Then install alma from CLI Setup and sign in through the connector from Step 4:

Terminal
alma login --proxy=almaforge.example.com:443

After login, read the resources the bootstrap drop-ins created:

Terminal
alma clusteralma get oidcconnector/corpalma get storageconfig

The commands should show the cluster named almaforge.example.com, the corp OIDC connector, and the state, audit, and recordings storage instances. If login succeeds but the read commands return forbidden, see Login succeeds but read commands return forbidden.

Connect resources​

The cluster is now ready to host infrastructure access. Add agents for each kind of target:

Each agent is another almad process with different service flags. See the per-access guides above for Token resources and configuration.

Operations​

After the cluster is running, the Operations guide covers the day-2 work:

  • Routine maintenance. Rolling upgrades, certificate rotation, scaling, and adding nodes.
  • Observability. Monitoring targets for /readyz, /healthz, and the Prometheus metrics endpoint.
  • Incident response. Break-glass access when SSO is down, plus access incident playbooks.
  • Data lifecycle. Backups, restore drills, and cluster decommissioning.

Access-request approvals integrate with Slack, PagerDuty, and Jira so requests, approvals, and notifications stay in the tools your team already uses.

Troubleshooting​

Certificate errors at the public hostname​

The TLS certificate on each host must cover almaforge.example.com and any application subdomains you serve. Replace the certificate and matching key on every host, then open a new TLS connection to each host to verify the renewed certificate. Roll the hosts after renewal unless ALMA_TLS_RELOAD_INTERVAL is configured. See operator-supplied certificates. Internal cluster CA rotation is a separate operation.

alma fails with certificate signed by unknown authority​

The workstation does not trust the CA that signed the cluster's TLS certificate. Install the cluster CA into the OS trust store:

Terminal
sudo cp ca.crt /usr/local/share/ca-certificates/almaforge-ca.crtsudo update-ca-certificates

Equivalent commands: update-ca-trust on Red Hat, security add-trusted-cert on macOS. A short-lived workaround for scripts is SSL_CERT_FILE=/path/to/ca.crt alma ..., which the error message itself suggests.

Load balancer health checks fail​

Skip this section if you use DNS round-robin. For a load-balanced deployment, the load balancer cannot reach a node's diagnostic endpoint, or the node is not ready. Set ALMA_DIAG_LISTEN on every host first. Then confirm /readyz returns 200 on each host and that the load balancer health-check port matches the address you bound ALMA_DIAG_LISTEN to (commonly :3000).

install.sh hangs at the package download step​

The host cannot reach https://get.almaforge.com to fetch the binary. Common causes:

  • The host has no outbound HTTPS path (private subnet without a NAT gateway, restrictive egress firewall, or air-gapped network).
  • A transparent proxy is intercepting TLS and presenting an untrusted CA. curl reports SSL certificate problem.
  • A corporate DNS resolver does not have https://get.almaforge.com.

Verify with:

Terminal
curl -fsSL -m 10 https://get.almaforge.com -o /dev/null -w '%{http_code}\n'

A 200 means the host can reach the repo. Anything else (timeout, Connection refused, certificate error) is the cause. Fix the egress path before retrying the installer.

almad fails to start with cannot connect to <state-backend>​

The host cannot reach the state backend on the addresses in 01-state-storage.yaml. Common causes:

  • Wrong endpoint addresses, or the backend cluster is unhealthy.
  • The host's client cert is not signed by the backend CA, or the CA path in the storage manifest is wrong.
  • A firewall is blocking the backend client port between the hosts and the backend.

For etcd specifically, verify from the host:

Terminal
etcdctl --endpoints=https://etcd-1.example.internal:2379 \    --cert=/etc/almaforge/etcd-client.crt \    --key=/etc/almaforge/etcd-client.key \    --cacert=/etc/almaforge/etcd-ca.crt \    endpoint health

For other backends, use the vendor's health-check tool against the endpoints in 01-state-storage.yaml.

alma cluster shows a hostname instead of your domain​

The cluster name is the first host's hostname. You skipped 00-cluster-name.yaml before first start, so the cluster adopted the first host's hostname and ClusterName is now immutable. There is no in-place rename. A different cluster name is not itself a TLS endpoint error. If the name must change, plan a new-cluster cutover and preserve backups before decommissioning the existing cluster. Set 00-cluster-name.yaml before starting the replacement.

Host starts but alma get proxy shows nothing​

The Proxy service started but is not advertising itself. Check:

  • ALMA_ADVERTISE is a hostname clients can resolve.
  • Look for registered Proxy lines in journalctl -u almad.

Sign-in fails with redirect_uri_mismatch​

The IdP's registered redirect URI does not match what almad produces from ALMA_ADVERTISE at runtime. The cluster generates https://<ALMA_ADVERTISE>/api/v1/oidc/callback. Register that exact URL on the IdP side. Trailing slash, port presence, and http vs https must all match. See Step 4 for the IdP-side registration.

The sign-in page shows no connector to pick​

The sign-in page renders one button per active connector, so an empty page means the cluster holds none. List them:

Terminal
sudo alma get oidcconnector

Run the command on a cluster host, where --auth-server can use the local admin identity. If the drop-in was applied but the cluster does not see it, verify the file is under /etc/almaforge/almad.d/ on every host and that apiVersion and kind name a connector resource. Connector changes reload automatically from the watched directory. Check the journal for reload errors. See OIDC connector guide for the full schema.

Login succeeds but read commands return forbidden​

The assigned roles do not grant the requested read operation. Inspect the current identity before changing the connector:

Terminal
alma status

status shows the resolved roles for the current session. Confirm that claimsToRoles maps the intended IdP claim (often email or groups) to the appropriate administrator role, then sign in again after any mapping change. See Built-in roles for available role targets and Claims, traits, and role templates for how OIDC claims reach roles.

References​

  • Declarative Configuration: full schema for StorageConfig, OIDCConnector, Token, Role, and target kinds.
  • Environment Configuration: every supported ALMA_* variable.
  • CLI Reference: alma and almad subcommands and flags.
  • Storage Guide: backend selection, etcd cluster layout, PostgreSQL sizing, DynamoDB IAM, audit and recording retention.
  • Networking Guide: load balancer reference, advertise vs listen, and wildcard DNS.
  • CLI Setup: installing alma on operator workstations and OS trust-store commands for the cluster CA.
  • Backups: restore drills and capacity planning.
  • Break-glass: recovery procedures when SSO is broken.
  • Upgrades: rolling upgrade flow and release-notes links.
  • Monitoring: Prometheus scrape targets and alert recipes.
  • Hardening Guide: production-grade filesystem modes, mTLS, secrets, session limits, and Web UI hardening.
  • Support plans: support plans, response times, severity levels, and professional services.