Skip to main content

Break-Glass Access

Break-glass access is the operator-of-last-resort path into a AlmaForge cluster when the normal sign-in flow is unavailable: SSO outage, expired admin credentials, or locked-out roles. Auth and its state backend must still be available.

There are two recovery paths, both authorized by the local admin identity on an Auth host and usable only by an operator with shell access to that host:

  • Command-line administration by running alma on an Auth host, for resource inspection, resource changes, and locks. This is the bulk of this page.
  • A browser administrator session with alma login --breakglass, for reaching the Web UI to configure the first SSO connector or repair a broken one. See Recover Web UI access.

Run alma on the host where almad runs. It loads the local admin identity from the data directory and talks directly to Auth, bypassing the Proxy Service. Use it only to recover the cluster control plane. It is meant for resource inspection and resource changes, not for SSH or SCP into managed nodes.

On the Auth host the client finds Auth at the default address automatically. If Auth listens on a non-default address, point the client at it with --auth-server=<host:port>. --auth-server is a generic address override, not an emergency-access mode of its own.

Before the incident​

Break-glass only works when access to a running Auth host and its local admin identity already exists.

  • At least two operators can SSH to every Auth host and use sudo to read /var/lib/almaforge.
  • The cluster CA fingerprint is recorded outside the cluster.
  • This runbook URL is linked from the incident response page.
  • A staging break-glass drill has run in the last quarter.
  • Auth host filesystem controls follow the Hardening Guide.

Break-glass requires a running Auth host. If every Auth host is down, first restore an Auth Service using the existing state backend. Restore the backend from backup only if its data is unavailable or unusable. See Backups.

TL;DR​

SSH to the Auth host and run the client with sudo so it can read the root-owned data directory:

Terminal
sudo alma get nodes

If the Auth Service uses a non-default data directory, point the client at that directory:

Terminal
sudo alma --data-dir=<data-directory> get nodes

Or set the environment variable for the command:

Terminal
sudo env ALMA_DATA_DIR="<data-directory>" alma get nodes

The default data directory is /var/lib/almaforge.

Working recovery paths​

Direct local-admin CLI access​

Use direct break-glass access when the Proxy is unavailable, SSO is broken, or no normal login profile exists on the host. The client loads the local admin identity from the Auth data directory and connects to Auth directly.

This mode is for administrative resource work with commands such as get, apply, and delete. SSH and SCP require a Proxy connection, so they are not available in break-glass CLI access.

Running on the Auth host, the client discovers the local admin identity and the local Auth address automatically, so the plain command works with no extra flags. Add --auth-server=<host:port> only when Auth listens on a non-default address.

In a multi-Auth cluster, choose any reachable Auth host. The local admin identity is per host, but resource reads and writes go through the shared cluster state backend.

Recover Web UI access​

Use alma login --breakglass when you need the Web UI itself, not the command line: to configure the first SSO connector on a new cluster, or to repair a broken connector when SSO sign-in is failing.

Run it on an Auth host:

Terminal
sudo alma login --breakglass
Break-glass recovery requested. Open this link within five minutes:
    https://example.com/login#breakglass=<claim>
It can be used once and opens a short administrator session as "breakglass".

The command authenticates with the same on-host admin identity as break-glass CLI access, mints a one-time recovery link, and exits. It writes no login profile and issues no certificates. Open the link in a browser, confirm once on the "Continue as breakglass" page, and the browser receives a short administrator session as the built-in breakglass user. Use it to open the connector editor and configure SSO.

A browser cannot start this flow. Only a caller holding the on-host admin identity can, so shell access to an Auth host is the authority. The link carries a one-time secret in its fragment. Open it yourself, do not forward it, and prefer a browser that does not sync history to a third party.

The recovery session is capped by the built-in admin role's session limit and cannot be renewed past it. If recovery takes longer than the cap, run the command again for a fresh link. To cut off a live session, lock the breakglass user. See Cut off compromised access.

Cut off compromised access​

Use locks when break-glass is part of an incident response and an account, role, login, access request, or server needs to be blocked without waiting for certificate expiry. Enforcement starts when the lock reaches the service handling the access, typically within seconds over a healthy Auth connection.

Lock a user:

Terminal
sudo alma create lock \  --target-user=<name> \  --message="<reason>"

Set an expiry with --expires=<rfc3339-time> or --lock-ttl=<duration> when the lock should lift automatically. For --expires, use UTC with a Z suffix, for example 2026-07-01T09:00:00Z. For the broader revocation runbook, see Revoking access.

Drill in staging​

Run this drill quarterly in staging and record the operator, host, time, and commands in the drill evidence.

Terminal
ssh <operator>@<staging-auth-host>sudo alma get nodessudo alma create token --services=ssh --expires=5msudo alma create lock \  --target-user=drill-user \  --message=drill \  --lock-ttl=5msudo alma get lock

After five minutes, confirm the drill lock has expired:

Terminal
sudo alma get lock

Guardrails​

Break-glass access is controlled by filesystem access to the Auth data directory. Anyone who can read /var/lib/almaforge can use the local admin identity through break-glass CLI access. Restrict access to the Auth host, protect the data directory, and review the filesystem baseline in the Hardening Guide.

On VM and bare-metal Auth hosts, the break-glass path needs /var/lib/almaforge/host_uuid and /var/lib/almaforge/proc/sqlite.db. The admin identity is the ids/admin/current record inside the process store, not a plaintext file to inspect or copy during a drill.

Break-glass CLI access has no dedicated audit event. Treat every use as a manual security event. Alert on Auth host shell access and sudo during incidents, then review follow-on audit events generated by the resource changes. Creating locks emits LockCreate. Deleting locks emits LockDelete.

The alma login --breakglass browser flow does emit login events for the breakglass user, so a recovered Web session is recorded in the audit log at both initiation and sign-in.

After break-glass​

Before closing the incident:

  1. Preserve shell history from the operator session.
  2. Preserve Auth daemon logs for the incident window.
  3. Export audit events for the incident window before normal access is restored.
  4. Preserve relevant filesystem snapshots if the Auth host or data directory might have been exposed.
  5. Review locks and remove any temporary lock whose incident purpose is complete.
  6. Confirm normal login works.
  7. Capture the reason, commands run, evidence locations, and follow-up work in the incident record.

For credential rotation after emergency access, see CA Rotation.

References​