Skip to main content

AlmaForge Server Access Guide

Give engineers SSO-backed SSH access to Linux servers across cloud and on-prem environments with AlmaForge. Users sign in once through SSO and use a short-lived certificate from the cluster CA for alma ssh connections. An agent on each host dials out to the Proxy and brokers connections through a reverse tunnel, so hosts stay off the public internet. This brings SSH into the same control plane as Kubernetes, applications, and databases:

  • Recorded terminals, tied to a real person. PTY output is captured for terminal playback through alma recording play, alongside structured JSON audit events keyed to the user's SSO identity. Interactive sessions emit session events, while non-interactive commands emit SessionExec events with the command and outcome. Capture follows your recording policy.
  • SSO mapped to OS logins: keep your native tools. Users sign in with SSO and connections use their short-lived certificate from the cluster CA. No user authorized_keys to rotate, no bastion key, no shared password. The native ssh, scp, rsync, and IDE Remote-SSH stack still works through an OpenSSH ProxyCommand.
  • Just-in-time elevation for sensitive hosts. Engineers request time-bounded access to production through the CLI or web UI. Slack notifies reviewers, and PagerDuty can supply on-call context. Approval grants a role for a fixed window. The cert expires when the window ends. See Access Requests.
  • SSH for CI/CD and AI agents without shared join secrets. Workload identity issues short-lived certs to pipelines, jobs, and agents with the same RBAC and recording controls. Provider-backed joins avoid shared bootstrap tokens. See Workload Identity.
  • Self-hosted, no telemetry, runs in air-gapped networks. The access plane runs inside your boundary. Agent hosts dial out to the Proxy through a reverse tunnel and need no public IP or inbound connection from it. See agentless OpenSSH for hosts that use their existing SSH server.

Architecture​

Server access uses three main components.

almad is the server daemon. One binary exposes every server-side role. The roles used for SSH access are:

  • almad --auth-service runs the Auth Service. It issues short-lived certificates, stores cluster state, and is the source of truth for identities, roles, and registered hosts. It stores the cluster's persistent state.
  • almad --proxy-service runs the Proxy Service, the public TLS endpoint at almaforge.example.com:443. It handles browser sign-in and brokers SSH connections through reverse tunnels to agents. By default, SSH sessions terminate at the host-side agent.
  • almad --ssh-service runs the SSH Service on each Linux host. It dials out to the Proxy, holds a reverse tunnel open, authenticates each inbound SSH connection against the cluster CA, performs the role check, opens the OS session, and writes the recording.

Roles compose freely. A small cluster can run --auth-service and --proxy-service in one almad process. Production deployments split roles across multiple instances and run Auth, Proxy, and SSH services in HA mode. See How it works for deployment shapes and the full role catalog.

alma is the CLI for users and workloads. It authenticates interactively through SSO (alma login) or non-interactively through workload identity. Once authenticated, alma ssh opens a session to a host, alma scp transfers files, and alma proxy ssh lets native OpenSSH clients route through the Proxy with an OpenSSH ProxyCommand stanza.

Server access topology

A user runs alma login before connecting. That uses the SSO flow and leaves alma with a short-lived certificate on disk. From there, alma ssh <login>@<host> authenticates with the user's SSH certificate. The Proxy routes the connection through the host's reverse tunnel. The host-side agent checks access and opens an OS session as the requested login, attaches the recorder, and forwards traffic in both directions. The session ends when the user disconnects.

For hosts where you cannot install almad, such as legacy systems, hardened images, or third-party appliances, register the existing OpenSSH server through the Proxy instead. See OpenSSH agentless mode.

Prerequisites​

  • A running AlmaForge cluster, with Auth Service and Proxy Service reachable at almaforge.example.com:443.
  • An SSO connector configured on that cluster, with at least one operator account that can sign in through the Proxy. See Single Sign-On for OIDC setup.
  • The alma CLI installed on an operator workstation with network access to the Proxy. See Install the CLI. The operator account must be able to apply Role and join-token resources. The admin preset role covers both.
  • A Linux host (Ubuntu 20.04+, Debian 11+, RHEL/CentOS 8+, or equivalent) with almad already installed and its systemd unit available. See Install AlmaForge.
  • Outbound network reachability from the host to the Proxy at almaforge.example.com:443. No inbound ports are required on the host.

The three steps below deploy the SSH Service on one host, grant a user access, and verify they can reach it. Optional configuration (host user auto-provisioning, agentless OpenSSH, auditd integration) lives in Advanced options.

Step 1. Deploy the SSH Service​

A join token authenticates the agent to the AlmaForge Auth Service the first time it starts. Its method block decides how the agent proves identity: a static random string, or a workload identity such as AWS IAM, a GCP service account, or an Azure managed identity. Workload-identity tokens carry no secret material, but they require the Auth Service to trust the cloud provider's identity signal, so they fit hosts running on a supported cloud.

Pick the join method that matches where the host runs.

A random string with a TTL. Useful for first install and for hosts without a supported workload-identity provider.

Terminal
alma create token --services=ssh

Default TTL is 30 minutes. The token is not single-use: every host you bring up during that window can consume the same value. Each agent trades the token for its own per-host identity, persisted under /var/lib/almaforge. Once a host has joined, the token is no longer used by it.

Override the TTL with --expires=24h for slower rollouts. Revoke a leaked or unused token with alma delete token/<client-id>. Pass --client-id=<name> to alma create token to set the resource name explicitly. The command then prints both client_id and client_secret. Set ALMA_CLIENT_ID and ALMA_CLIENT_SECRET on the host. You can also pass --client-id on the command line. Supply the secret through ALMA_CLIENT_SECRET, not a command argument.

--services=ssh already covers the SSH Service. To also serve applications or databases from the same almad process on this host, use the comma-separated form (alma create token --services=ssh,app,database) and set the matching ALMA_*_SERVICE toggle for each role.

Save the Proxy address, join client ID, client secret, SSH Service setting, and static labels in /etc/almaforge/almad.env. The systemd unit reads this file when almad starts. The file contains the client secret, so restrict access to root.

With ALMA_PROXY set, the agent joins through the Proxy and validates the Proxy's TLS certificate against the host's system trust store. Use a publicly trusted Proxy certificate or install its signing CA in the host's trust store before the join. This check protects the first connection, before the agent receives cluster keys.

Use ALMA_PROXY for the SSH agent in this guide. ALMA_CA_PIN applies to direct Auth joins, not Proxy joins, and ALMA_AUTH_SERVER cannot be set alongside ALMA_PROXY. See Hardening for the cluster's join trust configuration.

almad.env, a template to copy to /etc/almaforge/almad.env:

examples/servers/env/almad.env
# Environment file for the almad systemd unit.
#
# Path:        /etc/almaforge/almad.env
# Permissions: chmod 600 (the join token is sensitive)

# Proxy address (host:port).
ALMA_PROXY=almaforge.example.com:443

# Join token from `alma create token --services=ssh --client-id=<name>`.
# Default TTL: 30 minutes. The token is NOT single-use and stays
# valid for that window. After first connect, identity is persisted
# under /var/lib/almaforge and the token is no longer used.
# Revoke a leaked token with `alma delete token/<client-id>`.
ALMA_CLIENT_ID=paste-client-id-here
ALMA_CLIENT_SECRET=paste-client-secret-here

# A proxy join verifies the Proxy certificate against this host's
# system trust store. Use a publicly trusted certificate or install its
# signing CA in that store. ALMA_CA_PIN pins the Auth Service CA for a
# direct join selected by ALMA_AUTH_SERVER. If ALMA_PROXY is also set,
# the Proxy takes precedence and ALMA_CA_PIN is ignored.

# Enable the SSH Service role.
ALMA_SSH_SERVICE=true

# ALMA_LABELS sets operator-defined labels as comma-separated key=value
# pairs. They appear in audit events and `alma get nodes`. Use stable
# keys such as env, region, and team. Keep fast-changing values out of
# this setting. Render inventory labels into this value when you
# provision the host.
#
# The agent also advertises generated labels. It refreshes AWS and Azure
# tags from instance metadata once an hour and prefixes their keys
# with aws/ or azure/. Hetzner supplies hetzner/instance-id,
# hetzner/region, hetzner/availability-zone, hetzner/hostname, and
# hetzner/local-ipv4. Every host that sends heartbeats also advertises
# internal/expires. Role serverLabels matchers evaluate the full set.
ALMA_LABELS=env=prod,region=eu-central-1,team=sre

# Optional short identifier for this node (defaults to OS hostname).
# Must be unique within the cluster.
# ALMA_NODE_NAME=web-01
Terminal
chmod 600 /etc/almaforge/almad.envsystemctl enable --now almad

ALMA_LABELS sets operator-defined labels. Use stable keys such as env, region, and team, with values up to 63 characters of letters, digits, -, _ and ., as in env=production. Keep fast-changing values out of this setting. If labels come from an inventory system, render them into ALMA_LABELS when you provision the host.

The agent also advertises generated labels. On supported cloud instances, it refreshes labels from the instance metadata service once an hour. AWS and Azure instance tags appear with the aws/ or azure/ prefix, rewritten to the label character set: the AWS tag aws:cloudformation:stack-id arrives as aws/aws.cloudformation.stack-id. Write your matchers against the imported key. Hetzner metadata appears as hetzner/instance-id, hetzner/region, hetzner/availability-zone, hetzner/hostname, and hetzner/local-ipv4. Every host that sends heartbeats also advertises an internal/expires label containing the registration record's expiration time.

Matchers in Role.spec.allow.serverLabels evaluate the full label set on every connection. Imported keys work like keys from ALMA_LABELS. Named-key matchers evaluate only those keys. The "*": "*" selector matches every host.

Confirm the host completed the join handshake from the agent side:

Terminal
journalctl -u almad -n 50 | grep -Ei 'joined|registered'

Then confirm AlmaForge sees the host:

Terminal
alma get nodesHost    UUID                                  Public Address  Labels                                  Version------- ------------------------------------- --------------- --------------------------------------- -------web-01  3f0a6e2c-1d4b-4f3e-9c2a-5d7e8b9c0f1a                  env=prod,region=eu-central-1,team=sre   2026.1.26

The Public Address column is empty for tunneled hosts. The agent holds the reverse tunnel open and the Proxy reaches it through that tunnel. The host appears within one heartbeat (a few seconds). If it does not, see Troubleshooting.

Step 2. Grant a user access​

Two layers of authorization decide what a user does on a host. AlmaForge picks which hosts the user can reach and which OS logins they can assume via Role.serverLabels and Role.serverLogins. The OS decides what each login can do once AlmaForge has opened the session: file permissions, sudoers, capabilities. The host's local controls take over. AlmaForge never elevates a login past what the OS grants it.

An access denial can come from AlmaForge when no role selects the host or permits the requested login, or from the OS when that login lacks permission for a command.

Apply a Role and map an IdP group to it with the SSO connector's claimsToRoles. See OIDC for the full schema.

serverLogins and serverLabels values support trait templates ({{external.team}}, {{external.preferred_username}}, {{internal.logins}}), so a single role can scope each user to their own slice of the inventory based on IdP claims.

One role, every host, fixed OS logins. For demos, dev environments, or first-time setup. Tighten later.

ssh-basic.yaml:

examples/servers/roles/ssh-basic.yaml
# Quick-start SSH role. Use fixed OS logins on any registered host,
# then replace the wildcard label matcher before production use.
apiVersion: almaforge.com/v1
kind: Role
metadata:
  name: ssh-access
spec:
  allow:
    serverLabels:
      '*': '*' # Wildcard key and value matches every Node regardless of labels.
    serverLogins: # OS user accounts the role can assume on the target host.
      - ubuntu
      - ec2-user
  deny:
    serverLogins: # Hard block on root, even if another role grants it.
      - root

Map a workforce SSO group to that role:

ssh-sso-claims.yaml:

examples/servers/roles/ssh-sso-claims.yaml
# Map an IdP group to the SSH role through an OIDC connector.
# Replace okta with your connector name and engineers with the IdP
# group whose members should receive ssh-access.
apiVersion: almaforge.com/v1
kind: OIDCConnector
metadata:
  name: okta
spec:
  issuerURL: 'https://example.okta.com'
  clientID: '<client-id>'
  clientSecret: '<client-secret>'
  claimsToRoles:
    - claim: 'groups'
      value: 'engineers'
      roles: ['ssh-access']

Apply each YAML file from the tab you used:

Terminal
# Quick Startalma apply -f ssh-basic.yaml -f ssh-sso-claims.yaml
# SSO templatesalma apply -f ssh-sso.yaml

For the full Role schema, supported matchers, predicate syntax (serverLabelsExpression), and the complete trait-template surface, see the RBAC reference. For on-demand role escalation, see Access Requests.

Security note. A deny.serverLogins: [root] blocks the root login on every host the user can otherwise reach. It does not stop the user from running sudo -i once they are signed in as a permitted login. Block privilege escalation in OS sudoers, not in the AlmaForge role.

Step 3. Verify access​

After Step 1 registered the host and Step 2 gave the user a role that selects it, three entry paths reach the same SSH session:

  • alma ssh: the native AlmaForge client. Best for daily operator use, fan-out, file transfer, and port forwarding.
  • Native ssh through the Proxy: the standard OpenSSH client routed via alma proxy ssh as ProxyCommand. Best for IDEs, rsync, git+ssh, and anything that expects an ssh_config.
  • Web terminal: browser shell at the Proxy URL. Best for ad-hoc access from a laptop without alma installed.

All three honor the same role checks. If the user is rejected, the failure is at one of the two layers from Step 2. Proxy hides the host when no role's serverLabels selects it. The agent denies the login when the role's serverLogins does not include it. See Troubleshooting for the common causes.

Before using either CLI path, sign in:

Terminal
alma --proxy=almaforge.example.com:443 login

This authenticates against AlmaForge through SSO and writes a short-lived certificate to the alma profile directory. Browser sessions handle their own login through the same SSO flow.

Path 1: alma ssh​

AlmaForge CLI SSH flow

List the hosts the user can reach. The list is filtered by role: hosts whose labels do not match any role's serverLabels are hidden.

Terminal
alma get nodesHost    UUID                                  Public Address  Labels                                  Version------- ------------------------------------- --------------- --------------------------------------- -------web-01  3f0a6e2c-1d4b-4f3e-9c2a-5d7e8b9c0f1a                  env=prod,region=eu-central-1,team=sre   2026.1.26web-02  9b1f4d7a-2e6c-4a8b-b3d5-1e0f7c9a8b2d                  env=prod,region=eu-central-1,team=sre   2026.1.26db-01   2c8e7f1d-5b4a-4c9e-9f1d-7a2b6c0e5d8f                  env=prod,region=eu-central-1,team=dba   2026.1.26

Open an interactive shell, or run a one-shot command. The OS login (ubuntu here) must appear in the user's role serverLogins list. See Step 2. A login the role does not include is rejected at the Proxy.

Terminal
alma ssh ubuntu@web-01alma ssh ubuntu@web-01 'systemctl status nginx'

<login>@<label-expr> fans the command out across every matched host and prints results inline. This is the fastest way to answer fleet-wide questions without writing a playbook for a one-off:

Terminal
alma ssh ubuntu@env=prod 'uptime'alma ssh ubuntu@env=prod,team=sre 'systemctl is-active nginx'

Transfer files with alma scp, or point rsync at alma ssh:

Terminal
alma scp report.txt ubuntu@web-01:/tmp/rsync -avz -e "alma ssh" ./build/ ubuntu@web-01:/srv/app/

Forward ports with the usual SSH flags (-L, -R, -D):

Terminal
alma ssh -L 5432:localhost:5432 ubuntu@db-01

Unless a role sets sshFileCopy: true or portForwarding: true, the Proxy refuses the operation with a clear error.

Path 2: Native ssh for IDEs, rsync, git+ssh, Ansible​

Native ssh flow

Tools that expect a standard OpenSSH config, such as VS Code Remote-SSH, JetBrains Gateway, rsync, git+ssh, Ansible, and configuration management tools, work through the Proxy with one ProxyCommand stanza. After alma login, connect with the synthesized hostname:

Terminal
ssh ubuntu@web-01.almaforge.example.com

The OpenSSH stanza has this shape:

text
Host *.almaforge.example.com almaforge.example.com
    UserKnownHostsFile ~/.config/almaforge/known_hosts
    IdentityFile ~/.config/almaforge/profiles/almaforge.example.com/<sso-user>
    CertificateFile ~/.config/almaforge/profiles/almaforge.example.com/<sso-user>-ssh/almaforge.example.com-cert.pub
    HostKeyAlgorithms rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com

Host *.almaforge.example.com !almaforge.example.com
    Port 3022
    ProxyCommand alma proxy ssh --cluster=almaforge.example.com --proxy=almaforge.example.com:443 %r@%h:%p

The Host *.almaforge.example.com block wildcard-matches registered nodes and routes each match through alma proxy ssh. OpenSSH uses the short-lived key, certificate, and known-hosts file written by alma login. There is no DNS A record for web-01. The host name is <node-name>.<cluster-name>, synthesized by the Proxy from the registered node and the cluster's public address. It works the same way for scp, sftp, rsync, and IDEs that shell out to OpenSSH.

The cert's TTL is bound to the operator's AlmaForge login. When that session expires, ssh returns an authentication error and the operator re-runs alma login.

Path 3: Web terminal​

Web terminal flow

Open https://almaforge.example.com in a browser, sign in through SSO, and pick the host from the Servers list. The web terminal opens a full PTY in the browser, recorded the same way as a alma session.

This path needs no client install and no local cert. It is the fastest way to give a one-time approver, an auditor, or an on-call engineer a working shell from a fresh device.

Recordings from every path land in the cluster's configured session storage backend (local disk, S3, or GCS) and are owned by the cluster with no third party. Replay with alma recording play or from the web UI. See Storage Configuration for backends and retention controls.

Server access is working when the host appears in alma get nodes, the user's role selects it, and at least one of the three entry paths opens a recorded shell as an allowed OS login.

Advanced options​

The subsections below are optional features. Reach for them when you need a deployment shape the basic flow does not cover (legacy OpenSSH hosts), when role behavior needs more than label-and-login control (host user auto-provisioning, role options), or when the host needs to integrate with the Linux audit subsystem (auditd).

OpenSSH agentless mode​

OpenSSH agentless flow

Use this mode for hosts that already run sshd and where you cannot install almad (legacy systems, hardened images, third-party appliances). The Proxy brokers connections directly to sshd using short-lived certificates from the AlmaForge CA. No long-running agent is added to the host.

Tradeoffs vs. the SSH Service path:

  • Host labels are fixed by the registration record. The host cannot refresh them through an installed agent.
  • No auditd event injection on the host side.
  • The host's own sshd policy applies (PermitRootLogin, AllowUsers, etc.) on top of the AlmaForge role check.

If you have a choice, prefer the SSH Service. The agentless path is for hosts you genuinely cannot run an agent on.

The host needs OpenSSH sshd 7.4 or newer, with port 22 reachable from the AlmaForge Proxy.

Issue a node-typed token from your workstation:

Terminal
alma create token --services=ssh

Copy the almad binary to the host, then run the agentless join as root. The command updates the OpenSSH configuration to trust the AlmaForge CA, registers the host with the cluster, and writes the host identity under /var/lib/almaforge.

Keep the client secret out of command arguments. Arguments can appear in ps output and shell history. almad join openssh reads the secret only from ALMA_CLIENT_SECRET.

Create a root-owned file for the secret, set its mode to 0600, and use sudoedit to enter the secret without opening a root shell:

Terminal
sudo install -o root -g root -m 0600 /dev/null /root/alma-client-secretsudoedit /root/alma-client-secret

Pass the absolute path directly to sudo as an environment variable. The join restarts OpenSSH by default, using the systemd unit the host installs: sshd.service on Red Hat-family distributions and ssh.service on Debian-family ones. To schedule the restart yourself, add --no-restart-sshd to the command below.

Terminal
sudo ALMA_CLIENT_SECRET=/root/alma-client-secret \    almad join openssh \    --address=<openssh-host>:22 \    --proxy=almaforge.example.com:443 \    --client-id=<client-id> \    --join-method=static \    --labels=env=legacy,team=ops

Credentials created by alma create token use the static join method. Place ALMA_CLIENT_SECRET after sudo, as shown. The default sudo configuration discards variables exported before the command.

When ALMA_CLIENT_SECRET contains an absolute path, the command reads the file and trims its contents. Any other value, including a relative path, is treated as the secret itself.

The join runs once and exits. After it succeeds, the host has its own signed identity on disk. Delete /root/alma-client-secret when you no longer need it.

Confirm the join from your workstation:

Terminal
alma get nodes

The agentless host has an empty Version column because no AlmaForge process runs there to report a version. Connect to it as you would an SSH Service host:

Terminal
alma ssh <login>@<openssh-host>

The join prepends one line to /etc/ssh/sshd_config: Include /var/lib/almaforge/sshd.conf.

The rest of /etc/ssh/sshd_config remains unchanged. The included file points TrustedUserCAKeys, HostKey, and HostCertificate to paths under /var/lib/almaforge/openssh. The join stores the corresponding key material in that directory.

To remove the host, delete the prepended Include line and reload sshd. Then delete /var/lib/almaforge/sshd.conf and /var/lib/almaforge/openssh. Finally, remove the node record from your workstation:

Terminal
alma delete node/<host-id>

Auto-create host users​

By default, the OS login named in Role.serverLogins must already exist on the host. To skip that pre-provisioning step, enable host user auto-provisioning. The agent creates the OS account at session start, adds it to the listed hostGroups, writes any hostSudoers entries to /etc/sudoers.d, and (with createHostUserMode: drop) deletes the account at session end.

ssh-host-users.yaml:

examples/servers/roles/ssh-host-users.yaml
# SSH role with host user auto-provisioning. The agent creates the OS
# account at session start and, with mode=drop, deletes it at session
# end so you do not pre-create per-engineer logins on the host.
apiVersion: almaforge.com/v1
kind: Role
metadata:
  name: ssh-auto-users
spec:
  options:
    # drop: create at session start, delete at session end (transient).
    # keep: create at session start, leave the account in place after.
    # off:  do not auto-create. The login must already exist.
    createHostUserMode: drop
  allow:
    serverLabels:
      env: 'dev'
    serverLogins:
      - '{{external.preferred_username}}' # Per-user OS account, named from the IdP claim.
    hostGroups: # Groups the auto-created user is added to. Missing groups are created.
      - docker
      - developers
    hostSudoers: # Sudoers entries written under /etc/sudoers.d for the session.
      - 'ALL=(root) NOPASSWD: /usr/bin/systemctl restart nginx.service'

createHostUserMode accepts:

  • drop: create at session start, delete after the user's last session ends. Account cleanup does not undo changes made during a session.
  • keep: create at session start, leave the account in place after.
  • off: disable auto-creation (the default). The OS login must already exist.

Every role matching the target server has to opt in. An unset createHostUserMode counts as off, so one matching role that omits the field blocks account creation for the whole session. The built-in access role matches every server and omits it, so a user who holds it alongside the role above sees unknown user <login>. Scope the roles so only one matches, or set createHostUserMode on every matching role.

The agent invokes useradd, userdel, usermod, groupadd, getent, and visudo to manage users, so it must run as root on the host. The packaged systemd unit already does. Those binaries must be on the agent's $PATH. Each value of Role.serverLogins must conform to the host's username rules (see User/Group Name Syntax).

hostSudoers is a session-scoped privilege grant. Entries are written to /etc/sudoers.d for the duration of the session and removed at session end. They survive an agent crash and are cleaned up on next agent start. If you mint a sudoers rule that grants ALL=(root) NOPASSWD: ALL, you have given the user full root for that session. The cleanup at session end does not undo what they did with it.

Role options for SSH​

Role options apply per-session and override the cluster default. The fields below cover the controls operators tune most often. For the full list, see the RBAC reference.

ssh-options.yaml:

examples/servers/roles/ssh-options.yaml
# Reference SSH role exercising the role options operators tune
# most often. Trim to taste. Every option below has a sensible
# default, so this file is meant to be read field-by-field.
apiVersion: almaforge.com/v1
kind: Role
metadata:
  name: ssh-options-ref
spec:
  options:
    maxSessionTTL: 8h # Maximum credential lifetime.
    forwardAgent: true # Allow native SSH agent forwarding through the Proxy.
    portForwarding: true # Allow `-L`/`-R` TCP forwards.
    sshFileCopy: true # Allow SCP and SFTP. Omit to block file copy.
    clientIdleTimeout: 30m # Disconnect idle clients after this period. "never" disables.
    disconnectExpiredCert: true # End the session if the user's cert expires mid-session.
    maxSessions: 10 # Max channels per SSH connection. 10 matches OpenSSH.
    maxConnections: 0 # Cluster-wide concurrent session cap. 0 = no limit. Adds backend load when set.
    pinSourceIp: false # Embed the login IP in the cert. The cert refuses use from any other IP.
    recordSession:
      default: bestEffort # Fallback recording mode for protocols without an explicit setting.
      ssh: bestEffort # Per-protocol mode for SSH sessions.
  allow:
    serverLabels:
      env: ['dev', 'staging']
    serverLogins:
      - '{{external.preferred_username}}'

Notable behaviors:

  • portForwarding and sshFileCopy grant ssh -L/-R and SCP/SFTP. Neither is granted unless a role sets it to true, so a role that omits them allows interactive sessions only.
  • pinSourceIp: true binds the certificate to the operator's sign-in IP and refuses use from any other IP. Breaks if the IP changes mid-session (mobile networks, VPN reconnect).
  • recordSession.ssh: strict terminates the session if the recorder cannot write: full disk, lost connection to the storage backend. Use it on prod hosts where an unrecorded session is unacceptable. bestEffort keeps the session running and drops the recording.
  • maxSessionTTL caps the lifetime of certificates issued for this role (default 12h). Lower it for sensitive prod roles so a compromised cert expires sooner. Raise it for long-running workload-identity bots.

auditd integration​

When the host runs the Linux Auditing System (auditd) and the SSH Service has the privileges to talk to it, AlmaForge emits structured events into the audit log automatically. No configuration is needed. The events carry the AlmaForge username, role set, and session ID, so a downstream auditd rule can correlate kernel-level events with the AlmaForge audit trail and SIEM pipelines pick up SSH activity alongside everything else auditd captures.

Identity files for non-interactive SSH​

Scripts and CI jobs that should not run a browser-based SSO flow use an identity file, a single file holding a short-lived certificate plus the private key. An operator generates one once on their workstation by adding -o to the regular SSO login flow, then hands the file to the job. The job points alma at it through the global -i flag:

Terminal
# On the operator's workstation, after a normal SSO login:alma login --proxy=almaforge.example.com:443 -f file -o ./ci-identity.pem
# In the CI job:alma -i ./ci-identity.pem ssh ubuntu@web-01 'deploy.sh'

The file inherits the cert TTL and roles of the operator who generated it (capped per role by maxSessionTTL, which is 12h unless the role raises it). Treat it like any other secret: store it in a CI secret manager, mount it read-only into the job, and rotate before expiry.

For long-running automation, prefer the Workload Identity guide. It issues certs continuously without a human bootstrap step.

Login banner​

Set a cluster-wide login banner, such as legal disclaimers, change-freeze notices, and ownership reminders, through the cluster's AuthConfig resource. The banner is displayed on alma login and in the web UI before the user reaches the dashboard. It does not appear on each alma ssh session. For per-host banners, use sshd's Banner directive on the host itself.

login-banner.yaml:

examples/servers/auth/login-banner.yaml
# Cluster-wide login banner shown during user login.
# Apply this when every user must see the same legal or ownership notice.
apiVersion: almaforge.com/v1
kind: AuthConfig
metadata:
  name: cluster # AuthConfig is cluster-wide. The required name is cluster.
spec:
  messageOfTheDay: |
    Authorized access only. All sessions are recorded.
    See policy at https://wiki.example.com/access.

Apply with alma apply -f login-banner.yaml. Clear the banner by setting messageOfTheDay: "" and re-applying.

Troubleshooting​

alma get nodes shows nothing​

The user's roles do not match any host's labels. Run alma get nodes -o yaml as an admin to see how each host is actually labeled, then compare that output with the role's serverLabels matcher. If the admin also sees an empty list, no agent has joined yet. Check journalctl -u almad on the host for the join-handshake error.

alma ssh is denied for the requested OS login​

The Proxy returns access denied to <user> (where <user> is the requested OS login) when the role's serverLogins list does not include that user. Check alma get role/<role-name> -o yaml and confirm the expanded serverLogins value (after trait templating) contains the user the operator typed.

Token rejected with "only valid for the X join method"​

The Token's method block does not match the proof the host presents. alma create token creates static join credentials. For a platform-native join, apply a Token with the matching method block and set ALMA_JOIN_METHOD on the host. --services=ssh selects the service, not the proof method. Inspect alma get token/<client-id> -o yaml and compare it with the host's join environment. See the Token schema.

Session disconnects with "client certificate expired"​

When disconnectExpiredCert is enabled, an active session disconnects when the user's certificate expires. The TTL is bounded by maxSessionTTL on the role and the --ttl flag on alma login. Re-run alma login and reconnect. For long-running roles (CI bots, daemons), raise maxSessionTTL on the role or switch the workload to Workload Identity. Renewed credentials apply to new connections, not an already-open SSH session. See role options.

ssh denied with "node does not allow port forwarding"​

Returned when a user runs ssh -L, -R, or -D and no role they hold sets portForwarding: true. The interactive shell still works. Only the forward is denied. Either set portForwarding: true on a role the user has, or use a different role for forward-heavy workflows.

alma ssh denied with "lock in force"​

The user, the role, the device, or the host has an active Lock resource attached, blocking new sessions. Locks are how operators short-circuit access without touching the role definition. List active locks with alma get locks and remove the matching one with alma delete lock/<name>.

alma ssh rejected with "too many session channels"​

The connection reached the role's maxSessions limit for concurrent SSH session channels. Close unused channels or review that limit. When the user holds several roles, the smallest nonzero limit applies.

alma ssh succeeds but the session hangs​

The agent accepted the connection but cannot start a PTY. Common causes:

  • Host user auto-provisioning is enabled but useradd is missing or the agent does not have permission to invoke it.
  • The host's /etc/security/limits.conf rejects the new session (process or file-descriptor limit reached).
  • A failing recordSession.ssh: strict setting plus a full disk on the recording backend. The session is held until the recording write times out.

Check journalctl -u almad on the host for the specific error.

Failed to launch: fork/exec /proc/self/exe: permission denied​

The almad binary lacks read+execute for the OS user the session is opening as. Some hardened images strip world-execute on freshly installed binaries. Restore it:

Terminal
sudo chmod go+rx $(which almad)sudo systemctl restart almad

A node is still listed minutes after the host shut down​

alma get nodes reflects the last known heartbeat. A host that stops sending heartbeats remains visible until its heartbeat expires (10 minutes by default). This is heartbeat lag, not a configuration problem. To force-remove a host you have already decommissioned:

Terminal
alma delete node/<host-id>

To distinguish a transient gap from a real shutdown, check the expires field on the resource:

Terminal
alma get nodes -o json \    | jq '.[] | {hostname: .spec.hostname, expires: .metadata.labels["internal/expires"]}'

If the time-to-expire is closer than the normal heartbeat window (under two or three minutes), the host has stopped heartbeating.

Auto-created host user is not removed at session end​

createHostUserMode: keep is the most common cause. It is the documented behavior. If the role specifies drop and the user still persists, check journalctl -u almad for a userdel failure. Active processes owned by the user, an open file in their home directory, or userdel not being on the agent PATH all block cleanup.

OpenSSH agentless host fails with "no authorities for hostname"​

The host's sshd_config no longer trusts the active AlmaForge CA, typically after a CA rotation or after the cluster was reinstalled. Re-run almad join openssh on the host with a fresh token to refresh the trusted CA list, then restart sshd.

OpenSSH agentless host refuses login as root​

alma ssh root@<openssh-host> returns the host's own permission-denied error, distinct from the AlmaForge Proxy denial in the login-denied entry above. The agentless mode hands off straight to sshd, which enforces its own root-login policy. If /etc/ssh/sshd_config sets PermitRootLogin no or PermitRootLogin forced-commands-only, sshd rejects the session before any AlmaForge layer sees it. Either pick a non-root login, or change the setting to PermitRootLogin prohibit-password (which still blocks password auth and only allows certificate-based root login) and reload sshd. SSH Service hosts are not affected. almad --ssh-service opens the OS session itself and does not consult sshd_config.

References​

  • Declarative configuration: full schema for the Node resource and every other resource the SSH Service reads.
  • Environment configuration: every ALMA_* environment variable the SSH Service reads at startup, including ALMA_SSH_SERVICE and ALMA_LABELS.
  • RBAC reference: full schema for the Role resource, including serverLabels, serverLogins, hostGroups, hostSudoers, createHostUserMode, and the options block.
  • CLI reference: all alma ssh, alma scp, alma proxy ssh, alma apply, alma get, and almad --ssh-service flags.
  • Audit events: the record shape and the fields carried by the SessionStart, SessionExec, SessionPortForward, and SessionEnd events emitted for every SSH session.
  • Storage Configuration: backend choices for session recordings and audit events (local disk, S3, GCS, Azure Blob).
  • Agent topology: how many agent processes to run per resource pool, and how they join the cluster.
  • Break-glass access: recovery paths when SSO or the cluster is unreachable.
  • Agent fleet upgrades: when to upgrade agents relative to the cluster, and how to verify each one.
  • Access Requests: on-demand role escalation for time-bounded production access.