Skip to main content

AlmaForge Applications Access Guide

Publish internal web apps, REST APIs, and raw TCP services through AlmaForge without exposing them to the public internet. Users sign in once through SSO. An agent inside the upstream's network forwards each request, so only the agent needs reachability to the application. This brings internal apps into the same control plane as Kubernetes, servers, and databases:

  • One access plane for apps and infrastructure. Engineers use one path for apps, SSH, Kubernetes, and databases. Teams get one identity model and one audit trail instead of a separate access stack for each protocol.
  • Publish private apps without exposing them. Users connect through an AlmaForge URL while the upstream stays off the public internet. This works for apps, APIs, and TCP services. You do not need a VPN, inbound firewall rules, or per-app ingress.
  • Give your apps SSO without changing their code. AlmaForge handles SSO at the access layer, with no SSO library to integrate. Apps that need per-user authorization can use the signed identity and groups attached to each request. See JWT validation.
  • Grant access with context, not standing privilege. Users request access for a task, incident, or on-call event, with scope and expiry tied to that context. Review requests in the web UI, CLI, or API. Slack notifies reviewers, and PagerDuty can supply on-call context.
  • Audit app access by user, not by IP address. Session events tie the user's real SSO identity to the target app and upstream URL. HTTP request events add the method, path, query string, and response status, giving auditors a request-level trail without relying on raw IP logs. See Audit events.

Architecture​

Application access uses four main components.

almad is the server daemon. One binary exposes every server-side role. The roles used for application access are:

  • almad --auth-service runs the Auth Service. It issues short-lived certificates, stores cluster state, and is the source of truth for identities, roles, and registered resources. It stores the cluster's persistent state.
  • almad --proxy-service runs the Proxy Service, the public TLS endpoint at almaforge.example.com:443. It terminates user sessions (SSO redirects for browsers, mTLS for alma callers) and brokers connections to agents. It holds no long-term state.
  • almad --app-service runs the App agent. It connects private apps to AlmaForge by dialing out to the Proxy, holding a reverse tunnel open, and forwarding requests to upstream apps. One agent can serve many apps.

Roles compose freely. A small cluster can run --auth-service and --proxy-service in one almad process. Production deployments can split roles across multiple instances and run Auth, Proxy, and App services in HA mode. See How it works for deployment shapes and the full role catalog.

alma is the CLI for users and workloads. It authenticates interactively through SSO (alma login) or non-interactively through workload identity. Once authenticated, alma can either open a localhost proxy for an app or export an mTLS certificate for an HTTPS or TLS client to connect directly.

Application access topology

Browser flow. A user opens https://<app>.almaforge.example.com in a browser. The Proxy redirects to the SSO IdP, the user authenticates, and the IdP redirects back. It then asks the Auth Service for a short-lived certificate, stores the session in a cookie, and forwards the request through the reverse tunnel to the App agent. The agent connects to the upstream over HTTP or HTTPS and can attach a AlmaForge-signed JWT with the user's identity. The upstream can use that JWT for authorization without implementing SSO itself. The response returns through the same path. See Browser access for the user-facing steps.

Browser access flow

API flow. A user or workload starts with alma login. That uses the same SSO flow and leaves alma with a short-lived certificate on disk. From there, the caller authenticates each request over mTLS in one of two ways. Run alma proxy app <name> to open a localhost tunnel for native clients such as curl, psql, and grpcurl. See Local proxy:

Local proxy access flow

Or run alma apps config <name> to get the cert, key, and CA paths for an HTTPS or TLS client that connects to the Proxy directly. See mTLS access:

mTLS access flow

In both cases, the Proxy validates the client certificate and forwards the request through the reverse tunnel to the App agent. From there, the agent reaches the upstream the same way as in the browser flow.

JWT identity propagation. For HTTP upstreams, the agent signs a short-lived JWT and attaches it to forwarded requests. The token carries the user's identity, groups, and mapped AlmaForge traits as captured in their app session. The upstream verifies it against the JWKS endpoint and authorizes the request without implementing SSO itself. TCP upstreams do not receive a JWT. See JWT identity propagation for the claim format and a verifier example.

Prerequisites​

  • A running AlmaForge cluster, with Auth Service and Proxy Service reachable at almaforge.example.com:443.
  • An SSO connector configured on that cluster, with at least one operator account that can sign in through the Proxy. See Single Sign-On for OIDC setup.
  • A wildcard DNS record and a wildcard TLS certificate covering *.almaforge.example.com, so each app is reachable at <app-name>.almaforge.example.com. See Requirements for the DNS, TLS, and ACME details.
  • The alma CLI installed on an operator workstation with network access to the Proxy. See Install the CLI. The operator account must be able to apply AppTarget, Role, and join-token resources. The admin preset role covers all three.
  • A host for the App agent with network reachability to (a) the Proxy at almaforge.example.com:443 and (b) every upstream the agent will publish (HTTP URL or TCP host:port). The agent is the only component that needs to reach the upstream.

Every published app is one AppTarget resource served by an App agent. The four steps below deploy the agent, register the app, grant access, and verify the user can reach it. Optional configuration (dynamic labels, JWT propagation, header rewrites, upstream TLS) lives in Advanced options.

Step 1. Deploy the App agent​

A join token authenticates the agent to the AlmaForge Auth Service the first time it starts. Its method block decides how the agent proves identity: a static random string, or a workload identity such as AWS IAM, a GCP service account, or an Azure managed identity. Workload-identity tokens carry no secret material, but they require the Auth Service to trust the cloud provider's identity signal, so they fit agents running on a supported cloud.

Pick the join method that matches where the agent runs.

A random string with a TTL. Useful for first install and for agents running on a host without a supported workload-identity provider.

Terminal
alma create token --services=app

Default TTL is 30 minutes. Override with --expires=24h for slower rollouts. Revoke a leaked or unused token with alma delete token/<client-id>. Pass --client-id=<name> to alma create token to set the resource name explicitly. The command then prints both client_id and client_secret. Set ALMA_CLIENT_ID and ALMA_CLIENT_SECRET on the agent host, or pass --client-id on the command line. The secret has no flag of its own, so it never reaches ps.

--services=app covers application access. To also expose databases or Kubernetes through the same agent, add database or kubernetes to the --services list and set the matching ALMA_*_SERVICE toggle.

On the agent host, start almad with the App role enabled and the token:

Terminal
almad start \    --app-service \    --no-auth-service \    --no-proxy-service \    --client-id=<token-name> \    --proxy=almaforge.example.com:443

App agents join over a reverse tunnel, so they connect to the public Proxy address, not the Auth Service. Use --proxy on the command line or ALMA_PROXY in the environment. An agent host installs the same package as a cluster host, from the same APT, RPM, or tarball source. See Install the package for those steps. Nothing else on that page applies to an agent, which runs the App Service rather than Auth and Proxy and needs no storage backend, load balancer, or connector of its own. To run the agent in Kubernetes (StatefulSet, ServiceAccount, ConfigMap, Secret), follow the in-cluster procedure on the Kubernetes access page with --services=app in place of --services=kubernetes.

Step 2. Register an application​

Each app is registered through an AppTarget resource. This is the configuration: what you tell the cluster to serve. You can apply it centrally to the cluster, or drop it as a file on the agent host. Either way, the resource shape is the same. It only depends on the upstream protocol. Pick the tab that matches.

For an HTTP or HTTPS upstream. spec.uri is the URL the agent forwards traffic to. Users open the app at <name>.almaforge.example.com in a browser, or call it with curl after alma apps login <name>.

http-apptarget.yaml:

examples/applications/kinds/http-apptarget.yaml
# HTTP/HTTPS web app behind the App agent. Users open it at
# https://<name>.<cluster> in a browser.
apiVersion: almaforge.com/v1
kind: AppTarget
metadata:
  name: grafana # DNS-label name; users type this and it becomes <name>.<cluster>.
  labels: # Static labels matched by Role.appLabels for RBAC.
    env: prod
    team: sre
spec:
  uri: http://grafana.monitoring.svc:3000 # Upstream URL; scheme picks http / https / tcp.
  fqdn: grafana.almaforge.example.com # Externally visible hostname; must be covered by the Proxy's wildcard cert.

The four fields you set:

  • metadata.name: short identifier users type into alma apps login and the left-most label of the public URL (<name>.almaforge.example.com). Because the name becomes a DNS label, it must match [a-z0-9-]{1,63}: lowercase letters, digits, and hyphens, no leading or trailing hyphen, and no longer than 63 characters. Keep it stable across redeploys. It appears in audit events and CLI output.
  • metadata.labels: arbitrary key/value pairs used by Role.spec.allow.appLabels. No required schema.
  • spec.fqdn: externally visible hostname users hit. Default to <name>.almaforge.example.com to stay inside the wildcard certificate. Override only when you need a different domain (and remember the Proxy must hold a certificate for that exact name).
  • spec.uri: upstream the agent forwards traffic to. Either an http:///https:// URL or a tcp://host:port URI. May include a path such as https://kdash.svc.cluster.local/api/v1/....

Apply the resource through one of two paths, based on who owns the app inventory. Both paths end at the same place at runtime: the agent serves the upstream and emits an App heartbeat per target. They differ only in where the AppTarget is stored.

Centrally to the cluster. Run alma apply against the Auth Service from the operator workstation. The AppTarget is persisted in the cluster backend and listed by alma get apptarget. Every agent whose label matcher selects the target's labels picks it up on its next heartbeat (a few seconds), no restart needed.

Terminal
alma apply -f http-apptarget.yamlapptarget "grafana" has been created

The agent's label matcher is set with ALMA_MATCH_LABELS in the environment. Use this path when AlmaForge is the source of truth for app inventory.

This path also lets service teams self-register their own apps without operator involvement. Grant a team's role create, read, update, and delete on the apptarget resource, scoped to a label range the team owns through appLabels. The team then runs alma apply -f against the cluster from CI or a workstation, and any agent whose match labels select the team's range picks the target up on its next heartbeat. No agent restart, no operator-side change. See RBAC reference for the rule schema and label-matching semantics.

On the agent host. Drop the same YAML under /etc/almaforge/almad.d/ (/etc/almaforge/almad.d/ by default). The agent reads the directory at startup, with no apply call and no label match. Restart the agent to pick up changes. Use this path when the agent host is the source of truth: configuration management, baked image, or StatefulSet ConfigMap.

The catch: drop-in AppTarget files are not persisted in the cluster backend. They register inside the agent process only, and the agent emits an App heartbeat per drop-in target. As a result, alma get apps lists them, but alma get apptarget does not. The file on the host is the source of truth. To inspect what each agent is serving, use alma get apps -o yaml and read the hostname and host_id on each row.

For agents running inside a Kubernetes cluster, register each service with an explicit AppTarget. See manual in-cluster registration for the setup procedure.

Confirm the agent picked up the AppTarget:

Terminal
alma get appsHost         Name     Type Public Address           URI                                 Labels             Version------------ -------- ---- ------------------------ ----------------------------------- ------------------ -------almaforge-0  grafana  HTTP grafana.almaforge.example.com  http://grafana.monitoring.svc:3000  env=prod,team=sre  0.2.266

The new app should appear within one heartbeat (a few seconds for centrally-applied targets, immediately for file-drop targets after the agent restarts). If it does not, see alma get apps shows nothing.

Step 3. Grant a user access​

Two layers of authorization decide what a user does. AlmaForge picks the apps the user can launch via Role.appLabels. The upstream decides what they can do once connected. AlmaForge propagates identity (header rewrites, JWT) but does not grant verbs upstream. A 403 either comes from the Proxy (no role selects the app) or from the upstream (identity has no permission there).

Apply a Role and map an IdP group to it with the SSO connector's claimsToRoles. See OIDC for the full schema.

appLabels values support trait templates ({{external.team}}, {{internal.logins}}), so a single role can scope each user to their own slice of the inventory based on IdP claims.

One role, every app. For demos, dev environments, or first-time setup. Tighten later.

examples/applications/roles/apps-all.yaml
# Quick-start role: launch any app in the cluster. Tighten with
# label-scoped allows in production.
apiVersion: almaforge.com/v1
kind: Role
metadata:
  name: apps-all
spec:
  allow:
    appLabels:
      '*': '*' # Wildcard key and value matches every AppTarget regardless of labels.

Map a workforce SSO group to that role:

examples/applications/roles/apps-all-claims.yaml
# OIDCConnector fragment: map the IdP `engineers` group to the apps-all role.
spec:
  claimsToRoles:
    - claim: groups # OIDC claim name to read from the IdP token.
      value: engineers # Required value of the claim for this mapping to fire.
      roles: [apps-all] # AlmaForge roles granted on match.

Apply:

Terminal
alma apply -f roles.yaml

End-to-end:

text
operator → SSO IdP claims → AlmaForge connector
        → AlmaForge Role(s)
        → appLabels select an AppTarget
        → header / JWT rewrite carries identity to the upstream
        → upstream authorization
        → response

For trait templating ({{external.groups}}), deny rules, and predicate-based matchers (appLabelsExpression), see the RBAC reference. For the per-app token shape, see JWT identity propagation.

Step 4. Verify access​

A user with the right role reaches a published app through a browser, a local proxy, or an mTLS client. Unattended workloads use the local proxy or mTLS path with workload identity.

  • Browser access: open the app in a browser, authenticate via SSO at the Proxy. No CLI on the user's machine.
  • Local proxy: alma proxy app opens a tunnel on 127.0.0.1:<port>. Native clients (curl, nc, grpcurl) point at localhost. Best for ad-hoc work and tools that do not speak mTLS directly.
  • mTLS access: alma apps config prints cert, key, and CA paths. The caller's HTTPS client speaks mTLS straight to <app>.almaforge.example.com. Best for service-to-service traffic where both ends already authenticate with AlmaForge-issued certificates.

If the user is rejected, the failure is at one of the two layers named in Step 3. The Proxy returns 403 when no role's appLabels selects the app. The upstream returns 403 when the propagated identity has no permission there. See Troubleshooting for the common causes.

Browser access​

Open https://<app-name>.almaforge.example.com in a browser, or click the app in the AlmaForge Web UI at https://almaforge.example.com. If no SSO session is active, the Proxy redirects through the IdP and back. Once authenticated, the Proxy routes the request to the agent and the upstream UI loads, with no VPN client, no CLI, and no direct upstream exposure.

Browser access flow

Browser access depends on wildcard DNS and a wildcard TLS certificate covering *.almaforge.example.com. If spec.fqdn uses a hostname outside that wildcard, the Proxy must also hold a certificate for that exact name. See Requirements for the DNS, TLS, and ACME details.

Local proxy​

Local proxy access flow

Local-proxy access starts with a cluster login. alma login exchanges the SSO session for a short-lived AlmaForge certificate the CLI uses to fetch app sessions:

Terminal
alma login --proxy=almaforge.example.comalma get appsHost         Name        Type Public Address              URI                      Labels    Version------------ ----------- ---- --------------------------- ------------------------ --------- -------almaforge-0  orders-rpc  TCP  orders-rpc.almaforge.example.com  tcp://orders.svc:9090    env=prod  0.2.266

alma get apps lists the apps the current user's roles expose. If an app is missing, the problem is almost always RBAC. See Troubleshooting.

Log in to the app and start the tunnel:

Terminal
alma apps login orders-rpcalma proxy app orders-rpc --port 8443Proxying connections to orders-rpc on 127.0.0.1:8443

Point the native client at 127.0.0.1:8443. For the TCP orders-rpc target above:

Terminal
grpcurl -plaintext localhost:8443 listnc -v localhost 8443

For an HTTP AppTarget, alma proxy app exposes the same upstream over plain HTTP on localhost, so the equivalent call against an HTTP target named grafana is:

Terminal
alma proxy app grafana --port 8443curl http://localhost:8443/api/health

The tunnel stays in the foreground. Ctrl+C stops it. Omit --port to let alma choose a random local port and print it at startup.

alma apps logout <name> releases a single app's certificate when the work is done. alma apps logout with no argument logs out of every app in the current profile.

mTLS access​

mTLS access flow

mTLS access skips the localhost hop. The caller's HTTPS client speaks mTLS straight to <app>.almaforge.example.com using a client certificate the AlmaForge CA issued. Use this when you do not want a alma process running alongside the caller. The typical case is one service calling another, but a script with a real TLS client works the same way.

The caller authenticates first. Humans use alma login (as in Local proxy above). Unattended workloads use workload identity, which renews the certificate on a schedule with no operator in the loop. From there on, the steps are identical:

Terminal
alma apps login grafanaalma apps config grafanaName:     grafanaURI:      https://grafana.almaforge.example.com:443CA:       /home/alice/.almaforge/keys/almaforge.example.com/certs.pemCert:     /home/alice/.almaforge/keys/almaforge.example.com/alice-app/almaforge.example.com/grafana-x509.pemKey:      /home/alice/.almaforge/keys/almaforge.example.com/alice

Plug those paths into the caller's HTTPS client. With curl:

Terminal
curl \  --cert /home/alice/.almaforge/keys/almaforge.example.com/alice-app/almaforge.example.com/grafana-x509.pem \  --key /home/alice/.almaforge/keys/almaforge.example.com/alice \  --cacert /home/alice/.almaforge/keys/almaforge.example.com/certs.pem \  https://grafana.almaforge.example.com/api/annotations

alma apps config grafana --format=curl prints a ready-to-run curl command with the paths already filled in. --format=<uri|ca|cert|key|curl> prints just one field at a time, so config templates can splice individual paths in. See the CLI reference for the full subcommand.

The Proxy validates the caller's mTLS connection. The App agent then forwards the request to the upstream, where the app can validate the JWT as described in JWT identity propagation. The same verification path applies to human and workload identities.

Certificate lifetime is bound to the caller's session. For workload callers, refresh runs automatically through the Workload Identity renewal flow. For human callers, sign in again with alma login and run alma apps login <name> to refresh the app certificate.

Application access is working when the app appears in alma get apps and an authorized user can load its UI or complete a request through one of these paths.

Advanced options​

The subsections below are optional AppTarget features. Reach for them when an upstream needs richer label data than static metadata can express, when its same-origin checks or redirects don't survive the hop through the agent, or when it serves a TLS certificate the agent cannot verify.

Dynamic labels​

Static metadata.labels are fine when an app's classification never changes, but inventory data often comes from elsewhere: a tag on the underlying VM, the result of a health probe, or the namespace the upstream lives in. spec.dynamicLabels runs a shell command on the agent host on a fixed period and uses its stdout as the label value. Roles match dynamic labels with the same appLabels matcher used for static ones.

examples/applications/dynamic-labels/grafana.yaml
# AppTarget with two dynamic labels. The agent runs each `command` on its
# `period` and uses stdout as the label value. `command` is exec-style: there
# is no shell unless you spawn one with /bin/sh -c.
apiVersion: almaforge.com/v1
kind: AppTarget
metadata:
  name: grafana
  labels:
    env: prod
spec:
  uri: http://grafana.monitoring.svc:3000 # Upstream URL.
  fqdn: grafana.almaforge.example.com # Externally visible hostname.
  dynamicLabels: # Map of label name -> {period, command}; refreshed on the period.
    arch:
      period: 1h # Go duration; the command runs every hour.
      command: ['/bin/uname', '-m'] # Exec-style argv; argv[0] is the binary, no shell parsing.
    region:
      period: 5m # Refresh often enough to notice instance moves without hammering metadata on every heartbeat.
      command: [
          '/bin/sh',
          '-c',
          'curl -s http://169.254.169.254/latest/meta-data/placement/region',
        ] # Wrap in /bin/sh -c when you need shell features.

The agent invokes command directly. There is no shell unless you spawn one yourself with /bin/sh -c. The agent trims surrounding whitespace from successful output. If the command fails, the label contains the error and any captured output instead of its previous value. Account for that value in label-based access rules.

Choose a period that matches how fast the underlying value changes. Short periods (under a minute) are fine for cheap local commands but add load when the command hits a network. Long periods (tens of minutes) suit values that rarely change, such as region, machine type, OS release.

Kubernetes service discovery​

info

Kubernetes Service discovery is not available in this release.

Register in-cluster services with explicit AppTarget resources for now. Run the App agent inside the cluster by following the in-cluster procedure on the Kubernetes access page with --services=app in place of --services=kubernetes, then use Register an application to add each service you want to publish.

Skip TLS verification on the upstream​

By default the agent verifies an HTTPS upstream's TLS certificate against the host's trust store. Install the upstream's signing CA in that store when it uses a private CA. A plain HTTP upstream instead needs an http:// URI and does not perform a TLS handshake. Set insecureSkipVerify: true on the AppTarget to skip verification on the agent → upstream hop:

examples/applications/skip-tls/grafana.yaml
# AppTarget fragment: skip TLS verification on the agent → upstream hop.
# Safe when the agent and upstream share a private network (same cluster,
# VPC, or host). Unrelated to the Proxy → browser TLS handshake.
spec:
  uri: https://grafana.monitoring.svc:3000 # Upstream URL; https here means the agent dials TLS but accepts any cert.
  insecureSkipVerify: true # Disables agent-side cert verification of the upstream — agent → upstream only, not Proxy → browser.

Skipping verification disables the agent's check of the upstream's identity. A private network alone does not restore that check. Prefer a trusted certificate for production. The browser-to-Proxy TLS connection is unaffected by this setting.

Inject request headers​

spec.rewrite.headers injects, replaces, or removes headers on every request the agent forwards to the upstream. Use it to fix Origin and Host checks (Grafana, ArgoCD compare those headers to their configured external URL and reject requests carrying the upstream's internal hostname), or to pass identity to the upstream as custom headers.

Header values support trait templates ({{internal.*}}, {{external.*}}) for identity-header injection and the {{internal.jwt}} token for splicing the AlmaForge-signed JWT. See Inject the JWT into a custom header. The agent reserves these header names, and a rewrite rule cannot target them:

  • Alma-Access-Jwt-Assertion,
  • the standard X-Forwarded-* set,
  • X-Forwarded-Ssl,
  • X-App-Api-Error and X-App-Api-Info (internal diagnostics).

Some X-App-* names are agent-managed and therefore reserved. A rewrite rule naming any reserved header is ignored, so the upstream sees the agent's own value or no header at all, never yours. Other X-App-* names are yours to use.

AlmaForge removes all of these from the incoming request before it applies the rules above, so a value a user puts on the wire never reaches the upstream. The same removal covers the conventional authenticating-proxy headers X-Forwarded-User, X-Forwarded-Email, X-Forwarded-Groups, X-Forwarded-Preferred-Username, X-Forwarded-Access-Token, the X-Auth-Request-* set, X-Remote-User, X-Remote-Group, any header starting with X-Remote-Extra- or X-Authentik-, the Remote-User / Remote-Groups / Remote-Name / Remote-Email set, X-Forwarded-Client-Cert, and Grafana's X-Webauth-User, X-User-Email and X-User-Name. Underscores count as hyphens here, so X-Forwarded_User goes too. Those are not reserved, so a rewrite rule may set any of them, and the rule is applied after the removal.

examples/applications/rewrite/headers-grafana.yaml
# AppTarget fragment: rewrite Origin and Host so the upstream's same-origin
# check passes, and inject the operator's logins trait as a custom header.
spec:
  uri: https://grafana.monitoring.svc:3000
  fqdn: grafana.almaforge.example.com
  rewrite:
    headers: # List of headers injected, replaced, or set on every upstream request.
      - name: Origin # Tell the upstream the request came from its public URL, not the internal address.
        value: 'https://grafana.almaforge.example.com'
      - name: Host
        value: 'grafana.almaforge.example.com'
      - name: Alma-User # Custom identity header; Alma-Access-Jwt-Assertion, X-Forwarded-*, and selected X-App-* names are reserved.
        value: '{{internal.logins}}' # Trait template; supports {{internal.*}}, {{external.*}}, and {{internal.jwt}}.

Rewrite HTTP redirects​

The agent forwards traffic to the upstream's internal address, so upstreams that issue redirects often send the user to a hostname the browser cannot resolve (for example, Location: https://grafana.monitoring.svc/login). List those internal hostnames in spec.rewrite.redirect and the agent rewrites each Location header to the app's fqdn, landing the user on https://grafana.almaforge.example.com/login:

examples/applications/rewrite/redirect-grafana.yaml
# AppTarget fragment: rewrite the Location header on upstream redirects so
# the user lands on the public address instead of the internal hostname.
spec:
  rewrite:
    redirect: # List of internal hostnames whose Location header should be rewritten to fqdn.
      - grafana.monitoring.svc
      - grafana.monitoring.svc.cluster.local

Point at a sub-path​

If the upstream does not serve from /, put the full path in spec.uri. The agent does not prepend it to every request. Instead, when a user hits the bare public address (/), the agent issues a 302 redirect to https://<fqdn>/<uri-path>. The user's browser follows the redirect and from there every link the upstream emits already carries the prefix, so subsequent requests forward cleanly.

A typical case is Jenkins started with --prefix=/jenkins, which mounts the entire UI and API at /jenkins/.... Set spec.uri to include the same prefix so a hit to the bare hostname lands the user on the Jenkins home page:

examples/applications/kinds/sub-path-jenkins.yaml
# App that mounts on a sub-path. Jenkins started with `--prefix=/jenkins`
# serves its UI and API at /jenkins/...; spec.uri repeats the prefix so the
# bare public address lands the user on the Jenkins home page.
apiVersion: almaforge.com/v1
kind: AppTarget
metadata:
  name: jenkins
  labels:
    env: prod
spec:
  uri: http://jenkins.ci.svc:8080/jenkins # Include the same prefix Jenkins serves at.
  fqdn: jenkins.almaforge.example.com

JWT identity propagation​

AlmaForge runs a dedicated JWT CA alongside its host and user CAs. The Proxy publishes its public key at https://almaforge.example.com/.well-known/jwks.json, the agent signs a short-lived JWT with the active key and attaches it to forwarded HTTP requests, and the upstream verifies it against the JWKS endpoint. The token carries the AlmaForge username, roles, and traits, plus standard JWT claims (iss, aud, nbf, exp). Upstream APIs use the JWT to authenticate the user without speaking SSO themselves. This is the recommended way to expose a service that already supports bearer-token auth (Grafana, Elasticsearch, an internal REST API) through AlmaForge without giving every operator a long-lived API key.

Overview​

The agent signs a JWT with the cluster's JWT CA and attaches it to a fixed header on forwarded requests. The upstream validates the signature against the JWKS endpoint exposed by the Proxy and reads identity and roles directly from the claims. The upstream needs no SSO integration of its own. It trusts the AlmaForge JWT CA the same way it would trust an external identity provider.

End-to-end on a single request:

text
user → SSO sign-in to AlmaForge Web UI
     → AlmaForge Proxy verifies the session
     → agent rewrites the request, attaching the signed assertion
     → upstream receives the request plus the JWT header
     → upstream verifies the JWT against /.well-known/jwks.json
     → upstream reads sub, roles, traits from the claims
     → upstream authorizes and serves the response

The roles and traits in the token are the ones captured in the user's app session certificate, so they change when that session is reissued, not on the next request. There is no session cookie, no refresh handshake, and no logout call to make on the upstream side. To cut a user off before expiry, lock them. Enforcement depends on the lock reaching the Proxy and agent, typically within seconds over a healthy Auth connection. See Revoking access.

Token format​

The agent attaches the JWT in the fixed Alma-Access-Jwt-Assertion header on every upstream request. The header value is a compact-serialized JWT, three base64url-encoded sections separated by .:

  • Header: {"alg": "RS256", "typ": "JWT"}.
  • Payload: the claims described below.
  • Signature: RSA-SHA256 over the first two sections, signed with the AlmaForge JWT CA's active key.

Example assertion, truncated: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOlsiaHR0c.... Treat the value as opaque text: do not parse it as form input or split on . past the first two separators.

Example payload:

JSON
{
  "aud": ["http://grafana.monitoring.svc:3000"],
  "iss": "almaforge.example.com",
  "sub": "alice",
  "nbf": 1603835795,
  "exp": 1603943800,
  "iat": 1603835795,
  "username": "alice",
  "roles": ["dev"],
  "traits": {
    "logins": ["alice", "ubuntu"]
  }
}

Standard JWT claims:

  • iss: the AlmaForge cluster name (matches the proxy hostname for default deployments).
  • aud: a single-element array with the app's uri, the internal address the agent forwards to.
  • sub, username: the AlmaForge login name.
  • nbf, iat, exp: not-before, issued-at, and expiration as Unix timestamps. The TTL is bound to the operator's AlmaForge session.

AlmaForge-specific claims:

  • roles: the user's AlmaForge roles, as captured in their app session.
  • traits: the user's resolved trait map, from the same session.

spec.rewrite.jwtClaims on the AppTarget controls which of roles and traits make it into the payload. The four allowed values:

  • roles-and-traits (default): both roles and traits are present.
  • roles: only roles. Use when the upstream authorizes on roles alone and traits inflate the payload.
  • traits: only traits. Use when the upstream needs trait values (logins, groups) but not AlmaForge role names.
  • none: neither claim. Use when the upstream only needs identity (sub, iss, aud) and the roles / traits sets are large enough to push request headers past the upstream's size limit.

Choose the smallest claim set the upstream needs. Role access checks remain governed by the RBAC reference.

Inject the JWT into a custom header​

Upstreams that look for a specific header name (for example, Authorization: Bearer <token>) read the JWT through a header-rewrite rule. Use the {{internal.jwt}} template variable in spec.rewrite.headers to splice the AlmaForge-signed JWT into the header value:

examples/applications/jwt/inject-bearer.yaml
# AppTarget that splices the AlmaForge-signed JWT into a custom Authorization
# header. The agent regenerates {{internal.jwt}} on every request, so the
# upstream always sees a fresh assertion bound to the current operator session.
apiVersion: almaforge.com/v1
kind: AppTarget
metadata:
  name: orders-api
spec:
  uri: https://orders.api.internal
  fqdn: orders.almaforge.example.com
  rewrite:
    headers:
      - name: Authorization # Upstream expects "Bearer <token>" in Authorization; AlmaForge-signed JWT replaces a long-lived API key.
        value: 'Bearer {{internal.jwt}}' # {{internal.jwt}} is regenerated per request; do not cache or hard-code.

The agent signs {{internal.jwt}} for the app session, so the upstream sees an assertion bound to the operator session that requested it.

Validate the JWT​

AlmaForge exposes a JSON Web Key Set (JWKS) at the proxy's /.well-known/jwks.json path. Upstreams fetch and cache the keys, then verify each incoming JWT's signature, iss, aud, and exp.

Terminal
curl https://almaforge.example.com/.well-known/jwks.json

Example response:

JSON
{
  "keys": [
    {
      "kty": "RSA",
      "n": "xk-0VSVZY76QGqeN9TD-FJp32s8jZrpsalnRoFwlZ_JwPbbd5...",
      "e": "AQAB",
      "alg": "RS256"
    }
  ]
}

A correct verifier:

  1. Fetches and caches /.well-known/jwks.json from the trusted Proxy.
  2. Tries the published RSA signing keys. App assertions do not include a kid, so key selection must not require one.
  3. Verifies the RS256 signature.
  4. Asserts iss matches the AlmaForge cluster name.
  5. Asserts aud contains the app's uri.
  6. Asserts nbf <= now < exp.

Verify the signature. The presence of a header is not an identity claim you can act on: only the signature ties the name in the token to the user AlmaForge authenticated. An upstream that reads a username out of a header without verifying it is trusting whatever arrived, which is why AlmaForge removes the conventional identity headers from every incoming request.

If you rotated the AlmaForge JWT CA, restart upstreams or invalidate their JWKS cache so they pick up the new key.

Troubleshooting​

alma get apps shows nothing​

There are two resource kinds in play. An AppTarget is what you told the cluster about: the configuration. An App is what an agent actually picked up and is serving right now. The two should match, but they don't always, and an empty alma get apps means nothing has lined up yet.

Start by checking what is configured. Run alma get apptarget as an admin. Only centrally-applied targets show up here: the ones you pushed to the cluster with alma apply -f. Targets dropped as YAML files on an agent host are deliberately not listed: they live on the host, not in the cluster, and get apptarget only knows about cluster state. So if your app is missing, either you never applied it centrally, or you applied it on the agent host and should verify it through the next step instead.

Next, check what the agents found. Run alma get apps -o yaml as an admin. This is the full picture: every running agent, every target it is currently serving, and the hostname and host_id it is serving from. If the target is configured but no row appears, no agent's match labels select it. Every centrally-applied target needs at least one agent whose ALMA_MATCH_LABELS selects the target's metadata.labels. Check the agent logs and the target's labels. If the target was dropped on the agent host but no row appears, the agent never parsed the file: check the agent log for a parse error and confirm the file lives under /etc/almaforge/almad.d/ and ends in .yaml.

Finally, if the admin can see the app but a specific user cannot, it's an RBAC mismatch. Compare the admin's alma get apps -o yaml output with the user's role appLabels matcher. The user sees only the rows their role selects.

Browser cannot resolve the domain​

The user opens <app>.almaforge.example.com and the browser fails before any TLS handshake: DNS_PROBE_FINISHED_NXDOMAIN, ERR_NAME_NOT_RESOLVED, or "site can't be reached". This is plain DNS: nothing answers for <app>.almaforge.example.com.

DNS lookup failure

The Proxy expects a wildcard A (and/or AAAA) record covering *.almaforge.example.com that points at the Proxy, so every app's public address resolves without per-app DNS work. See Requirements for the DNS setup.

Browser shows a TLS certificate error​

The user opens the app at <app>.almaforge.example.com and the browser itself blocks the page with a certificate warning (mismatched hostname, untrusted issuer, expired). This is the browser-to-Proxy hop, not anything inside AlmaForge.

Browser-to-Proxy TLS failure

The Proxy needs a wildcard TLS certificate that covers *.almaforge.example.com from a CA the browser trusts. See Requirements for the DNS, TLS, and ACME details.

Page shows "untrusted or self-signed certificate"​

The browser reaches <app>.almaforge.example.com fine. The browser-to-Proxy TLS is healthy, but the page rendered by the Proxy reports Unable to access "<app>.almaforge.example.com". This may happen if your Proxy is using untrusted or self-signed certificate.

The break is on the agent-to-upstream hop:

Upstream TLS handshake failure

Inspect the agent logs to identify a failing upstream TLS handshake. If the upstream serves plain HTTP, use an http:// URI. If it serves HTTPS with a private CA, install that CA in the agent's trust store. Skipping certificate verification does not make an HTTP server speak TLS. See Skip TLS verification on the upstream.

Agent cannot reach the upstream​

Requests to the app return 502 Bad Gateway or 503 Service Unavailable, and the agent's logs show connection failures against spec.uri.

Agent-to-upstream network failure

Confirm DNS, NetworkPolicy or security-group reachability from the agent's pod or VM, and check that spec.uri matches the upstream's actual address.

CSRF or CORS errors on a web app​

Apps that rely on Origin and Host checks (Grafana, ArgoCD) compare those headers against their configured external URL. Because the agent forwards traffic to the upstream's internal address, the upstream sees Host: grafana.monitoring.svc:3000 instead of Host: grafana.almaforge.example.com and rejects the request. Rewrite both headers to the public address. See Inject request headers.

Upstream redirects break the URL​

The upstream issues a redirect to its internal hostname (for example, Location: https://grafana.monitoring.svc/login) and the user lands on a URL the browser cannot resolve. Add the internal hostnames to spec.rewrite.redirect. See Rewrite HTTP redirects.

JWT validation fails​

Verify the iss claim matches the AlmaForge cluster name, the aud claim matches the app's uri, and the JWKS endpoint at https://almaforge.example.com/.well-known/jwks.json returns a signing key that verifies the assertion. App assertions do not carry a kid. After a JWT CA rotation, follow the cache-refresh guidance in Validate the JWT.

Upstream returns 431 or "Request header fields too large"​

The JWT header may exceed the upstream's header-size limit when a user has many roles or large traits. Check the limit on the upstream and any intervening HTTP proxy. Increase that configured limit or shrink the payload by setting spec.rewrite.jwtClaims on the AppTarget to roles, traits, or none.

Cert TTL too short for long-running CLI commands​

The app certificate inherits the user's AlmaForge session TTL. Long curl jobs or large file transfers can outrun it. Raise maxSessionTTL on the user's role if a longer lifetime fits policy, or sign in again and run alma apps login <name> before starting the job. Certificate renewal does not extend an already-open connection's original credentials.

Isolate the problem with an echo app​

When a real app returns 502s, a blank page, broken redirects, "JWT invalid", or strange cookies, the symptoms alone do not tell you whether the issue is in AlmaForge (Proxy, Agent, JWT, RBAC) or in the upstream itself. The fastest way to bisect is to stand up a known-good echo upstream and register it as a parallel AppTarget on the same agent.

Run mendhak/http-https-echo (or any equivalent HTTP echo container) on the same host as the agent:

Terminal
podman run -d --name echo -p 127.0.0.1:8080:8080 \    mendhak/http-https-echo:latest

Then register an AppTarget that points at it:

examples/applications/troubleshoot/echo.yaml
# Diagnostic AppTarget. Pair with `mendhak/http-https-echo` running on the
# agent host (see "Isolate the problem with an echo app" in the docs).
# Use to bisect failures: if `echo` works and a real app does not, the
# AlmaForge path is healthy and the problem is in the upstream itself.
apiVersion: almaforge.com/v1
kind: AppTarget
metadata:
  name: echo
  labels:
    env: prod
spec:
  uri: http://127.0.0.1:8080 # Echo container exposed on the agent's loopback.
  fqdn: echo.almaforge.example.com

Then test it:

Terminal
alma apps login echoalma apps config echo --format=curl

Run the printed curl command. It includes the client certificate and key required for mTLS access.

The echo server returns the exact request the agent forwarded: headers, body, and the AlmaForge-signed JWT. Look for the JWT assertion header in the echo output. From there:

  • If echo works and the real app does not, AlmaForge is healthy. The problem is in the real upstream: its auth, its config, or its same-origin / redirect handling. Look at Origin, Host, Location in the echo output to see what the agent is actually sending.
  • If echo also fails, the problem is on the AlmaForge path itself: Proxy, Agent, RBAC, or DNS/TLS. Walk back through the other troubleshooting entries above, this time against a target with no upstream of its own to blame.

For agent operations, Agent topology explains how many processes to run per resource pool. Break-glass access covers recovery when SSO or the cluster is unreachable. The RBAC reference carries the appLabels rules that decide which registered apps a user reaches.

References​

  • Configuration reference: full schema for the AppTarget resource and the agent's static config drop-in directory.
  • Environment reference: every ALMA_* environment variable the agent reads at startup.
  • RBAC reference: Role schema, the full set of appLabels matching rules, and trait templates.
  • CLI reference: all alma apps, alma proxy app, alma apply, alma get, and almad --app-service flags.
  • Kubernetes access: in-cluster agent deployment (StatefulSet, RBAC, env Secret, join-method tabs) shared by Kubernetes, applications, and database access.