Workload Identity

Workload identity lets agents, CI jobs, and services authenticate with credentials their platform already provides, then receive short-lived AlmaForge certificates. Start with First successful join or the GitHub Actions example.

Why workload identity

Non-human workloads need access to infrastructure too. A nightly job runs psql against a production database. A Kubernetes agent registers itself with the control plane. A CI pipeline runs alma kube login prod && kubectl apply -f manifest.yaml to roll out a release. The question is the same in every case: who is this caller, and what roles does it get?

The common answer is a long-lived API key. That key ends up in build secrets, .env files, ConfigMaps, and image layers. It rarely rotates. When it leaks, the blast radius is everything the key could ever do, and the audit log cannot always tell which workload the key came from.

Workload identity replaces that pattern. Every non-human caller proves who it is using an identity its platform already issues:

• An AWS STS caller identity.
• A GCP metadata-server ID token.
• An Azure managed-identity attestation.
• A Kubernetes ServiceAccount JWT.
• A GitHub Actions, GitLab CI, CircleCI, or Spacelift OIDC token.

The cluster verifies the attestation against the platform, checks it against a Token allow rule, and issues a short-lived certificate.

The model matches human sign-in. A person proves identity through an SSO provider and receives a short-lived certificate. A workload proves identity through its platform and receives a short-lived certificate.

Who uses it

Three kinds of caller present a workload identity to AlmaForge:

• Agents are almad processes that join the cluster to proxy access to a resource: an SSH host, Kubernetes cluster, database, or internal web app. Agents run inside the private network of the resources they proxy and dial outbound to the Proxy. No inbound ports are required on the agent host.
• Bots and CI runners consume the AlmaForge API. A nightly backup job that calls alma get, a Terraform plan that applies Role manifests, or a release pipeline that opens a local database proxy can all run as workloads.
• The alma CLI itself can run as a workload. When a script or pipeline sets the workload identity environment variables and runs a command, alma joins without an interactive SSO session.

All three go through the same Token resource and the same join handshake. Once the workload has its certificate, it is governed by the same RBAC, audit, and lock machinery as any other authenticated caller. There is no parallel service account system to keep in sync.

First successful join

A general checklist that applies to every method:

• Pick the join method that matches the workload's platform. Use static only if no platform-native method fits.
• For a bot, create the user RBAC roles the bot will impersonate. Agents skip this step.
• Apply a Token whose spec.<method>.allow pins one source identity.
• For a bot, run alma create bot <name> --roles=<list> --token-name=<token> to bind the bot to the token.
• In the workload's environment, export ALMA_PROXY, ALMA_CLIENT_ID, and ALMA_JOIN_METHOD. Add ALMA_CLIENT_SECRET for the static method only.
• Run alma login on the workload side with those variables set, then inspect the resulting profile with alma status. status reads the local profile and does not initiate a join.

The worked example below walks through this for a GitHub Actions job. The join method sections explain the platform-specific allow rules.

Worked example: GitHub Actions to PostgreSQL

A nightly job in my-org/orders needs to run read-only queries against a dev PostgreSQL instance. End-to-end:

1. Create the bot's user RBAC role.

role-db-readonly.yaml:

examples/workload-identity/walkthrough/role-db-readonly.yaml

# Role for a bot that reads orders.* from dev PostgreSQL databases.
# The bot impersonates this role. It does not grant any access by
# itself until a Token references the bot via spec.botName.
apiVersion: almaforge.com/v1
kind: Role
metadata:
  name: db-readonly
spec:
  allow:
    databaseLabels:
      env: dev # Restrict to databases labeled env=dev.
    databaseNames: # Database names this role may access.
      - orders
    databaseUsers: # PostgreSQL roles this user may assume on the DB.
      - readonly

Terminal

alma apply -f role-db-readonly.yaml

2. Create the join Token.

token-ci-orders-readonly.yaml:

examples/workload-identity/walkthrough/token-ci-orders-readonly.yaml

# Token that lets the ci-orders-readonly bot join from a specific
# GitHub Actions workflow on my-org/orders. Pin every claim the
# workflow already controls (repository, workflow name, ref) so a
# leaked OIDC token from a feature-branch workflow cannot use this
# token.
apiVersion: almaforge.com/v1
kind: Token
metadata:
  name: ci-orders-readonly
spec:
  roles:
    - system:bot
  botName: ci-orders-readonly # Must exist before the first join.
  github:
    allow:
      - repository: my-org/orders # owner/repo of the GitHub workflow.
        workflow: orders-db-checks # GitHub workflow name, not the filename.
        ref: refs/heads/main # Git ref the workflow must run on.
        refType: branch # branch or tag.

Terminal

alma apply -f token-ci-orders-readonly.yaml

3. Create the bot bound to that token.

Terminal

alma create bot ci-orders-readonly \    --roles=db-readonly \    --token-name=ci-orders-readonly

4. Wire the workflow. Add this to .github/workflows/db-checks.yml:

github-workflow.yml:

examples/workload-identity/walkthrough/github-workflow.yml

# .github/workflows/db-checks.yml
# Nightly GitHub Actions job that opens a short-lived database
# connection using AlmaForge workload identity.
#
# permissions.id-token: write is the GitHub side of the contract. It
# tells the runner to mint an OIDC token the job can present. The
# ALMA_* env vars tell alma which Token to use and how to identify
# the workload.
name: orders-db-checks
on:
  schedule:
    - cron: '17 2 * * *'
  workflow_dispatch:
permissions:
  id-token: write
jobs:
  query:
    runs-on: ubuntu-latest
    env:
      ALMA_PROXY: almaforge.example.com:443
      ALMA_CLIENT_ID: ci-orders-readonly
      ALMA_JOIN_METHOD: github
    steps:
      - name: install alma
        run: curl -fL https://get.almaforge.com/install.sh | sh
      - name: run nightly query
        run: |
          alma db login --db-user=readonly --db-name=orders dev-orders-pg
          alma db connect --db-user=readonly --db-name=orders dev-orders-pg \
            <<< "SELECT count(*) FROM orders;"

The first step fetches alma. The second uses workload identity to obtain a database certificate (alma db login), then runs the query (alma db connect).

5. Trigger the workflow. A successful first join emits these audit events:

text

TokenCreate            ci-orders-readonly
BotJoin                ci-orders-readonly  method=github
CertificateCreate      user=bot-ci-orders-readonly
DatabaseSessionStart   user=bot-ci-orders-readonly db=dev-orders-pg

The setup is working when the workflow authenticates as bot-ci-orders-readonly and completes the permitted database query. Before using it for production, restrict the Token's github.allow rule with the applicable environment, actor, and ref values.

The Token resource

Every workload joins through a Token. The token declares which join method is allowed, which system roles the resulting certificate carries, and any platform-specific allowlist.

token.yaml:

examples/reference/configuration/token.yaml

# Token example for the declarative configuration reference.
# Adjust metadata.name and spec values for your cluster before applying.
apiVersion: almaforge.com/v1
kind: Token
metadata:
  name: node-bootstrap
spec:
  # System roles granted to the joining service after a successful join.
  roles:
    - system:ssh

  # The join method is selected by which sub-message is set.
  # Set one of: static, ec2, aws, github, gitlab, circleCI,
  # spacelift, kubernetes, azure, gcp. This token uses the EC2
  # instance-identity-document method via spec.ec2.
  ec2:
    # Allow rules restrict where the token may be used from.
    allow:
      - awsAccount: '123456789012'
        awsRegions:
          - us-east-1
        awsRole: telemetry-node

  # Suggested labels that newly enrolled resources should set on themselves.
  # Each value is a list, so one key can suggest several values.
  suggestedLabels:
    environment: [prod]
    region: [us-east-1]

Key fields:

• spec.roles lists the system roles the joined workload receives. Every system role carries the system: prefix on the wire, for example system:ssh (SSH agents), system:proxy, system:kubernetes, system:app, system:database, or system:bot (CI jobs and other bot workloads). Use the smallest set the workload actually needs. Do not mix system roles (system:*) with user RBAC role names (bare identifiers) in one token. The validator rejects mixed roles.
• spec.botName is required when spec.roles includes system:bot. The token lets the named bot join. The bot's own Role controls which user RBAC roles the job may impersonate.
• spec.<method> picks the join method. Exactly one method block must be present: static, aws, ec2, azure, gcp, kubernetes, github, gitlab, circleCI, or spacelift. The presence of the block is the selector. There is no separate joinMethod field to keep in sync.
• spec.<method>.allow is a list of allow rules that pin the source identity. Platform-native methods constrain joins by account, project, service account, namespace, and similar fields.
• spec.suggestedLabels is an optional set of labels the joining workload is asked to set on itself, so RBAC rules can target it.

See the Token reference for the full schema.

Runtime inputs

Agents, bots, and alma read the same environment variables when they present a token:

Variable	Meaning
ALMA_PROXY	Cluster proxy address (host:port). Required for agents and for alma workload first contact.
ALMA_CLIENT_ID	The metadata.name of the Token to use. Required.
ALMA_CLIENT_SECRET	The plaintext secret for spec.static. Setting it implies the static method when ALMA_JOIN_METHOD is unset. Other methods ignore it.
ALMA_JOIN_METHOD	One of static, aws, ec2, azure, gcp, kubernetes, github, gitlab, circleci, spacelift. Required for non-static methods.
ALMA_CA_PIN	Auth Service CA pins for a direct join through ALMA_AUTH_SERVER. Joins through ALMA_PROXY use the system trust store.
AZURE_CLIENT_ID	For the azure method only, when the VM has more than one user-assigned managed identity. Matches the standard Azure SDK variable.

Workload identity engages only when ALMA_PROXY, ALMA_CLIENT_ID, and either ALMA_CLIENT_SECRET or ALMA_JOIN_METHOD are set. With only some of those set, alma falls through to the interactive SSO path instead, and almad refuses to start.

The server access guide shows how an agent receives these through /etc/almaforge/almad.env. The in-cluster agent guide shows the Kubernetes equivalent. For CLI usage, export them in the CI job environment.

CLI workload sessions

alma is a workload too. When the env vars above are set, running any command that contacts the cluster joins as a workload instead of looking up a cached SSO profile:

Terminal

export ALMA_PROXY=almaforge.example.com:443export ALMA_CLIENT_ID=ci-orders-pgexport ALMA_JOIN_METHOD=githubalma proxy db dev-orders-pg --port 5432

The first alma command creates a normal local profile marked as workload identity. The client ID, join method, bot certificate, and working certificate are stored in the client profile store so later commands can refresh without an interactive login. The ALMA_CLIENT_SECRET value is not written to the profile.

Short commands refresh the workload profile before making an RPC when the working certificate is near expiry. Long-running local proxies, such as alma proxy db and alma proxy kube, check certificate freshness on new downstream connections and refresh before handing the connection to the protocol proxy.

When a workload-identity environment and a cached alma login profile both exist, workload identity wins for commands that contact the cluster. Running alma login with workload identity variables set creates or refreshes the workload profile. Running it without those variables keeps the normal interactive SSO flow for humans.

To render credentials to disk for consumers that read certificates from fixed paths, use alma export -c <file>. See the Credential Export guide for output configuration and renewal.

Join methods

Pick the platform-native method wherever the workload runs there. Platform-native joins verify the platform's identity proof and require no shared secret on disk. Fall back to static only when no platform method fits.

Static

A pre-shared client_id / client_secret pair. client_id is the token name. client_secret is set out-of-band and presented through ALMA_CLIENT_SECRET. The cluster stores only sha256(client_secret) in spec.static.secret, so a leaked manifest cannot impersonate the workload.

ssh-node-token.yaml:

examples/configuration/infrastructure-as-code/ssh-node-token.yaml

# Static join token for an SSH Service agent named ssh-node-01.
# Store the unhashed client secret outside this file and pass it to the
# agent as ALMA_CLIENT_SECRET.
apiVersion: almaforge.com/v1
kind: Token
metadata:
  name: ssh-node-01 # Agent ALMA_CLIENT_ID value.
spec:
  roles:
    - system:ssh # Service role granted to the joining SSH agent.
  static:
    # SHA-256 of the ALMA_CLIENT_SECRET value.
    secret: 9e7a4b3c6f2d1e8a5b0c4d9e2f1a3b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a

To compute spec.static.secret, hash the chosen secret:

Terminal

printf %s "<your-secret>" | sha256sum | awk '{print $1}'

Use static joins for evaluation, for agents running outside any supported cloud, or when no other method fits. Rotate the secret on a schedule and treat the value as sensitive. Set a short metadata.expires (e.g., one hour) on bootstrap tokens so a leaked secret cannot be reused indefinitely for new joins.

Static also works for bot joins. Set spec.roles: [system:bot] and spec.botName: <name> instead of an agent service role, then create the bot against the token as in Worked example.

AWS IAM

The workload signs an sts:GetCallerIdentity request with its existing AWS credentials. AlmaForge verifies the signature against AWS STS and matches the resulting account and ARN against spec.aws.allow.

Any IAM principal works: an EC2 instance role, an EKS pod role (IRSA), an ECS task role, or a Lambda execution role. The role does not need any attached policies. Only the sts:GetCallerIdentity call is required, and that is allowed by default. The AlmaForge Auth Service does not need its own AWS credentials.

The join method value is aws. The legacy iam value is not a supported alias.

aws-iam.yaml:

examples/servers/tokens/aws-iam.yaml

# Token for SSH Service agents that join the cluster with AWS IAM.
# Pin awsAccount and awsARN to the exact role the host assumes.
apiVersion: almaforge.com/v1
kind: Token
metadata:
  name: ssh-node
spec:
  roles:
    - system:ssh # Token is only valid for SSH Service hosts.
  # Presence of the aws block selects the "aws" join method, which
  # verifies the host via the AWS IAM caller identity.
  aws:
    allow:
      - awsAccount: '123456789012' # AWS account that owns the host identity.
        awsARN: 'arn:aws:sts::123456789012:assumed-role/almaforge-node/*' # Assumed-role ARN. Wildcards widen the trust boundary.

The awsARN field supports wildcards (*, ?) to match assumed role session names. Pin to the narrowest ARN pattern that still covers your fleet.

AWS EC2

The workload submits its signed EC2 Instance Identity Document from the instance metadata service. The Auth Service calls ec2:DescribeInstances to confirm the instance is running and matches spec.ec2.allow. Use this when the agent runs directly on EC2 and you want the join scoped to a specific instance role or region.

The Auth Service needs an IAM principal with ec2:DescribeInstances on the target accounts. The Auth host picks those credentials from the standard AWS chain (instance profile, environment variables, or the shared credentials file). The server also checks for reuse of a stolen Instance Identity Document, so an EC2 instance that already has an agent registered will be refused until the prior server record is gone.

When in doubt between aws and ec2, prefer aws. It works from EC2, EKS, ECS, Lambda, and on-prem hosts with AWS credentials, and the Auth Service needs no AWS permissions of its own.

Azure

The Azure VM presents its managed-identity attested-data document and an access token. AlmaForge verifies both against Azure AD and matches the VM's subscription and resource group against spec.azure.allow.

azure.yaml:

examples/servers/tokens/azure.yaml

# Token for SSH Service agents that join with Azure managed identity.
# The allow block restricts joins to one subscription and resource group.
apiVersion: almaforge.com/v1
kind: Token
metadata:
  name: ssh-node
spec:
  roles:
    - system:ssh # Token is only valid for SSH Service hosts.
  # Presence of the azure block selects the "azure" join method, which
  # verifies the host via Azure managed-identity attestation.
  azure:
    allow:
      - subscription: '11111111-1111-1111-1111-111111111111' # Azure subscription containing the VM.
        resourceGroups:
          - 'servers-rg' # Restrict joins to this resource group.

The VM needs a managed identity with permission to read its VM resource. No other permissions are required. The AlmaForge Auth Service does not need its own Azure credentials.

GCP

The GCE VM fetches an ID token from the metadata server, signed by Google. AlmaForge verifies the signature and matches the VM's project, location, and service account against spec.gcp.allow.

gcp.yaml:

examples/servers/tokens/gcp.yaml

# Token for SSH Service agents that join with GCE metadata attestation.
# Pin projectIDs and serviceAccounts to the exact host identity.
apiVersion: almaforge.com/v1
kind: Token
metadata:
  name: ssh-node
spec:
  roles:
    - system:ssh # Token is only valid for SSH Service hosts.
  # Presence of the gcp block selects the "gcp" join method, which
  # verifies the host via GCE metadata-server attestation.
  gcp:
    allow:
      - projectIDs:
          - 'my-project' # GCP project hosting the agent.
        serviceAccounts:
          - 'almaforge-node@my-project.iam.gserviceaccount.com' # Service account the VM runs as.

Any service account on the VM works. No IAM roles are required on the joining VM, and the Auth Service does not need GCP credentials.

Kubernetes

A pod presents the JWT mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. AlmaForge verifies the token and matches the resulting namespace:serviceaccount against spec.kubernetes.allow.

spec.kubernetes.type picks the validation backend:

In-cluster (default)

Use this when the agent runs in the same Kubernetes cluster as the Auth Service. Auth calls the local TokenReview API to verify the JWT. The Auth ServiceAccount must hold the system:auth-delegator ClusterRole so it can call TokenReview. The AlmaForge Kubernetes overlay grants this by default.

kubernetes-token.yaml:

examples/workload-identity/kubernetes-token.yaml

# Token for an in-cluster Kubernetes agent.
# The agent proves identity with its mounted ServiceAccount JWT.
apiVersion: almaforge.com/v1
kind: Token
metadata:
  name: kube-agent # Agent ALMA_CLIENT_ID value.
spec:
  roles:
    # Grant only the system roles the agent actually runs. An unused
    # role widens the trust boundary for no gain.
    - system:kubernetes # Allow joining Kubernetes agents.
    - system:app # Allow joining Application agents.
    - system:database # Allow joining Database agents.
  kubernetes:
    # Validation backend.
    #   in_cluster  - Auth calls the local TokenReview API. Use this
    #                 when Auth runs in the same Kubernetes cluster
    #                 as the joining agent. Default.
    #   static_jwks - Auth verifies the JWT against a pinned issuer
    #                 JWKS configured under spec.kubernetes.staticJWKS.
    #                 Use this when the agent runs in a different
    #                 Kubernetes cluster that Auth cannot reach.
    type: in_cluster
    allow:
      - serviceAccount: almaforge:agent # namespace:name of the joining ServiceAccount.

Out-of-cluster (static JWKS)

Use this when the agent runs in a different Kubernetes cluster that Auth cannot reach over the API. Pin the remote cluster's issuer JWKS on the Token. Fetch it once with kubectl --context=<remote> get --raw /openid/v1/jwks and copy the JSON into spec.kubernetes.staticJWKS.jwks.

kubernetes-static-jwks-token.yaml:

examples/workload-identity/kubernetes-static-jwks-token.yaml

# Token for a Kubernetes agent running in a different cluster from
# Auth. Auth cannot call TokenReview on the remote cluster, so it
# verifies the agent's ServiceAccount JWT against a JWKS that the
# remote cluster's apiserver publishes at /openid/v1/jwks.
apiVersion: almaforge.com/v1
kind: Token
metadata:
  name: remote-kube-agent # Agent ALMA_CLIENT_ID value.
spec:
  roles:
    - system:kubernetes # Allow joining Kubernetes agents.
  kubernetes:
    type: static_jwks # Verify ServiceAccount JWTs against the pinned JWKS below.
    staticJWKS:
      # Fetch from the remote cluster's apiserver:
      #   kubectl --context=<remote> get --raw /openid/v1/jwks
      # Replace this placeholder with the value that command returns.
      jwks: |
        {"keys":[{"use":"sig","kty":"RSA","kid":"replace-me","alg":"RS256","n":"replace-me","e":"AQAB"}]}
    allow:
      - serviceAccount: almaforge:agent # namespace:name of the joining ServiceAccount.

Re-fetch the JWKS and update the Token if the remote cluster rotates its service-account signing keys.

GitHub Actions

The GitHub Actions runner fetches an OIDC token from the runner's ID-token endpoint. AlmaForge verifies the signature against the GitHub JWKS and matches the workflow's claims against spec.github.allow.

github-token.yaml:

examples/workload-identity/github-token.yaml

# Token for a GitHub Actions workload.
# The system:bot role requires botName. Configure that bot's impersonation role
# separately so it can request the RBAC roles the job needs.
apiVersion: almaforge.com/v1
kind: Token
metadata:
  name: ci-orders-pg # CI job ALMA_CLIENT_ID value.
spec:
  roles:
    - system:bot # CI jobs join as bots, then impersonate allowed RBAC roles.
  botName: ci-orders-pg # Existing Bot resource this token is scoped to.
  github:
    # enterpriseServerHost: ghe.example.com # Set for GitHub Enterprise Server.
    allow:
      - repository: my-org/orders # owner/repo that may use this token.
        repositoryOwner: my-org # Extra guard against repository transfer.
        workflow: production deploy # GitHub workflow name.
        ref: refs/heads/main # Git ref the workflow must run on.
        refType: branch # Accepted values are branch or tag.
        environment: production # GitHub deployment environment.

At least one of repository, repositoryOwner, or sub must be set on every allow rule, so a misconfigured token cannot match every workflow in GitHub. Pin as many additional fields as fit your threat model. A repository alone lets any workflow in that repo join, while pinning workflow, ref, and environment narrows it to one production deploy path.

The runner must make GitHub's ID-token request environment variables available to the job. Set permissions: id-token: write in the workflow and export ALMA_JOIN_METHOD=github before running alma.

The workflow allow field matches GitHub's workflow name claim. Use the workflow's top-level name: value, not the .github/workflows/*.yml filename.

GitLab CI

GitLab does not auto-inject a usable OIDC token for this flow. The workflow must declare an id_tokens block, name the slot GITLAB_JWT, and set the audience to the AlmaForge proxy URL.

gitlab-ci.yml:

examples/workload-identity/gitlab-ci.yml

# GitLab job that exposes an OIDC token for AlmaForge.
# The slot must be named GITLAB_JWT because the client reads that variable.
deploy:
  variables:
    ALMA_PROXY: almaforge.example.com:443
    ALMA_CLIENT_ID: ci-platform-deploy
    ALMA_JOIN_METHOD: gitlab
  id_tokens:
    GITLAB_JWT:
      aud: https://almaforge.example.com
  script:
    - alma proxy db dev-orders-pg --port 5432

alma reads the token from the GITLAB_JWT env var that GitLab injects, calls the cluster, and AlmaForge verifies the signature against the GitLab JWKS and matches the pipeline claims against spec.gitlab.allow.

gitlab-token.yaml:

examples/workload-identity/gitlab-token.yaml

# Token for a GitLab CI workload.
# The system:bot role requires botName. Configure that bot's impersonation role
# separately so it can request the RBAC roles the job needs.
apiVersion: almaforge.com/v1
kind: Token
metadata:
  name: ci-platform-deploy # CI job ALMA_CLIENT_ID value.
spec:
  roles:
    - system:bot # CI jobs join as bots, then impersonate allowed RBAC roles.
  botName: ci-platform-deploy # Existing Bot resource this token is scoped to.
  gitlab:
    # domain: gitlab.example.com # Defaults to gitlab.com.
    allow:
      - projectPath: platform/deploy # group/project, with glob support.
        ref: main # Git ref the job must run on.
        refType: branch # Accepted values are branch or tag.
        refProtected: true # Only protected branches or tags may join.
        environment: production # GitLab environment name.
        pipelineSource: push # Pipeline source claim, such as push.

At least one of sub, projectPath, namespacePath, or ciConfigRefURI must be set on every allow rule, so a misconfigured token cannot match every pipeline in the GitLab instance. projectPath and namespacePath accept glob characters (*, ?). Combine them with refProtected: true to keep the credential out of feature-branch pipelines.

CircleCI

CircleCI injects the CIRCLE_OIDC_TOKEN_V2 env var for jobs that run with at least one CircleCI context. alma reads it, and AlmaForge verifies the signature against the CircleCI JWKS for the configured organization, then matches the run's project and context against spec.circleCI.allow.

circleci-token.yaml:

examples/workload-identity/circleci-token.yaml

# Token for a CircleCI workload.
# CircleCI exposes CIRCLE_OIDC_TOKEN_V2 only for jobs with at least
# one context.
apiVersion: almaforge.com/v1
kind: Token
metadata:
  name: ci-mobile-release # CI job ALMA_CLIENT_ID value.
spec:
  roles:
    - system:bot # CI jobs join as bots, then impersonate allowed RBAC roles.
  botName: ci-mobile-release # Existing Bot resource this token is scoped to.
  circleCI:
    organizationID: 11111111-1111-1111-1111-111111111111 # CircleCI organization UUID.
    allow:
      - projectID: 22222222-2222-2222-2222-222222222222 # CircleCI project UUID.
        contextID: 33333333-3333-3333-3333-333333333333 # Required CircleCI context UUID.

organizationID is required. Allow rules can pin a projectID, a contextID, or both. Pin both for production-impact tokens so the credential is bound to a specific project running with a specific context.

Spacelift

Spacelift injects the SPACELIFT_OIDC_TOKEN env var into a run. alma reads it, and AlmaForge verifies the signature against the Spacelift JWKS for the configured tenant hostname, then matches the run's space, stack, and scope against spec.spacelift.allow.

spacelift-token.yaml:

examples/workload-identity/spacelift-token.yaml

# Token for a Spacelift workload.
# The system:bot role requires botName. Configure that bot's impersonation role
# separately so it can request the RBAC roles the run needs.
apiVersion: almaforge.com/v1
kind: Token
metadata:
  name: ci-tf-network # Spacelift run ALMA_CLIENT_ID value.
spec:
  roles:
    - system:bot # Automation joins as a bot, then impersonates allowed RBAC roles.
  botName: ci-tf-network # Existing Bot resource this token is scoped to.
  spacelift:
    hostname: example.app.spacelift.io # Tenant hostname, without scheme or path.
    allow:
      - spaceID: root # Spacelift space ID.
        callerID: network-prod # Stack or module ID.
        callerType: stack # Accepted values are stack or module.
        scope: write # Accepted values are read or write.

At least one of spaceID or callerID must be set on every allow rule. hostname is required and must not include a scheme or path. Use scope: read for plan-only tokens that should never apply changes, and scope: write for apply runs.

Scope, renewal, and revocation

A workload identity is bounded by:

• The join allow rule declared on the Token.
• The system roles declared on the Token. The certificate cannot carry a system role the token did not authorize.
• For system:bot tokens, the bot's impersonation Role. The token lets the bot join, but that Role controls which user RBAC roles the bot can request.
• The certificate TTL. The workload re-joins or rotates on its own before expiry.
• The cluster lock set. Lock a workload identity with alma create lock --target-user=bot-<bot-name> (see Revoking access).

Renewal depends on the join method. Static bot joins receive renewable bot certificates and the one-time static Token is deleted after first use. Non-static bot joins re-attest to the platform when the bot certificate is stale, using the recorded token name and join method. Either way, already-issued certificates remain valid until they expire unless a lock invalidates them.

Deleting a Token blocks new first-contact joins. For non-static bot profiles, it also blocks future re-attestation. To block existing credentials, create a lock. Enforcement starts as the lock reaches the services handling the workload's access, typically within seconds over a healthy Auth connection.

Audit

Workload access emits the same audit stream as other access paths. Token creation emits TokenCreate, bot joins emit BotJoin, certificate issuance emits CertificateCreate, resource use emits protocol-specific events, and locks emit LockCreate. See the Audit Reference for the event catalog.

Troubleshooting

unable to validate cluster CA

This error means the host's system trust store cannot validate the Proxy certificate. A self-signed certificate is the usual cause.

Configure the Proxy with a publicly trusted certificate, or add its signing CA to the host's trust store. Use alma auth pin for direct Auth Service joins. Proxy joins always use the system trust store.

token <name> not found

ALMA_CLIENT_ID does not match any Token's metadata.name. The client ID is the token name itself, not a hex value or a secret.

Compare exactly:

Terminal

alma get token/<name>

If the token does not exist, create it. If it does, check that ALMA_CLIENT_ID in the workload's environment matches its metadata.name character-for-character.

GitHub OIDC join rejected, no audit event fires

The workflow is not allowed to mint an ID token. GitHub refuses to hand the OIDC token to the runner, so the request never reaches the cluster.

Add id-token: write to the workflow or job permissions:

github-id-token-permissions.yaml:

examples/workload-identity/troubleshooting/github-id-token-permissions.yaml

# GitHub Actions permission needed before a job can request an OIDC
# token. Add it at workflow scope or job scope.
permissions:
  id-token: write

If the join still fails and audit does fire, the cause is on the cluster side. Check the Token's spec.github.allow rules against the actual repository, workflow, and ref of the run.

GITLAB_JWT environment variable missing

GitLab is not exposing an OIDC token to the job under the name alma expects.

Add an id_tokens: block to the job and name the slot GITLAB_JWT:

gitlab-id-token.yml:

examples/workload-identity/troubleshooting/gitlab-id-token.yml

# GitLab CI id_tokens block for AlmaForge workload identity. The slot
# must be named GITLAB_JWT because the client reads that environment
# variable.
id_tokens:
  GITLAB_JWT:
    aud: https://almaforge.example.com # Audience must match the proxy URL.

The audience must match the AlmaForge proxy URL. A wrong audience makes the JWKS verification succeed but the audience check fail.

CIRCLE_OIDC_TOKEN_V2 ... must be present

CircleCI only injects CIRCLE_OIDC_TOKEN_V2 into jobs that run with at least one CircleCI context attached. A job without a context has no OIDC token to present.

Attach a context to the job in .circleci/config.yml:

circleci-context.yml:

examples/workload-identity/troubleshooting/circleci-context.yml

# CircleCI workflow fragment that attaches a context. CircleCI only
# injects CIRCLE_OIDC_TOKEN_V2 into jobs with at least one context.
workflows:
  release:
    jobs:
      - deploy:
          context:
            - almaforge-deploy # Context name that enables OIDC token injection.

cannot add bot: user <name> already exists

A prior alma create bot run left the bot in place. The bot creation flow refuses to overwrite an existing User resource.

Inspect the existing bot before replacing it:

Terminal

alma get bots

If it is the intended bot, use its existing configuration. Delete and recreate it only when replacing that identity is intentional. Deletion removes its bound role and can interrupt workloads using it:

Terminal

alma delete bot/<name>alma create bot <name> --roles=<role-list> --token-name=<token>

The User and bound Role are removed together when the bot is deleted. The Token is independent and survives bot deletion.

EC2 join refused on second attempt

The cluster checks for reuse of stolen EC2 Instance Identity Documents. If a prior agent already joined from the same instance, a new join with the same IID is refused while the old server record is still present.

Stop the agent on the EC2 instance:

Terminal

sudo systemctl stop almad

If the data directory contains a host ID from a different join method, remove only /var/lib/almaforge/host_uuid before starting the agent.

From an admin shell with cluster access, remove the stale server record:

Terminal

alma delete node/<id>

Start the agent again:

Terminal

sudo systemctl start almad

static_jwks: must be set when type is "static_jwks"

The Kubernetes Token declares type: static_jwks but spec.kubernetes.staticJWKS.jwks is empty.

Fetch the JWKS from the remote cluster's apiserver:

Terminal

kubectl --context=<remote> get --raw /openid/v1/jwks

Paste the JSON into spec.kubernetes.staticJWKS.jwks as a single string. Re-apply the Token. Re-fetch and re-apply if the remote cluster rotates its service-account signing keys.

In-cluster Kubernetes join returns TokenReview permission denied

The AlmaForge Auth ServiceAccount is missing the built-in system:auth-delegator ClusterRole, so it cannot call the TokenReview API to verify the joining pod's JWT.

Apply the ClusterRoleBinding from the AlmaForge Kubernetes overlay. The bundled clusterrolebinding.yaml binds it to the almaforge ServiceAccount.

Joined but permission denied opening a resource

The workload received its certificate, but its bound RBAC Role does not allow the target. The bot's user role is what controls which resources it can reach.

Inspect the bot's role and the target's labels:

Terminal

alma get role/<role-name>alma get db/<db-name>

Compare databaseLabels, databaseNames, and databaseUsers on the role against the database's labels and configured users.