Skip to main content

AlmaForge Kubernetes Access Guide

Connect Kubernetes clusters to AlmaForge for SSO-backed access through kubectl. Users receive short-lived credentials and mapped Kubernetes identities. The agent reaches private API servers while Kubernetes RBAC continues to control what those identities can do.

  • Unify access across all Kubernetes clusters. Put EKS, GKE, AKS, and self-hosted clusters behind one access path instead of managing separate access workflows for each platform.
  • SSO mapped into Kubernetes RBAC, with no static credentials. Users sign in with SSO, AlmaForge maps identity to Kubernetes users and groups. kubectl obtains short-lived credentials through the exec plugin instead of a static kubeconfig or long-lived token, while the Kubernetes API server still makes the final allow-or-deny decision through its own bindings. See RBAC.
  • Keep existing tools. Engineers keep kubectl, kubeconfig, and the rest of their normal Kubernetes workflow. No retraining, no separate Kubernetes client, and no separate operator path for day-to-day work. The AlmaForge CLI supplies credentials to kubectl. See Connect to Kubernetes.
  • Expose private clusters and internal services through managed access. The agent dials out from the cluster or VM, so Kubernetes APIs, in-cluster services, internal applications, and databases stay off the public internet while remaining reachable through one controlled access path.
  • Record what actually happened. Every Kubernetes request is tied to a human or workload identity. Interactive kubectl exec and attach terminals support playback. Audit events capture non-interactive commands and port-forward activity. Capture follows your recording and audit policy.

Prerequisites​

  • A running AlmaForge cluster with Auth Service and Proxy Service reachable at almaforge.example.com:443.
  • An SSO connector configured on that cluster, with at least one operator account that can sign in through the Proxy. See Single Sign-On for OIDC setup.
  • The alma CLI installed on an operator workstation with network access to the Proxy. See Install the CLI. The operator account must be able to apply KubernetesTarget, Role, and join-token resources. The admin preset role covers all three.
  • A target Kubernetes cluster running version 1.24 or later, with cluster-admin access for the initial RBAC setup.
  • kubectl 1.24 or later on operator workstations.
  • For the remote cluster path, a Linux VM with almad already installed, its systemd unit already installed, and network reachability to both the target cluster API server and the Proxy at almaforge.example.com:443. See Install AlmaForge.

In-cluster agent​

There are two ways to connect a Kubernetes cluster to AlmaForge:

  1. Run the agent inside the cluster.
  2. Run the agent outside the cluster and point it at the cluster with a kubeconfig.

The in-cluster path is the recommended default, and this section covers it. If you need to run the agent outside the cluster, for example on a VM, use Remote cluster.

almad runs inside the Kubernetes cluster you want to protect and connects out to your AlmaForge Proxy Service. From there it can broker access to the Kubernetes API itself, with fine-grained control per namespace, resource, and verb, and to applications, databases, or other services that stay reachable only over in-cluster networking. Nothing needs to be exposed externally. The protected targets stay inside Kubernetes, and the agent is the path back to them.

In-cluster agent topology

The next steps set up the agent and bring this cluster online in AlmaForge. The RBAC section later is about user access: which users can reach this cluster and what they can do with kubectl.

Enroll from the web UI instead​

The steps below apply each manifest by hand, which is the path to take when you want to read or adapt them. To have them generated instead, open Resources, choose Enroll resource, and pick Kubernetes. The wizard issues the join token, then gives you two commands: one applies the manifests the Proxy serves at /install.yaml, the other applies this cluster's own Secret and KubernetesTarget.

The served manifests carry the scoped role from Step 3, not the quick-start cluster-admin binding, and they are the same for every cluster. The wizard then waits until the cluster registers, so you see it join rather than checking by hand.

Step 1. Verify in-cluster connectivity​

Connect to the target cluster with your existing cluster-admin path. AlmaForge is not in front of this cluster yet, so use the cloud provider's credential flow or whatever native admin kubeconfig you already use:

  • GKE: gcloud container clusters get-credentials
  • EKS: aws eks update-kubeconfig
  • AKS: az aks get-credentials --admin
  • Other clusters: an emergency admin kubeconfig, client certificate, or pre-positioned cluster-admin credential.

This is the same path described in Operations > Break-glass access. Keep it working after install. It is how you recover the cluster if the agent or RBAC ever get into a bad state.

Confirm the kubeconfig works and has the permissions this guide needs:

Terminal
kubectl cluster-infokubectl auth can-i create namespacekubectl auth can-i create clusterrolebinding

cluster-info proves the API server is reachable and the current context is valid. The two can-i checks confirm you have the cluster-admin scope required to apply the rest of the manifests.

Step 2. Create the in-cluster Namespace​

Namespaced agent objects live in one namespace. Apply it first. ClusterRoles and ClusterRoleBindings are cluster-scoped and remain outside that namespace.

01-namespace.yaml:

examples/kubernetes/in-cluster-agent/01-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: almaforge

Step 3. Create the ServiceAccount​

The ServiceAccount needs a ClusterRole for impersonation, resource discovery, exec/attach proxying, and access to its own state Secrets. The Kubernetes Secret backend persists per-pod identity in <pod>-state Secrets in the agent's namespace, so Secret CRUD on that namespace is required.

Pick the quick-start path for a fast setup, or the production path for least-privilege scoping. The user-side group bindings live in RBAC.

Grants the agent cluster-admin. One binding covers impersonation, discovery, and state Secret access.

02-rbac-quick-start.yaml:

examples/kubernetes/in-cluster-agent/02-rbac-quick-start.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: almaforge
  namespace: almaforge
---
# Quick-start RBAC: grants the agent cluster-admin so it can
# impersonate any user or group, read any resource, and manage its
# own state Secrets. For least-privilege bindings, use
# 02-rbac-advanced.yaml. See the Kubernetes RBAC section of the
# docs for the user-side group bindings.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: almaforge-cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: almaforge
    namespace: almaforge

Step 4. Create the ConfigMap with the KubernetesTarget​

The agent reads AlmaForge resource YAML from /etc/almaforge/almad.d. The ConfigMap mounts at that path, where each data key becomes a resource file. The agent loads KubernetesTarget files at startup. Restart the agent after changing those targets. On a VM, the equivalent is a YAML file in /etc/almaforge/almad.d.

03-configmap.yaml:

examples/kubernetes/in-cluster-agent/03-configmap.yaml
# Drop-in resources for the in-cluster agent. This ConfigMap mounts at
# /etc/almaforge/almad.d, where almad reads resource files and
# hot-reloads on change. No restart needed.
apiVersion: v1
kind: ConfigMap
metadata:
  name: almaforge-resources
  namespace: almaforge
data:
  # Registers this Kubernetes cluster. metadata.name is the cluster
  # name users type into `alma kube login`. metadata.labels
  # are what AlmaForge role matchers and ALMA_MATCH_LABELS evaluate
  # against. Empty spec means "use the Pod's mounted ServiceAccount
  # token".
  kubernetes.yaml: |
    apiVersion: almaforge.com/v1
    kind: KubernetesTarget
    metadata:
      name: in-cluster
      labels:
        env: dev
        region: local
    spec: {}

  # Optional: register an in-cluster HTTP service as an AppTarget so
  # users can reach it through the AlmaForge proxy with `alma app
  # login <app-name>`. Requires ALMA_APP_SERVICE=true on the agent
  # (set in 04-env-secret.yaml).
  #
  # app.yaml: |
  #   apiVersion: almaforge.com/v1
  #   kind: AppTarget
  #   metadata:
  #     name: <app-name>
  #     labels:
  #       app: <app-name>
  #   spec:
  #     uri: http://<service>.<namespace>.svc.cluster.local:<port>

metadata.name becomes the cluster name users type into alma kube login. metadata.labels are arbitrary key/value pairs. There is no required schema. The example uses env and region, but you can choose any labels that fit your role matchers. spec: {} means "use the agent's ambient Kubernetes credentials". In this deployment that means the pod's mounted ServiceAccount token. For a remote agent it means the kubeconfig at ALMA_KUBECONFIG.

The same ConfigMap also includes a commented-out AppTarget example for registering an in-cluster HTTP service. Uncomment and customize it to expose an internal app through the same agent. For the full application flow, see Applications access.

Step 5. Issue a join token​

A join token authenticates the agent to the AlmaForge Auth Service the first time it starts. Its method block decides how the agent proves identity: a static random string, or a workload identity such as AWS IAM, a GCP service account, or an Azure managed identity. Workload-identity tokens carry no secret material, but they require the Auth Service to trust the cloud provider's identity signal, so they fit clusters running on a supported cloud.

Pick the join method that matches where the agent runs.

A random string with a TTL. Useful for first install and for clusters without a supported workload-identity provider.

Terminal
alma create token --services=kubernetes,app,database --format=text

Default TTL is 30 minutes. The token is not single-use: both StatefulSet replicas can consume the same token. Each trades it for its own per-pod identity, persisted in <pod>-state Secrets. Once both replicas have joined, the token is no longer used.

Override the TTL with --expires=24h for slower rollouts. Revoke a leaked or unused token with alma delete token/<client-id>. Pass --client-id=<name> to alma create token to set the resource name explicitly. With --format=text, the command prints client_id on the first line and client_secret on the second. Set ALMA_CLIENT_ID and ALMA_CLIENT_SECRET on the agent Pod (e.g., in the almaforge-env Secret).

The example environment enables Kubernetes, App, and Database Services, so the token grants all three. For a Kubernetes-only agent, use --services=kubernetes and disable the App and Database Services in the environment Secret.

Step 6. Create the Secret with environment variables​

The StatefulSet loads a single Secret with envFrom, so every key becomes a container environment variable. Use it to pass the proxy address, the join token reference, and the per-role toggles in one place. In the example file, the required keys are:

  • ALMA_PROXY: the AlmaForge Proxy address.
  • ALMA_CLIENT_ID: the client_id printed by alma create token in Step 5.
  • ALMA_CLIENT_SECRET: the client_secret printed by alma create token in Step 5, used only by the static-token method. Workload-identity methods (kubernetes, AWS, GCP, Azure) carry no secret and instead set ALMA_JOIN_METHOD with just ALMA_CLIENT_ID, as the example file does.
  • ALMA_KUBERNETES_SERVICE=true: turns on the Kubernetes Service role.

The bundled Secret uses Kubernetes-native joining to a Proxy in the same Kubernetes cluster. For the static-token path above, set ALMA_PROXY to your Proxy address, set ALMA_JOIN_METHOD=static, and replace ALMA_CLIENT_ID and ALMA_CLIENT_SECRET with the issued values. For Kubernetes-native joining, configure the Kubernetes join method and its Token before deploying the agent.

04-env-secret.yaml:

examples/kubernetes/in-cluster-agent/04-env-secret.yaml
# Bundles the join configuration with the rest of the agent's
# environment. The StatefulSet loads this Secret with envFrom, so
# every key becomes a container environment variable.
apiVersion: v1
kind: Secret
metadata:
  name: almaforge-env
  namespace: almaforge
stringData:
  # AlmaForge Proxy address (host:port). The agent connects here
  # over mTLS and registers itself with the Auth Service through
  # the same socket. This points at the in-cluster proxy headless
  # Service published by the reference cluster bundle.
  ALMA_PROXY: 'almaforge.almaforge-cluster.svc:8443'

  # Kubernetes join method: the agent proves identity with its
  # mounted ServiceAccount JWT, which the Auth Service validates
  # through the Kubernetes TokenReview API. No shared secret is
  # carried, so only ALMA_CLIENT_ID is set (the Token resource's
  # metadata.name). The kube-agent Token shipped by the cluster
  # bundle allows the almaforge:almaforge ServiceAccount created in
  # 02-rbac-advanced.yaml.
  ALMA_JOIN_METHOD: 'kubernetes'
  ALMA_CLIENT_ID: 'kube-agent'
  # Static-token join method instead: drop ALMA_JOIN_METHOD above,
  # run `alma create token --services=kubernetes,app,database` to mint
  # a token, set ALMA_CLIENT_ID to the printed client_id, and uncomment
  # ALMA_CLIENT_SECRET below with the printed client_secret.
  # ALMA_CLIENT_SECRET: '<paste-client-secret-here>'

  # The proxy serves a web TLS leaf minted by the cluster's private
  # cert-manager CA. trust-manager distributes that CA into every
  # namespace as the almaforge-ca ConfigMap. Mount it (see
  # 05-statefulset.yaml) and point SSL_CERT_FILE at the bundle so
  # the agent trusts the proxy on first connect.
  SSL_CERT_FILE: '/etc/ssl/almaforge/ca-bundle.pem'

  # Enable the Kubernetes Service role: brokers kubectl traffic to
  # the in-cluster API server using the Pod's ServiceAccount.
  ALMA_KUBERNETES_SERVICE: 'true' # Sic: only strings are supported here.
  # Enable the Application Service role: brokers HTTP(S) access to
  # in-cluster apps and services declared as AppTarget resources.
  ALMA_APP_SERVICE: 'true'
  # Enable the Database Service role: brokers connections to
  # in-cluster databases declared as DatabaseTarget resources.
  ALMA_DATABASE_SERVICE: 'true'
  # SSH is not needed on an in-cluster agent.
  ALMA_SSH_SERVICE: 'false'

  # Resource label selector, fanned out to the app, database, and
  # Kubernetes services. Without it those services start but watch no
  # dynamic targets, so AppTarget / DatabaseTarget / KubernetesTarget
  # resources registered in the cluster are never brokered. '*=*' makes
  # this agent serve every target in the cluster. Narrow it (for
  # example env=prod) to shard targets across several agents.
  ALMA_MATCH_LABELS: '*=*'

  # Diagnostic endpoint listen address. Required for the
  # `/healthz` and `/readyz` probes the StatefulSet uses.
  ALMA_DIAG_LISTEN: '0.0.0.0:3000'

Step 7. Create the StatefulSet​

The agent runs as a StatefulSet so each replica gets a stable pod name (almaforge-0, almaforge-1). The Kubernetes Secret backend activates automatically when the agent detects an in-cluster ServiceAccount token. It stores per-pod identity in <pod>-state Secrets keyed off the pod's hostname, so stable names mean stable host UUIDs across restarts. No env vars are required to wire it up.

A headless Service is required by the StatefulSet spec.

The container image is quay.io/almaforge/almaforge:2026.1.26, which resolves to quay.io/almaforge/almaforge:<version> at docs build time. For production, pin the image to a sha256 digest:

Terminal
crane digest quay.io/almaforge/almaforge:2026.1.26sha256:...

Then set image: quay.io/almaforge/almaforge:2026.1.26@sha256:... in the StatefulSet. For private registries, add an imagePullSecrets entry referencing your registry credentials Secret.

05-statefulset.yaml:

examples/kubernetes/in-cluster-agent/05-statefulset.yaml
# Headless Service is required by the StatefulSet spec for stable
# pod DNS. Mirrors modules/almaforge-k8s/agent.tf.
apiVersion: v1
kind: Service
metadata:
  name: almaforge
  namespace: almaforge
spec:
  clusterIP: None
  selector:
    app.kubernetes.io/instance: almaforge
  ports:
    - name: diag
      port: 3000
---
# StatefulSet (not Deployment) so each replica gets a stable pod
# name (almaforge-0, almaforge-1). The Kubernetes Secret backend
# keys per-pod identity off the pod's hostname, so stable names mean
# stable host UUIDs across restarts.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: almaforge
  namespace: almaforge
spec:
  replicas: 2
  serviceName: almaforge
  selector:
    matchLabels:
      app.kubernetes.io/instance: almaforge
  template:
    metadata:
      labels:
        app.kubernetes.io/name: almaforge
        app.kubernetes.io/instance: almaforge
        app.kubernetes.io/component: agent
    spec:
      serviceAccountName: almaforge
      automountServiceAccountToken: true
      # Spread replicas across nodes (and zones if available) so a
      # single-node failure cannot take both pods. ScheduleAnyway
      # keeps the StatefulSet schedulable on small/dev clusters.
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/instance: almaforge
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/instance: almaforge
      containers:
        - name: almaforge
          # For production, pin the image to an immutable digest:
          # image: quay.io/almaforge/almaforge:latest@sha256:<digest>
          image: quay.io/almaforge/almaforge:latest
          imagePullPolicy: IfNotPresent
          ports:
            - name: diag
              containerPort: 3000
          envFrom:
            - secretRef:
                name: almaforge-env
          volumeMounts:
            - name: config
              mountPath: /etc/almaforge/almad.d
              readOnly: true
            - name: cluster-ca
              mountPath: /etc/ssl/almaforge
              readOnly: true
          resources:
            requests:
              cpu: '100m'
              memory: '256Mi'
            limits:
              memory: '512Mi'
          livenessProbe:
            httpGet:
              path: /healthz
              port: diag
            initialDelaySeconds: 10
            periodSeconds: 30
          readinessProbe:
            httpGet:
              path: /readyz
              port: diag
            initialDelaySeconds: 5
            periodSeconds: 10
      volumes:
        - name: config
          configMap:
            name: almaforge-resources
        # CA bundle trust-manager distributes into every namespace.
        # Lets the agent trust the proxy's cert-manager web TLS leaf
        # on first connect (see SSL_CERT_FILE in 04-env-secret.yaml).
        - name: cluster-ca
          configMap:
            name: almaforge-ca
---
# PodDisruptionBudget keeps at least one replica serving during
# voluntary disruptions (node drains, cluster upgrades, autoscaler
# moves). Pair with topologySpreadConstraints above so the kept
# replica is on a different node from the one being drained.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: almaforge
  namespace: almaforge
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: almaforge

Step 8. Apply the manifests and verify​

The previous steps walked through the manifests one by one for explanation. To deploy, save them into one directory and apply everything at once. Kubernetes applies them in lexical filename order, which is why the prefixes 01- through 05- matter:

Terminal
kubectl apply -f .

Wait for both replicas to roll out and become Ready before treating the deployment as live. kubectl apply returning success only means the API accepted the objects, not that the agent is serving:

Terminal
kubectl -n almaforge rollout status statefulset/almaforge --timeout=120skubectl -n almaforge get pods -l app.kubernetes.io/instance=almaforgeNAME           READY   STATUS    RESTARTS   AGEalmaforge-0    1/1     Running   0          90salmaforge-1    1/1     Running   0          80s

Confirm each replica completed the join handshake and the SelfSubjectAccessReview preflight:

Terminal
kubectl -n almaforge logs -l app.kubernetes.io/instance=almaforge --tail=50 | \    grep -Ei 'joined|registered|impersonate'

Then confirm AlmaForge sees the cluster:

Terminal
alma get kubeCluster        Labels                       Version-------------- ---------------------------- -------prod-eu-central env=prod,region=eu-central  2026.1.26

Run an end-to-end check from a workstation logged in to AlmaForge. A successful access shows up in the audit trail:

Terminal
alma kube login <cluster-name>kubectl get pods -A

Confirm the request in the audit event stream. To verify terminal recording as well, open and close an authorized kubectl exec -it session, then list recordings with alma recording ls --last=10m.

The cluster can take a short time to appear. Restart the StatefulSet after changing file-mounted targets. After the cluster appears and both replicas are Ready, revoke the static join token:

Terminal
alma delete token/<client-id>

Workload-identity tokens (IAM/GCP/Azure) stay in place because the agent re-authenticates from ambient credentials on every rejoin. See Replica lifecycle for when a fresh token is needed again.

Remote cluster​

Use this mode when the agent runs outside the target cluster. The recommended in-cluster path is in In-cluster agent.

almad is deployed outside the Kubernetes cluster and reaches the target cluster's API server over the network using a kubeconfig file. One agent can hold kubeconfigs for many clusters and serve them all at once. The kubeconfig provides access to the Kubernetes API, not a network tunnel to every in-cluster service. Use it for managed clusters where you don't control Pods, fleets of small clusters served by a shared agent, or deliberate separation of duties. Publishing in-cluster apps or databases requires their own enabled service and network reachability from the agent. A kubeconfig alone does not provide that network path.

Remote cluster topology

The examples below assume the agent runs on a Linux VM with systemd.

At a high level, the remote setup has four parts: cluster-side impersonator RBAC, a kubeconfig file on the VM that uses that identity, an env file with the join token and kubeconfig path, and a KubernetesTarget resource that registers the cluster in AlmaForge. User access comes later in RBAC.

Step 1. Verify remote cluster connectivity​

Same checks as Step 1 of the in-cluster agent setup, run against every target cluster the agent will serve.

Step 2. Create the remote Namespace​

Same shape as Step 2 of the in-cluster agent setup, applied to each target cluster.

01-namespace.yaml, applied to each target cluster:

examples/kubernetes/remote-agent/01-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: almaforge

Step 3. Create the ServiceAccount and impersonator RBAC​

The VM-hosted agent still needs an identity inside each target cluster. Give it a ServiceAccount and a ClusterRole that can impersonate users and groups, run the startup preflight, and proxy Kubernetes subresources such as exec, attach, and port-forward.

02-rbac.yaml, applied to each target cluster:

examples/kubernetes/remote-agent/02-rbac.yaml
# ServiceAccount and impersonator RBAC for the remote-agent path.
# Apply this to each target cluster the VM-hosted agent will serve.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: almaforge
  namespace: almaforge
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: almaforge-impersonator
rules:
  - apiGroups: ['']
    resources: ['users', 'groups', 'serviceaccounts']
    verbs: ['impersonate']
  - apiGroups: ['']
    resources: ['pods', 'nodes']
    verbs: ['get', 'list', 'watch']
  - apiGroups: ['']
    resources: ['pods/exec', 'pods/attach', 'pods/portforward']
    verbs: ['create']
  - apiGroups: ['authorization.k8s.io']
    resources: ['selfsubjectaccessreviews']
    verbs: ['create']
  - apiGroups: ['rbac.authorization.k8s.io']
    resources: ['clusterroles', 'clusterrolebindings', 'roles', 'rolebindings']
    verbs: ['get', 'list', 'watch']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: almaforge-impersonator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: almaforge-impersonator
subjects:
  - kind: ServiceAccount
    name: almaforge
    namespace: almaforge

The user-side group bindings still live in RBAC.

Step 4. Build the kubeconfig​

Build a kubeconfig that authenticates as the almaforge ServiceAccount you just created. The file needs three values:

  1. The target cluster's API server address.
  2. The target cluster's CA certificate in base64 form.
  3. A token for the almaforge ServiceAccount.

Get those values from a workstation already pointed at the target cluster:

Terminal
kubectl config view --raw --minify -o jsonpath='{.clusters[0].cluster.server}'kubectl config view --raw --minify -o jsonpath='{.clusters[0].cluster.certificate-authority-data}'kubectl -n almaforge create token almaforge

03-kubeconfig.yaml, a template to copy to the VM and fill in:

examples/kubernetes/remote-agent/03-kubeconfig.yaml
# Kubeconfig used by the VM-hosted Kubernetes agent.
# Replace the placeholders with the target cluster's API server,
# CA certificate, and the ServiceAccount token from Step 2.
apiVersion: v1
clusters:
  - cluster:
      certificate-authority-data: <base64-cluster-ca>
      server: https://<api-server-hostname>:6443
    name: <cluster-name>
contexts:
  - context:
      cluster: <cluster-name>
      user: almaforge
    name: <cluster-name>
current-context: <cluster-name>
kind: Config
preferences: {}
users:
  - name: almaforge
    user:
      token: <service-account-token>

Copy 03-kubeconfig.yaml to prod.yaml and replace the three placeholders with the API server address, CA data, and ServiceAccount token you just retrieved.

This path is simple, but the token from kubectl create token is not a permanent credential. The API server decides its lifetime. For long-lived VM agents, plan to rotate that token before it expires or use an exec-based kubeconfig instead.

For managed clusters, the kubeconfig may also use an IAM-backed exec plugin (aws-iam-authenticator, gke-gcloud-auth-plugin, kubelogin) instead of a static token. The agent uses whatever authentication mechanism the kubeconfig defines. If the kubeconfig calls an exec plugin, that binary must also be installed on the VM running the agent and reachable in its execution environment.

Copy the resulting file to a path under /var/lib/almaforge on the VM. The agent reads it via ALMA_KUBECONFIG (set in Step 5).

Terminal
mkdir -p /var/lib/almaforge/kubeconfigscp prod.yaml /var/lib/almaforge/kubeconfigs/chmod 600 /var/lib/almaforge/kubeconfigs/prod.yaml
One agent, many clusters

One agent can serve many clusters at once from a single kubeconfig. The example above shows one cluster for clarity. To add more, append additional clusters, users, and contexts entries to the same kubeconfig (or merge multiple kubeconfigs with KUBECONFIG=a.yaml:b.yaml kubectl config view --flatten), then create one KubernetesTarget per context in Step 6. The context name is what AlmaForge registers as the cluster name, so make it match the target's metadata.name.

Step 5. Issue a join token and write the env file​

Terminal
alma create token --services=kubernetes

Write /etc/almaforge/almad.env with the proxy address, the token, and the kubeconfig path. The systemd unit loads this file. Restrict permissions because the token is sensitive. The example file sets:

  • ALMA_PROXY: the AlmaForge Proxy address.
  • ALMA_CLIENT_ID: the client_id printed by alma create token in this step.
  • ALMA_CLIENT_SECRET: the client_secret printed by alma create token in this step.
  • ALMA_KUBERNETES_SERVICE=true: turns on the Kubernetes Service role.
  • ALMA_KUBECONFIG: the kubeconfig path from Step 4.
  • ALMA_MATCH_LABELS: only needed for the centralized-apply path in Step 6.

almad.env, a template to copy to /etc/almaforge/almad.env (then chmod 600):

examples/kubernetes/remote-agent/almad.env
# Environment file for the almad systemd unit on a VM.
# Path: /etc/almaforge/almad.env
# Permissions: chmod 600 (the join token is sensitive).

# AlmaForge Proxy address (host:port). The agent connects here
# over mTLS and registers with the Auth Service through the same
# socket.
ALMA_PROXY=<proxy-hostname>:443

# Join token from `alma create token --services=kubernetes --client-id=<name>`.
# Default TTL is 30 minutes. The token is NOT single-use and
# stays valid for that window. The Auth Service exchanges it
# for real credentials on first connect; identity is then
# persisted in the agent's data dir and the token is no longer
# used. Revoke a leaked token with
# `alma delete token/<client-id>`.
ALMA_CLIENT_ID=<paste-client-id-here>
ALMA_CLIENT_SECRET=<paste-client-secret-here>

# Enable the Kubernetes Service role.
ALMA_KUBERNETES_SERVICE=true

# SSH is not needed on a Kubernetes agent.
ALMA_SSH_SERVICE=false

# Path to the kubeconfig staged in Step 1. The agent uses this
# identity to impersonate AlmaForge users against the target
# cluster's API server.
ALMA_KUBECONFIG=/var/lib/almaforge/kubeconfigs/prod.yaml

# Required only when KubernetesTarget resources are applied
# centrally (Step 3). Drop this line if you use the drop-in path.
ALMA_MATCH_LABELS=env=<env>
Terminal
chmod 600 /etc/almaforge/almad.env

Step 6. Register the cluster​

For VMs, the simplest default is centralized apply. Apply the KubernetesTarget to the AlmaForge Auth Service from any host with alma admin access. The resource lives in the Auth Service backend, and any agent whose ALMA_MATCH_LABELS selects the target's labels picks it up.

ALMA_MATCH_LABELS is the selector the agent uses for centrally applied KubernetesTarget, AppTarget, and DatabaseTarget resources. When one agent serves several resource kinds, the same selector applies to all of them.

kubernetestarget.yaml, applied via alma apply:

examples/kubernetes/remote-agent/kubernetestarget.yaml
# Registers a remote Kubernetes cluster with AlmaForge. Apply
# centrally with `alma apply -f` (recommended for VMs)
# or drop into /etc/almaforge/almad.d/ on the agent host. The shape is
# identical to the in-cluster mode.
#
# - metadata.name is the cluster name users type into
#   `alma kube login`.
# - metadata.labels are matched by AlmaForge role kubernetesLabels
#   and by ALMA_MATCH_LABELS on agents that pick up centrally
#   applied targets.
# - spec: {} means "use the kubeconfig at ALMA_KUBECONFIG on the
#   agent host".
apiVersion: almaforge.com/v1
kind: KubernetesTarget
metadata:
  name: prod-us-east
  labels:
    env: prod
    region: us-east-1
spec: {}
Terminal
alma apply -f kubernetestarget.yamlkubernetestarget "<cluster-name>" has been created

Alternatively, place the same YAML in /etc/almaforge/almad.d on the remote-agent VM. The agent reads these targets at startup. Restart it after changing a file-mounted KubernetesTarget. Use this path when configuration management owns the resource file and the VM is the source of truth. Remove ALMA_MATCH_LABELS from almad.env in this setup. The selector applies only to centrally managed resources.

Step 7. Start the agent and verify​

Install almad and its systemd unit on the remote-agent VM before starting the service. See Install the package.

Terminal
systemctl enable --now almadalma get kubeCluster        Labels                       Version-------------- ---------------------------- -------prod-eu-central env=prod,region=eu-central  2026.1.26

The cluster can take a short time to appear. Centrally applied target changes are watched automatically. File-mounted target changes require systemctl restart almad. Registration is complete when alma get kube lists it.

RBAC​

This section covers the access-control chain: define an AlmaForge Role, map it to Kubernetes groups, then bind those groups on the cluster.

Minimal happy path:

  1. Create one AlmaForge role that grants access to one labeled cluster and impersonates one Kubernetes group.
  2. Bind that Kubernetes group to a native ClusterRole or RoleBinding on the cluster.
  3. Assign the AlmaForge role to a user or SSO group.
  4. Log in with alma login, then alma kube login <cluster-name>, and run kubectl.

Overview​

Two kinds of RBAC are in play. Agent RBAC is the ServiceAccount and ClusterRole that let almad itself talk to the API server, as configured in Step 3 of the in-cluster agent setup. User RBAC is everything in this section: the AlmaForge Role resources that decide which Kubernetes identity each operator gets, plus the matching native bindings that grant that identity real verbs.

This section covers user RBAC. It assumes agent RBAC is already in place from the deployment step.

Access to a registered cluster is governed by a two-layer model:

  1. AlmaForge decides which Kubernetes identity the user gets. A AlmaForge Role names one or more Kubernetes groups and users. The AlmaForge Kubernetes Service forwards each kubectl request with impersonation headers that assume those groups and that user.
  2. Kubernetes decides what that identity can do. A native RoleBinding or ClusterRoleBinding must bind the impersonated group to a ClusterRole that grants real verbs on real resources. AlmaForge never grants Kubernetes verbs directly. It only controls who impersonates whom.

If either layer is missing, the user gets a 403.

Kubernetes impersonation, in one paragraph

"Impersonation" is a built-in Kubernetes mechanism. A request that includes Impersonate-User or Impersonate-Group HTTP headers is processed as if it came from the named user or group, not from whoever actually authenticated the connection. The Kubernetes API server runs the impersonated identity through its native RBAC. AlmaForge uses this mechanism so that operators authenticate to AlmaForge once with SSO, but the API server still sees standard Kubernetes principals on every request.

The end-to-end chain:

text
operator → SSO IdP claims → AlmaForge connector
        → AlmaForge Role(s)
        → kubernetesGroups + kubernetesUsers
        → impersonation headers on the kubectl request
        → Kubernetes RBAC (RoleBinding / ClusterRoleBinding)
        → verb on resource

On the AlmaForge side, a Role controls:

  • which clusters it makes available (kubernetesLabels)
  • which objects inside those clusters are visible (kubernetesResources)
  • which Kubernetes groups and users get impersonated (kubernetesGroups, kubernetesUsers)

On the Kubernetes side, a binding maps each impersonated group to real verbs. AlmaForge does not create these bindings. Adding a new group to kubernetesGroups requires a matching RoleBinding or ClusterRoleBinding on every target cluster.

The agent's own RBAC (the ServiceAccount and ClusterRole that let it impersonate) was already configured in Step 3 of the in-cluster agent setup. For remote agents, apply the 02-rbac.yaml pattern from Step 3 of the remote cluster setup to each target cluster.

The agent runs a SelfSubjectAccessReview preflight at registration and logs a loud warning if impersonate is missing. Fix the RBAC before users notice.

Step 1. Define an AlmaForge Role​

A Role resource names which Kubernetes groups and users authenticated operators get impersonated as, and optionally filters which clusters and which objects are visible.

kube-access.yaml, applied via alma apply:

examples/kubernetes/roles/kube-access.yaml
apiVersion: almaforge.com/v1
kind: Role
metadata:
  name: kube-access
spec:
  allow:
    kubernetesLabels:
      env: prod
    kubernetesResources:
      - kind: pod
        namespace: 'app-*'
        name: '*'
        verbs: ['get', 'list', 'watch']
    kubernetesGroups:
      - developers
    kubernetesUsers:
      - '{{external.email}}'

The four Kubernetes-specific fields:

  • kubernetesLabels picks which clusters the role makes available. It is matched at request time against the live Kubernetes heartbeat, the resource visible in alma get kube. Supports exact, glob (prod-*), regex (^.*$), and list values. Use kubernetesLabelsExpression for predicate logic beyond key/value matching.
  • kubernetesResources filters which objects inside matched clusters the user can see and act on. Each entry has kind, namespace, name, verbs. Anything not listed is denied.
  • kubernetesGroups is the list of Kubernetes groups AlmaForge impersonates on every request. Always automatic. Values from all matching roles are unioned.
  • kubernetesUsers is the impersonated Kubernetes user. With one value, auto-selected. With many, the operator picks one with --as. Empty falls back to the AlmaForge username.

For the full schema, supported kinds, verbs, label-expression syntax, and trait templates ({{external.email}}, {{internal.kubernetes_users}}), see the RBAC Reference.

Map SSO traits into Kubernetes identity​

SSO claims supply the traits used to resolve Kubernetes identities. The templates let one role resolve a different identity for each user.

This example expects claims named kubernetes_users and kubernetes_groups. If your provider uses different names, update the templates to match.

{{internal.<name>}} and {{external.<name>}} read the same trait map. The internal. prefix accepts only the supported variable names. Other names produce an unsupported-variable error. Use dot notation for a plain identifier. Use bracket notation when the claim name contains a hyphen, colon, slash, or another character that is invalid in an identifier.

group-member.yaml, applied via alma apply:

examples/kubernetes/roles/group-member.yaml
apiVersion: almaforge.com/v1
kind: Role
metadata:
  name: group-member
spec:
  allow:
    kubernetesUsers:
      - '{{internal.kubernetes_users}}'
    kubernetesGroups:
      - '{{internal.kubernetes_groups}}'

A user whose resolved kubernetes_groups trait contains developers and viewers will be impersonated as both Kubernetes groups, with their kubernetes_users trait supplying the impersonated user. No per-user role edits are needed when group membership changes.

Impersonate a Kubernetes ServiceAccount​

For automation that needs a stable Kubernetes identity (CI bots, controllers running through AlmaForge workload identity), use the system:serviceaccount:<namespace>:<name> form in kubernetesUsers. The ServiceAccount and its native bindings already exist on the cluster. The AlmaForge role decides who gets to impersonate it.

kube-bot-access.yaml, applied via alma apply:

examples/kubernetes/roles/kube-bot-access.yaml
apiVersion: almaforge.com/v1
kind: Role
metadata:
  name: kube-bot-access
spec:
  allow:
    kubernetesLabels:
      env: prod
    kubernetesResources:
      - kind: deployment
        namespace: 'ci-*'
        name: '*'
        verbs: ['get', 'list', 'patch']
    kubernetesUsers:
      - 'system:serviceaccount:ci-system:deploy-bot'

Step 2. Bind the impersonated groups in Kubernetes​

Every Kubernetes group named in any kubernetesGroups field must be bound to a real ClusterRole (or Role) on every target cluster. AlmaForge does not create these bindings.

For the example above, bind developers to the built-in view ClusterRole cluster-wide.

developers-binding.yaml, applied via kubectl apply:

examples/kubernetes/bindings/developers-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: almaforge-developers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
  - kind: Group
    name: developers
    apiGroup: rbac.authorization.k8s.io

For a namespace-scoped binding, swap ClusterRoleBinding for RoleBinding and add metadata.namespace.

The built-in system:masters group is already bound to cluster-admin on every Kubernetes cluster. If a AlmaForge role impersonates system:masters, no extra binding is needed. Treat this as a break-glass path, not a normal design pattern. It grants cluster-admin to anyone who receives that impersonated group, so prefer a dedicated group bound to a narrower ClusterRole or namespace-scoped RoleBinding.

Step 3. Assign the role to users​

Map an IdP claim, such as group membership, to kube-access through the OIDC connector's claimsToRoles. The user then signs in again to receive credentials with the new role. See Single Sign-On.

For on-demand role escalation, see Access Requests.

Step 4. Combine roles and apply deny rules​

A user with multiple roles is evaluated as follows. AlmaForge first selects every role whose kubernetesLabels matches the target cluster, then unions kubernetesGroups and kubernetesUsers from each allow block. Visible objects are the union of every allow.kubernetesResources across matching roles. A matching deny.kubernetesGroups or deny.kubernetesUsers entry subtracts from the impersonation header. A matching deny.kubernetesResources entry denies access to that resource for the request. Deny on other role fields, for example rules, is still a whole-request veto. This lets you grant broad access through one role and punch specific holes through another:

kube-viewer.yaml, applied via alma apply:

examples/kubernetes/roles/kube-viewer.yaml
apiVersion: almaforge.com/v1
kind: Role
metadata:
  name: kube-viewer
spec:
  allow:
    kubernetesLabels:
      '*': '*'
    kubernetesResources:
      - kind: pod
        namespace: '*'
        name: '*'
        verbs: ['get', 'list', 'watch']
    kubernetesGroups:
      - viewer

nginx-exec.yaml, applied via alma apply:

examples/kubernetes/roles/nginx-exec.yaml
apiVersion: almaforge.com/v1
kind: Role
metadata:
  name: nginx-exec
spec:
  allow:
    kubernetesLabels:
      '*': '*'
    kubernetesResources:
      - kind: pod
        namespace: '*'
        name: 'nginx-*'
        verbs: ['exec']
    kubernetesGroups:
      - exec-and-logs

deny-redis-exec.yaml, applied via alma apply:

examples/kubernetes/roles/deny-redis-exec.yaml
apiVersion: almaforge.com/v1
kind: Role
metadata:
  name: deny-redis-exec
spec:
  deny:
    kubernetesResources:
      - kind: pod
        namespace: '*'
        name: 'redis-*'
        verbs: ['exec']
    kubernetesGroups:
      - exec-and-logs

A user with all three roles can list pods everywhere as viewer, exec into nginx-* as exec-and-logs, and hit an access denial for kubectl exec against redis-* pods. The deny rule only affects the matching exec target. It does not remove the user's broader read access.

For the full evaluation order, predicate logic, and trait templates, see the Access Controls reference.

Security considerations​

Narrow AlmaForge roles with broad Kubernetes groups. When you restrict a role's kubernetesResources to a subset of namespaces or names, you are restricting what AlmaForge shows the user. You are not restricting what the impersonated Kubernetes group can read upstream. List operations exploit this gap.

Suppose a user has a role that grants pod access only in the default namespace but impersonates system:masters.

narrow-system-masters.yaml:

examples/kubernetes/roles/narrow-system-masters.yaml
apiVersion: almaforge.com/v1
kind: Role
metadata:
  name: narrow-system-masters
spec:
  allow:
    kubernetesLabels:
      '*': '*'
    kubernetesResources:
      - kind: pod
        namespace: 'default'
        name: '*'
        verbs: ['get', 'list', 'watch']
    kubernetesGroups:
      - system:masters

And a second role that grants pod access cluster-wide through a narrow group:

broad-default-viewer.yaml:

examples/kubernetes/roles/broad-default-viewer.yaml
apiVersion: almaforge.com/v1
kind: Role
metadata:
  name: broad-default-viewer
spec:
  allow:
    kubernetesLabels:
      '*': '*'
    kubernetesResources:
      - kind: pod
        namespace: '*'
        name: '*'
        verbs: ['get', 'list', 'watch']
    kubernetesGroups:
      - default-viewer

When the user runs kubectl get pods -A, the AlmaForge Kubernetes Service forwards the list request to the API server as system:masters + default-viewer, gets back every pod in every namespace, and then filters the response to what either role's kubernetesResources allows. Because the second role allows namespace: "*", nothing is filtered out. The user sees every pod in the cluster, even though the system:masters role was supposed to be scoped to default.

The fix is to keep the narrow-scoped AlmaForge role narrow on both axes: tighten the Kubernetes group too.

narrow-default-viewer.yaml:

examples/kubernetes/roles/narrow-default-viewer.yaml
apiVersion: almaforge.com/v1
kind: Role
metadata:
  name: narrow-default-viewer
spec:
  allow:
    kubernetesLabels:
      '*': '*'
    kubernetesResources:
      - kind: pod
        namespace: 'default'
        name: '*'
        verbs: ['get', 'list', 'watch']
    kubernetesGroups:
      - default-viewer

Rule of thumb: if a role restricts namespaces in kubernetesResources, the Kubernetes group it impersonates must be restricted to the same namespaces by a RoleBinding. Don't mix scoped AlmaForge filters with cluster-wide Kubernetes groups and expect the filter to hold.

Connect to Kubernetes​

The user workflow is two commands: log in to AlmaForge, then log in to the Kubernetes cluster. After that, regular kubectl commands keep working through the exec plugin.

Log in to AlmaForge​

Terminal
alma --proxy=almaforge.example.com:443 login

Authenticates against AlmaForge through SSO and stores short-lived certificates in the alma profile directory for follow-up commands.

List Kubernetes clusters​

Terminal
alma get kubeCluster        Labels                       Version-------------- ---------------------------- -------prod-eu-central     env=prod,region=eu-central     2026.1.26staging-eu-central  env=staging,region=eu-central  2026.1.26

The list is filtered by your AlmaForge roles. Clusters whose labels do not match any role's kubernetesLabels are hidden.

Connect to a single cluster​

Terminal
alma kube login prod-eu-centralLogged into Kubernetes cluster "prod-eu-central". Try 'kubectl version' to test the connection.

This adds a kubeconfig context to your default kubeconfig (~/.kube/config, or the path in $KUBECONFIG) and switches the active context to it. The context uses alma as a credential exec plugin. It supplies a valid short-lived certificate for requests through the AlmaForge Proxy, reusing cached credentials while they remain valid.

The alma binary must be on the operator's $PATH for kubectl to find it as the exec plugin. If the alma session expires mid-kubectl, the plugin returns an authentication error and the operator must re-run alma login.

After login, regular kubectl works as usual:

Terminal
kubectl get pods -n app-apikubectl exec -it webapp-0 -- /bin/shkubectl port-forward svc/api 8080:80kubectl logs -f deployment/api

Switch between clusters​

Each alma kube login adds (or refreshes) a context and switches the active context to it. Logging in to a different cluster is the shortest way to switch:

Terminal
alma kube login staging-eu-centralkubectl get pods               # now runs against staging

Every previously logged-in cluster also remains available as a standard kubeconfig context. The context name format is <almaforge-cluster>-<kubernetes-cluster>. For example, if your AlmaForge cluster name is root and the Kubernetes cluster is prod-eu-central, the context becomes root-prod-eu-central. List, switch, or pin contexts with regular kubectl commands:

Terminal
kubectl config get-contextskubectl config use-context root-staging-eu-centralkubectl --context=root-prod-eu-central get nodes

Generate a kubeconfig with every cluster at once​

--all logs in to every cluster the user has access to in one shot. Each lands as its own kubeconfig context. Use it on laptops, CI runners, or scripts that hop across many clusters without re-prompting:

Terminal
alma kube login --all

Default context names follow <almaforge-cluster>-<kube-cluster>. Override with --set-context-name and the {{.KubeName}} template. For example, this drops the AlmaForge-cluster prefix:

Terminal
alma kube login --all --set-context-name='{{.KubeName}}'

--all is mutually exclusive with --labels and --query. Use those for a subset:

Terminal
alma kube login --labels='env=prod'alma kube login --query='labels["env"] == "prod" && labels["region"] != "us-east-1"'

Whether you log in to one cluster or to many, alma kube login writes or refreshes a stanza in your kubeconfig that hands credential issuance back to alma through the client.authentication.k8s.io/v1beta1 credential exec plugin contract.

kubectl invokes alma kube credentials when it needs credentials, receives a short-lived X.509 certificate, and uses it to call the AlmaForge Proxy. The certificate's TTL is bound to the operator's AlmaForge login. When that session expires, the plugin returns an authentication error and the operator re-runs alma login. There are no long-lived secrets in this file: the CA is public and credential retrieval is delegated to the binary.

The alma binary must remain available at the path recorded in the kubeconfig. If you move the binary, regenerate the kubeconfig with alma kube login so the command field points at the new path.

Show kubeconfig stanza

kubeconfig-stanza.yaml, the reference shape written by alma kube login:

examples/kubernetes/kubeconfig-stanza.yaml
# Reference shape of the kubeconfig stanza that `alma kube login`
# writes (or refreshes) under ~/.kube/config. The exec plugin hands
# credential issuance back to `alma` on every kubectl call,
# so there are no long-lived secrets in this file: the CA is public
# and short-lived X.509 certificates are minted on demand.
apiVersion: v1
kind: Config
clusters:
  - name: <proxy-cluster>
    cluster:
      server: https://<proxy-hostname>:443
      certificate-authority-data: <base64 cluster CA>
      tls-server-name: kube-almaforge-proxy-alpn.<proxy-cluster>
contexts:
  - name: <proxy-cluster>-<kube-cluster>
    context:
      cluster: <proxy-cluster>
      user: <proxy-cluster>-<kube-cluster>
      extensions:
        - name: almaforge.kube.name
          extension: <kube-cluster>
users:
  - name: <proxy-cluster>-<kube-cluster>
    user:
      exec:
        apiVersion: client.authentication.k8s.io/v1beta1
        command: alma
        args:
          - kube
          - credentials
          - --kube-cluster=<kube-cluster>
          - --proxy=<proxy-hostname>:443

Success check​

Run one allowed command and one denied command. Replace app-api with a namespace your role should allow, and keep kube-system or another namespace your role should not allow:

Terminal
kubectl get pods -n app-apikubectl get pods -n kube-systemError from server (Forbidden): pods is forbidden...

The setup is working if all of the following are true:

  1. alma get kube shows the cluster you just registered.
  2. alma kube login <cluster-name> writes a kubeconfig context without error.
  3. The first kubectl get pods works in a namespace your role allows.
  4. The second kubectl get pods returns forbidden in a namespace your role does not allow.

Transient local proxy (no kubeconfig changes)​

alma proxy kube does not touch your default kubeconfig. Instead it:

  1. Starts a local TCP proxy on 127.0.0.1:<random-port>.
  2. Writes a fresh kubeconfig file to a temporary path that points at that local address.
  3. Prints the path so you can export KUBECONFIG=... and run kubectl in that shell only.

When you Ctrl+C the command, the proxy stops and the temp kubeconfig is discarded. Your ~/.kube/config is never touched.

Terminal
alma proxy kube prod-eu-central

Use it when:

  • You don't want to leave state in ~/.kube/config (CI, one-off scripts, shared workstations).
  • A corporate HTTPS proxy interferes with the direct connection from kubectl to the AlmaForge Proxy. The local proxy isolates the workaround from kubectl.
  • Tooling expects kubectl to talk to a fixed local address.

You can pass cluster names, --labels, or --query to choose which clusters to proxy. With no arguments, the local proxy serves the clusters you have already loaded with alma kube login.

Troubleshooting​

alma get kube shows nothing​

The user's roles do not match the cluster labels. Run alma get kube -o yaml as an admin to see how the cluster is actually labeled, then compare that output with the role's kubernetesLabels matcher.

kubectl get pods returns forbidden​

A forbidden response can come from either access-control layer. Check that the AlmaForge role permits the resource and verb through kubernetesResources, then check the native RoleBinding or ClusterRoleBinding for the impersonated Kubernetes identity. See RBAC for the complete chain.

Agent logs say impersonate is forbidden​

The agent's ServiceAccount, or the identity in its kubeconfig, is missing the impersonate verb on users, groups, or serviceaccounts. Grant it via the ClusterRole in RBAC.

Agent logs Failed to test the necessary Kubernetes permissions​

The agent's preflight SelfSubjectAccessReview check failed. The agent still registers the cluster, but user requests hit the same error at runtime. Either the impersonate verb is missing, or the create verb on authorization.k8s.io/selfsubjectaccessreviews is missing. Fix it before users hit request-time denials.

A centrally applied KubernetesTarget never shows up in alma get kube​

No agent matches the target's labels. Every centrally applied target needs at least one agent whose ALMA_MATCH_LABELS selects it. Check agent logs and the target's metadata.labels. Agents using only the drop-in path do not need a match label, but they also do not pick up centrally applied targets.

alma kube login succeeds but kubectl exec is denied​

The exec verb must be listed explicitly in a matching kubernetesResources entry. verbs: ["get", "list", "watch"] allows kubectl get pods but not kubectl exec. Add exec, and portforward if you use kubectl port-forward, to the role.

Agent on GKE Autopilot logs cannot impersonate "system:masters"​

GKE Autopilot blocks impersonation of the built-in system:masters group at admission, even when the agent has the impersonate verb. Use a regular Kubernetes group bound to cluster-admin instead. In the AlmaForge Role, replace kubernetesGroups: [system:masters] with a group like almaforge-cluster-admin, then bind it on the cluster.

gke-autopilot-cluster-admin.yaml, applied via kubectl apply:

examples/kubernetes/bindings/gke-autopilot-cluster-admin.yaml
# Workaround for GKE Autopilot, which blocks impersonation of the
# built-in `system:masters` group at admission. Bind a regular group
# to `cluster-admin` and reference it from the AlmaForge Role's
# kubernetesGroups instead.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: almaforge-cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: Group
    name: almaforge-cluster-admin
    apiGroup: rbac.authorization.k8s.io

Agent fails to join with no authorities for hostname​

The agent's persisted identity is signed by a CA the cluster no longer trusts (typically after a CA rotation or after the AlmaForge cluster was reinstalled). The agent will not re-join automatically because it still holds a stored identity.

For the in-cluster StatefulSet, delete the affected pod's <pod>-state Secret, mint a fresh join token, update ALMA_CLIENT_ID and ALMA_CLIENT_SECRET in the almaforge-env Secret, and restart the pod. The healthy replica keeps serving in the meantime.

For an external agent, stop the agent, delete its data directory (default /var/lib/almaforge), update its join token, and start it again.

Operations​

Break-glass access​

Keep one native cluster-admin break-glass path outside AlmaForge so you can deploy or repair AlmaForge itself, fix a bad RoleBinding, remove a broken KubernetesTarget, or disable the agent if you lock yourself out through impersonation RBAC.

That path should stay independently permissioned at cluster-admin. It must keep working even when AlmaForge RBAC, impersonation, or target configuration is broken.

For managed Kubernetes, that usually means the cloud provider's own credential flow:

  • GKE: gcloud container clusters get-credentials
  • EKS: aws eks update-kubeconfig
  • AKS: az aks get-credentials --admin
  • Other clusters: keep an emergency admin kubeconfig, client certificate, or other pre-positioned cluster-admin credential.

High availability​

For the in-cluster deployment in this page, the example StatefulSet already runs two replicas. That is the default high-availability shape. Each replica gets its own stable pod name and its own persisted identity in a <pod>-state Secret. With two replicas, Kubernetes can restart, reschedule, or roll one pod while the other keeps serving, assuming the cluster can schedule both replicas. For real fault isolation, make sure the cluster can place the two replicas separately instead of landing both on the same failure domain.

For the non-Kubernetes path, run two agents on two separate hosts. Point both at the same target clusters and, if you use centralized apply, give both hosts matching ALMA_MATCH_LABELS so both agents pick up the same KubernetesTarget. This gives you a second serving path when one host is down for maintenance or fails unexpectedly.

Replica lifecycle​

Each replica joins with a short-lived join token, receives its own long-lived host identity certificates (SSH and TLS), and stores that identity in its <pod>-state Secret. That Secret is not application data. It is the replica's re-creatable identity state.

You do not need to back up <pod>-state as a separate disaster-recovery asset. If a replica loses its Secret, mint a fresh join token, update ALMA_CLIENT_ID and ALMA_CLIENT_SECRET in the shared almaforge-env Secret, and restart only the affected pod so it can join again and receive a new identity. The healthy replica keeps serving from its own <pod>-state Secret and does not need to re-join.

The agent updates the Secret when its stored identity changes. During CA rotation it can also store temporary replacement identity and rotation state in the same Secret until the rotation finishes.

Once both replicas are healthy, revoke or delete the join token. Keep a live join token only when you are intentionally bringing up a new replica or recovering one whose <pod>-state Secret was lost.

Kubeconfig lifecycle​

This applies only to the external-agent path. Treat a kubeconfig that embeds the output of kubectl create token as a bootstrap or break-glass path, not as the preferred steady state. That kubeconfig inherits the token's server-controlled lifetime. Replace it before expiry, or use an exec-based or cloud-IAM-backed kubeconfig so the VM does not depend on a pasted token.

Observability​

The agent exposes /healthz and /readyz on the diag port (3000 in the StatefulSet example). Kubernetes uses them for the liveness and readiness probes. Operators can curl them too.

For now, watch:

  • StatefulSet Ready count (alert if it stays below replicas longer than your normal rollout or pod-restart envelope).
  • alma get kube returning the cluster (alert if absent).
  • Pod logs for impersonate is forbidden or Failed to test the necessary Kubernetes permissions lines.

Upgrades​

For the in-cluster deployment, upgrade by bumping the image version in the StatefulSet and applying the change. Kubernetes handles the rolling update. Because the example runs two replicas, one pod can update while the other keeps serving. Watch the rollout and wait for both replicas to become Ready before treating the upgrade as complete. Plan that upgrade as part of the same AlmaForge release rollout, and follow the release notes for any version-specific ordering or compatibility requirements.

If the new version misbehaves, roll back with kubectl rollout undo:

Terminal
kubectl -n almaforge rollout undo statefulset/almaforgekubectl -n almaforge rollout status statefulset/almaforge

For the non-Kubernetes path, upgrade one agent host at a time. Stop the first agent, install the new version, start it again, and confirm it is serving the cluster. Then repeat on the second host. Do not upgrade both remote agents at once unless you are prepared for a temporary loss of Kubernetes access through AlmaForge. Keep the remote agents aligned with the same tested AlmaForge release unless the release notes say otherwise.

Uninstall​

Delete the KubernetesTarget first so AlmaForge stops advertising the cluster.

For the in-cluster deployment, delete the agent workload and related objects. If you deployed the example manifests as-is, deleting the entire almaforge namespace removes its StatefulSet, ServiceAccount, Role, RoleBinding, ConfigMap, and Secrets. It also removes any other workloads or data in that namespace, so check its contents first. Delete the agent's ClusterRole and ClusterRoleBinding separately. Those cluster-scoped objects survive namespace deletion.

For the external-agent path, remove the systemd unit or other agent process, delete the kubeconfig file from the host, and revoke or remove the credential it used. That may mean deleting a pasted ServiceAccount token, revoking cloud-backed access, or removing another pre-positioned cluster credential.

Operators can also delete stale kubeconfig contexts with kubectl config delete-context.

References​

  • Declarative configuration: full schema for KubernetesTarget and Kubernetes.
  • Environment configuration: environment variables and the /etc/almaforge/almad.d/ file-drop.
  • RBAC Reference: full schema for the Role resource, including kubernetesLabels, kubernetesResources, kubernetesGroups, and kubernetesUsers.
  • CLI reference: all alma kube, alma apply, alma get, and almad --kubernetes-service flags.
  • Applications access: publish in-cluster HTTP, REST API, and TCP services through the same agent.