Skip to main content

Upgrades

An upgrade replaces the almad binary (or container image) on each replica, one at a time. Existing storage remains in place, but a release can migrate stored state. Check its release notes and take a backup before upgrading. In the recommended deployment, each cluster replica runs Auth and Proxy in one process: upgrade those replicas first, then upgrade agents.

Version policy​

AlmaForge versions use MAJOR.MINOR.PATCH numbering.

Before v1.0​

A 0.x release can include breaking changes to configuration, stored state, or protocols. Read the release notes for each target release and complete any migration steps before upgrading. Patch releases contain compatible bug fixes and security updates.

Upgrade one minor release at a time, including clients and agents before moving to the next release. For example, upgrade from v0.7 to v0.8, then to v0.9. Skipping a minor release is not supported before v1.0.

From v1.0​

From v1.0, compatibility is a release promise. Breaking changes require a new major version. Minor releases add compatible features, and patch releases contain compatible bug fixes and security updates. Read the release notes before each production upgrade.

Upgrade cadence​

Use a regular cadence so upgrades are not rare, high-risk events:

  • Apply patch releases within 30 days. Treat security fixes as same-day changes when the advisory calls for it.
  • Apply minor releases on a quarterly cadence after staging validation.
  • Plan major releases separately. They may require resource-schema or environment-file edits.
  • Do not bundle a state-backend upgrade with a AlmaForge upgrade in the same maintenance window. etcd, PostgreSQL, DynamoDB, and Firestore each have their own upgrade procedure and compatibility window.

Compatibility between components​

During a rolling upgrade, different replicas run different versions simultaneously. The following rules keep the cluster stable during that window.

  • Cluster replicas upgrade before clients and agents. Finish every almad replica before upgrading workstation alma clients or the agent fleet. Newer clients and agents may use protocol features not yet present on an older cluster replica.
  • A 0.x cluster supports its own release and the previous minor release's clients and agents. For example, v0.8 supports v0.7 and v0.8 clients and agents. v0.9 supports v0.8 and v0.9. The server does not currently enforce this minimum version, so a successful connection does not prove an older version is supported. v1.0 accepts clients and agents from the final 0.x release for the transition. The ongoing 1.x support window will be published with v1.0.
  • An agent cannot run a newer major version than the cluster. When an agent connects, it pings Auth and compares the reported major version before continuing. If the agent is newer, it refuses to start. This is why the cluster upgrades first.
  • One replica at a time. Wait for each restarted replica to pass /readyz and stabilize before touching the next. The L4 LB pulls the restarting replica out of rotation automatically while it restarts.
  • Clients and agents follow promptly. Bring them to the cluster's release before beginning the next minor upgrade.

With three healthy replicas and available storage and load balancing, new connections use the replicas that remain in service. On Linux, SIGHUP supports a zero-downtime upgrade: the replacement process inherits the listening sockets and accepts new connections when ready, while the old process serves existing connections until they finish. For a restart, drain active sessions first. Connections still active when the process exits are dropped. See high-availability failure modes.

Pre-flight​

Do this before any production binary or image changes. It applies to patch, minor, and major upgrades.

  1. Snapshot the state backend and verify the snapshot before proceeding.

    Terminal
    # etcdetcdctl snapshot save almaforge-<timestamp>.dbetcdutl snapshot status almaforge-<timestamp>.db
    # PostgreSQLpg_dump -Fc -f almaforge-<timestamp>.dump <database-name>pg_restore -l almaforge-<timestamp>.dump >/dev/null
    # DynamoDBaws dynamodb create-backup --table-name <table-name> --backup-name almaforge-<timestamp>aws dynamodb describe-backup --backup-arn <backup-arn>
    # Firestoregcloud firestore export gs://<bucket>/almaforge-<timestamp>

    Use the command that matches your backend. For restore detail, see Backups.

  2. Confirm the target version passed staging or canary validation. Run the checks from Verification against that environment before touching production.

  3. Confirm monitoring is paging on /readyz, process restarts, and audit-store write errors from the daemon logs.

  4. Announce the maintenance window to stakeholders.

  5. Write down rollback criteria before the window opens. Decide who can call rollback and how long a replica may stay unhealthy before you stop the rollout.

Maintenance windows​

During the window:

  • Freeze role changes, target registrations, and new agent joins unless they are part of the upgrade.
  • Drain one Linux host or Kubernetes pod at a time.
  • Keep the rollback owner present until every almad replica and the first agent batch verify on the new version.
  • Treat a stuck replica as a rollback decision, not as background work to debug while the rest of the fleet advances.

Upgrade order​

  1. almad fleet (one replica at a time, verify before next)
  2. Agent fleet (on their own schedule, after the cluster is on the new version)

Each replica should be fully on the new version before the next restarts. Never skip verification between replicas. Never start the agent fleet before every cluster replica is on the new version.

Upgrade on Linux​

The upgrade procedure is the same as a rolling restart with a new binary. On package-based installs, the DEB and RPM maintainer hooks run systemctl daemon-reload, then restart almad.service when the service is already active. Drain the node from your load balancer before the package install if you need to stop new sessions first. Tarball installs do not have package scripts, so you restart the daemon manually after replacing the binary.

Cluster daemon fleet​

Repeat the steps below for every cluster host, one at a time:

Terminal
sudo apt install almaforge=2026.1.26     # Debian/Ubuntu# orsudo dnf install almaforge-2026.1.26     # RHEL/Rocky/Fedora

The package scripts restart an active daemon. Verify the restarted process before moving to the next replica:

Terminal
sudo systemctl is-active --quiet almad && echo activesudo journalctl -u almad -n 30 | grep -Ei 'started|joined|ready'curl -sf http://localhost:3000/readyz && echo ready

For a tarball install, replace the binary path used by your systemd unit and then restart explicitly:

Terminal
sudo install -m 0755 almad /usr/local/bin/almadsudo systemctl restart almad

Verify the host is on the new version before moving to the next:

Terminal
alma get authalma get proxy

Both commands list the merged Auth+Proxy process from each replica. All entries should show the target version before continuing.

Between replicas, wait for the restarted host to pass /readyz, then wait your runbook's stabilization interval before moving on. Sixty seconds is a reasonable starting point for small clusters.

Agent fleet​

Agents upgrade on their own schedule. The cluster does not gate on agent versions once every almad replica is upgraded. Upgrade each agent with the same exact-version package install or tarball replacement pattern, verifying after each one:

Terminal
alma get nodes                # SSH agentsalma get kube                 # Kubernetes agentsalma get db                   # Database agents

If your hosts use unattended package upgrades, hold the package after the planned upgrade and unhold it only during the next maintenance window:

Terminal
sudo apt-mark hold almaforge

Upgrade on Kubernetes​

The kustomize overlay shipped with the Kubernetes deployment guide declares one StatefulSet named almad with three replicas. To upgrade, update the image tag in your forked overlay and apply:

warning

Never run :latest in production. Pin the exact image tag in your overlay, and change that tag deliberately during the upgrade window.

Terminal
cd <your-overlay>kustomize edit set image quay.io/almaforge/almaforge:2026.1.26kubectl apply -k .

Watch the rollout. Use separate terminal windows so a blocked rollout is visible before the timeout:

Terminal
# window 1: rollout statuskubectl rollout status statefulset/almad -n almaforge-clusterWaiting for 3 pods to be ready...statefulset rolling update complete 3 pods at revision almad-<hash>...
# window 2: pod watchkubectl -n almaforge-cluster get pods \    -l app.kubernetes.io/name=almaforge --watch
# window 3: events when a pod stallskubectl -n almaforge-cluster get events \    --field-selector involvedObject.name=almad-0

The StatefulSet uses the default RollingUpdate strategy. It replaces one pod at a time and waits for the new pod to pass the /readyz probe before continuing. No manual pod-by-pod intervention is needed.

Verify after the rollout completes:

Terminal
kubectl get pods -n almaforge-clusteralma get authalma get proxy

Agents running outside Kubernetes upgrade on their own schedule per the Linux agent procedure above.

Automatic agent updates​

Not available in this release. Upgrade agents manually using the Agent fleet procedure after the cluster upgrade is complete.

Verification​

After upgrading every replica, confirm the client and cluster versions:

Terminal
alma versionalma status | grep Versionalma get auth -o yaml | grep versionalma get proxy -o yaml | grep version

Every almad replica should show the same version. If the client is older than the cluster, expect missing CLI features. Upgrade the client to match before declaring the window complete.

Also confirm the diagnostic endpoint is healthy on every host:

Terminal
curl -sf http://almaforge-1.almaforge.example.com:3000/readyz && echo okcurl -sf http://almaforge-2.almaforge.example.com:3000/readyz && echo okcurl -sf http://almaforge-3.almaforge.example.com:3000/readyz && echo ok

Rollback​

Rollback is the same procedure in reverse: install the previous package on one host at a time, restarting and verifying after each. The rollback order is also reversed: agents first, then the cluster replicas.

Check that the previous version can read the current state before a rolling rollback. Keep healthy replicas serving and verify each replacement before moving on. If restoring the backend is required, stop all replicas that use it and follow the offline recovery procedure.

On Kubernetes, rollback with:

Terminal
kubectl rollout undo statefulset/almad -n almaforge-clusterkubectl rollout status statefulset/almad -n almaforge-cluster

StatefulSets retain pod-template revisions in ControllerRevision objects. rollout undo selects a previous revision. Also restore the image tag in your managed overlay so the next apply keeps that version.

Use this decision tree before relying on rollout undo:

  1. If the new pods are running but the cluster is misbehaving, undo the rollout and verify.
  2. If the new pods are in CrashLoopBackOff, confirm the previous image is still pullable, inspect the failed pod logs, then undo the StatefulSet.
  3. If the older binary cannot read state written by the new version, stop every replica using that backend. Preserve the current state, restore the snapshot from Pre-flight, then redeploy the old image. Changes made after the snapshot are lost from the restored state.

If a failed pod prevents rollback from progressing, first revert the StatefulSet template, then delete only the pod still using the failed revision so the controller can recreate it. See Kubernetes' StatefulSet rollback guidance.

note

State in the backend (roles, connectors, tokens, target registrations) is not rolled back with the binary. If an upgrade wrote a new schema version to the backend, downgrading the binary may not be possible without a backend restore. Check the release notes for the version you are rolling back from before proceeding.

Troubleshooting​

Partial upgrade left in mixed state​

If an upgrade was interrupted midway, identify why it stopped before changing another replica. When the new version is healthy and the interruption was operational, resume one replica at a time. When the new version is failing, use the rollback criteria agreed in Pre-flight. Keep agents on the old version until the cluster upgrade is complete.

Agent is too new for the cluster​

If a newly upgraded agent fails to join with a version-check error, check that every cluster replica is fully on the new version. The agent compares its local major version with the version returned by Ping. A single remaining cluster replica on the old major version can cause intermittent failures when the LB routes to it.

Terminal
alma get authalma get proxy

Every row should show the target version before any agent attempts to join.

Linux package restart happened before drain​

The Linux package scripts restart almad.service automatically when the service is already active. If you installed the package before removing the node from the LB target set, the node may restart while it still receives new connections.

Terminal
sudo journalctl -u almad -n 80 | grep -Ei 'started|listening|ready|error'curl -sf http://localhost:3000/readyz && echo ready

If the node is healthy, leave it on the new version and continue with the next node after the stabilization interval. If it is not healthy, keep it out of rotation and roll that one node back before touching the next replica.

LB drain or health check is stuck​

An L4 target may stay in service because /readyz is still passing, or stay draining because long-lived SSH or database sessions have not closed. Check the LB target state, the local readiness endpoint, and your active-session view before forcing progress.

Terminal
curl -sf http://<node-host>:3000/readyz && echo ready

Continue only when no new sessions are routed to the node and the active sessions are either drained or explicitly approved to disconnect. If you cannot drain within the window, abort the rollout and return the node to service on its current version.

Kubernetes pod stuck or in CrashLoopBackOff​

If kubectl rollout status blocks, identify the failing pod before running rollout undo:

Terminal
kubectl -n almaforge-cluster get podskubectl -n almaforge-cluster describe pod almad-0kubectl -n almaforge-cluster logs almad-0 --previouskubectl -n almaforge-cluster get events \    --field-selector involvedObject.name=almad-0

If the new pod fails readiness because of a transient dependency, keep the rollout paused and fix that dependency. If it fails on configuration, schema, missing secret, or image-pull errors, abort the rollout and use the Kubernetes rollback decision tree above.

Resource or connector rejected on startup​

A release with breaking configuration changes may reject a field that an older binary accepted in a drop-in manifest. OIDCConnector and StorageConfig resources are common places to see this because they are loaded during Auth startup.

On Linux:

Terminal
sudo journalctl -u almad -n 80 | grep -Ei 'unknown|invalid|schema|OIDCConnector|StorageConfig'

On Kubernetes:

Terminal
kubectl -n almaforge-cluster logs almad-0 --previous \    | grep -Ei 'unknown|invalid|schema|OIDCConnector|StorageConfig'

Patch the manifest in source control, apply it to the same replica or overlay, then restart only that replica. If the old binary cannot read the patched resource and the new binary cannot start, stop the rollout and follow Rollback. Restore a backend only when its stored state requires recovery, not merely because a file on disk is invalid.

Cluster daemon fails to start after an upgrade​

A release with breaking changes may add required configuration fields or change backend schema expectations. Read the release notes for the target version and check:

  1. Drop-in manifests under /etc/almaforge/almad.d/ for any deprecated fields the new binary rejects.
  2. The daemon log for the specific error:
Terminal
sudo journalctl -u almad -n 50 | grep -i "error\|unknown\|invalid\|schema"
  1. If the binary refuses to start due to an unexpected backend state, restore the backend from a pre-upgrade snapshot. See Backups for the restore procedure.

Rolling back after a schema change​

Rolling back is potentially unsafe if the new binary already wrote schema changes to the backend. The binary can start, but reads and writes may fail or behave unexpectedly. The safe path is:

  1. Stop every replica on the new version.
  2. Restore the backend from a pre-upgrade snapshot.
  3. Install the old binary on every host and restart.

Further reading​

  • Cluster on Linux: Linux-based upgrade context and the Operations section.
  • Cluster on Kubernetes: kustomize overlay and StatefulSet rollout mechanics.
  • High Availability: why one-replica-at-a-time rolling keeps the cluster serving.
  • Scaling: adding replicas to the cluster and the storage backend.
  • CA Rotation: rotating the cluster CA after an upgrade or key-exposure incident.
  • Backups: snapshot and restore before upgrades.