Skip to main content

Hetzner Cloud Integration for AlmaForge

On a Hetzner Cloud server, almad reads the Hetzner instance metadata service on startup and attaches the server's identity fields as labels to every resource the agent registers: SSH nodes, applications, databases, and Kubernetes clusters. The labels are prefixed with hetzner/ to distinguish them from ordinary static labels.

There is no config knob to turn on. When almad starts on a Hetzner Cloud server, it auto-detects the cloud, probes the metadata service, and starts the importer. The agent recognizes Hetzner by reading the kernel DMI data, specifically /sys/class/dmi/id/sys_vendor starting with Hetzner and product_name starting with vServer. If either probe does not match, the importer stays idle and the agent runs with only its static labels.

TL;DR​

  1. Install almad on a Hetzner Cloud server and join it to a AlmaForge cluster using any join method. No Hetzner- specific steps are required (Prerequisites).
  2. List the node. The hetzner/* labels should already be there (Verify).

Prerequisites​

Before you start, make sure you have:

  • One almad agent running on a Hetzner Cloud server and joined to a AlmaForge cluster. See SSH server enrollment to install and join an agent.
  • A working alma CLI pointed at the cluster so you can list resources after the agent starts.
  • Outbound reachability from the server to http://169.254.169.254/hetzner/v1/metadata. The Hetzner metadata service is available on every Hetzner Cloud server by default, and no authentication is required to read it from the server itself.

Verify labels on the agent​

After the agent starts, list the node. Each Hetzner metadata field appears as a hetzner/<field> label. The table view truncates long label lists, so pass -o yaml to see every label at once:

Terminal
alma get node -o yamlapiVersion: almaforge.com/v1kind: Servermetadata:  labels:    env: demo    region: hel1    hetzner/availability-zone: nbg1-dc3    hetzner/hostname: demo-1    hetzner/instance-id: "12345678"    hetzner/region: eu-central  name: 080dcdae-ee84-487b-82fb-9de120fcc554spec:  useTunnel: true  hostname: demo-1.example.com

In this example the agent has two static labels (env and region) set in its config and four hetzner/* labels imported from the Hetzner metadata service. Notice that region: hel1 and hetzner/region: eu-central coexist cleanly: the static region label is whatever the operator chose to type in the agent config, and the imported hetzner/region label is whatever Hetzner returns from the metadata service. The hetzner/ prefix exists to keep the two namespaces from colliding.

One of the five possible fields, hetzner/local-ipv4, is absent from this example on purpose: the server in the example has no private network attached, so the Hetzner metadata service does not return a local-ipv4 value, and the agent silently skips it. Fields that are present on the server show up. Fields that are absent do not.

The five fields the agent reads are fixed, and the values come directly from the Hetzner metadata service:

  • hetzner/instance-id: the numeric Hetzner Cloud server ID. Useful for correlating a node in AlmaForge with the same server in the Hetzner Cloud console or in hcloud server list.
  • hetzner/region: the Hetzner region, for example eu-central or us-east. Useful as a match label on roles that should only grant access to servers in a particular region.
  • hetzner/availability-zone: the datacenter identifier, for example fsn1-dc14 or nbg1-dc3. Finer-grained than the region field.
  • hetzner/hostname: the hostname Hetzner returns from the metadata service. This is the server name set in Hetzner Cloud, not the Linux hostname on disk.
  • hetzner/local-ipv4: the private IP address Hetzner assigns on the server's default interface.

The importer refreshes these fields on a fixed one-hour schedule, and the first fetch runs at startup.

The same labels flow to every other resource the agent registers. If the host runs the application service, the database service, or the Kubernetes service alongside SSH, each resource they register receives the same hetzner/* label set.

Troubleshooting​

hetzner/* labels never appear​

From the server itself, confirm the metadata service responds:

Terminal
curl -sS http://169.254.169.254/hetzner/v1/metadata/instance-id12345678

If that returns the server's ID, the metadata service is healthy and the agent should be able to read it. Restart almad to force a fresh fetch. The first action the importer takes after startup is a full fetch.

If the curl command hangs, returns nothing, or returns a non- numeric value, the Hetzner metadata service is not reachable from the server. Check the server's network configuration, in particular any egress filtering on the metadata IP 169.254.169.254 and any process listening on that address locally (some cloud-init setups intercept the metadata endpoint).

If the agent does not detect Hetzner at startup at all, check the DMI data:

Terminal
cat /sys/class/dmi/id/sys_vendorHetznercat /sys/class/dmi/id/product_namevServer

Both values must match. The detection is strict on purpose: almad does not want to probe a non-Hetzner metadata service on a lookalike host.

almad also logs the unreachable-metadata case once per process lifetime at warning level: Could not fetch Hetzner instance metadata.

Labels are stale​

The importer refreshes on a fixed one-hour schedule. A field that changed a few minutes ago may not be visible yet. To force a refresh, restart almad.

In practice, none of the five fields change on a running server without operator intervention: the instance ID, region, and availability zone are fixed for the life of the server, the hostname requires a Hetzner Cloud API call to change, and the local IPv4 address only changes if you attach or detach a private network.

Operations​

Hetzner Cloud labels set via the API are not imported​

The Hetzner Cloud API supports free-form labels on a server (hcloud server label add, labels in Terraform). These Hetzner-side user labels live in the Hetzner Cloud control plane and are exposed through the Hetzner Cloud API, not through the local metadata service on the server. almad reads only the local metadata service, so Hetzner-side user labels are not imported today. If you need env=prod-style labels on an agent running on Hetzner, set them as static labels in the agent's configuration or pass them through a join token.

Blast radius of the imported fields​

The agent reports hetzner/* labels from the metadata it reads on the host. These labels help select resources in a role. They are not host attestation and do not establish that a host with compromised root access is trustworthy. Restrict host administration and combine label matching with the target identities and permissions in your access roles.

Refresh cadence​

The one-hour refresh interval is fixed in the agent and not configurable. A metadata change reaches AlmaForge after the next successful fetch. Restarting the agent requests a fresh fetch without waiting for the interval and reuses existing certificates. A restart can interrupt sessions on that host.

Interaction with static labels​

Static labels set in the agent's configuration and hetzner/* labels coexist. Static labels take precedence if the same key is configured in both places. Reserve hetzner/ keys for the importer so a manual value does not mask the metadata value.

References​