Skip to main content

AWS EC2 Integration for AlmaForge

On an AWS EC2 instance, almad reads the instance's tags from the EC2 Instance Metadata Service (IMDS) and attaches them as labels to every resource the agent registers: SSH nodes, applications, databases, and Kubernetes clusters. Each tag key is prefixed with aws/ to distinguish it from ordinary static labels.

There is no config knob to turn on. When almad starts on an EC2 instance, it auto-detects the cloud, probes IMDS, and starts the importer. If the instance is not on EC2, or if IMDS is unavailable, the importer stays idle and the agent runs normally with only its static labels.

TL;DR​

  1. Enable instance metadata tags on the EC2 instance (Step 1). AWS ships this off by default.
  2. Apply your EC2 tags in the AWS console, CLI, or your IaC tool. They show up as aws/<key> labels on the agent's resources (Step 2).

Prerequisites​

Before you start, make sure you have:

  • One almad agent running on an Amazon EC2 instance and joined to a AlmaForge cluster.
  • Permission to toggle instance metadata tags on that EC2 instance, either in the AWS console or through the AWS CLI. This is a per-instance setting. No IAM policy change is required on the agent.
  • A working alma CLI pointed at the cluster so you can list resources after applying tags.

Step 1. Enable instance metadata tags​

EC2 ships with instance metadata tags disabled, so by default IMDS does not expose the instance's tags to anything running on the instance, including almad. Turn the setting on before you expect labels to appear.

Restart behavior

Instances on the Nitro system pick up new, changed, and deleted tags while running. Everything else must be restarted after the first time you enable the setting. See the AWS documentation on working with tags in IMDS.

Option A: AWS Console​

To launch a new instance with instance metadata tags enabled:

  1. On the launch page, open Advanced details near the bottom.
  2. Confirm that Metadata accessible is Enabled.
  3. Set Allow tags in metadata to Enabled.

To modify an existing instance:

  1. From the instance summary, open Actions > Instance settings > Allow tags in instance metadata.
  2. Check Allow and save.

Option B: AWS CLI​

On launch:

Terminal
aws ec2 run-instances \    --image-id image-id \    --instance-type instance-type \    --metadata-options "InstanceMetadataTags=enabled" \    ...

On an existing instance:

Terminal
aws ec2 modify-instance-metadata-options \    --instance-id i-0123456789abcdef0 \    --instance-metadata-tags enabled

Step 2. Verify labels on the agent​

Tag the instance in the AWS console, the AWS CLI, or your IaC tool. almad fetches all tags once on startup and then on a one-hour interval, so give it up to an hour to pick up new or changed tags. Restarting the agent re-runs the fetch immediately.

List the node to confirm the labels landed. Each EC2 tag appears as an aws/<key> label. The table view truncates long label lists, so pass -o yaml to see every label at once:

Terminal
alma get node -o yamlapiVersion: almaforge.com/v1kind: Servermetadata:  labels:    env: prod    aws/Name: db-01    aws/Team: platform    aws/CostCenter: rd-42  name: 5c1f...spec:  useTunnel: true  hostname: db-01.example.com

Tag values pass through unchanged. Tag keys that are not valid AlmaForge label keys are skipped with a debug-level log on the agent and do not appear in the output. This is rare but worth knowing about if a tag you expected is missing.

The same labels flow to any other resource the agent registers. If the host runs the application service, the database service, or the Kubernetes service as well as SSH, each resource they register receives the same aws/... label set.

Troubleshooting​

aws/ labels never appear​

Start with IMDS. From the instance itself, confirm the tags endpoint responds:

Terminal
TOKEN=$(curl -sX PUT "http://169.254.169.254/latest/api/token" \    -H "X-aws-ec2-metadata-token-ttl-seconds: 60")curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \    http://169.254.169.254/latest/meta-data/tags/instance

If that returns a 404, instance metadata tags are still disabled on the instance. Redo Step 1. On non-Nitro instances, restart the instance after enabling the setting the first time.

almad also logs this case once per process lifetime at warning level: Could not fetch EC2 instance's tags, please ensure 'allow instance tags in metadata' is enabled on the instance.

Labels are stale​

The importer refreshes tags on a fixed one-hour schedule. A tag you added five minutes ago may not be visible yet. To force a refresh, restart almad. The first action the importer takes after startup is a full fetch.

Operations​

Blast radius of a bad tag​

Tags enter the agent as static labels, so any role or access rule that matches on aws/... picks up whatever value IMDS returns. Access rules that grant access when a label matches a specific value use the imported label after the agent refreshes and publishes it. A tag edit does not revoke access immediately. For an urgent cutoff, use a lock. Treat write permission on EC2 tags as equivalent to write permission on the labels those tags drive. Scope the AWS IAM permission ec2:CreateTags and ec2:DeleteTags on production instances accordingly.

Refresh cadence and determinism​

The one-hour refresh interval is fixed in the agent and not configurable. A tag change on EC2 propagates to AlmaForge after the next successful refresh and resource update. Restarting the agent triggers a new fetch, and reuses existing certificates, but can interrupt sessions on that host.

Interaction with static labels​

Static labels set in the agent's configuration and aws/... labels coexist. Static labels take precedence if the same key is configured in both places. Reserve aws/ keys for the importer so a manual value does not mask the EC2 tag. See RBAC label matching to use imported labels in roles.

References​