Monitoring and signals

almad exposes health and metrics endpoints on two configurable listeners:

• /healthz and /readyz - liveness and readiness, on the address ALMA_DIAG_LISTEN binds. It has no default, so the endpoint is off until you set one. This page uses :3000 throughout.
• /metrics - Prometheus exposition format, on the address ALMA_METRICS_LISTEN binds. An address given without a port gets :3081. While the variable is unset, the diagnostic listener serves /metrics itself, so binding only ALMA_DIAG_LISTEN still exposes metrics.

The diagnostic port is what your load balancer health-checks. The metrics port is what Prometheus scrapes. Both should stay on private networks.

Scrape configuration

Enable both endpoints in /etc/almaforge/almad.env:

/etc/almaforge/almad.env

ALMA_DIAG_LISTEN=0.0.0.0:3000
ALMA_METRICS_LISTEN=0.0.0.0:3081

The metrics listener serves mutual TLS or plaintext, with nothing in between. To require a certificate from the scraper, set all three of these next to ALMA_METRICS_LISTEN:

env

ALMA_METRICS_TLS_CERT=/etc/almaforge/metrics.crt
ALMA_METRICS_TLS_KEY=/etc/almaforge/metrics.key
ALMA_METRICS_CLIENT_CA=/etc/almaforge/scraper-ca.crt

Setting only some of them fails at startup rather than quietly serving plaintext. Prometheus then needs a tls_config presenting a certificate that ALMA_METRICS_CLIENT_CA issued. Full descriptions are in the environment reference.

Add each almad metrics listener to your Prometheus targets:

prometheus-static.yaml:

examples/monitoring/prometheus-static.yaml

# Static Prometheus scrape targets for the almad metrics endpoint.
# Use hostnames reachable from your Prometheus server.
- job_name: almad
  static_configs:
    - targets:
        - 'almaforge-1.internal:3081'
        - 'almaforge-2.internal:3081'
        - 'almaforge-3.internal:3081'

On Kubernetes, the example bundle ships a podmonitor.yaml template (commented out in kustomization.yaml). Uncomment it after installing the Prometheus operator.

For the full metric catalog, see the Metrics Reference.

Signals to watch

These are the metrics most likely to catch a problem before users do:

Signal	Metric	What it means
Storage backend write latency	backend_write_seconds (p99)	State backend (etcd, PostgreSQL, DynamoDB, or Firestore) is the bottleneck
Host certificate generation pressure	auth_generate_requests_throttled_total	Auth is throttling host and service certificate generation
Audit emit failures	audit_failed_emit_events	Audit event emission is failing. Investigations may have gaps
Active SSH sessions	proxy_ssh_sessions_total	Proxy load envelope
File-descriptor pressure	process_open_fds / process_max_fds	Proxy or Auth is close to running out of sockets
Process CPU	rate(process_cpu_seconds_total[5m])	Sustained pressure means add replicas or bigger nodes
Recording upload latency	s3_requests_seconds, gcs_event_storage_uploads_seconds	Session recordings backing up at the object store

See the Scaling Guide for the full breakdown of what each signal means and how to act on it.

Alerts that earn their keep

Wire these as a starting set. Tune the thresholds for your shape:

• /readyz returning non-200 for more than 30 seconds. The process exists but is not serving. Page on-call.
• audit_failed_emit_events rate above zero for more than 5 minutes. Audit event delivery is failing. Page on-call, preserve daemon logs for the affected interval, and investigate missing records. Document any audit gap and report it under your audit policy.
• auth_generate_requests_throttled_total rate sustained above zero. Host and service certificate generation is being throttled. Check Auth load and state backend latency before resizing Auth.
• LockCreate events (from your audit sink, not Prometheus). Every lock is a security event worth a notification.
• process_open_fds / process_max_fds above 0.7 for any replica. Resize before the limit is hit.

For audit event codes and fields, see Audit Events.

Where to look first

When an alert fires:

1. Check /readyz on every replica. A non-200 response identifies process state, not the failed dependency.
2. Check journalctl -u almad -n 200 on the affected host for the exact error.
3. Cross-reference with the Troubleshooting guide for the failing surface.
4. Check the state backend's own health (etcd member-list, Postgres primary status, DynamoDB throttling metrics).

References

• Metrics Reference - full Prometheus metric catalog.
• Audit Reference - event codes and fields.
• Scaling - how to act on each signal.
• Troubleshooting - symptom-to-cause index.