Skip to main content

PagerDuty Integration for AlmaForge

AlmaForge can file a PagerDuty incident for every access request in your cluster. When a request is created in AlmaForge, the plugin looks up the target PagerDuty service from a request annotation and opens a notification incident. When the request is approved, denied, or expires, the matching incident is resolved too.

It can also approve a request on its own when the requester is already on call for one of the annotated services, so an incident responder elevates without waiting for a reviewer. That is opt-in. See On-call auto-approval.

The integration is built into almad. You do not install a separate plugin, run a callback service, or expose a public URL to PagerDuty. Configure it with a PagerdutyConnector resource, either under Integrations in the Web UI or with alma apply. Both paths store the same resource.

TL;DR​

  1. Mint a PagerDuty REST API key (Step 1).
  2. Identify the target service and on-call service (Step 2).
  3. Apply the example PagerdutyConnector from Step 3.
  4. Annotate the AlmaForge roles that should drive PagerDuty behavior (Step 4).
  5. File a test access request to confirm the incident lands in the service (Step 5).

Limitations​

The integration uses PagerDuty as a notification channel and as the on-call source of truth. It is one-way for reviews: acking or resolving an incident in PagerDuty does not approve or deny the matching AlmaForge access request. Decisions other than the on-call auto-approval must still be made in AlmaForge, either in the Web UI or with alma request review.

Only one PagerDuty integration is supported per cluster. The connector must be named pagerduty. Any PagerdutyConnector applied under a different name is ignored.

On-call auto-approval matches AlmaForge users to PagerDuty users by email address only. A AlmaForge user whose login name is not a valid email address is skipped, even if a PagerDuty user with a matching name exists. There is no trait mapping, alias table, or fallback lookup.

The approval review is authored by the built-in Pagerduty system role, not by the requester or the matched PagerDuty user. AlmaForge enforces this. A plugin cannot impersonate a human approver. The business identity of the approval lives in the review reason text, which always names the real requester and the matched on-call service.

Prerequisites​

Before you start, make sure you have:

  • A running AlmaForge cluster with an Auth Service that you can reach over the network and a working alma CLI pointed at it.
  • A requester allowed to request access and a different user allowed to review that request. See Just-in-Time Access.
  • Outbound HTTPS egress from almad to https://api.pagerduty.com (or https://api.eu.pagerduty.com for EU customers) from every auth host. The integration makes no inbound connections, so no firewall or NAT rules on the AlmaForge side are required.
  • A PagerDuty account with at least one service that AlmaForge should file incidents against. A dedicated service called something like almaforge-access-requests keeps the noise separated from your operational alerts.
  • A PagerDuty user whose identity the plugin will use for REST calls. Create a dedicated bot user whose audit trail you want to see in PagerDuty.
  • For on-call auto-approval to work, each AlmaForge user who should be eligible must have a PagerDuty user with the same email address as their AlmaForge login name.

Step 1. Create a REST API key​

Sign in to PagerDuty as an admin. Open Integrations → API Access Keys → Create New API Key. Give it a descriptive label (for example, almaforge cluster prod), leave the Read-only box unchecked, and save the key value somewhere safe. PagerDuty only shows it once. Access to stored connector secrets requires permission to read secrets.

Treat the API key as a password

Anyone with the REST API key can create, modify, and resolve incidents across your PagerDuty account. Store it in the same secret manager you use for production credentials. Do not commit it to Git.

REST API key, not integration key

The integration needs a REST API key, not a PagerDuty service integration key (also called a routing key or integration key). Integration keys are service-scoped and only let you send events through the Events API. The REST API key is account-scoped and lets the plugin read services, create incidents, and query on-call schedules.

Step 2. Identify the target services​

Decide which PagerDuty service should receive incident notifications for access requests. Also decide which services the plugin should consult when checking whether the requester is on call.

Most clusters use the same service for both. Larger setups split noisy notifications from the critical on-call rotations.

Note the service names. You will reference them by name from your access-request annotations, not by PagerDuty service ID. The lookup ignores case, so the annotation does not have to match the casing PagerDuty shows.

Step 3. Apply a PagerdutyConnector​

Write a YAML file describing the connector:

pagerduty.yaml:

examples/integrations/pagerduty.yaml
apiVersion: almaforge.com/v1
kind: PagerdutyConnector
metadata:
  name: pagerduty
spec:
  # Replace both values below before applying.
  userEmail: almaforge-bot@example.com
  apiKey: replace-me

For a quick setup, keep apiKey inline as shown above. For production, keep the secret out of the manifest and load it from a file or environment variable instead. See Secret management in this page's Operations section.

Replace:

  • userEmail with the email of the PagerDuty user whose identity the plugin should assume for REST calls. This value is sent as the From: header on every PagerDuty API request.
  • apiKey with the REST API key from Step 1.

The connector above only files incidents. To also let an on-call requester through without a reviewer, add approveOnCall: true to the spec. Read On-call auto-approval before you do.

Apply the connector:

Terminal
alma apply -f pagerduty.yaml
One connector per cluster

Only one PagerDuty integration is supported per cluster. The connector must be named pagerduty. Any PagerdutyConnector applied under a different name is ignored.

EU region

EU-region PagerDuty customers must override spec.apiURL to https://api.eu.pagerduty.com. The US default https://api.pagerduty.com does not serve EU accounts.

Step 4. Annotate access requests with PagerDuty services​

The plugin uses access-request annotations to decide which PagerDuty service to file an incident against and which services to check for on-call membership. Add the annotations to the AlmaForge roles that should drive PagerDuty behavior, under spec.allow.request.annotations.

pagerduty-role-annotations.yaml:

examples/integrations/pagerduty-role-annotations.yaml
# A role that drives the PagerDuty integration. The connector holds the
# credentials. The role decides who gets paged, so different roles can
# page different teams. A role without these annotations opens no
# incident and logs nothing.
apiVersion: almaforge.com/v1
kind: Role
metadata:
  name: prod-oncall-access
spec:
  allow:
    request:
      # The roles a holder of this role may request. The annotations
      # below apply to requests for these roles.
      roles: ['access']
      annotations:
        # The PagerDuty service an incident is opened on. Matched
        # against the service name, not its ID, and the comparison
        # ignores case. Only the first value is used.
        pagerduty_notify_service: ['Prod Access Requests']
        # The services checked to decide whether the requester is
        # currently on call. Every value is checked, and one match is
        # enough. Read only when the connector sets
        # approveOnCall: true, and the requester is matched to a
        # PagerDuty user by email address alone.
        pagerduty_services: ['Prod On-Call']

In the Web UI the same field is on the role itself, under Access requests, Advanced, Annotations.

When a user requests access, the plugin creates an incident in the Prod Access Requests service. If the requester is on call in Prod On-Call, the plugin submits an approval review for the request.

The annotation keys are fixed. Use pagerduty_notify_service and pagerduty_services exactly as written.

Step 5. Test end-to-end​

The plugin polls the PagerdutyConnector resource once a minute and reconciles its state against what it observes. It starts on first appearance, restarts on resource-version change, and stops on delete. Apply, edit, and delete are all picked up within about 60 seconds with no daemon restart.

Tail the almad logs on your auth host and look for the plugin's start lines:

Terminal
journalctl -u almad -f | grep pagerduty_service...  pagerduty_service: connector found; starting plugin  api_url=https://api.pagerduty.com  user_email=almaforge-bot@example.com...  Plugin is ready

Now create an access request tied to a role that carries the PagerDuty annotations. Confirm that an incident appears in the target service:

Terminal
alma request create --roles access --nowait --reason "pagerduty smoke test"

As a different user with permission to review this request, approve or deny it in AlmaForge. Use the request ID printed by the create command:

Terminal
alma request review <request-id> --approve --reason "smoke test ok"

The matching PagerDuty incident transitions to resolved with a note summarizing the approver and the reason.

On-call auto-approval​

The plugin can auto-approve access requests from users who are currently on call, using PagerDuty as the source of truth for the rotation. This path is off unless you ask for it. Set spec.approveOnCall: true on the connector to turn it on. The check then runs once per incoming request, immediately after the notification incident is created:

  1. The plugin reads the pagerduty_services annotation from the access request. It is a list of PagerDuty service names.
  2. It takes the requester's AlmaForge login name, which must be a valid email address. Requests from non-email usernames are logged and skipped. It looks up the matching PagerDuty user via GET /users?query=<email>. The match is case-insensitive on the email field. There is no trait lookup, no alias mapping.
  3. For each annotated service, the plugin resolves its escalation policy and calls PagerDuty's on-call endpoint to check whether the user is currently in the rotation for that policy.
  4. If at least one service matches, the plugin submits an approval review on behalf of its built-in system role. The approval reason in the AlmaForge audit log names the requesting user, the matched PagerDuty user, and the on-call services.

Each auto-approval leaves an AccessRequestReview event in the AlmaForge audit log, attributed to the built-in Pagerduty system role rather than a human reviewer. Filter on that event type when you need to reconcile which requests were unblocked by an on-call rotation versus a manual approval. The review reason field carries the matched PagerDuty user and services, so a single audit query answers "who was on call when this was approved?" without cross-referencing PagerDuty.

If any step does not match, the plugin leaves the request pending and the notification incident is still created. Most of those skips are logged at DEBUG, so they are invisible at the default log level. Raise it with ALMA_LOG_LEVEL=debug when you need to see why a request was not auto-approved. The requester can still be approved manually by another reviewer in AlmaForge.

Leave spec.approveOnCall unset and the plugin never submits a review. It still creates notification incidents, and every request is approved manually in AlmaForge.

Troubleshooting​

API key is wrong or revoked​

Unauthorized (HTTP 401) from PagerDuty. The REST API key is invalid, expired, or revoked. Check the value of spec.apiKey or the external secret source behind it. Mint a fresh key in PagerDuty and re-apply the connector.

Invalid From header​

Invalid From header from PagerDuty. The spec.userEmail is not a real PagerDuty user on this account. PagerDuty requires the From: header on mutating REST calls to name an existing user. Confirm the value in spec.userEmail matches a real user in your PagerDuty directory.

Token file cannot be read​

no such file or directory or permission denied on plugin start. The file named by a ${file:PATH} reference in apiKey does not exist or is not readable by root. A file over the 64 KiB cap is reported as exceeds 65536 bytes. Check file ownership and permissions on the auth host.

On-call auto-approval does not fire​

Create the request, confirm the notification incident is created, then read the auth journal for pagerduty_service. Only two of the causes below are logged at WARN. The rest are logged at DEBUG, so start by setting ALMA_LOG_LEVEL=debug on the auth host and reproducing.

Logged at WARN, visible at the default level:

  • The requester's AlmaForge login name is not a valid email address.
  • None of the services named in the pagerduty_services annotation resolve to a real PagerDuty service.

Logged at DEBUG, silent unless you raise the level:

  • No PagerDuty user has that email address. The plugin matches by email exactly, case-insensitive.
  • The requester is not on call in any of the services listed in the pagerduty_services annotation. The plugin honors the configured escalation policy, not a "soft" on-call list.
  • The annotation is missing from the role or request.

Silence at the default level therefore does not mean the check never ran. Confirm spec.approveOnCall is set to true, since it is off unless you set it.

Plugin starts and immediately exits​

The most common causes are an unreachable api.pagerduty.com (for example, an egress problem from the auth host), a wrong spec.apiURL override, or a missing EU region override. Tail the auth journal for the pagerduty_service component to see the underlying error. The plugin retries on the next poll tick (about 60 seconds), so a transient network problem self-recovers.

Operations​

Lifecycle​

The PagerDuty plugin lives inside the almad binary and is enabled by default on every process that runs the Auth Service. To turn it on or off for a specific host, pass --pagerduty-service or --no-pagerduty-service on the almad start command line, or set ALMA_PAGERDUTY_SERVICE to true or false. Both forms work under systemd drop-ins, container environments, and Kubernetes pod specs.

The connector is reloaded every 60 seconds. Applying it starts the plugin. Editing it triggers a restart on the next reload. Deleting it stops the plugin. None of these transitions require a almad restart.

High availability​

The plugin runs on every auth process, including in multi-node clusters. Each access request produces exactly one PagerDuty incident regardless of how many auth processes are running, and resolution transitions work correctly even if the auth host that created the original incident has since restarted.

Egress requirements​

Auth hosts require outbound TCP/443 to api.pagerduty.com (or api.eu.pagerduty.com for EU customers). No inbound connectivity is required, and the integration does not open any listening sockets.

Sidecar deployment​

The plugin can also run as a dedicated sidecar process: a standalone almad instance that joins the existing cluster solely to bridge PagerDuty. This topology is appropriate when the auth nodes themselves cannot reach PagerDuty and a separate host is used as the egress point.

On the auth cluster, mint a join token that grants the Pagerduty system role:

Terminal
alma create token --services=pagerduty --expires=1h

On the sidecar host, start almad with the token and the PagerDuty role enabled. No configuration file is required:

Terminal
export ALMA_CLIENT_SECRET=/path/to/client-secretalmad start \    --auth-server=auth.example.com:443 \    --client-id=<client-id> \    --pagerduty-service

The same parameters can be supplied as environment variables for container or systemd deployments:

Terminal
export ALMA_AUTH_SERVER=auth.example.com:443export ALMA_CLIENT_ID=<client-id>export ALMA_CLIENT_SECRET=/path/to/client-secretexport ALMA_PAGERDUTY_SERVICE=truealmad start

The sidecar joins the cluster as system:pagerduty, receives an identity certificate from the auth service, polls the PagerdutyConnector/pagerduty resource over the standard auth gRPC channel, and runs the plugin locally.

Secret management​

For production, keep the REST API key outside the manifest. spec.apiKey can load it from a file or environment variable. See Resource value expansion for the exact syntax and field support. Rotate the key by updating the underlying secret and re-applying the connector. No almad restart is required.

Identity and permissions​

The PagerDuty integration runs under a built-in system role called Pagerduty. almad grants it the minimum set of permissions it needs and nothing more: list and read on ClusterName, list and readwithsecrets on PagerdutyConnector so it can load its API key, read on CertificateAuthority, and list, read and update on AccessRequest and AccessPluginData. It also carries a review condition covering every role, which is what lets the on-call auto-approval path submit a review at all. No hand-authored RBAC resources are required, and the role cannot be used to sign in as a human.

Every PagerDuty API call the plugin makes carries spec.userEmail in the From: header. In the PagerDuty audit log every incident, annotation, and on-call query appears to come from that user. Pick a name that makes the source obvious, for example almaforge-bot@example.com.

Monitoring​

The auth service logs plugin lifecycle events (start, stop, restart on connector change) and PagerDuty REST API errors under the pagerduty_service component. Tail the journal with:

Terminal
journalctl -u almad -f | grep pagerduty_service

A successful start prints connector found; starting plugin followed by Plugin is ready. Connector changes log connector changed; restarting plugin. PagerDuty API errors are logged with the HTTP status code and the description PagerDuty returned. A successful auto-approval logs Successfully submitted a request approval at INFO. Skipped requests are logged at DEBUG apart from an invalid requester email and an unresolvable service name, which are WARN.

Every AlmaForge access request and review already produces a structured audit event in the auth service journal. The plugin lifecycle lines above are usually enough to confirm what the plugin did in response.

Removal​

Delete the connector resource:

Terminal
alma delete PagerdutyConnector/pagerduty

The plugin stops on the next polling tick, within about 60 seconds, with no almad restart. Existing PagerDuty incidents are left in place. The integration never deletes incidents on its own.

To revoke the REST API key entirely, delete it in the PagerDuty account settings.

References​