Observability for Directory Sync

Ory Polis exports metrics, traces and logs through OpenTelemetry. Configure the exporters with the OpenTelemetry environment variables. The shape of the metrics, and how to query them, is described in Observability for Enterprise SSO, the traces in Traces and the logs in Logs: each operation under SCIM requests and webhooks is also a span with the same attributes, and a log record when it ends, and a webhook delivery is a client span that carries the trace on to the webhook endpoint. The Directory API counters are metrics only.

SCIM requests and webhooks

Name	Attributes	Description
dsync.request, .duration	resource, method, code, outcome, error.type	SCIM requests. outcome is error when the response status is 400 or above.
webhook.delivery, .duration	event_type, batch, code, error.type, outcome	Webhook deliveries. Directory sync's user and group events are the event_type="directory-sync" series; the connection lifecycle events share the metric under event_type="connection".
dsync.webhook.event	event, outcome, error.type	Each directory sync event a delivery carried, so a batch counts every event in it; a retried event is counted at every attempt
dsync.webhook.dropped	event, reason	Queued events deleted unsent, because their directory was deleted, is gone, inactive or has no webhook
dsync.sync, .duration	outcome, error.type	Syncs of one directory from its provider, such as Google Workspace, on each sync run

Directory API

Name	Attributes	Description
dsync.connection.create		Directory sync connection create requests
dsync.connection.get		Directory sync connection get requests
dsync.connection.delete		Directory sync connection delete requests
dsync.event.dead_lettered		Events moved to the dead-letter store: rejected with 400 or 422, or out of delivery retries
dsync.event_batch.failed		Connections whose every queued event failed to deliver, counted once per connection per batch run
dsync.event	event	Changes to users, groups and group membership, whether or not a webhook is configured

Attribute values

Attribute	Values
resource	users, groups, discovery (ServiceProviderConfig, Schemas and ResourceTypes), unknown
method	The HTTP method of the SCIM request, in lower case, such as post, and other for a method outside the standard ones, as on the Ory Go services' request metrics
code	The HTTP status returned to the SCIM client, or by the webhook endpoint, named as on the Ory Go services' request metrics; unknown for a webhook delivery that received no response
error.type	On dsync.request, the RFC 7644 scimType of the error response, such as uniqueness, invalidSyntax or invalidValue, else its status code, such as 401 or 404, and _OTHER when Polis failed before answering. On dsync.sync, the status of the provider's response, such as 401 for a revoked grant, else the transport error, or scim_request when the sync ran but some of the SCIM requests it made failed. On webhook.delivery and dsync.webhook.event, the status code of a response outside 2xx, such as 503, else the transport error of a delivery that received no response, such as ECONNREFUSED or ETIMEDOUT, else _OTHER
batch	true when one delivery carried several events, which happens when webhook batching is configured
event_type	directory-sync for user and group events, connection for the sso.* and dsync.* connection lifecycle events
event	user.created, user.updated, user.deleted, group.created, group.updated, group.deleted, group.user_added, group.user_removed: the event type the webhook receives
reason	directory_deleted: the directory was deleted while they waited; directory_not_found: the directory no longer exists when the batch job runs; directory_inactive: it was deactivated; webhook_not_configured: its webhook URL or secret was removed

Logs

Every operation above emits a log record when it ends, as described in Logs. Logs are an Enterprise feature, exported only with a valid license key.

warning

Unless DSYNC_TELEMETRY_REDACT is true, the log records carry the SCIM request bodies and webhook event payloads, with the users' and groups' attributes; see Payloads.

Directory sync adds:

• The record of every SCIM request and every provider sync carries the directory it was for: polis.tenant, polis.product, polis.dsync.directory.id, polis.dsync.directory.name, polis.dsync.directory.type and polis.dsync.directory.active. A request to a deactivated directory is refused with a 403, so it fails with error.type 403 and polis.dsync.directory.active false.
• Every change directory sync makes is a record of its own: see Changes.
• Every change to a directory itself, such as created, updated or deactivated, is a record too: see Connection changes, and Finding a connection to list and search directories from them.
• Every webhook delivery says what it delivered, and each event it carried, or that the queue dropped, is a record of its own: see Webhook deliveries.
• A provider sync whose SCIM requests failed carries polis.dsync.sync.failed_requests, how many did, and says how many of how many in its body; each failed request is a polis.dsync.request record of its own, under the sync's trace.

Changes

Each change to a user, a group or a group's members is a log record named after the event the webhook receives: polis.dsync.user.created, polis.dsync.user.updated, polis.dsync.user.deleted, polis.dsync.group.created, polis.dsync.group.updated, polis.dsync.group.deleted, polis.dsync.group.user_added and polis.dsync.group.user_removed.

A change is recorded where it is made, so it is logged whether or not the directory has a webhook, and inside the SCIM request that made it: the record carries that request's trace ID and span ID. A Google Workspace sync makes its changes through the same SCIM requests, under its dsync.sync span, so a sync run leads to every change it made. A request that changes nothing, such as a create refused because the user exists, records no change.

Attribute	On	Description
polis.dsync.event.id	Every change	A random ID for the change, which every record of its webhook delivery carries too; it is never sent to the webhook
polis.tenant, polis.product, polis.dsync.directory.*	Every change	The directory, as on the request
user.id	User and membership changes	The ID Polis gave the user, the id the webhook receives
user.email	User and membership changes	The user's email address
user.name	User and membership changes	The user name the IdP knows the user by, the SCIM userName
user.full_name	User and membership changes	The user's first and last name
polis.dsync.user.active	User and membership changes	Whether the user is active in the directory; a deactivation is a user.updated with false
polis.dsync.group.id, polis.dsync.group.name	Group and membership changes	The group

Webhook deliveries

A directory with a webhook has each change sent to it: as the change is made, inside the SCIM request, or, when webhook batching is configured, by a background job that sends the queued events of each directory in one request and retries those that fail on its next run.

Each delivery is a webhook.delivery span and a polis.webhook.delivery log record, as above, with:

Attribute	Description
polis.tenant, polis.product, polis.dsync.directory.*	The directory whose events were delivered
server.address, server.port	The host and port of the webhook endpoint; never the full URL, whose query can hold a secret
polis.dsync.webhook.event_count	How many events the delivery carried
polis.dsync.event.id	The event ID of the change, when the delivery carried one event, as every delivery without batching does
polis.dsync.webhook.log.id	The ID of the webhook event log entry the delivery wrote, the event ID the Admin Portal shows; only when the directory logs its webhook events

Each event a delivery carried is also a polis.dsync.webhook.event record, under the delivery's span, so that one event can be found in a batch. It carries the event type, outcome, code and, on a failure, error.type of the delivery, the directory and the log entry, and:

Attribute	Description
polis.dsync.event.id	The event ID of the change, absent for an event queued before Polis recorded event IDs
polis.dsync.webhook.attempt	Which attempt at delivering the event this was, from 1, counting every delivery: a retry after a failure, and a repeat of a delivery whose queue entry would not delete. Only a batched event gets a second attempt; without batching an event is sent once, and the retries the transport makes within that send are part of the same attempt

A failed event is an ERROR record. The outcome is that of the HTTP response: any 2xx is a success, and a delivered event, as the batch job and the webhook event log count it too.

Deleting a directory deletes its queued events with it, and when the batch job finds that an event's directory no longer exists, was deactivated or no longer has a webhook, it deletes the event unsent. Each such event is a WARN record named polis.dsync.webhook.dropped, with its event type, the reason, its polis.dsync.event.id and what is known of its directory: only its ID when it no longer exists. Each run of the batch job is a span, dsync.event_batch, so the deliveries of one run, their events and the events it drops share the run's trace ID.

An application that embeds Polis with its own event callback, the dsync.callback option of the npm library, sends its own webhooks: Polis records no delivery for it, and passes the callback the event ID as context.eventId, its second argument.

Payloads

Attribute	On	Description
polis.scim.request	polis.dsync.request	The body of the SCIM request the IdP sent, as JSON, without its password attributes or the PATCH operations that set a password, once the request has authenticated
polis.dsync.event.payload	Every change	The event exactly as the webhook receives it, as JSON, with the IdP's raw attributes

They are on log records only, never on spans or metrics, and each is cut at 64 KiB.

warning

Unless DSYNC_TELEMETRY_REDACT is true, the payloads are exported with the logs: SCIM request bodies and webhook event payloads, which hold personal data, reach your log backend. The Helm chart sets it to true by default (monitoring.dsync.redact.enabled); a deployment that sets its own environment leaves it unset, and so exports them. Set it to true to leave every payload out; the IDs, names and email addresses of the users and groups are still logged.

Following a user

To find out whether a user is being synced, and what happened to them:

1. Search the change records by user.email, or by user.id when the webhook payload names it.
2. Read the changes in order: when the user was created, deactivated, added to or removed from each group.
3. Follow a record's trace ID to the SCIM request that made the change and its outcome, or to the sync run that made it. The request's polis.scim.request body is there only when DSYNC_TELEMETRY_REDACT is not true; the Helm chart sets it to true, so the body is left out by default there.

The records answer what happened, not what is true now: which groups a user is in, or how many users a group has, comes from the directory itself, and the log holds only what happened since logs were exported.

Following a webhook event

To find out whether a change reached the webhook:

1. Find the change, as in Following a user, and take its polis.dsync.event.id.
2. Search the polis.dsync.webhook.event records by that ID: one per attempt, in order, each with its outcome and status code. No record yet means the event is still waiting in the batch queue, or that its directory has no webhook; a polis.dsync.webhook.dropped record means it will not be sent.
3. Follow a record's polis.dsync.webhook.log.id to the entry in the Admin Portal, or its trace ID to the delivery, and without batching to the SCIM request that made the change.

To go the other way, from an event in the Admin Portal, search polis.dsync.webhook.log.id for its event ID.

Example queries

Share of webhook deliveries that failed, over five minutes:

sum(rate(webhook_delivery_total{event_type="directory-sync", outcome="error"}[5m]))
  / sum(rate(webhook_delivery_total{event_type="directory-sync"}[5m]))

Changes by event type, over an hour:

sum by (event) (increase(dsync_event_total[1h]))

Events that failed to deliver, by event type and cause, over an hour:

sum by (event, error_type) (increase(dsync_webhook_event_total{outcome="error"}[1h]))

Queued events dropped unsent, by reason:

sum by (reason) (increase(dsync_webhook_dropped_total[1h]))

SCIM error responses by resource and status:

sum by (resource, code) (rate(dsync_request_total{outcome="error"}[5m]))

Upgrading from Polis 26.4 and earlier

Polis 26.4 and earlier named every metric with a jackson. prefix. The directory sync metrics it reported have these successors, each of which now also carries app, which is polis, and version. The other metrics are listed in Observability for Enterprise SSO.

Before	In PromQL before	Now	In PromQL now
jackson.dsync.connection.create	jackson_dsync_connection_create_total	dsync.connection.create	dsync_connection_create_total
jackson.dsync.connection.get	jackson_dsync_connection_get_total	dsync.connection.get	dsync_connection_get_total
jackson.dsync.connection.delete	jackson_dsync_connection_delete_total	dsync.connection.delete	dsync_connection_delete_total
jackson.dsync.events_batch.failed	jackson_dsync_events_batch_failed_total	dsync.event_batch.failed	dsync_event_batch_failed_total

The other directory sync metrics, from dsync.request to dsync.webhook.dropped, are new and have no predecessor.