Observability for Enterprise SSO

Ory Polis exports metrics, traces and logs through OpenTelemetry. Configure the exporters with the OpenTelemetry environment variables.

How the metrics are shaped

Every metric is reported by the polis meter, and the names below are the OpenTelemetry names; a Prometheus backend turns the dots into underscores and appends a suffix, so oauth.authorize is queried as oauth_authorize_total.

Names carry no service prefix. As on the Ory Go services, every point carries app, which is polis, and version, the Polis version, and those are what tell Polis's series from another service's: select them with {app="polis"}.

Attribute names are snake_case, as on the Go services' metrics. The database pool metrics take their names from the OpenTelemetry semantic conventions, but their attributes follow the same rule.

An operation such as an authorize request records two instruments that carry the same attributes:

• a counter, <operation>, with an outcome attribute that is success or error
• a histogram, <operation>.duration, in seconds, with the same buckets as the Ory Go services, from 10 ms to 20 s

Because a request and its error share one attribute set, an error ratio can be computed per protocol or login type, and the sum over outcome is the total number of requests. An attribute that could not be determined before a request failed, such as the protocol of a request whose connection was never resolved, is reported as unknown rather than left out.

A failed single sign-on or logout operation also carries error.type, which says why it failed; the values are listed in Failure types. A success carries no error.type, so the success series are unaffected and the error series split by cause.

Single sign-on

Name	Attributes	Description
oauth.authorize, .duration	protocol, login_type, outcome, error.type	OAuth authorize requests
oauth.response, .duration	protocol, login_type, outcome, error.type	SAML and OIDC responses from the IdP handled on the OAuth flow
oauth.token, .duration	protocol, login_type, outcome, error.type	OAuth token requests
oauth.userinfo, .duration	protocol, login_type, outcome, error.type	OAuth userinfo requests
oauth.logout, .duration	protocol, logout_type, outcome, error.type	Single logout requests Polis sends to a SAML IdP
oauth.logout.response, .duration	protocol, logout_type, outcome, error.type	Single logout responses from a SAML IdP

Identity federation

Name	Attributes	Description
idfed.authorize, .duration	protocol, login_type, outcome, error.type	Identity federation authorize requests
idfed.response, .duration	protocol, login_type, outcome, error.type	IdP responses handled on the identity federation flow
idfed.logout, .duration	protocol, logout_type, outcome, error.type	Identity federation logout requests from the service provider
idfed.logout.response, .duration	protocol, logout_type, outcome, error.type	Identity federation logout callbacks from the upstream IdP

A federated request that arrives through the OAuth endpoints is counted under both oauth.* and idfed.*, with the same outcome, so the OAuth series remain the total of everything the endpoint served.

Access tokens in the userinfo request URI

oauth.userinfo.query_token counts the userinfo requests that carry the access token in the request URI rather than the Authorization header, so you can see whether anything would break before enabling OPENID_USERINFO_REJECT_QUERY_TOKEN. The series to watch is presentation_rejected="false" with outcome="success": those requests work today and are the ones that stop working when the setting is on. A token rejected on its own merits is already failing and counts under outcome="error" instead.

Name	Attributes	Description
oauth.userinfo.query_token	presentation_rejected, outcome	Userinfo requests carrying the access token in the request URI

Connection API

Name	Attributes	Description
connection.create		SSO connection create requests
connection.get		SSO connection get requests
connection.delete		SSO connection delete requests
connection.event	event	Changes to SSO connections and directories, such as sso.created or dsync.deactivated; see Connection changes
webhook.delivery, .duration	event_type, batch, code, error.type, outcome	Deliveries of the sso.* and dsync.* connection lifecycle events to the configured webhook, as the event_type="connection" series; see Observability for Directory Sync for the attributes

The create, get and delete counters count API requests, whether or not they succeed; connection.event counts the changes that were made.

HTTP requests

Each request that matches a Polis route is also recorded under the names and attributes the Ory Go services (Hydra, Kratos, Keto and the rest) use for theirs, so Polis appears on the dashboards and alerts that read one request metric from every service. The two share the latency buckets above.

Name	Attributes	Description
http.requests, .duration	code, method, endpoint, app, version	Requests served, by status code, method and route; app is polis

In Prometheus these are http_requests_total and http_requests_duration_seconds, the same as the Go services'. endpoint is the route template rather than the request path, with a path parameter written {param}: a request to /api/setup/abc123/sso-connection is recorded under /api/setup/{param}/sso-connection. method is lower case, and one of the methods the Go services name; anything else is other. A request that matched no route, a static asset or an unmatched 404, is not recorded.

Database connection pool

Reported for PostgreSQL-compatible databases, following the OpenTelemetry semantic conventions for database clients. The values are read from the pool each time metrics are collected.

Name	Attributes	Description
db.client.connection.count	db.client.connection.pool.name, db.client.connection.state	Connections in the pool, by state
db.client.connection.max	db.client.connection.pool.name	Maximum number of connections the pool may open
db.client.connection.pending_requests	db.client.connection.pool.name	Requests waiting for a connection

Attribute values

Attribute	Values
app	polis, on every metric
version	The Polis version, on every metric
outcome	success, error
protocol	saml, oidc, saml-federation, oidc-federation, unknown
login_type	sp-initiated, idp-initiated, unknown
logout_type	sp-initiated: Polis accepts single logout from the service provider only
error.type	See Failure types
presentation_rejected	true when a userinfo request was refused before the token was read, false otherwise
db.client.connection.pool.name	The application_name of the database connection, jackson unless set in the connection URL
db.client.connection.state	idle for a connection open but not in use, used for one serving a query
event	On connection.event: sso.created, sso.activated, sso.deactivated, sso.deleted, the same four for dsync, and sso.updated and dsync.updated, which have no webhook

Failure types

The error.type of a failed single sign-on or logout operation is one of a fixed set of values. The OAuth error codes keep the meaning RFC 6749 and RFC 6750 give them; the others say where in Polis the request failed.

Value	The request failed because
invalid_request	it was missing a required parameter or carried one Polis does not accept, such as no state or no redirect URL
invalid_client	the client, or the identity federation app, is unknown, or its credentials are wrong
invalid_grant	the authorization code is unknown, expired or already used, its PKCE verifier does not match, or the redirect URI differs from the authorize request's
invalid_token	the access token presented to userinfo is unknown or expired
unsupported_grant_type	the token request asked for a grant other than authorization_code
unsupported_response_type	the authorize request asked for a response type other than code
connection_not_found	no SSO connection matches the tenant, product, client ID or IdP
connection_inactive	the SSO connection is deactivated
redirect_not_allowed	the redirect URL, ACS URL or target_link_uri is not registered on the connection or app
session_not_found	the response or callback does not belong to a request Polis started, or its session has expired or been used
idp_initiated_disabled	an IdP-initiated login reached a connection that does not allow it
idp_error	the IdP answered with an error, such as an OIDC error parameter or a logout status other than success
saml_invalid	the SAML document is malformed, such as one that declares a DTD or names no issuer
saml_validation	the SAML response failed validation: its signature, audience, validity period, request ID or replay check
saml_request	Polis could not build the SAML request to the IdP
saml_response	Polis could not build the SAML response to the service provider, on identity federation
oidc_request	Polis could not build the OIDC request to the IdP, usually because the IdP's discovery document could not be fetched
oidc_token_exchange	exchanging the OIDC authorization code with the IdP, or reading the ID token it returned, failed
misconfigured	the deployment or the connection is incomplete, such as missing JWT signing keys or an IdP without single logout
_OTHER	none of the above describes it

Traces

When a traces endpoint is configured, each request to Polis opens a request span, and the sampler decides whether its trace is recorded and exported. The request span is opened by Next.js and named after the route, GET /api/oauth/authorize say, and continues a trace that arrives in a traceparent header, so a login can be followed from the application that started it. The same span is where the HTTP request metrics are measured, so they are recorded whether the trace is sampled or not. Under the request span:

• Every operation in the tables above is a span with the same name as its metric, oauth.authorize say, carrying the same attributes as the metric apart from app and version; the span's resource names the service in service.name and carries service.version, so a slow or failing series can be followed to the traces behind it. A failure sets the span status to error with the reason as its message, and when Polis has the error, records it with its stack as an exception event. An operation that is also recorded under a second metric, such as a response on the identity federation flow, names it in polis.operation.also. Single sign-on and logout spans also carry the login and logout attributes, which have too many values to be metric attributes, apart from user.id and user.email, which are on log records only.
• The calls Polis makes to identity providers (metadata fetches, OIDC token and userinfo exchanges) and to directory sync webhooks are client spans with the OpenTelemetry HTTP attributes, and carry a traceparent header on to the other side.
• Database queries are client spans with the OpenTelemetry database attributes, for PostgreSQL, MySQL and MariaDB, SQL Server, MongoDB, DynamoDB and Redis.
• A change to an SSO connection or a directory is a span named after its webhook event under connection, connection.sso.created say, and a run of the directory sync webhook batch job is a dsync.event_batch span. They have no metric of their own: they hold the records and webhook deliveries of that change or run together, in one trace, even where no request span is around them, as for the batch job, which runs on a timer.

Before a span is exported, Polis strips the parts of URLs that can carry secrets: setup tokens, authorization codes, state values, SAML messages, access tokens and SCIM filters. The request span carries no http.target; http.route and http.method identify the request. On the spans of calls Polis makes, url.full and http.url keep the scheme and host, and the path, any credentials and every query value are replaced with REDACTED, so a webhook delivery is exported as https://hooks.example.com/REDACTED?token=REDACTED. url.path and url.query are redacted the same way.

Spans use the polis tracer and are reported by the Ory Polis service, as the other Ory services report theirs (Ory Hydra, Ory Kratos); OTEL_SERVICE_NAME changes it.

Sampling

Sampling follows OTEL_TRACES_SAMPLER; with the default every request that starts at Polis is traced, so set a ratio on a busy deployment. With a ratio sampler only that share of traces is recorded and exported, while the HTTP request metrics still count the requests whose traces were dropped.

Logs

info

Logs are an Enterprise feature: Polis exports them only with a valid license key. Metrics and traces need none.

warning

Log records carry payloads, personal data included, unless redaction is turned on. Set SSO_TELEMETRY_REDACT and DSYNC_TELEMETRY_REDACT to true to leave them out. Unset or false, SAML responses, IdP profiles, OIDC ID tokens, SCIM request bodies and webhook event payloads reach your log backend. The Helm chart sets both to true by default; a deployment that sets its own environment must set them itself.

Every operation in the tables above, and every directory sync operation, emits one log record when it ends, through the polis logger:

• The event name is the name of the operation's metric with a polis. prefix, polis.oauth.response say.
• The severity is INFO for a success and ERROR for a failure. A failure's body is the reason it failed, and when Polis has the error, the record carries it as the OpenTelemetry exception.type, exception.message and exception.stacktrace attributes.
• The attributes are the span's: the metric attributes, outcome and error.type included, and for single sign-on and logout the login and logout attributes. The user attributes, user.id and user.email, are on the log record only. Single sign-on records also carry the payloads of the requests and responses.
• The record carries the trace ID and span ID of its operation span, so a record leads to the trace behind it.

Log records are not sampled. Where OTEL_TRACES_SAMPLER keeps a share of the traces, every login still has its log records, and those whose trace was kept lead to it. When only logs are exported, Polis still gives every record a trace ID and span ID, continuing a trace that arrives in a traceparent header, so the records of one request share a trace ID and can be matched to the traces of the application that called Polis.

Login and logout attributes

Attribute	On	Description
polis.login.id	The operations of a login	One ID, a random UUID, for every request of one login; see Following a login
polis.logout.id	The operations of a logout	One ID, a random UUID, for the request and the response of one logout
polis.tenant, polis.product	Where known	The tenant and product of the connection or app
polis.connection.client_id	Where known	The client ID of the SSO connection the login or logout went through
polis.idfed.app.id	Identity federation	The ID of the identity federation app
polis.idp.provider	Where known	The provider name of the IdP, as configured on the connection
polis.idp.issuer	SAML responses	The issuer of the SAML response or logout response, without any credentials a URL carries
polis.redirect.origin	Where known	The origin of the URL the login or logout returns to: scheme, host and port, never the path or query
user.id	From the IdP response on	The ID of the user the IdP vouched for; on a SAML logout, the NameID being logged out
user.email	From the IdP response on	The email address of the user the IdP vouched for
polis.sso_trace.id	Failures that save an SSO trace	The ID the SSO trace of the failure is saved under: the ID the Admin Portal shows, and the one the end user sees when the error shows one

user.id and user.email are logged whenever logs are exported; SSO_TELEMETRY_REDACT does not remove them, because they are what a login is looked up by. They are never put on spans: find a login's log record by the user, and follow its trace ID to the spans. An attribute that is not known when the operation ends, such as the user of a request that failed before the IdP answered, is left out.

A failure is recorded before its SSO trace is saved, so a slow or unavailable trace store never holds up the failure's metrics, span and log record. An error that shows the end user the trace ID waits up to two seconds for the save; token, userinfo and identity federation errors, which show no trace ID, do not wait for it. A trace that could not be saved is missing from the Admin Portal even though its polis.sso_trace.id was logged.

Payloads

Single sign-on log records carry what the request or response held, so a login can be read without reproducing it:

Attribute	On	Description
polis.saml.request	polis.oauth.authorize, polis.idfed.authorize, polis.oauth.logout, polis.idfed.logout	The SAML request: the AuthnRequest or LogoutRequest Polis sends to the IdP, or on identity federation, the one the service provider sent
polis.saml.response	polis.oauth.response, polis.oauth.logout.response	The SAML response or logout response the IdP sent, as XML
polis.idp.profile	polis.oauth.response	The profile Polis extracted from the IdP response, as JSON
polis.oidc.id_token	polis.oauth.response on OIDC	The ID token the OIDC IdP issued

They are logged for successes and failures alike, on log records only, never on spans or metrics. Each value is cut at 64 KiB, and a cut value ends with a note of how many characters were dropped.

warning

Unless SSO_TELEMETRY_REDACT is true, the payloads are exported with the logs: SAML responses, IdP profiles and OIDC ID tokens, which hold personal data, reach your log backend. The Helm chart sets it to true by default (monitoring.sso.redact.enabled); a deployment that sets its own environment leaves it unset, and so exports them. Set it to true to leave every payload out.

SSO_TELEMETRY_REDACT covers the log records only; SSO_TRACES_REDACT does the same for the SSO traces in the Admin Portal. The access and refresh tokens the OIDC IdP issues are never logged, not even in the IdP profile of an identity federation app that includes them in its assertion: they are credentials that may still be valid when the log is read.

Following a login

A login is several requests to Polis, and so several traces: the authorize request, the IdP response, the token exchange and the userinfo request. All of their records carry the same polis.login.id. To read a login:

1. Find a record of it, by user.email, user.id, polis.tenant or polis.connection.client_id, or by polis.sso_trace.id when the end user reports the ID shown in the error.
2. Search for its polis.login.id to get every step of the login, in order, with the outcome and error.type of each.
3. Follow a record's trace ID to the trace, when it was sampled, for the calls Polis made to the IdP and the database.

An IdP-initiated login gets its ID when the IdP response arrives, and an authorize request that redeems the code of an IdP-initiated login continues that login's ID.

Limitations

• A login that passes through the IdP selection page, or through the app selection of an IdP-initiated login, starts again with a new polis.login.id once the user chooses.
• A logout is not linked to the login it ends: nothing of a login is kept that a later logout could be tied to. Match the two by user.id and polis.connection.client_id, and by time.
• Single logout is initiated by the service provider only; Polis does not accept a logout the IdP starts.
• A SAML response rejected because it declares a DTD is not logged as a payload.

Connection changes

Each change to an SSO connection or a directory is a log record, whether or not a webhook is configured:

Record	When
polis.connection.sso.created, polis.connection.dsync.created	The connection or directory was created
polis.connection.sso.updated, polis.connection.dsync.updated	Its configuration changed: its name, redirect URLs, IdP metadata or webhook, say
polis.connection.sso.deactivated, polis.connection.sso.activated, and the same for polis.connection.dsync.*	It was deactivated or activated again
polis.connection.sso.deleted, polis.connection.dsync.deleted	It was deleted; deleting by tenant and product records each connection it deleted

An update that changes nothing, such as the same metadata uploaded again, is not recorded, nor is a change of activation alone, which has a record of its own, nor the rotation of a Google directory's tokens. The records name what changed, never a secret: no client secret, SCIM token, webhook URL or webhook secret.

Attribute	On	Description
polis.tenant, polis.product	Every change	The tenant and product of the connection or directory
polis.connection.client_id	SSO connections	The connection's client ID
polis.connection.name	SSO connections	The connection's name
polis.connection.active	SSO connections	Whether the connection is active after the change; an inactive one refuses logins
protocol	SSO connections	saml or oidc
polis.idp.provider	SSO connections	The provider name of the IdP, as on login records
polis.idp.issuer	SSO connections	The IdP the connection trusts: the SAML entity ID, or the OIDC issuer, from the connection's metadata or else its discovery URL, without any credentials a URL carries
polis.dsync.directory.*	Directories	The directory's ID, name, type and whether it is active, as on directory sync records
polis.dsync.webhook.configured, polis.dsync.webhook.log.enabled	Directories	Whether the directory has a webhook, with its URL and secret set, and whether it keeps a log of its webhook deliveries
polis.connection.changed	Updates	The names of the fields the update changed, sorted, such as ["description","name","redirectUrl"]; never their values

When WEBHOOK_URL is set, each change but an update is also sent there, as a webhook.delivery span and polis.webhook.delivery log record that names the event, the endpoint's server.address and server.port, never its full URL, and the connection or directory by tenant, product and client ID or directory ID. A delivery that fails is an ERROR record; it does not fail the request that made the change, which is saved by then. The transport retries a failed delivery three times, three seconds apart, within the same delivery, and it is not sent again after that. See Delivery.

Finding a connection

The records tell what happened to each connection, and so what exists:

• To list the connections of a tenant, take the latest record of each polis.connection.client_id with that polis.tenant; a polis.connection.sso.deleted one means it is gone. The same by polis.dsync.directory.id lists directories.
• To search them, filter those records by polis.connection.name, polis.idp.provider, polis.idp.issuer, protocol or polis.connection.active.
• To see what changed and when, read the records of one connection in order, with polis.connection.changed on each update. A change's record and its webhook delivery share a trace ID, that of the API request that made the change.

The records hold what happened since logs were exported: a connection that has not changed since then has none. The Admin Portal and the API list what exists now. Records do not say who made a change, which the API does not know.

Example queries

Error ratio of authorize requests per protocol, over five minutes:

sum by (protocol) (rate(oauth_authorize_total{app="polis", outcome="error"}[5m]))
  / sum by (protocol) (rate(oauth_authorize_total{app="polis"}[5m]))

Failed IdP responses by cause, over five minutes:

sum by (protocol, error_type) (rate(oauth_response_total{outcome="error"}[5m]))

95th percentile latency of the IdP response handling:

histogram_quantile(0.95, sum by (le, protocol) (rate(oauth_response_duration_seconds_bucket[5m])))

Connections in use against the pool limit:

sum(db_client_connection_count{db_client_connection_state="used"}) / sum(db_client_connection_max)

Directory sync metrics are described in Observability for Directory Sync.

Lifecycle webhooks that failed, by status, over an hour:

sum by (code) (increase(webhook_delivery_total{event_type="connection", outcome="error"}[1h]))

Upgrading from Polis 26.4 and earlier

Polis 26.4 and earlier named every metric with a jackson. prefix, counted errors on separate .error counters and reported the database pool on gauges of its own. Each of those metrics has a successor, listed here. Every metric now also carries app, which is polis, and version, so a query that matched on the prefix can match on app="polis" instead.

Before	In PromQL before	Now	In PromQL now
jackson.oauth.authorize	jackson_oauth_authorize_total	oauth.authorize, summed over outcome	sum(oauth_authorize_total)
jackson.oauth.authorize.error	jackson_oauth_authorize_error_total	oauth.authorize with outcome="error"	oauth_authorize_total{outcome="error"}
jackson.oauth.response	jackson_oauth_response_total	oauth.response, summed over outcome	sum(oauth_response_total)
jackson.oauth.response.error	jackson_oauth_response_error_total	oauth.response with outcome="error"	oauth_response_total{outcome="error"}
jackson.oauth.token	jackson_oauth_token_total	oauth.token, summed over outcome	sum(oauth_token_total)
jackson.oauth.token.error	jackson_oauth_token_error_total	oauth.token with outcome="error"	oauth_token_total{outcome="error"}
jackson.oauth.userinfo	jackson_oauth_userinfo_total	oauth.userinfo, summed over outcome	sum(oauth_userinfo_total)
jackson.oauth.userinfo.error	jackson_oauth_userinfo_error_total	oauth.userinfo with outcome="error"	oauth_userinfo_total{outcome="error"}
jackson.idfed.authorize	jackson_idfed_authorize_total	idfed.authorize, summed over outcome	sum(idfed_authorize_total)
jackson.idfed.authorize.error	jackson_idfed_authorize_error_total	idfed.authorize with outcome="error"	idfed_authorize_total{outcome="error"}
jackson.idfed.response	jackson_idfed_response_total	idfed.response, summed over outcome	sum(idfed_response_total)
jackson.idfed.response.error	jackson_idfed_response_error_total	idfed.response with outcome="error"	idfed_response_total{outcome="error"}
jackson.idfed.logout	jackson_idfed_logout_total	idfed.logout, summed over outcome	sum(idfed_logout_total)
jackson.idfed.logout.error	jackson_idfed_logout_error_total	idfed.logout with outcome="error"	idfed_logout_total{outcome="error"}
jackson.connection.create	jackson_connection_create_total	connection.create	connection_create_total
jackson.connection.get	jackson_connection_get_total	connection.get	connection_get_total
jackson.connection.delete	jackson_connection_delete_total	connection.delete	connection_delete_total
jackson.db.connections.max	jackson_db_connections_max	db.client.connection.max	db_client_connection_max
jackson.db.connections.total	jackson_db_connections_total	db.client.connection.count, summed over db.client.connection.state	sum(db_client_connection_count)
jackson.db.connections.idle	jackson_db_connections_idle	db.client.connection.count with db.client.connection.state="idle"	db_client_connection_count{db_client_connection_state="idle"}
jackson.db.connections.waiting	jackson_db_connections_waiting	db.client.connection.pending_requests	db_client_connection_pending_requests

The attributes carried over keep their names and values: protocol, login_type and logout_type. The pool gauges' db_name is now db.client.connection.pool.name. The directory sync metrics are listed in Observability for Directory Sync.

Two things change beyond the names:

• The request counters now carry outcome, so the sum over outcome is what the request counter counted before, and its outcome="error" series is what the .error counter counted. A query that read a request counter without grouping by outcome still gets the total.
• A failed federated request was counted on either the OAuth or the identity federation .error counter, depending on the flow. It is now an outcome="error" series under both oauth.* and idfed.*, as the request itself already was.