Running multiple instances

Ory Polis keeps its state in the database, so you can run as many instances as you need behind a load balancer and scale them up and down freely. There is one thing worth getting right the first time you point instances at a new, empty database: the SAML signing certificate.

Why the first start is different

Polis creates a certificate only when it has not been given one. If no instance is configured with a certificate, the first one to start against an empty database generates a self-signed SAML signing certificate, stores it, and signs SAML requests and responses with it. Every instance that starts afterwards reads that same certificate back out of the database, so they all sign with the same key, and the public key you publish in your SP metadata matches all of them.

An instance that is configured with a certificate uses it and never reads or writes the stored one. That makes the two sections below alternatives rather than a mixture: pick one and apply it to every instance.

If several instances start at the same moment against a database that has no certificate yet, each can find nothing stored, generate its own, and write it. Only one of those writes survives. An instance left holding one of the others carries on signing with a private key the database no longer holds, so its signatures cannot be verified against the metadata your identity providers were configured with, and SSO fails for whichever requests happen to reach that instance until it is restarted.

Instances in one process coordinate with each other, so this only applies to instances starting as separate processes, containers or pods.

There are two ways to avoid it. Supplying your own certificate is the better one for automated deployments.

Supply your own certificate

Set PUBLIC_KEY and PRIVATE_KEY (both base64 encoded) and Polis uses that certificate instead of generating one. No instance writes a certificate, start order stops mattering entirely, and you end up with a signing key you control, can back up, and can rotate deliberately rather than one that appeared on a pod you no longer have.

Generate a pair and encode it:

openssl req -x509 -newkey rsa:2048 -keyout key.pem -out public.crt -sha256 -days 365 -nodes

cat public.crt | base64
cat key.pem | base64

See PUBLIC_KEY and PRIVATE_KEY for the full description. If you embed Polis as a library, the same values are certs.publicKey and certs.privateKey.

Give the same pair to every instance, and do not mix configured and unconfigured ones against a single database. Because a configured instance never touches the stored certificate, an unconfigured instance beside it finds an empty database, generates a certificate of its own, and signs with a key your identity providers were never given. Two instances configured with different pairs fail the same way.

Or start one instance first

If you would rather let Polis generate the certificate, leave PUBLIC_KEY and PRIVATE_KEY unset everywhere, bring up a single instance against the new database, and wait for it to be ready before starting the others. Once the certificate is in the database, starting instances is no longer order-dependent.

You can confirm it exists by fetching the SP metadata from the running instance and checking that it carries an X509Certificate:

curl https://<your-polis-url>/.well-known/sp-metadata

After that, scale up.

Restarts and upgrades

None of this applies to a database that already holds a certificate. Rolling restarts, rolling upgrades and autoscaling against an existing database are unaffected, whatever order instances come up in.

Setting or clearing PUBLIC_KEY and PRIVATE_KEY on a running deployment is a different matter: it changes which certificate those instances sign with. Roll that change out to every instance together, and give your identity providers the new public key.