ZITADEL in production

Practices for running ZITADEL as the identity layer behind a product, and for a consultancy standing up an isolated instance per client rather than rediscovering the same failures on every engagement.

ZITADEL is excellent software with an unusually punishing first hour, and the reason is consistent: the things that break are off by default, and nothing warns you. Not the deployment settings, not the token settings, not the claims your application needs. Each one fails with an error that points somewhere else.

Everything here was verified against a live instance, not read from documentation. The one exception is references/untested.md, which is research only and says so at the top. Keep that boundary intact: when something in that file gets exercised for real, move it out.

The one rule

The external domain and the TLS mode are decided before first boot, and they are effectively permanent.

ExternalDomain is written into the database during the setup phase. From it ZITADEL derives the instance domain, the default organization's domain, and every login name (user@zitadel.<domain>). Changing it later requires rerunning setup, and existing login names keep the old domain.

--tlsMode silently overrides ZITADEL_EXTERNALSECURE, so getting it wrong produces a system whose configuration disagrees with reality in ways that surface as unrelated browser errors.

Point a real DNS record at the host, set the mode correctly, and boot once. Everything else is recoverable. This is not.

The trap that costs everyone an hour

--tlsMode on the start command sets both TLS.Enabled and ExternalSecure, and the flag wins over the environment variable:

--tlsMode	TLS.Enabled	ExternalSecure
disabled	false	false
external	false	true
enabled	true	true

Behind any TLS-terminating proxy the correct combination is:

--tlsMode external
ZITADEL_EXTERNALPORT=443
ZITADEL_EXTERNALSECURE=true

Setting ZITADEL_EXTERNALSECURE=true while leaving --tlsMode disabled does nothing at all. The symptom is the console failing with [unknown] Failed to fetch after a login that worked perfectly.

The single best diagnostic in the whole system:

curl -s https://<domain>/ui/console/assets/environment.json

Both api and issuer must be https. If api is http while issuer is https, ExternalSecure is effectively false: check the flag, not the variable. They disagree because api comes from configuration while issuer comes from the request, where the proxy's X-Forwarded-Proto wins.

Full recipe in references/deployment.md, every observed failure in references/troubleshooting.md.

Off by default, and nothing tells you

After deployment, the next hour goes here. A freshly registered application does not produce what an application needs, and each gap presents as a code bug.

Default	What it breaks	Fix
Auth Token Type = Bearer	Access token is opaque. No local JWKS verification is possible, so every request needs an introspection round trip	Set JWT
Profile claims absent from ID token	Your session's user object is empty. Looks exactly like a broken profile mapping	☑ Include user's profile info in the ID Token
Roles not asserted	The roles claim is missing entirely	☑ on the app and Assert Roles on Authentication on the project
No SMTP	Verification, password reset and invitations fail silently. Every new user is stuck	Configure SMTP
Access token lifetime = 12 hours	A revoked user keeps working for half a day	5 to 15 minutes, plus refresh tokens

Two console behaviours make this worse. The application page saves per section, so changing a dropdown and saving elsewhere silently does nothing. And the tell is in the metadata: if Changed still equals Created, nothing was persisted.

The four decisions that matter

1. One instance per client, never one shared

For an agency the answer is per-client isolation, ideally in the client's own cloud account. In the order the reasons actually bite:

Handover. Engagements end. A client's own deployment is handed over in an afternoon; extracting one tenant from a shared database is an unpaid migration project. Passkeys make it worse, since WebAuthn credentials are bound to the relying-party ID, which is the domain, so they never survive a move.

Blast radius. One shared instance compromised is every client compromised.

Legal. Shared makes you a data processor for every client's personal data.

Coupling. Shared outages, shared maintenance windows, and your most conservative client gating everyone else's upgrades.

The objection is "that is twelve deployments to maintain", and it is backwards. Twelve identical deployments from one template let you canary upgrades on your own instance and roll client by client. Shared forces big-bang upgrades.

Note that virtual instances give clean domains and admin boundaries but store everything in one database. Good operational separation, not a legal boundary.

2. Reserve the organization layer for the client

The hierarchy is Instance to Organization to Project to Application to Users, and it offers exactly two tenancy levels:

Layer	Represents
Instance	One client company (auth.clientname.com)
Organization	That client's tenants, or one default org if single-tenant
Project	A product or app boundary inside the client
Application	The OIDC or SAML client

Do not use organizations to separate your clients from each other. The moment a client's product needs per-customer branding, SSO or delegated administration, that layer has to be free.

3. Project is the trust boundary, not application

Verified on a live token: an access token issued to one application carries the client IDs of every application in the project, plus the project ID.

"aud": ["<web app id>", "<mobile app id>", "<project id>"]

Two consequences:

• Validate audience against the project ID. Then every present and future

client type is accepted without touching the API. Hardcoding one client ID means the next platform you add fails validation.

• Applications in a project implicitly trust each other's tokens. If a token

must be valid for the mobile app but rejected by an internal admin API, they belong in separate projects.

Decide this per client: one project with many applications (shared trust, simple), or separate projects per trust domain.

4. Where organizations actually live

This decides your schema, so settle it before the first migration.

In ZITADEL when the client's tenants need their own SSO, identity providers or branding. Roles arrive as claims and the admin console is free. Cost: creating an organization during signup is a remote call that cannot join your transaction.

In your application database when tenants are logical groupings. Transactional, joinable, fast. Cost: you rebuild the organization model and give up per-tenant SSO unless you later mirror.

B2B SaaS whose customers are companies: the first. Internal tools and consumer apps: the second, and considerably less work.

Never put your code in the credential path

The browser authenticates directly against ZITADEL. Your API only verifies tokens and never sees a password.

Never build an endpoint that accepts a username and password and forwards them. That is the resource-owner-password grant, deprecated in OAuth 2.1, and it breaks MFA, social login and enterprise SSO simultaneously.

Match the application's authentication method to what your client library can actually send, not to the security ranking. Private Key JWT is stronger and correct for a BFF you write yourself. It is impossible for a library that only knows client secrets, and the resulting error, empty client assertion, names what ZITADEL expected rather than what you misconfigured.

Standing up a new client

Steps 1 and 2 are the irreversible ones.

1. DNS A record for auth.<client>.<tld>, resolving before anything starts
2. --tlsMode external, EXTERNALPORT=443, EXTERNALSECURE=true
3. Generate POSTGRES_PASSWORD and a 32-character ZITADEL_MASTERKEY; store the

masterkey in a secrets manager, labelled with its deployment

1. Deploy, wait for the healthcheck
2. curl .../ui/console/assets/environment.json, both URLs https
3. Log in, change the bootstrap admin password
4. Configure SMTP
5. Register applications, then fix every default in the table above
6. Shorten the access token lifetime; enable refresh tokens
7. Add traefik.http.services.<svc>.loadbalancer.server.scheme=h2c before any

Terraform provisioning

1. Provision with the Terraform provider, not by clicking
2. Verify a restore, with the masterkey, into a scratch instance

Anti-patterns that cost the most

• Booting on a generated hostname. Platforms mint a new one per deploy;

ZITADEL fixes its domain at setup. Everything you create dies on the next redeploy.

• One shared instance across clients. A liability decision disguised as an

infrastructure saving.

• Organizations used for client separation. Burns the layer the client needs.
• A backup without the masterkey. Restores perfectly, decrypts nothing.
• --tlsMode disabled behind a TLS-terminating proxy.
• Bearer instead of JWT access tokens. Forces introspection on every request.
• Hardcoding a client ID as the expected audience. Breaks the next platform.
• Caching introspection results with a TTL. That is what a short-lived JWT

already is, rebuilt with more moving parts.

• A client secret in a mobile binary. It is not a secret.
• An embedded WebView for login. Breaks SSO, password managers and passkeys.
• Proxying credentials through your own API.
• Editing config without rerunning setup. A surviving volume keeps every value

you are trying to change.

The rest

• references/integration.md - token settings,

audience, verification, introspection, JIT provisioning

• references/mobile.md - native clients, PKCE,

redirect schemes, session lifetimes

• references/testing.md - proving a deployment and an

application actually work, including without a client app

• references/operations.md - upgrades, backups,

scaling

• references/deployment.md and

references/troubleshooting.md

• references/untested.md - **documentation only, not

yet verified**: service accounts and machine-to-machine, the limits of ZITADEL's authorization model, external IdPs and SAML federation, and Actions v2

Two things from that last file are worth knowing even before verifying them. Calling ZITADEL's own APIs needs the reserved scope urn:zitadel:iam:org:project:id:zitadel:aud, without which the token is rejected no matter what roles the account holds. And ZITADEL's authorization model stops at project-scoped roles, so "can this user edit this record" is always your application's problem, never a claim you can ask for.

Maintainability

Twelve isolated instances is the right call for the reasons above, and it moves the cost from "extracting a tenant" to "keeping twelve deployments the same". That second cost is the one you actually have to manage.

Provision with Terraform, never by clicking. Step 11 is the maintainability step and it is the one that gets skipped, because clicking works on the first instance. Console configuration is unreviewable, unrepeatable, and undiffable: there is no way to answer "why is this client's token lifetime different" or "does client nine have the roles asserted" except by opening nine consoles. Configuration in a repo turns both into a diff. The settings table above is exactly the set of defaults you will otherwise get wrong once per client.

Drift is the standing risk. Every instance is a copy, and copies diverge. Keep one module with per-client variables rather than twelve edited copies, and when something is fixed, apply it everywhere rather than where it was noticed.

Upgrade continuously, canary on your own instance. The argument for many deployments only pays off if you use it: upgrade your own first, then roll client by client. The failure mode is the opposite, where nobody upgrades because each one is a manual risk, and eighteen months later every instance is several majors behind on the component holding the passwords.

Verify the restore, not the backup. A backup without the masterkey restores perfectly and decrypts nothing, which is a fact you want to learn on a scratch instance rather than during an incident. Put it on a schedule, because an untested restore is a hypothesis and it decays.

Keep the tested and untested boundary intact. references/untested.md exists so that the rest of this skill can be trusted without qualification. When something in it gets exercised for real, move it out. A document that mixes verified and assumed guidance is one a reader has to second-guess line by line, which is the same as not having it.

See the maintainability skill for reversible versus irreversible decisions.

Copy: no em-dashes

Never use an em-dash (U+2014, the long dash) in user-facing strings or in documentation you generate. It is the loudest tell that text was generated. A full stop or a colon is almost always the better edit.

Sources

See references/sources.md for the reading list, the upstream issues referenced here, and which claims are opinion rather than documentation.