Troubleshooting

Every failure observed bringing up a real instance, with the symptom you actually see and the cause it points nowhere near.

Start here

curl -s https://<domain>/ui/console/assets/environment.json

{"api":"https://<domain>","issuer":"https://<domain>","clientid":"..."}

Both must be https. This one call collapses most of the table below into a single observation, and it is the first thing to run when anything looks wrong.

The two fields can disagree because they are built differently: api comes from configuration, issuer comes from the request, where the proxy's X-Forwarded-Proto wins. So api: http alongside issuer: https is the precise signature of ExternalSecure being false while the proxy serves HTTPS.

Two more diagnostics worth knowing:

# what the container is ACTUALLY running, and when it last restarted
docker inspect $(docker ps -qf name=zitadel) \
  --format '{{json .Config.Cmd}}{{println}}{{.State.StartedAt}}'

# does the transport work? distinguishes instance resolution from h2c
curl -si -X POST https://<domain>/zitadel.management.v1.ManagementService/GetMyOrg \
  -H 'Content-Type: application/json' -d '{}'

For the second: a 401 or any response carrying a grpc-status header means the transport is healthy. A 404 means instance resolution. A 502 or protocol error means h2c.

The table

Symptom	Actual cause	Fix
Console shows [unknown] Failed to fetch; devtools network tab says blocked:csp	api is http while the page is https, so every XHR is cross-scheme	--tlsMode external
Login works, then everything breaks at /ui/console/users/me	Same cause. The login flow is server-rendered HTML with no XHR, so it is unaffected; users/me is the first route that calls the API	Same
Console keeps failing after a correct redeploy	The browser cached environment.json	Hard reload (Ctrl+Shift+R)
API routes 404 while static assets load fine	ZITADEL resolves virtual instances by Host header and the stored instance domain does not match	Redeploy with the correct ExternalDomain and the volume deleted
Config change appears to do nothing	ExternalDomain/Port/Secure need the setup phase to rerun; a surviving Postgres volume keeps the old values	docker compose -p <project> down -v
ExternalSecure=true set but api still http	--tlsMode disabled on the command line overrides it	Change the flag
Terraform or an SDK fails while the console works	Console uses gRPC-Web over HTTP/1.1; native gRPC needs HTTP/2	Add loadbalancer.server.scheme=h2c
could not create email channel ... Errors.SMTPConfig.NotFound	No SMTP configured. Password reset, verification and invitations all fail silently	Console: Settings, Notifications, SMTP. Or DefaultInstance.SMTPConfiguration
Passkeys cannot be registered	WebAuthn requires a secure context, and ExternalSecure=false means cookies are not marked secure	--tlsMode external
all predefined address pools have been fully subnetted	Docker ran out of bridge subnets after repeated redeploys. Not a ZITADEL problem	See below
missing translation ... PasswordChange.Footer	Cosmetic gap in v4.16.0 email templates	Ignore
invalid_client / empty client assertion at the token endpoint	The application's auth method is Private Key JWT but the client library sent a client secret. The error names what ZITADEL expected, not what you set	Match the app's auth method to the library. See integration.md
Login succeeds but the session's user object is empty	Profile claims are not in the ID token by default. Requesting the profile scope is not enough	☑ Include user's profile info in the ID Token
Roles claim missing after ticking the app checkbox	The project also needs Assert Roles on Authentication, and roles must exist and be assigned	Both, plus real role assignments
Console setting looks changed but behaves as before	The application page saves per section. Tell: Changed still equals Created	Save on that specific section
An API cannot read email or name from the access token	Access tokens never carry profile claims, only the ID token and introspection do	Call userinfo or introspect once per new user
A new client platform fails audience validation	The API hardcodes one client ID; audience is project-wide	Validate against the project ID
Code:XXXXXX visible in container logs	SMTP is unconfigured, so the notification failed and the template arguments (including the init code) were logged	Configure SMTP. Treat those logs as containing credentials

Why blocked:csp is the giveaway

ZITADEL emits a CSP header whose connect-src lists 'self' plus the bare hostname. When the console is served over HTTPS but reads api: http://... from environment.json, its XHRs target a different scheme, and CSP refuses on both counts:

• 'self' is the https origin and does not match an http URL
• a schemeless host-source only matches the document's own scheme, or an

http-to-https upgrade. Never a downgrade.

So the request never leaves the browser: 0.0 kB, 1 ms, blocked:csp. The CSP header itself is correct. The api URL is not.

This is also why the failure looks like it begins after login. Login is server-rendered navigation with no restricted XHR at all.

Docker address pool exhaustion

Not ZITADEL, but you will hit it while iterating. Docker carves a subnet per bridge network from default pools of 172.17.0.0/12 in /16s plus 192.168.0.0/16 in /20s: roughly 31 networks, then failure. Every compose app takes one, and failed deploys leave theirs behind.

docker network ls | wc -l
docker network prune -f

Permanent fix in /etc/docker/daemon.json, then restart Docker (which bounces every container on the host):

{
  "default-address-pools": [
    { "base": "172.17.0.0/12", "size": 24 },
    { "base": "10.201.0.0/16", "size": 24 }
  ]
}

Carving /24s instead of /16s takes you from about 31 networks to several thousand, and 254 addresses is ample for a compose stack. Check the second range does not collide with your provider's private networking or a VPN.

Known upstream issues

• #12065 HTTP used to

retrieve the discovery document despite ExternalSecure=true, behind Traefik. Open. Documented workaround is terminating TLS at ZITADEL itself (--tlsMode enabled with a mounted certificate).

• #4475 The older Traefik

variant of the same complaint.

• #9912 environment.json

listing the internal IP instead of the external domain. Fixed.

Before blaming #12065, check the flag. If issuer resolves to https then the proxy headers are correct and ZITADEL is honouring them, which means a http api is configuration, not a bug. Genuine #12065 territory is when the running command says external, the container restarted recently, and api is still http.

A note on method

Several of these produce errors that point nowhere near the cause: a CSP error for a TLS mode setting, a transport-layer suspicion for a scheme mismatch, a "works then breaks" pattern for something that was wrong from the start.

Check environment.json first, then the running container's actual command, and only then form a theory. Both are one command and neither can mislead you.