Troubleshooting¶

Known failure modes, each as cause and fix. They are grouped by when you hit them: standing the stack up, enrolling agents, and running in production.

Installation and startup¶

The stack will not come up: `FATAL: sorry, too many clients already`¶

Some services start; the rest fail to acquire database connections and exit. Nothing reaches a healthy state. Which services survive varies between restarts, because it is a race for a fixed pool of connections.

Cause. max_connections is below what the stack reserves. It reserves 103 — and PostgreSQL's default is 100, so an unmodified database is already three short. HikariCP fills each pool to its maximum eagerly rather than on demand, so the demand is immediate and permanent rather than gradual, and probe-scheduler alone reserves 58 of the 103 (its pool is derived from its 50 dispatch workers). See Scaling for the full table.

Fix. Start PostgreSQL with max_connections=160, as the bundled Compose file does:

command: ["postgres", "-c", "max_connections=160"]

On a database you did not start from that Compose file, set it in postgresql.conf and restart — max_connections is not reloadable, a SIGHUP will not take. Confirm with SHOW max_connections; rather than assuming.

If you cannot raise it, lower the demand instead: reduce SCHEDULER_DISPATCH_WORKERS on probe-scheduler (its pool follows) and DB_POOL_SIZE on the services that honour it. Do not pin DB_POOL_SIZE on the scheduler alone — a pool below the worker count turns the shortage into 30-second connection timeouts and lost probes instead of a clean startup failure.

DB_POOL_SIZE does not apply everywhere

aggregate-worker, metrics-service and realtime-service pass maximumPoolSize = 5 in code, which overrides the environment variable. Their pools are 5 whatever you set. Count them as 5 each when budgeting.

See Scaling.

Postgres will not start: `could not access file "timescaledb"`¶

Only after upgrading from a release whose Compose file pulled timescale/timescaledb:latest-pg16. Postgres exits immediately with FATAL: could not access file "timescaledb": No such file or directory.

Cause. That image wrote shared_preload_libraries = 'timescaledb' into postgresql.conf inside the data volume when the volume was first initialised. The setting lives in the volume, not the image, so it survives the switch to a plain postgres image — which does not ship the library.

Fix. That volume is also a PostgreSQL 16 cluster, and the current stack runs PostgreSQL 18, which cannot open it — so the way through is the dump-and-restore in Upgrading: dump with the old image, restore into a fresh volume. The dump may carry a CREATE EXTENSION timescaledb line, which is safe to delete. Nothing in the schema used the extension, so nothing is lost.

If you are staying on a PostgreSQL 16 image for now, removing the setting from the existing volume is enough, and does not need the database to be running:

docker compose down
# The Compose project is named "tracedown", so the volume is prefixed.
# Confirm yours with: docker volume ls
docker run --rm -v tracedown_tracedown-pgdata:/pg alpine \
  sed -i "s/^shared_preload_libraries = 'timescaledb'/#&/" /pg/postgresql.conf

Postgres will not start: `there appears to be PostgreSQL data in`¶

Only after upgrading to a release whose Compose file pulls postgres:18-alpine while keeping a volume that an older image initialised. The container exits with Error: in 18+, these Docker images are configured to store database data in a format which is compatible with "pg_ctlcluster" and names the directory it found.

Cause. The volume holds a PostgreSQL 16 cluster. The 18 image keeps its cluster in a version-specific subdirectory (/var/lib/postgresql/18/docker) and refuses to start next to data it did not create, rather than silently initialising an empty database beside it.

Fix. Restore a dump into a fresh volume — the procedure is in Upgrading. The refusal is the safe outcome: the old data is untouched until you remove the volume yourself.

The Docker build fails at a COPY step¶

Cause. The backend's Docker build context is the parent of the repository root, so the repository must sit at core/tracedown-core-backend inside a containing directory. Cloned anywhere else, the build fails with a "not found" error at the first COPY step.

Fix. Clone into the tree shown in Requirements.

realtime-service will not start¶

Cause. Its DATABASE_URL, DATABASE_USER, DATABASE_PASSWORD and REDIS_A_URL are mandatory HOCON substitutions with no defaults. A missing one fails config resolution at startup rather than degrading at runtime.

Fix. Set all four. See Configuration.

Agents and certificates¶

An agent registers but never receives work¶

The agent is up, enrolled, and idle.

Cause. The scheduler dials the agent — the agent does not poll for work. It must therefore be reachable inbound from the scheduler. An agent behind NAT with no inbound route registers successfully and then never hears from anyone, which is why this looks like a scheduling bug rather than a networking one.

Fix. Give the scheduler a route to the agent's URI. Also confirm the agent is active and passing health challenges — see Monitoring Tracedown and Probe Agents.

"CA root not initialized — run --agent-bootstrap first"¶

Cause. No active CA row exists in the database.

Fix. Run --agent-bootstrap, which creates it. This is exactly what the compose tracedown-ca-init step does:

./bin/api-gateway --agent-bootstrap <agent-slug>

See Certificate Authority.

A bootstrap token is rejected¶

Cause. Tokens are single-use with a 1 hour TTL. A reused or stale token is refused.

Fix. Generate a fresh one with --agent-bootstrap.

An agent refuses to enrol: "could not authenticate the gateway"¶

Cause. The agent authenticates the gateway before the bootstrap token leaves the process, and it could not. Usually the gateway's certificate comes from a private CA the agent's system trust store does not carry, or is self-signed. Registration was abandoned before the token was sent, so nothing was leaked and the token is still unused — but it is still ticking towards its 1-hour expiry.

Fix. Set PROBE_AGENT_BOOTSTRAP_CA_BUNDLE to that CA's PEM bundle, or PROBE_AGENT_BOOTSTRAP_PIN_SHA256 to the fingerprint of the certificate the gateway presents. See Authenticating the gateway at enrolment.

An agent refuses to enrol in production over `http://`¶

Cause. DEPLOYMENT_ENV=production and PROBE_AGENT_SCHEDULER_URL is not https://, so the token would cross the wire in the clear. The agent raises instead. PROBE_AGENT_INSECURE_SKIP_BOOTSTRAP_TLS_VERIFY is refused in production too, so it is not a way round this.

Fix. Point the agent at an https:// URL that terminates TLS in front of the gateway. The nginx.conf and apache.conf shipped in docker/deploy/ already proxy the enrolment paths; a vhost you wrote yourself may not, and a request that misses them is answered with the dashboard's index.html rather than by the gateway. See Reaching enrolment over https. On a private network where the agent and the gateway share the Docker network, the alternative is to leave DEPLOYMENT_ENV unset on the agent container, which is what the local stacks do.

Agent renewal skipped with an unknown-slug warning¶

Cause. The agent cannot determine its own slug. It is normally persisted at /certs/agent-slug.txt; if that file is absent — typically a lost or recreated cert volume — renewal has nothing to renew against and skips.

Fix. Set PROBE_AGENT_SLUG explicitly.

Variables decrypt as garbage, or agent mTLS fails after a config change¶

Cause. PLATFORM_AES_KEY differs between api-gateway, probe-scheduler and notification-dispatcher. All three must share the exact same value — the gateway encrypts, the other two decrypt. The dispatcher needs it for org variables referenced from webhook URLs.

Fix. Make the value identical across all three. Note that data encrypted under a previous key is not recoverable by setting a new one — see Secrets & Encryption.

Running¶

Users see runs marked skipped and a dispatch_capacity banner. A skipped run never reached an agent; three different faults produce one, and the banner says which. The two entries after this one cover the others.

Cause. Either the dispatch queue overflowed (SCHEDULER_DISPATCH_QUEUE_SIZE, default 100000 — it must exceed your largest per-tick fleet), or a service's next tick came due while its previous tick was still waiting in the queue. Both are capacity sheds; a run suppressed by the service's queue policy is dropped silently and does not appear as skipped.

Fix. For overflow, raise the queue size — entries are tiny. For sustained capacity pressure, add agents. Resist raising SCHEDULER_DISPATCH_WORKERS: it is the global backpressure limit, and raising it risks congestion collapse against slow targets. See Scaling.

Probes are skipped with no agent available¶

The alert is no_eligible_agent.

Cause. The tick found nothing to dispatch to. Either no agent is currently passing its health challenge, or the affected services are restricted to agents that are not — a service pinned to one agent has nowhere to go the moment that agent stops being eligible.

Fix. Check agent health first (Settings → Agents, or Monitoring Tracedown); a fleet that looks empty to the scheduler is often one failing challenge away from working. If the fleet is healthy, check the affected services' agent pickers — see Services.

Probes are skipped although the agents look healthy¶

The alert is agent_dispatch_failed.

Cause. Agents were eligible — they passed a challenge within the last minute — and every one of them was tried without a single one taking the run. The connection was refused, the handshake did not complete, or the job was turned away. This is what an agent lost between health rounds looks like: a container killed, a network path closed, or a certificate the scheduler no longer trusts.

Fix. Confirm the agent processes are running and reachable inbound from the scheduler on port 8443, and that their certificates are current. See Probe Agents and Certificate Authority.

Runs are recorded with status `error`¶

Cause. The run did not evaluate. Either the script or the executor failed, or the agent took the job and broke while running it — it answered HTTP 500, or with something that is not a probe result. Such a run is never re-dispatched: the agent may already have called the monitored target, and retrying would probe it twice for one tick.

Fix. The diagnostic is stored with the run, on its Raw result tab. If it points at the agent rather than the script, the agent's own logs have the rest. See Reading Results.

Agent statuses are frozen and health checks report as unavailable¶

The alert is health_token_unavailable, and it names an endpoint rather than an agent.

Cause. A health challenge requires the agent to fetch a one-time token from the gateway. The scheduler could not reach that endpoint itself, or could not store the token in Redis A to begin with — so the round proves nothing about the agent and is discarded as inconclusive rather than held against it. Usually the gateway is down, Redis A is unreachable, or GATEWAY_URL on the scheduler points somewhere the scheduler cannot actually reach.

Fix. Check the gateway and Redis A, then GATEWAY_URL — its default http://localhost:8080 is not the gateway's own port and must be set explicitly. See Configuration and Monitoring Tracedown.

The mirror-image fault reads as agent_down

GATEWAY_URL has to be reachable by the agents too — the scheduler hands each one GATEWAY_URL plus /internal/health/token/{challengeId} to fetch. A value only the scheduler can resolve, such as the internal Docker address against an agent on another host, produces the opposite symptom: the scheduler reaches the endpoint, so the round is not excused, and the agent is marked down instead. Agents off the Docker network need the public https URL there, which the shipped vhosts proxy — see Reaching enrolment over https.

Aggregation or retention work is duplicated¶

Cause. More than one aggregate-worker replica is running. It has no distributed lock — its jobs are plain coroutine loops that assume they are alone.

Fix. Run exactly one replica. See Retention & Aggregation.

Emails are not sent¶

Cause, the common one. EMAIL_PROVIDER defaults to console, which only logs the message.

Cause, the subtle one. The variable is set on the wrong service. email-service is the only process that talks to a mail provider — the gateway and the notification-dispatcher publish envelopes onto the email_queue in Redis A and never open an SMTP connection of their own. Provider settings (EMAIL_PROVIDER, EMAIL_FROM_ADDRESS, EMAIL_SMTP_*, EMAIL_RESEND_API_KEY, EMAIL_MAILGUN_*) therefore only do anything on email-service; set on any other service they are ignored, and mail keeps going to a log.

Fix. Configure the provider on email-service and check its logs, not the sending service's. See Configuration.

The usage window is shorter than expected¶

Cause. RESULT_RETENTION_DAYS caps it — you cannot display history that has been deleted. The gateway's value must match the aggregate-worker's, or the dashboard advertises a window the worker has already pruned.

Fix. Set RESULT_RETENTION_DAYS identically on both. See Retention & Aggregation.

Probes against a domain are limited to 3 calls, save no bodies, and run no more than every 5 minutes¶

Cause. TRUSTED_DOMAIN_MODE is false and the target domain is unverified. The anti-abuse policy then caps the script at 3 calls, disables body saving, and enforces a 5-minute minimum interval. Every tick the policy withholds is recorded as a skipped result whose reason names the rule — unverified_throttle (ran too soon after the previous run), unverified_max_calls (more than 3 calls) or unverified_includes (includes() against an unverified domain) — so the gap is explicable from the history; no alert is raised for them.

Fix. Verify the domain (Settings → Domains), or set TRUSTED_DOMAIN_MODE=true if every target is one you control — this turns the ownership checks off. Verification is the default; it signals good-faith use of the platform. See the User Manual and Configuration.