Scaling¶

Tracedown scales along three mostly independent axes: how many probes the scheduler can have in flight, how many probes the agent fleet can execute, and how many database connections the JVM services collectively demand. They fail in different ways and have different levers. This page covers each in turn, plus the resource overlay and the Redis split.

The single most important thing on this page is the replica table below. Get it wrong and you get duplicated work rather than an error message.

Replica safety¶

Not every service is safe to run more than once. The distinction is whether the service coordinates — through Redis, or through a claim in the database — or simply assumes it is alone.

Service	Replicas	Why
probe-scheduler	Multiple — safe	Quartz uses a RAM job store, so every replica schedules independently. Cross-replica safety comes from a Redis-backed lock taken per service before dispatch.
aggregate-worker	Exactly one	No distributed lock. Jobs are plain coroutine loops.
api-gateway	Multiple — safe	Stateless HTTP server. What the API keeps between requests — `Idempotency-Key` answers, open event-feed reads — lives in Redis A, so every replica must share it.
result-ingestor	Multiple — safe	Consumes the result queue with an atomic blocking pop; each result is taken by exactly one replica.
notification-dispatcher	Multiple — safe	Claims the outbox rows it reads in the same statement that reads them, with an expiring lease. Each row is delivered by exactly one replica.
email-service	Multiple — safe	Consumes the email queue with an atomic blocking pop.
metrics-service	Exactly one	The scrape side is stateless, but the nudge listener increments Redis counters, and pub/sub delivers each nudge to every replica — N replicas multiply every counter by N.
realtime-service	Multiple — safe	Each replica fans pub/sub events out to its own connected sockets.

Why the scheduler is safe¶

Each replica runs its own Quartz scheduler and will independently decide that a service is due at the same cron tick. The safety net is QueuePolicyManager, which does a Redis SET NX on probe_active:{serviceId} before dispatching. Exactly one replica acquires the key; the others lose. A losing replica does not error — it applies the service's queue policy, so the run is either skipped or enqueued.

This works only because all replicas coordinate through the same Redis A instance. Point two schedulers at different Redis instances and both will dispatch every run.

All scheduler replicas must share Redis A

The lock is the only thing preventing duplicate dispatch. REDIS_A_URL must resolve to the same instance for every replica.

Why the worker is not¶

The aggregate-worker's hourly aggregation, daily aggregation, retention, purge and session cleanup jobs are plain coroutine loops on an interval. There is no lock and no leader election. A second replica does not corrupt anything, but it duplicates every aggregation and retention pass — wasted database load, and retention deleting rows another replica is mid-aggregation over. Run exactly one. See Retention & Aggregation.

Why the dispatcher is safe¶

The notification-dispatcher used to poll the outbox for unpublished rows, deliver, and only then mark them published, with no locking of any kind. Two replicas raced the same rows and both delivered — duplicate emails, duplicate webhook calls, silently. It now claims the rows it reads: one statement both selects and claims a batch (claimed_by / claimed_at, with FOR UPDATE SKIP LOCKED), so no row is ever visible as work to a second replica. The Redis nudge is still a broadcast, and that is now harmless: every replica wakes up, and they take disjoint batches.

The claim is a lease rather than a held lock, because delivery does network I/O and must not hold a database transaction open across it. It is released by marking the row published, and expires after DISPATCHER_CLAIM_LEASE_SECONDS (default 120) so a replica that is killed mid-batch does not strand its rows — another picks them up once the lease runs out.

Delivery is at-least-once

It always was. A replica that dies between delivering and marking a row published re-delivers it after the lease expires. What keeps that from reaching anyone twice is notification_log: every recipient and webhook URL dispatched to for a probe result is recorded before dispatch and checked before dispatching again, so a redelivery completes what was missed and repeats nothing.

Two things stay per replica rather than per platform, because each replica runs its own webhook delivery pool: the circuit breaker that fast-fails a repeatedly failing endpoint, and the bounded delivery queue. A dead endpoint is probed once per replica per open window, and DISPATCHER_WEBHOOK_QUEUE_CAPACITY is the per-replica bound.

One ordering rule is worth knowing about because it caps per-service throughput on purpose: a replica never claims a row while an older undelivered row for the same service is outstanding. That is what keeps a recovery notification from overtaking the failure it recovers from. Different services run fully in parallel — the fan-out is across services — but one service's events are always handled in sequence.

When a notification cannot be delivered at all¶

Because one service's events are handled in sequence, an event that can never be delivered would otherwise sit at the head of its service's queue forever and every later alert for that service would queue behind it — silently. So the dispatcher eventually gives up on one.

DISPATCHER_MAX_EVENT_AGE_MINUTES (default 360, six hours) is the age past which a delivery that has already failed is abandoned: the event is marked handled without being delivered, logged at error with the service, the probe result, the age and the failing exception, and raised to the organization as a notification_dropped alert, so "why did I get no alert" has an answer in the warning log rather than being invisible. The events behind it then proceed, and are delivered or abandoned on their own merits in turn.

A dispatcher outage does not cost you alerts

The age is only ever consulted after a delivery attempt has failed, so a dispatcher that was simply down delivers its whole backlog when it comes back, however old that backlog is. Nothing deliverable is dropped for being old. What the setting bounds is how long an undeliverable event may block its service — and, in the other direction, how long a systemic outage (the database or Redis A unreachable, so every delivery fails alike) may last before the oldest events start being abandoned. Six hours is comfortably past any deploy, restart or dependency outage that gets attention, and short enough that one poisoned event cannot silence a service for a day.

The age is measured from when the probe ran, not when the row was written, so a backlog anywhere upstream ages an event the same way — which is the point: an alert about a probe from this morning has stopped being an alert.

Setting it to 0 or less disables giving up entirely and restores unbounded blocking. Quiet hours and the per-recipient cooldown are unaffected either way: they filter recipients inside a successful delivery, so an event they silence is handled normally and never reaches this path.

When a large backlog does drain, every queued failure for a service is delivered in sequence. For email that mostly collapses on its own — the per-recipient cooldown is opened by the first and drops the rest of the same kind — but webhooks are not cooldown-gated, so a bound endpoint receives the whole burst.

Why metrics-service is not¶

metrics-service has the shape the dispatcher used to have, on its write path. Serving scrapes is stateless, but its nudge listener increments counters in Redis for every probe result it hears about, and the nudge is pub/sub — every replica hears about every result. Two replicas double every counter the scrape endpoint later reports. Run one.

Scheduler throughput¶

Three settings govern the scheduler's capacity. Only one of them is a throughput lever, and it is not the one people reach for.

Variable	Default	Meaning
`SCHEDULER_DISPATCH_WORKERS`	`50`	Concurrent in-flight dispatches
`SCHEDULER_DISPATCH_QUEUE_SIZE`	`100000`	Queued service ids awaiting a worker
`SCHEDULER_THREAD_POOL_SIZE`	`10`	Quartz threads

Dispatch workers are backpressure, not throughput¶

Each dispatch worker awaits the agent's full probe round-trip before taking the next item. That makes SCHEDULER_DISPATCH_WORKERS the platform's global backpressure limit: it caps how many connections Tracedown opens against your targets at once, across every service and every agent.

It is 50 deliberately. Raising it lets the scheduler overwhelm a slow target — the observed failure is congestion collapse and runaway latency against a single endpoint over the public internet. Every probe then reports degraded latency that your API is not actually exhibiting, so you have converted a capacity problem into a monitoring accuracy problem.

Add agents, do not raise the worker count

If you need more probe throughput, add agents. Agents parallelise execution without increasing the concurrency any single target sees from a single scheduler. Raise SCHEDULER_DISPATCH_WORKERS only when you know the targets absorb the extra concurrency — for example, an internal fleet on a LAN.

Queue size¶

The queue holds service ids — roughly 50 bytes each — so the default of 100000 costs almost nothing. It must exceed your largest per-tick fleet: the number of services that can come due on the same cron tick. On overflow the excess is shed, and those runs are recorded as skipped. Your users see the probe reported as over capacity, and the platform raises a dispatch_capacity system alert — see Monitoring Tracedown.

Agent throughput¶

PROBE_AGENT_MAX_CONCURRENCY (default 256) sizes the agent's dedicated probe execution pool. This exists because Python's default asyncio executor pool is only min(32, cpu_count + 4) threads. When probing over the public internet, where each probe spends most of its wall time blocked on the network rather than on CPU, that default caps throughput far below what the host can handle.

Size it from your own numbers:

max_concurrency ≈ peak_probes_per_second × avg_probe_seconds

A fleet doing 20 probes/second against targets averaging 2 seconds needs ~40 concurrent slots. The 256 default has substantial headroom; raise it only when you have measured a queue building at the agent. See Probe Agents.

Database connections¶

This is the constraint that actually bites first on a small host.

HikariCP fills its pool to maximum eagerly rather than lazily, so a pool size is a reservation, not a ceiling you might one day reach. The connections are taken at startup and held idle for the life of the process. Budget them as spent:

Service	Pool	Set by
api-gateway	10	`DB_POOL_SIZE`
result-ingestor	10	`DB_POOL_SIZE`
notification-dispatcher	10	`DB_POOL_SIZE`
probe-scheduler	58	derived from `SCHEDULER_DISPATCH_WORKERS` (50) + 8 headroom
metrics-service	5	hard-coded
aggregate-worker	5	hard-coded
realtime-service	5	hard-coded
email-service	0	holds no database
Total	103

103 is above PostgreSQL's default max_connections of 100, which is why the bundled Compose file starts the database with postgres -c max_connections=160 rather than leaving the default in place. Without that, services fail to acquire connections and exit and the stack never reaches a healthy state.

160 leaves ~57 spare, which is the budget for everything else that connects: a psql session, the pg_dump in Backup & Restore, and above all extra replicas. A second gateway and a second scheduler add 68 by themselves — more than the spare. Raise max_connections before scaling out, or shrink the pools first.

DB_POOL_SIZE tunes the pool per service. But it does not apply everywhere:

Three services ignore DB_POOL_SIZE

aggregate-worker, metrics-service and realtime-service pass maximumPoolSize = 5 explicitly in code, which overrides the environment variable. The resource overlay sets DB_POOL_SIZE for them anyway, which is misleading — those values have no effect. Their pools are 5 regardless. Budget accordingly.

The gateway's pool also serves the API's event feed: each look a waiting read makes is a short transaction, and the feed never takes more than a third of the pool at once, so a burst of feed reads cannot starve the rest of the API. Between looks a waiting read holds no connection.

Sizing the scheduler's pool¶

probe-scheduler is over half the budget on its own, and it is the one pool you should not trim in isolation. It is derived from dispatch concurrency because every dispatch worker can want a connection at the same instant — that is what "concurrent dispatches" means — and a worker that cannot get one loses the probe outright rather than waiting for a slot.

So if you need the scheduler smaller, lower SCHEDULER_DISPATCH_WORKERS and let the pool follow it. Pinning DB_POOL_SIZE below the worker count instead just converts contention into 30-second connection timeouts and lost probes — the same shortage, reported later and less clearly.

The resource-limits overlay¶

The repository ships an overlay that approximates a small production VM (~8 vCPU / ~7 GB for the whole stack), so load tests surface OOM kills, CPU throttling and GC pressure instead of quietly borrowing the whole dev machine.

docker compose -f docker-compose.yml -f docker-compose.limits.yml up -d

It sets three things: per-service CPU and memory caps, JAVA_TOOL_OPTIONS=-XX:MaxRAMPercentage=60 so each JVM sizes its heap from the container cap rather than the default 25%, and per-service DB_POOL_SIZE (with the caveat above).

Per-service limits in the overlay

Service	CPUs	Memory	`DB_POOL_SIZE`
tracedown-postgres	2	2g	—
tracedown-redis-a	0.5	512m	—
tracedown-gateway	1	768m	6
tracedown-scheduler	1	768m	6
tracedown-ingestor	1	512m	4
tracedown-dispatcher	0.5	512m	3
tracedown-email	0.25	384m	2
tracedown-metrics	0.5	512m	3 (ignored)
tracedown-worker	0.5	512m	3 (ignored)
tracedown-realtime	0.5	384m	2 (ignored)
tracedown-nginx	0.5	128m	—

The probe agent is not compose-managed, so the overlay cannot limit it. Cap it on the running container:

docker update --cpus 2 --memory 3g --memory-swap 3g tracedown-agent-dev-agent

The event feed¶

GET /api/public/v1/events is a long-poll: a read can stay open for up to 30 seconds waiting for an event. Two things follow for a deployment that scales out.

Proxy timeouts. Every proxy and load balancer between clients and the gateway must allow a response 30 seconds to start, plus a margin. The shipped nginx.conf and apache.conf rely on their servers' defaults of 60 seconds, which is enough; a load balancer with an idle timeout under 30 seconds cuts long-polls short, and clients see errors instead of empty pages.

Shared bounds. Open reads are bounded at 2 per API key, 6 per user and 24 per organization, counted in Redis A across every gateway replica, and each replica takes at most 512 at once, 24 of them for one organization. With Redis A unreachable, each replica falls back to bounding a key to 2 on its own. A waiting read holds neither a thread nor a database connection — it waits for a nudge on Redis A — so open reads cost memory and a socket each, not pool capacity.

Splitting Redis¶

Tracedown addresses Redis in three roles — A (operational queues, outbox nudges, pub/sub; AOF-persisted), B (cache), and C (optional hierarchy cache). By default REDIS_A_URL, REDIS_B_URL and REDIS_C_URL all point at a single instance, and for most deployments that is enough.

The separation lives at the URL layer, not the data layer, so splitting them out is purely a configuration change: stand up the instances and repoint the URLs. The compose file carries commented-out service blocks for B and C ready to uncomment.

Split when you hit one of these:

Sustained throughput above roughly 1k probe results per minute.
You need failure isolation — cache eviction should never risk dropping queued results or outbox events.
You want to tune persistence and eviction independently. A wants AOF; B and C want --save "" with an allkeys-lru maxmemory policy, since everything in them is reconstructible.

That last point is the real argument. Sharing one instance forces one persistence policy onto data with opposite durability requirements: either you pay AOF cost for a cache, or you risk losing queued results to an eviction.