Response

Confidence

• Overall: moderate. I am fairly confident that the tenant_retry_policy flag path was needed for the sustained outage, and much less sure how it produced database connection exhaustion.
• Higher confidence (facts that line up): exhausted jobs show policy="" with delay_ms=0 and 38-41 ms budgets; the same job (319) retried with a 1000 ms delay and succeeded within seconds of the flag going off; the API rollback and a worker restart did not clear the problem while the flag stayed on; recovery began within two minutes of the flag being disabled.
• Lower confidence (inference): that zero-backoff retries are what saturated the 300 DB connections; that the worker scale-up made it worse (it is correlated, not shown); and that the flag was the origin rather than an amplifier of an earlier failure. The logs are a 12-line sample, with only one worker and one job showing the before/after change and no control group. The unrelated-infrastructure and API-only-release hypotheses are weakened, not ruled out.

Next two pieces of data I would request

1. Flag audit and policy resolution data. The change history for tenant_retry_policy in eu-west and US (when it was enabled, scope, percentage, and role of the approver), plus, per tenant, the resolved retry policy value and where it came from. This fixes the onset time against the 09:02 deploy and the 09:07 symptoms, shows why the policy was empty and for how many tenants, and gives a real dose-response comparison between eu-west and the US 2% cohort.
2. Time-aligned DB and retry telemetry, 09:00-10:20 UTC, both regions. Per-minute DB connections by client and pool (API versus workers), pool wait times, lock waits and idle-in-transaction counts, next to per-minute counts of retries, delay_ms distribution, and retry_budget_exhausted by policy value, and the error that caused the first failed attempt. This tests the weakest link (retries to connection exhaustion), shows which happened first (pool waits at 09:06:56 or the zero-delay retries), and shows who held the connections.

Actions: eu-west queue stall

Owners are roles, not people. Actions follow from the working hypothesis in INCIDENT.md (empty retry policy under tenant_retry_policy removed backoff and, with the scale-up, saturated the DB). That hypothesis is probable but unproven, so several actions below are written to confirm it as well as to fix it.

Priority key: C = containment, do now. P0 = before the flag is enabled anywhere again. P1 = within about two weeks. P2 = within the quarter.

1. Immediate containment

Flag disablement at 09:43 in eu-west and the API rollback at 09:24 are already done (per timeline). The items below close the remaining exposure.

C1. Verify the flag is really off everywhere it matters in eu-west

• Owner: On-call SRE (with Feature-flag platform owner).
• Do: check the flag at every level (global, region, per-tenant overrides, cached snapshots). Restart or redeploy one worker canary and confirm it starts with feature_flag=false. The 09:31 restart came up with feature_flag=true, so start-up-time flag reads are a known trap.
• Verification signals: worker start-up log shows feature_flag=false; zero retry_budget_exhausted policy="" lines and no delay_ms=0 retries across a full hour of normal load; retries show non-zero delay_ms.
• Rollback: re-enabling the flag is the rollback, and it should not be done until P0 items are complete.

C2. Turn the flag off in US as well (or confirm the exposed 2% are unaffected first, without delaying the switch-off)

• Owner: On-call SRE (executes), Release manager (approves).
• Do: same code, flag on for 2% of tenants, so the same defect may be silently present. In parallel, search US logs for policy="", delay_ms=0, and retry_budget_exhausted since the release, by tenant.
• Verification signals: US logs show no empty-policy exhaustion after the switch-off; if any existed before, list the affected tenants for C5.
• Rollback and side effects: turning it off reverts those tenants to default exponential retry (their pre-release behaviour). Ask the Product owner whether any of the 2% cohort was enabled deliberately for a customer commitment, and inform them if so.

C3. Freeze: keep the API rollback in place and block re-release of 2026.09.23.1 and any flag change for tenant_retry_policy

• Owner: Release manager.
• Verification signals: pipeline gate or flag lock is active; change calendar shows the freeze.
• Rollback considerations: confirm the rolled-back API version can read any data the new version wrote (for example tenant policy rows or config); the timeline does not say whether the release had schema or data changes. If it did, get the Database owner to confirm compatibility before any further rollback or roll-forward. Lift the freeze only when P0 is verified.

C4. Return worker count to its pre-incident level in controlled steps, if still at 90

• Owner: On-call SRE (with Capacity owner).
• Do: check DB connection use first. Reduce in steps (for example 90 to 70 to 50 to 40) while watching job age.
• Verification signals: DB connections stay well under 300 (target under about 70% at peak); job age and queue depth stay flat through each step; no pool-wait warnings.
• Rollback: if job age rises after a step, go back up one step. Do not scale above the connection budget: scale-up during the incident coincided with the queue growing faster (correlation, but enough reason for caution).

C5. Find and reconcile jobs affected by the incident window

• Owner: Backend engineering lead (with Support lead for customer-facing follow-up).
• Do: for eu-west 09:00-10:30 UTC (and US per C2), list every job that logged retry_budget_exhausted, its final state, and whether it touched a non-idempotent side effect (payment capture, email, webhooks). Check for duplicates and for jobs left in a non-terminal state.
• Verification signals: reconciliation report shows created jobs equal terminal-state jobs; zero unexplained non-terminal jobs; zero duplicate side effects, or a list of them.
• Rollback considerations: replays cannot be undone. Replay only with idempotency keys, run a dry-run first, and rate-limit replays so that they do not re-saturate the DB.

2. Corrective actions (prioritized)

P0-1. Reproduce and confirm the mechanism

• Owner: Performance/reliability engineer with the Queue service engineer.
• Do: in a staging environment with production-like DB connection limits, enable the flag with a tenant that has a missing, empty, and invalid policy. Inject transient job failures. Record retry delay, exhaustion rate, and DB connection use. Also test whether retries acquire a connection per attempt.
• Verification signals: repro shows policy="", delay_ms=0, exhaustion in tens of ms, and connection growth (confirms H1 and I3). If connection growth does not appear, the working hypothesis is wrong or incomplete, and the RCA must be reopened (see H2 and H4 in INCIDENT.md).
• Rollback: staging only. No production impact.

P0-2. Fix policy resolution: never resolve to an empty policy

• Owner: Queue service engineering lead.
• Do: fall back to the configured default (exponential) when a tenant policy is missing, blank, or invalid. Validate tenant policies at load time and alert on invalid ones. Add tests for each config shape.
• Verification signals: tests pass; the P0-1 repro no longer shows empty policy or zero delay; staging run with the flag at 100% shows only non-zero delays.
• Rollback: ship the fix with the flag still off. If the fix itself misbehaves, revert the change; the flag stays off, so there is nothing else to unwind. Note that tenants who genuinely had a blank policy will now get the default; confirm none intended "no retry".

P0-3. Add a retry floor and storm guard

• Owner: Queue service engineering lead.
• Do: enforce a minimum delay and jitter regardless of policy; when a job's budget is exhausted, requeue with backoff or dead-letter it instead of making it immediately available again; cap retry attempts per second per tenant.
• Verification signals: fault-injection test shows bounded retry rate and stable DB connections; production metric of exhausted-budget jobs per minute stays near zero.
• Rollback: make the floor and cap configuration values. If they delay legitimate work, tune the numbers rather than removing the guard.

P0-4. Staged re-enablement plan with health gates

• Owner: Release manager with the Feature-flag platform owner.
• Do: re-enable only after P0-1 to P0-3 are verified. Sequence: internal tenants, then 2%, 10%, 50%, 100%, one region at a time (US first). Halt automatically on retry exhaustion, DB pool wait, or job-age thresholds.
• Verification signals: each stage has a soak period with clean metrics; the auto-halt has been tested with a synthetic bad flag.
• Rollback: a documented one-step kill switch for the flag, tested before stage one.

P1-1. DB connection budget and isolation

• Owner: Database/platform reliability lead.
• Do: compute worst-case connections (pool size x workers x replicas, for API and workers) against the 300 limit at maximum autoscale; introduce per-client caps or a connection pooler; reserve headroom for the API so that worker trouble cannot starve job creation and checkout.
• Verification signals: worst-case calculation stays under about 80% of the limit; a load test at 90 workers keeps API connection acquisition healthy.
• Rollback: introduce the pooler or caps region by region (US first), keep the direct path available, and change pool sizes one pool at a time.

P1-2. Detection: alert on causes as well as symptoms

• Owner: Observability lead.
• Do: alerts for DB connection use above 80%, db pool wait exceeded, retry_budget_exhausted rate, and "backlog growing while worker CPU falls". Annotate dashboards with flag and deploy events.
• Verification signals: replaying the incident's metrics (game-day or backfill) fires a cause-level alert within about 2 min of onset (the actual first alert took about 4 min and was symptom-only); the alert links to recent flag changes.
• Rollback: alerts can be muted or re-thresholded. Track noise for two weeks and tune.

P1-3. Runbook and autoscaler guard for backlog incidents

• Owner: On-call lead / Incident process owner.
• Do: add to the queue-backlog runbook: check recent flag and config changes first; check DB connections and pool waits before scaling; if scale-up makes the queue worse, stop and revert the scale. Bound the autoscaler by the DB connection budget.
• Verification signals: tabletop exercise walks the incident and reaches the flag within 10 min; attempts to scale beyond the cap are refused or warned.
• Rollback: the cap is configuration and can be raised with an approval step.

P1-4. Blameless post-incident review

• Owner: Engineering manager (facilitator), with representatives from Queue, SRE, Release, and Database.
• Do: reconcile the timeline (especially flag enable time), review decision points and the information responders had, and track actions to closure.
• Verification signals: review published, actions ticketed with owners and dates, and the P0/P1 items above are closed or scheduled.
• Rollback: not applicable.

P2-1. Improve log content for retry and DB behaviour

• Owner: Queue service engineer.
• Do: include region, build version, flag state, resolved policy and its source (tenant versus default), the error that caused the first failure, DB pool acquire time, and job state after budget exhaustion in the retry logs.
• Verification signals: a game-day exercise can answer the "unknown" questions in INCIDENT.md section 7 from logs alone.
• Rollback: if log volume is a concern, sample at INFO but keep WARN and ERROR.

P2-2. Separate risky behaviour changes from deploys

• Owner: Release engineering lead.
• Do: avoid enabling a flag for a large share of a region in the same window as its code deploy; require a recorded flag-enable step (time, scope, approver role) in the change log so the timeline can always answer "when was it turned on".
• Verification signals: change log contains flag events for the next release; incident reviews no longer list "flag enable time unknown".
• Rollback: process change only; revert by policy decision.

Service disruption in our EU-West region

Between about 09:07 and 10:18 UTC, some customers served from our EU-West region may have seen slower checkouts and delays in background processing.

What we know: Job processing slowed and a backlog built up. We rolled back a recent release and switched off a related setting that controls how failed tasks are retried. Processing began recovering shortly after that setting was switched off, and the backlog was fully cleared at 10:18 UTC.

What we are still working out: The evidence points to that retry setting as the most likely trigger, but our investigation is not complete and we have not confirmed the exact chain of events. We are also checking whether any individual tasks need to be reprocessed, and will contact affected customers directly if so.

We plan to keep the setting off until we have verified a fix. We apologize for the disruption and will share a further update when our review is complete.

Incident analysis: eu-west queue stall (release 2026.09.23.1)

Sources: TIMELINE.md (cited as T) and LOGS.txt (cited as L plus line number). Convention: [F] = observed fact, stated in a source. [I] = inference, my reasoning on top of the facts. Nothing tagged [I] should be read as established.

Status at time of writing: service recovered (backlog cleared 10:18 UTC). Root cause is probable, not proven.

1. Impact

Observed [F]

• Region: eu-west. Alerts fired for job age and checkout latency at 09:11 (T).
• Queue depth rose from 09:07 and worker CPU fell from 65% to 18% (T).
• The database reached its configured maximum of 300 connections at 09:20 (T). At 09:19:52 api-03 logged that no connection slots were available (L7).
• New job creation was degraded until the API rollback at 09:24, after which it "normalizes" (T).
• Backlog stayed stuck after the rollback and cleared at 10:18 (T).
• US region: same release, flag enabled for 2% of tenants, no alert fired (T).

Derived durations [F, arithmetic on T]

• Onset to alert: about 4 min (09:07 to 09:11). The first log evidence of the problem is 09:06:58 (L1).
• Alert to mitigation that worked (flag off): 32 min (09:11 to 09:43). Onset to flag off: 36 min.
• Onset to backlog cleared: about 71 min (09:07 to 10:18). Flag off to backlog cleared: 35 min.

Not known

• Number of tenants and customers affected, number of checkouts that failed or timed out (versus merely slowed), and whether any jobs were dropped, dead-lettered, or executed more than once.
• Whether the US 2% cohort was silently affected (no alert is not the same as no impact).
• The incident date. The release tag suggests 2026-09-23 but T does not state a date.

2. Concise timeline (UTC)

Time	Event	Source	Type
09:02	API deploy 2026.09.23.1, adds tenant-specific retry policy	T	F
?	tenant_retry_policy flag enabled in eu-west (time and % of tenants not recorded)	none	gap
09:06:58	job 81 (tenant acme) retries 3x with delay_ms=0, then retry_budget_exhausted policy="" after 41 ms	L1-L4	F
09:07	Queue depth begins rising; worker CPU 65% to 18%	T	F
09:07:01	worker-22: DB pool wait exceeded 5000 ms, active=10 idle=0	L5	F
09:07:03	worker-08: job 92 (tenant beta) succeeds first attempt, 830 ms	L6	F
09:11	Alerts: job age, checkout latency	T	F
09:14	Workers scaled 40 to 90; queue depth then rises faster	T	F
09:19:52 / 09:20	api-03 out of DB connection slots; DB at 300 max connections	L7, T	F
09:24	API rolled back; job creation normalizes; backlog stays stuck	T	F
09:31:10	worker-51 restarted: config_retry_default="exponential" feature_flag=true	L8	F
09:31-09:35	Restarted pool processes normally for about 4 min, then stalls	T	F
09:35:12	worker-51: job 319 retry_budget_exhausted policy="" after 38 ms	L9	F
09:43	Flag tenant_retry_policy disabled; recovery begins within 2 min	T	F
09:43:07-10	worker-51 sees feature_flag=false; job 319 retries with delay_ms=1000, succeeds on attempt 2 in 190 ms	L10-L12	F
10:18	Backlog cleared	T	F

3. Observed facts versus inference

Observed [F]

1. In both retry_budget_exhausted log lines (L4, L9) the policy is an empty string, and the budget was spent in 38-41 ms.
2. In the only job with per-attempt detail while the flag was on (job 81), every retry had delay_ms=0 (L1-L3).
3. The worker start-up config shows a default retry policy of exponential (L8), yet the exhausted jobs report policy="".
4. After the flag was turned off, job 319, which had exhausted its budget at 09:35 (L9), retried with a 1000 ms delay and succeeded (L11-L12).
5. Rolling back the API fixed job creation but not the backlog (T).
6. Restarting a worker pool with the flag on gave about 4 minutes of normal processing, then a stall, and the first exhaustion line after restart appears 4 min 2 s after start-up (T, L8, L9).
7. Scaling workers 40 to 90 was followed by faster queue growth, and DB connections hit 300 six minutes later (T).
8. Worker CPU fell rather than rose during the stall (T), and a worker was waiting 5 s on its DB pool at the start (L5).

Inferred [I]

• I1. With the flag on, the policy lookup returns an empty value for at least some tenants or jobs and the code does not fall back to the exponential default. (Supported by facts 1-4; the reason for the empty value, such as tenant config with no policy or a bug in the new code, is not shown.)
• I2. An empty policy means zero backoff, so failing jobs are retried instantly, burn their budget in tens of milliseconds and are put back in the queue. (Facts 2 and 4 support this; that job 319 was picked up again later suggests exhausted jobs are re-queued rather than dropped, but the logs do not say.)
• I3. This churn saturated the shared DB connection limit, which is what starved workers (low CPU, 5 s pool waits) and later the API. This link is the weakest part of the chain. No log ties retries to connection usage.
• I4. Scaling to 90 workers made it worse by adding more consumers competing for a fixed 300 connections (fact 7). Correlation only; queue depth could also have risen faster simply because more workers produced more exhausted retries.
• I5. The stall after restart (fact 6) is consistent with the restarted workers reaching jobs that need a retry, or with DB saturation returning. The data cannot separate these.

4. Most likely root cause

Enabling the tenant_retry_policy code path (shipped in 2026.09.23.1) produced an empty retry policy that removed retry backoff. Jobs that hit a transient failure were retried immediately and exhausted their budget, creating a sustained retry loop that, together with the worker scale-up, exhausted the database connection limit and stalled workers and the API.

Confidence: moderate that the flag-enabled path was necessary for the sustained outage; low to moderate on the exact mechanism between retries and connection exhaustion (I3).

Why this hypothesis ranks first: it is the only one that explains the empty-policy log lines, the zero delays, the lack of recovery after the API rollback, the relapse after the worker restart, the same-job before/after change at the flag flip, and recovery within two minutes of the flag flip.

What it does not explain: what caused the first failed attempts. attempt=1 result=retry means something failed before any retry. If that something was DB pool contention (L5 shows a 5 s wait starting at about 09:06:56, which is not clearly after L1-L4), then missing backoff amplified an existing problem instead of creating it, and the original trigger is still unidentified.

5. Contributing factors

Each is supported by evidence but is a contributor, not proven cause.

1. No fallback or validation for an empty policy (I1; facts 1 and 3).
2. Shared DB connection ceiling (300) with no isolation between API and workers, so worker trouble also hurt job creation and checkout (L7, T). A pool showing active=10 while the DB max is 300 suggests pool sizes multiply quickly with worker count: 40 workers x 10 would already be 400 if pools are per worker. [I, unverified: pool scope is not stated.]
3. Containment made things worse: scaling workers 40 to 90 accelerated queue growth. This was a reasonable step given the alerts, but no signal told responders that the database, not worker capacity, was the constraint. [I; process gap, not individual fault]
4. Detection covered symptoms only: alerts came from job age and checkout latency, 4 min after onset. The DB pool warning at 09:07:01 and the retry exhaustion did not alert (nothing in T says they did).
5. Flag change not correlated with the incident timeline: it took 32 min from alert to flag off, and the flag is not mentioned until 09:43. The timeline does not say when it was enabled. [I]
6. Two changes in one window (code deploy and flag exposure), and rollback of the API did not remove the flag-controlled behaviour on workers, which delayed diagnosis. [I]
7. Worker restart is not a useful remedy when the cause is configuration, and it appeared to work for 4 min (fact 6), which may have encouraged further restart attempts. [I]

6. Hypotheses, with evidence for and against

#	Hypothesis	Evidence for	Evidence against or weakening	Assessment
H1	Flag-enabled empty policy causes zero-backoff retry churn, which saturates the DB	L1-L4, L9 (empty policy, delay 0, 38-41 ms); L11-L12 (same job gets 1000 ms delay and succeeds after flag off); recovery within 2 min of flag off (T); restart relapse with flag on (L8-L9); US 2% exposure with no alert (T)	Only one worker and one job show the before/after change (n=1, sampled logs, no control); no log links retries to DB connections; CPU fell instead of rising; first-failure cause unknown; the US comparison is confounded by traffic, scale and tenant mix	Most likely
H2	The API release, independent of the flag, exhausted DB connections (leak, bigger pool, extra queries)	Onset 5 min after deploy; API rollback normalized job creation; api-03 out of slots (L7)	Backlog persisted after rollback; recovery followed a flag change, not a code change; retry logs are worker-side. The rollback may also just have restarted pods and freed connections	Partly retained for the API-side impact only; cannot be excluded
H3	Scale-up to 90 workers was itself the cause of exhaustion	Depth rose faster after 09:14; 300 reached at 09:20	Onset (09:07) and pool wait (09:07:01) precede scale-up by 7 min	Aggravator, not origin
H4	Independent eu-west infrastructure fault (DB, network, noisy neighbour)	DB pool waits and connection ceiling are infrastructure-flavoured; US quiet	Recovery followed the flag flip without infrastructure action; same-job change at flag flip. No infra event data supplied	Not excluded, lower likelihood
H5	Stuck or degraded worker processes	Backlog stuck after rollback	Restart gave only 4 min of normal work, then relapse (T, L8-L9)	Unlikely
H6	Recovery at 09:43 was coincidental or a delayed effect of the 09:24 rollback	Flag flip came 19 min after rollback; traffic could have changed	Worker-51 stalled at 09:35, after the rollback; recovery began within 2 min of flag off; job 319 changed behaviour within 3 s of the config change	Weakened, not eliminated (no control group)

Correlation cautions

• The flag flip and recovery are closely timed, which is strong temporal evidence, but there was no controlled test, and several changes happened in the same hour (deploy, scale-up, rollback, restart, flag off).
• "US had the flag on for only 2% and no alert" supports a dose-response story but does not prove it, because alert thresholds may simply not have been reached.
• Job 319 succeeding on attempt 2 after the flag change may also reflect the underlying transient error clearing (for example DB pressure easing). The 1000 ms delay proves the policy changed; it does not prove the policy was the only thing that changed.

7. What remains unknown

1. When the flag was enabled in eu-west and for what share of tenants (T is silent; deploy was 09:02, symptoms 09:07).
2. Why the policy was empty (tenant with no policy configured, blank default in new code, config parse failure) and how many tenants were affected.
3. What the first-attempt failures were (error type) and whether they were caused by DB pool waits.
4. Who held the 300 connections (API, workers, which pools, idle-in-transaction, lock waits), and whether retries acquire a connection per attempt.
5. What happens to a job after retry_budget_exhausted: re-queued, dead-lettered, or failed; and whether any side effects (charges, emails) ran more than once.
6. Whether workers run 2026.09.23.1 or only share the flag, and whether the release contained schema or data changes that affect rollback.
7. Whether the worker count is still at 90, and why the pool stalled 4 min after restart rather than immediately.
8. Whether the US 2% cohort experienced the same behaviour.
9. Whether the CPU drop reflects workers blocked on the DB (my reading) or something else.

8. Data-quality limits

• LOGS.txt has 12 lines from 4 hosts. It is clearly a sample, so absence of a pattern in it means nothing, and selection bias is possible.
• Both exhausted-retry lines have empty policy, but there is no exhausted-retry line with a non-empty policy to contrast against.
• Timeline entries have minute resolution; log entries have second resolution. Ordering within the same minute (for example L1-L5 versus the 09:07 queue-depth event) cannot be resolved from this data.

Task 09 — Incident analysis

Analyze TIMELINE.md and LOGS.txt. Produce:

• INCIDENT.md: impact, concise timeline, most likely root cause, contributing factors, evidence supporting or weakening each hypothesis, and what remains unknown.
• ACTIONS.md: immediate containment plus prioritized corrective actions with owners expressed as roles, verification signals, and rollback considerations.
• CUSTOMER_UPDATE.md: a plain-language update under 180 words. Do not claim certainty beyond the evidence or blame an individual.

Separate observed facts from inference. Avoid treating correlation as proof. Write RESPONSE.md with your confidence level and the next two pieces of data you would request. Work only in this directory.

Queue incident timeline (UTC)

• 09:02 deploy api 2026.09.23.1; change adds tenant-specific retry policy.
• 09:07 queue depth begins rising in eu-west; worker CPU falls from 65% to 18%.
• 09:11 alerts fire for job age and checkout latency.
• 09:14 on-call increases workers from 40 to 90; queue depth rises faster.
• 09:20 database connections reach configured maximum of 300.
• 09:24 API deploy rolled back; new job creation normalizes, backlog still stuck.
• 09:31 one worker pool restarted; it processes jobs normally for four minutes, then stalls.
• 09:43 feature flag tenant_retry_policy disabled; recovery begins within two minutes.
• 10:18 backlog cleared.
• US region uses the same release but had the flag enabled for only 2% of tenants; no alert fired there.

Response notes

Confidence

Medium-high, about 7/10, that the defect to fix is the tenant_retry_policy path: with the flag on, job 81 and job 319 exhausted their retry budgets in 41ms and 38ms under policy="" and, for job 81, delay_ms=0. With the flag off on the same worker, job 319 used delay_ms=1000, failed once, and succeeded. The API rollback did not clear the backlog, and a worker restarted after that rollback was still on the flag and hit the same pattern. The US copy of the release at 2% exposure did not alert.

Medium-low, about 4/10, that this path is also a full explanation of the CPU drop (65% to 18%) and of database connections hitting 300. Those facts are real, they start after the first bad retry line, and the queue grew faster after workers went from 40 to 90. The log excerpt has two exhausted jobs, no connection time series, and no count of how common policy="" was. A concurrent database problem can still sit underneath H2 in INCIDENT.md. I am not treating the 09:43 recovery correlation as proof that nothing else changed at that minute.

Overall confidence in the working theory (flag-gated zero-delay retries as the trigger, connection saturation as a later amplifier, scale-up as a contributing factor): medium-high. Enough to hold the flag off and to write P1 against the empty-policy path. Not enough to call the database chapter closed.

Next two pieces of data

1. eu-west worker retry outcomes from 09:00 to 10:30 UTC, at attempt grain or rolled up by minute, tenant, and worker. Fields: result, delay_ms, policy, elapsed_ms, error reason, and final disposition (completed, requeued, dead-lettered, or dropped). This shows whether empty policy and zero delay dominated the backlog or only the two jobs in the excerpt, why attempts were retrying, and whether work was lost. Job 319 was not lost; the rest are unknown. If almost no other job has policy="" or delay_ms=0, H1 stops being a regional root cause and becomes a rare bad path that needs a different explanation for the queue.

2. Database connections per minute from 09:00 to 10:30 UTC, split by workers, API, and other, plus the eu-west time and tenant percentage for tenant_retry_policy. This is the check on the amplifier. If worker connections climb only after 09:06 and jump again after the 09:14 scale-up, the retry path and the scale-up earn the connection chapter. If connections were already near 300 before 09:06, H2 moves up and the retry logs are no longer the start of the story. The flag percentage is the other half of the US contrast: the "2% did not alert" observation only supports a dose effect if eu-west exposure was actually much higher, and the sources do not say that it was.

Actions

The backlog was clear at 10:18 UTC and the flag was already off. Containment below is what keeps that mitigation in place. Corrective items are ordered by how directly they close the failure mode in the logs. Owners are roles.

Each item has a verification signal and a rollback. "Rollback" of containment does not mean re-enabling the flag. Re-enabling tenant_retry_policy is what recreates the condition that preceded the stall.

Immediate containment

C1. Leave tenant_retry_policy off in every region

Owner: feature-flag owner, confirmed by the incident commander.

eu-west recovery followed the disable. The US was on the same release at 2% and did not alert; that is not a reason to keep a 2% canary on an empty-policy path. Hold the US at off as well until P1 is verified.

Verify: flag evaluations are off for a sample of tenants in eu-west and the US; workers log feature_flag=false; no new retry_budget_exhausted line with policy="" and elapsed_ms under 100; queue depth and oldest job age stay at the pre-09:02 baseline for 60 minutes.

Rollback: do not turn the flag back on as a rollback. If a named tenant had an explicit non-default retry contract, keep serving config_retry_default (exponential) for that tenant until P1 ships. Record the tenant with customer communications. Turning the flag on again is a new ramp (see P3), not an undo.

C2. Do not add workers while database connections are near the cap

Owner: SRE on-call.

The 09:14 move from 40 to 90 workers was followed by faster queue growth, then by the 300-connection ceiling at 09:20. CPU had already fallen, so the queue was not waiting on idle CPU.

Verify: worker count is not raised while connections are at or near 300; remaining connection slots reserved stays at zero on API logs; pool-wait warnings (wait_ms at the 5000ms ceiling, idle=0) are not climbing.

Rollback: if oldest job age grows and connections have clear headroom (well under 300, API errors absent), add a small worker step and recheck connections before the next step. If connections climb or queue growth accelerates, return to the previous worker count. That scale-down is the rollback.

C3. Confirm the backlog drained and look for silent loss

Owner: queue engineering.

Job 319 survived inner-budget exhaustion and later succeeded. Other jobs may not have.

Verify: queue depth at baseline, oldest job age within the usual bound, and dead-letter or drop counts compared with the hour before 09:02. A higher dead-letter count is residual customer impact to list by tenant.

Rollback: this check is read-only. If a replay is started and queue age or connections move the wrong way, stop the replay. Replay is not part of containment.

Prioritized corrective actions

P1. Empty or missing tenant policy must use the exponential default

Owner: worker/queue engineering.

worker-51 loaded config_retry_default="exponential" with the flag on, then exhausted job 319 in 38ms under policy="". After the flag went off, the same job used delay_ms=1000 and succeeded on the next attempt. The safe behavior is the one observed with the flag off: a non-zero backoff, and the configured default when the tenant policy is empty. Also reject a computed delay of 0 when retry budget remains, unless a policy explicitly allows zero delay and that case is tested.

Verify: automated cases for flag on + empty policy, flag on + a real tenant policy, and flag off. In staging, a tenant with no policy logs a non-zero delay consistent with exponential backoff and completes. Canary one tenant that has a validated policy; compare retry_budget_exhausted rate and job age with a control tenant. Ship this behind the same flag, still defaulting off, and enable only after these checks.

Rollback: turn tenant_retry_policy off. That path is the one that recovered job 319. Keep the previous worker build deployable until the canary window passes. A bad custom policy for a pilot tenant is rolled back by removing that tenant from the flag, not by raising the regional worker count.

P1. One containment control for this change: the worker flag

Owner: release management, with worker engineering.

At 09:24 the API rollback normalized new job creation and left the backlog stuck. At 09:31:10 a new worker still had feature_flag=true. The runbook for 2026.09.23.1 and any retry-policy change should name a single first containment step: disable tenant_retry_policy, then confirm config_changed feature_flag=false on workers. API version rollback is a separate step and did not drain the queue.

Verify: a staging drill that rolls back the API build and shows the worker flag is unchanged until someone disables it. The on-call checklist lists the flag first. The drill record is attached to the release.

Rollback: runbook and checklist only. If a later design moves policy fully into the API artifact, update the runbook in the same change. Until then, the flag remains the proven switch.

P2. Alert and halt on the signature that showed up at 09:06:58

Owner: SRE, with worker engineering.

Job-age and checkout-latency alerts fired at 09:11, about four minutes after the first empty-policy exhaustion. Add a signal for retry_budget_exhausted where elapsed_ms is under 100 or policy is empty, and a signal for database connections approaching 300. A flag ramp halts when either fires.

Verify: in staging, force an empty policy and confirm the signal trips inside one evaluation interval and that a ramp would stop. Production dashboards show the new series. The ramp doc lists both halt conditions.

Rollback: if the halt blocks a legitimate rollout, disable the auto-halt and keep the page. Tune the threshold if it flaps on healthy exponential retries. Do not delete the signal to unblock a ramp.

P2. Cap worker database connections below the server maximum

Owner: database platform, with SRE.

The server maximum is 300 and it was reached. One worker already reported active=10 idle=0 and a 5s wait. API processes also need connections (api-03 could not get one). Set a fleet budget so worker pools plus API pools plus an admin reserve stay under the server maximum. Prefer a smaller per-worker pool or a shared worker cap over raising max_connections. Raising the ceiling without a budget repeats this failure at a higher number and puts more memory pressure on the database.

Verify: documented sum of pool ceilings is under 300, or a measured peak during a staging retry storm stays under the cap while an API job-create still acquires a connection. On the next ramp, connections remain under the cap and the reserved-slot error stays at zero.

Rollback: restore the previous pool sizes by config if database wait time regresses and the cap is clearly idle. Scale the cap one step at a time. A full return to uncapped pools is the rollback if the new cap blocks ordinary throughput with connections still well under 300.

P3. Log the retry reason, tenant, policy, and delay on every attempt

Owner: worker/queue engineering.

Attempt lines for jobs 81 and 319 say result=retry and do not say why. The policy field appears only on the exhaustion warning. Add reason, tenant, policy, and delay_ms to every retry and exhaustion line so the next incident can tell a missing policy from a downstream error.

Verify: a forced failure in staging emits all four fields. A query can group exhaustions by reason and tenant.

Rollback: logging only. If a metric label with tenant or reason cardinality is too expensive, drop that label from metrics and keep the fields in logs.

P3. Ramp by tenant percentage only after policies are non-empty

Owner: release management.

The US at 2% did not alert; eu-west with the flag on (percentage unstated) did. Next enablement steps through tenants that have a validated, non-empty policy, one region at a time, with the P2 halt conditions armed. Start at or below the US exposure that stayed quiet, and only after P1 is in the build.

Verify: the rollout plan lists steps, halt signals, and the policy pre-check. A drill shows the ramp stopping when the empty-policy signal fires. The first production step is a single-tenant canary with a known policy.

Rollback: set the percentage to zero (flag off). That is the containment already observed at 09:43. Partial rollback is "remove the last tenant step," then confirm exhaustion rate and job age.

P3. Change the scale-up rule when CPU falls and the queue rises

Owner: SRE.

At 09:07 CPU fell from 65% to 18% as the queue rose. Adding workers at 09:14 coincided with faster queue growth. The runbook should send that pattern to database pool waits and to the retry-exhaustion signal before any scale-up. An autoscaler, if one exists, should refuse to add workers when connections are near 300 or when CPU is falling while queue age is rising.

Verify: runbook updated and reviewed by on-call. If automation is changed, a staging case with a full connection cap does not add workers, and a case with high CPU plus queue growth still does.

Rollback: manual override so an operator can add workers when connections have headroom and the refusal is wrong. The override expires at the end of the incident. Restoring the old scaler is the rollback if the new guard blocks a known-good scale-up.

Suggested order

Hold C1–C3 now. Then P1 (fallback, and the runbook that points at the flag). Then P2 (detect the signature, and cap connections) before any new ramp. P3 (richer logs, canary plan, scale-up rule) can proceed in parallel once P1's test cases exist. Do not re-enable the flag to "confirm the theory" in production; use the staging cases in P1 and the two data pulls in RESPONSE.md.

From 09:07 to 10:18 UTC, background jobs in our eu-west environment fell behind. The queue grew, workers became much less busy, and the database hit its connection limit. An API server could not get an ordinary connection. Alerts started at 09:11 UTC. The queue was clear by 10:18 UTC.

A 09:02 UTC release added a per-customer retry policy. Logs show some jobs retrying with no pause and stopping in under a second. Other jobs still finished. After an API rollback, new job creation returned to normal and the backlog stayed. At 09:43 UTC we turned the new retry switch off. One job then waited about a second and completed, and the queue began to recover within two minutes.

Our US environment ran the same release with the switch on for about 2% of customers and raised no alert. We cannot yet say how many customers were affected, or whether any jobs were dropped or only delayed. That retry switch stays off while we find out. We are sorry for the delay.

eu-west queue incident

Scope of this note: TIMELINE.md and LOGS.txt only. Times are UTC. The log file is a short excerpt, so a missing line is weak evidence that an event did not happen.

Labels: Observed means the timeline or a log line states it. Inference means a reading that goes beyond those lines. Correlation is recorded as correlation.

Impact

Observed

• In eu-west, queue depth started rising at 09:07 and worker CPU fell from 65% to 18%. Alerts for job age and checkout latency fired at 09:11.
• Database connections reached the configured maximum of 300 at 09:20. At 09:19:52, api-03 logged db remaining connection slots reserved.
• The backlog stayed stuck after the 09:24 API rollback and was cleared at 10:18. Processing was degraded for about 71 minutes (09:07–10:18). Recovery is recorded as starting within two minutes of 09:43, so most of the remaining time was drain, not a continued stall.
• Impact was uneven at the start of the excerpt. At 09:06:58, job 81 (tenant=acme) exhausted its retry budget in 41ms. At 09:07:03, job 92 (tenant=beta) succeeded on attempt 1 in 830ms.
• The same release ran in the US with tenant_retry_policy enabled for 2% of tenants. No alert fired there.
• Job 319 exhausted its retry budget at 09:35:12 and the same job id succeeded at 09:43:10. For this job, inner-budget exhaustion left the job available for a later delivery.

Not established. No customer counts, error rates, revenue, data-loss totals, or dead-letter totals appear in the sources. The timeline does not define whether "checkout latency" is a customer action or a worker job checkout. Regional queue delay and database connection exhaustion are established; the customer-visible surface is not.

Timeline

Time	Observed event	Source
09:02	API deploy 2026.09.23.1. The change adds a tenant-specific retry policy.	Timeline
09:06:58	worker-17, job 81, tenant=acme: attempts 1–3 are result=retry with delay_ms=0, then retry_budget_exhausted with policy="" and elapsed_ms=41.	Logs
09:07	eu-west queue depth starts rising. Worker CPU falls from 65% to 18%.	Timeline
09:07:01	worker-22: db pool wait exceeded wait_ms=5000 active=10 idle=0.	Logs
09:07:03	worker-08, job 92, tenant=beta: attempt 1 result=success elapsed_ms=830.	Logs
09:11	Alerts fire for job age and checkout latency.	Timeline
09:14	On-call raises workers from 40 to 90. Queue depth rises faster after that.	Timeline
09:19:52	api-03: db remaining connection slots reserved.	Logs
09:20	Database connections reach the configured maximum of 300.	Timeline
09:24	API deploy rolled back. New job creation normalizes. Backlog still stuck.	Timeline
09:31	One worker pool restarted. It processes jobs normally for four minutes, then stalls.	Timeline
09:31:10	worker-51 startup: config_retry_default="exponential" feature_flag=true.	Logs
09:35:12	worker-51, job 319: retry_budget_exhausted policy="" elapsed_ms=38. This is 4m02s after that worker's startup line.	Logs
09:43	Feature flag tenant_retry_policy disabled. Recovery begins within two minutes.	Timeline
09:43:07	worker-51: config_changed feature_flag=false.	Logs
09:43:09	worker-51, job 319: attempt 1 result=retry delay_ms=1000.	Logs
09:43:10	worker-51, job 319: attempt 2 result=success elapsed_ms=190.	Logs
10:18	Backlog cleared.	Timeline
(same release)	US flag coverage 2% of tenants. No alert there.	Timeline

The "four healthy minutes" for the restarted pool are a timeline statement. The excerpt has no successful job line from worker-51 between 09:31:10 and 09:35:12.

Most likely root cause

Inference, working theory. With tenant_retry_policy on, some jobs were retried under an empty policy (policy="") and delay_ms=0. Three attempts and budget exhaustion finished in 41ms (job 81) and 38ms (job 319). A retryable failure that clears only after a short wait would burn the whole budget before a later attempt can succeed.

The flag is the control that matches the only before/after behavior change in the logs. On worker-51, feature_flag=true at startup, then the empty-policy exhaustion at 09:35:12. At 09:43:07 the flag is false. Two seconds later job 319 waits delay_ms=1000, still fails once, and succeeds on attempt 2 in 190ms. The startup line shows config_retry_default="exponential" was already loaded while the flag was true, so the default existed and was not what those exhausted attempts used.

This is the best fit. It is not proof. The excerpt shows two exhausted jobs, and no line shows a non-empty policy value. The policy field appears only on those two warnings.

Why this is ahead of the alternatives. The API rollback at 09:24 normalized new job creation and left the backlog stuck. worker-51 still started with feature_flag=true at 09:31:10, after that rollback, and then hit the same exhaustion pattern. The backlog recovery recorded in the timeline follows the flag change, not the rollback and not the worker restart. The US ran the same release at 2% flag exposure and did not alert, which fits a flag-gated failure better than a release-wide failure. That regional contrast is suggestive only: the eu-west exposure percentage is not in the sources.

Hypotheses

H1. Empty tenant policy and zero delay under the flag stalled job progress

Support (observed).

• Both budget-exhaustion lines have policy="" and sub-50ms elapsed time. The only attempt lines tied to job 81 use delay_ms=0 and share one timestamp.
• Job 319's post-disable attempts use delay_ms=1000, about one second apart, and the second attempt succeeds. The first attempt is still result=retry, so the job had a real retryable failure and completed once a non-zero delay was used.
• worker-51 had the exponential default configured and the flag on, then exhibited the empty-policy exhaustion. After feature_flag=false on that same worker, the delay changed.
• The failure appears on two workers (17 and 51), before and after a pool restart, so it is not one crashed process.
• tenant=acme hit the fast-fail path; tenant=beta succeeded in the same minute. That is consistent with a tenant-specific policy, and it is also consistent with two unrelated outcomes. One success and one failure cannot size the tenant split.

Weaken.

• Two jobs are a small sample. They may be the only jobs in this state, or the excerpt may have kept the interesting lines.
• Nothing in the logs says why an attempt's result was retry. The zero delay may have amplified a separate fast error (lock, conflict, dependency), or the empty policy may itself be what the retry layer treats as a retry.
• policy="" is logged only on the exhaustion warning. A logger that prints an empty field on every exhaustion would look the same. The delay change at the flag flip is the stronger observation; the empty string is the accompanying field.
• The restarted pool is described as healthy for four minutes while the flag was already true. A defect that breaks every flagged job immediately does not explain four healthy minutes. A defect that affects some tenants, or that appears once a particular job is dequeued, does.
• Recovery "within two minutes" of the flag change is a timeline correlation. The sources do not list every change made at 09:43.

Status. Leading hypothesis. Fix and containment should assume this path is unsafe until the two data gaps in RESPONSE.md are closed.

H2. The database connection ceiling was the primary fault, and the retry lines are fallout

Support (observed).

• worker-22 blocked for 5000ms with active=10 idle=0 at 09:07:01, three seconds after job 81's exhaustion.
• api-03 reported reserved remaining connection slots at 09:19:52, and connections hit the configured maximum of 300 at 09:20.
• Worker CPU fell while the queue rose, which fits workers blocked on a pool wait better than a CPU-bound spin. Job 81 itself finished in 41ms, so that particular job was not the 5s wait; the wait is a second worker.
• Raising workers from 40 to 90 was followed by faster queue growth, then by the connection ceiling. Adding consumers made the backlog worse, which is what a saturated shared limit tends to do.

Weaken.

• Job 92 succeeded at 09:07:03, two seconds after the pool-wait warning, in 830ms. The database was still completing work.
• Job 81's three retries were already delay_ms=0 with an empty policy before the pool-wait line. A full database does not, by itself, set policy="" or delay_ms=0.
• The recorded recovery action that lines up with jobs succeeding is the flag change. No database failover, pool resize, or connection kill is in the timeline. Job 319 then completed in 190ms, so the database accepted that work immediately after the flag flip. Connections may have drained because workers stopped the bad path; the sources do not show a connection graph, so that drain is an inference.
• The global ceiling is a late event (09:20), thirteen minutes after the first empty-policy exhaustion and after the scale-up. It explains a widened blast radius better than the first minute of the incident.

Status. Real multiplier, weak as the originating cause. Early local pool waits and the later server-wide ceiling belong in the narrative as observed damage and as a reason the scale-up backfired.

H3. The API deploy broke processing independent of the feature flag

Support (observed).

• The queue moved at 09:07, five minutes after API deploy 2026.09.23.1.
• After the 09:24 rollback, new job creation normalized.

Weaken.

• The backlog stayed stuck while worker-51 was still on feature_flag=true.
• The same release in the US at 2% flag exposure did not alert.
• The behavior change in the log follows config_changed feature_flag=false, after the rollback had already happened.

Status. The deploy is the change that introduced tenant-specific retry. The rollback improved job creation and did not restore the backlog. Treat "roll back the API artifact" and "turn off tenant_retry_policy" as different actions. Only the second one lines up with processing recovering.

H4. Raising workers from 40 to 90 caused the incident

Support (observed). Queue depth was already rising at 09:07. After the 09:14 scale-up it rose faster, and connections hit 300 at 09:20.

Weaken. The scale-up is a response, six minutes after the queue started rising and after the empty-policy log line. It cannot be the start of the incident.

Status. Contributing factor. More workers can hold more database connections. Inference: if a busy worker holds on the order of the 10 active connections seen on worker-22, a jump from 40 to 90 workers can press a 300-connection server cap even when 40 workers had been under it. The sources do not state the pool maximum, so the arithmetic is conditional.

H5. One bad tenant (acme) wedged the region

Support (observed). The only named tenant on an exhaustion line is acme. beta succeeded in the same minute.

Weaken. Job 319 has the same policy="" exhaustion and, in this excerpt, no tenant field. That same job succeeded after the flag flipped, which a permanently bad tenant payload would not do by a flag change alone. Connection limits and a regional queue are broader than one tenant. The US contrast tracks flag exposure, which is a different axis than a single account.

Status. Possible that tenants with a missing policy were the ones harmed. Not supported as "acme alone wedged the fleet." How many tenants had an empty policy is unknown.

Contributing factors

Observed

• Worker count went from 40 to 90 while the queue was already growing. Depth then grew faster, and database connections reached 300.
• The API rollback and the worker flag were independent. After the rollback, a restarted worker still logged feature_flag=true and later exhausted a retry budget in 38ms.
• The configured database maximum (300) was actually reached, and an API process logged that remaining slots were reserved. Worker trouble and API trouble met at that limit.
• Inner retry budget can be spent in under 50ms when delay_ms=0. Job 319 shows the other side: a 1000ms delay was enough for a later attempt to succeed.

Inference

• An empty tenant policy did not fall back to config_retry_default="exponential". The default was loaded on the worker that then used delay_ms=0.
• Zero delay removes the wait that job 319 needed between a failed attempt and a successful one.
• There was no effective guardrail in the recorded response that stopped the flag while policy="" and sub-50ms exhaustions were happening. The successful containment was manual and came at 09:43, about 36 minutes after the first such log line.
• eu-west flag exposure was high enough to stall the region, given that disabling it preceded recovery and that 2% US exposure did not alert. The eu-west percentage itself is not written down.

Unknown

• eu-west flag percentage, and the time the flag was turned on relative to the 09:02 deploy.
• How many jobs and tenants logged policy="" or delay_ms=0, and the error reason behind result=retry.
• Whether policy="" means a missing tenant record, a serialization bug, or a log field that is unpopulated on the exhaustion path.
• Disposition of jobs other than 319: completed later, requeued, dead-lettered, or dropped. Job 319 survived to a later success. That does not generalize.
• What "new job creation normalizes" means for processing. Creation returning to a normal rate can coexist with workers still failing to drain.
• Why the restarted pool was healthy for about four minutes. Tenant mix, queue order, and pool warmup are all open.
• Connection counts split by workers versus API, minute by minute. Without that, H2 stays a plausible amplifier rather than a measured one.
• Whether any database operation (kill, failover, setting change) happened and was left out of the timeline.
• US job-level failure rate. No alert is not the same as no empty-policy retries.
• The customer-facing meaning of the checkout-latency alert, and any SLO or loss numbers.
• Whether another change landed at 09:43 together with the flag.

Fact and inference, separated

Safe to treat as fact: the deploy time and version, the CPU and queue description, the alert time, the worker scale-up and faster queue growth, the 300-connection ceiling, the API slot error, the rollback's failure to unstick the backlog, the flag disable time, recovery starting within two minutes of that disable, backlog clear at 10:18, US 2% exposure with no alert, and every field in the log lines above.

Working theory, not fact: empty policy under the flag skipped the exponential default, zero delay exhausted a budget that a 1s backoff could survive, that path was common enough to stall eu-west, and the resulting churn plus the scale-up drove connections to 300. The flag-off success on job 319 is the strongest single piece of evidence, and it is still one job plus a timeline correlation.

Task 09 — Incident analysis

Analyze TIMELINE.md and LOGS.txt. Produce:

• INCIDENT.md: impact, concise timeline, most likely root cause, contributing factors, evidence supporting or weakening each hypothesis, and what remains unknown.
• ACTIONS.md: immediate containment plus prioritized corrective actions with owners expressed as roles, verification signals, and rollback considerations.
• CUSTOMER_UPDATE.md: a plain-language update under 180 words. Do not claim certainty beyond the evidence or blame an individual.

Separate observed facts from inference. Avoid treating correlation as proof. Write RESPONSE.md with your confidence level and the next two pieces of data you would request. Work only in this directory.

Queue incident timeline (UTC)

• 09:02 deploy api 2026.09.23.1; change adds tenant-specific retry policy.
• 09:07 queue depth begins rising in eu-west; worker CPU falls from 65% to 18%.
• 09:11 alerts fire for job age and checkout latency.
• 09:14 on-call increases workers from 40 to 90; queue depth rises faster.
• 09:20 database connections reach configured maximum of 300.
• 09:24 API deploy rolled back; new job creation normalizes, backlog still stuck.
• 09:31 one worker pool restarted; it processes jobs normally for four minutes, then stalls.
• 09:43 feature flag tenant_retry_policy disabled; recovery begins within two minutes.
• 10:18 backlog cleared.
• US region uses the same release but had the flag enabled for only 2% of tenants; no alert fired there.

Confidence and next evidence

Overall confidence: moderate that the enabled tenant retry-policy path was the primary incident trigger and that DB contention amplified the impact. Zero-delay retries and empty-policy budget exhaustion, followed by delayed retry/success and regional recovery after flag disable, support that assessment. API rollback alone did not clear the stall.

High confidence in the events explicitly recorded in the supplied timeline/logs, including DB pressure and backlog clearance; these are sparse records, not an independent audit. Low confidence in the precise implementation defect, the direction of causality between retries and DB pressure, the reason for the four-minute restart reprieve, and the extent of customer impact. The evidence does not establish a connection leak, an infinite retry loop, data loss, or fully unaffected US traffic.

The next two pieces of data I would request, in order:

1. A correlated retry/queue/DB trace and metrics window covering 08:55–10:30 UTC on the incident day. Include original retry errors, effective policy and delay, attempts/budget/requeue decisions, connection acquisition and release, DB session owners/query and lock waits, arrivals/completions, and oldest-job age. Prioritize jobs 81 and 319 and an unaffected comparison such as job 92, with tenant/flag/region context and final outcomes. This would distinguish a DB-first fault from retry-driven pressure, test resource retention, and quantify whether queue recovery represented successful processing. If that historical detail was not retained, state the gap rather than inferring it from aggregate counts.
2. The retry-policy code/configuration change and effective rollout history. Include release diffs, tenant policy values and validation/fallback semantics, flag exposure by region, worker/API versions, and effective configuration around the 09:24 rollback, 09:31 restart, and 09:43 flag disable. This would show what policy="" means, whether the exponential default was bypassed, whether rollback left workers on the suspect path, and whether regional exposure explains the different signals.

All four requested deliverables are provided: INCIDENT.md, ACTIONS.md, CUSTOMER_UPDATE.md, and RESPONSE.md. This analysis proposes operational actions; it does not claim they were executed or that missing evidence was collected.

Containment and corrective actions

These are recommendations based on the supplied records, not actions executed during this analysis. The record already shows API rollback at 09:24, flag disable at 09:43, and backlog clearance at 10:18. Confirm present state before making further changes. Priority order is P0, P1, then P2; role owners below are proposed assignments.

Immediate containment — P0

Action	Owner role	Verification signal	Rollback / operational considerations
Keep tenant_retry_policy disabled pending validation; verify effective state in API and workers across all regions, including US, and record the deployed versions.	Incident commander with application on-call	Every participating process reports flag off; effective retry delays follow the known-good policy. Attempts per completed job, oldest-job age, checkout latency, and DB waits return toward the pre-incident baseline.	Do not infer worker state from an API rollback. Keep an independently usable flag kill switch; re-enable only through the gated rollout below.
Halt further scale-up. Set a worker concurrency/pool budget that leaves DB headroom for API and other clients; reduce workers gradually from 90 if still at that level and supported by measurements. Throttle retry concurrency or intake if necessary.	Queue platform on-call with database on-call	DB has reserved headroom, pool waits and connection errors decline, and completions exceed arrivals while a backlog exists. Track oldest-job age to ensure throttling does not starve old work.	The earlier 40-worker count is a reference, not a proven safe target. Change one control at a time. Revert the last reduction if useful throughput falls without relieving DB pressure, while retaining the validated connection cap. Do not raise the DB limit without capacity evidence.
Preserve available logs/config history and inspect DB sessions, lock waits, and connection ownership. Use targeted draining/restarts only if stuck resources persist after flag disable; reconcile failed/exhausted jobs before any replay.	Database on-call with worker service on-call	Specific blocked or retained resources are identified and released; healthy processing persists beyond the observed four-minute restart reprieve. Job reconciliation finds each affected job in an accounted-for state.	Avoid blanket session termination or repeated restarts. Respect queue leases and in-flight transactions; replay only with verified idempotency/deduplication. Stop targeted intervention if duplicate processing or error rates increase.
Confirm recovery and customer scope from job outcomes and service metrics. Proposed exit gate: at least 30 continuous minutes at pre-incident operating ranges for queue age, latency, DB waits, and retry rate, with all affected jobs accounted for.	Incident commander with observability and support leads	Backlog stays controlled after clearance; affected-job reconciliation and checkout outcome counts are recorded; US telemetry is checked despite no alert.	This is a proposed gate, not evidence that it was met. If metrics regress, retain containment and resume incident response. Customer communications must distinguish clearance from verified job correctness.

Prioritized corrective work

Priority and action	Owner role	Verification signal / acceptance criterion	Rollback considerations
P1 — Establish the failure mechanism before finalizing the fix. Correlate effective policies, original retry errors, attempts, and DB acquisition/release behavior for affected and unaffected jobs. Review policy/config diffs and rollback scope.	Worker service lead with database engineering	A reproducible case or correlated production trace explains policy selection, retry timing, and connection behavior. Record whether the DB was the initiating fault or a downstream bottleneck; revise the incident hypothesis if evidence contradicts it.	Preserve original evidence. Use isolated replay with sanitized inputs; do not enable the suspect path broadly to reproduce it.
P1 — Make retry policy selection and failure handling safe. Validate empty, missing, invalid, and overridden policies; choose an explicit documented safe fallback or rejection path. Bound attempts and total retry time; use backoff/jitter for transient failures and release resources before waiting.	Retry-policy/service engineering	Regression tests cover each invalid/config transition, an unavailable DB, budget exhaustion and requeue behavior. A representative load test shows bounded attempts, correct terminal handling, and no connection retention during backoff. Verify fallback behavior at startup and after reload.	Ship behind the existing disabled flag. If a canary regresses, disable it and return to the verified policy. Audit jobs that reached retry limits before replaying them.
P1 — Bound aggregate DB demand and make overload controls safe. Budget connection pools across APIs/workers/other clients and cap worker concurrency. Make scaling decisions account for DB waits, throughput, and headroom as well as queue depth.	Platform engineering with database engineering	Capacity/load tests at the former 40- and 90-worker scales demonstrate bounded total demand with explicit DB reserve; under DB slowdown, backpressure activates without runaway queue retries or API starvation.	Introduce limits incrementally; monitor throughput and oldest-job age. Restore a previous validated limit if needed while preserving aggregate headroom.
P1 — Verify configuration propagation and gate re-enablement. Log effective policy provenance/version, test API rollback with active workers, and cover worker startup, config refresh, and flag disable. Canary by tenant and region under representative load.	Release engineering with worker service engineering	All workers converge to flag changes; integration tests prove rollback/disable behavior. Canary includes transient DB failures and sustained load for at least the proposed 30-minute observation gate, exceeds the four-minute reprieve, and spans a full config-refresh cycle. Stop on abnormal retry rates, waits, job age, or checkout latency.	Keep flag disable independent of binary deployment; retain known-good binaries/config. Disable on any gate failure and verify propagation before expanding exposure.
P2 — Add diagnostic metrics and an evidence-based runbook. Measure attempts per completed job, selected policy/fallbacks, enqueue/completion rates, oldest-job age, pool waits, and DB occupancy by service and region. Correlate tenant/flag state through sampled logs.	Observability engineering with SRE	A fault-injection exercise detects zero-delay retry pressure and DB contention, identifies exposed cohorts, and guides responders to safe concurrency/flag controls. Compare regions with load and exposure context.	Stage alert thresholds to avoid noise; roll back noisy alert rules without removing underlying measurements. Control tenant-label cardinality and sensitive log content.

P1 behavior fixes and re-enablement depend on the mechanism investigation. The capacity and visibility work can proceed independently using the confirmed DB-pressure evidence. Final root-cause closure requires evidence for the mechanism; backlog clearance alone is insufficient.

Between 09:07 and 10:18 UTC, our eu-west region experienced a processing backlog. Monitoring also detected increased checkout latency. Recovery began within two minutes after we disabled a recently introduced retry-policy feature at 09:43 UTC, and the backlog cleared at 10:18 UTC.

Our current assessment is that the feature likely contributed to repeated job attempts and pressure on database connections. We have not yet confirmed the exact cause or the full number of customers affected. The US region did not trigger an alert, but that alone does not rule out impact there.

Our next steps are to verify affected job outcomes, confirm sustained recovery, and validate safeguards for retries and database capacity before re-enabling the feature. The available records do not establish whether any jobs were lost or processed more than once. We’re sorry for the disruption and will share further findings once verified.

Queue incident analysis

Scope and impact — observed facts

Sources: TIMELINE.md and LOGS.txt only. Times are UTC; the incident date is not explicitly stated. The release identifier is not sufficient to establish that date.

• In eu-west, queue depth began rising at 09:07 and the backlog cleared at 10:18: approximately 71 minutes from the first reported queue degradation to clearance, not necessarily 71 minutes of uniform customer impact.
• Job-age and checkout-latency alerts fired at 09:11. This establishes delayed processing and a checkout-latency signal; affected customer counts, actual latency, failed checkout counts, and financial impact are unavailable.
• Worker CPU fell from 65% to 18%. Database connections reached the configured maximum of 300 at 09:20; an API connection-allocation error was logged at 09:19:52.
• Impact was not universal: job 92 for tenant beta succeeded at 09:07:03. The US region ran the same release with the flag enabled for 2% of tenants and had no alert. Absence of an alert does not establish absence of degradation.
• The record does not establish lost jobs, duplicate processing, data corruption, or their absence. Backlog clearance alone does not verify correct completion of every job.

Concise timeline — observed facts

Time (UTC)	Recorded event
09:02	API release 2026.09.23.1 deployed, adding a tenant-specific retry policy.
09:06:58	Worker 17 logged three retries for acme job 81 with delay_ms=0, then budget exhaustion with policy="" and elapsed_ms=41.
09:07–09:11	eu-west queue depth rose and worker CPU fell. At 09:07:01, worker 22 reported a 5,000 ms DB pool wait with 10 active and zero idle connections. Job-age and checkout-latency alerts fired at 09:11.
09:14	Workers increased from 40 to 90; queue depth subsequently rose faster.
09:19:52–09:20	API logged unavailable DB connection slots; DB connections reached the configured maximum of 300.
09:24	API deployment rolled back. New job creation normalized, but the backlog remained stuck.
09:31–09:35	One worker pool restarted and processed jobs normally for four minutes, then stalled. Worker 51 started with default exponential and flag enabled; at 09:35:12 it logged budget exhaustion for job 319 with policy="" and elapsed_ms=38.
09:43–09:45	Retry-policy flag disabled at 09:43; recovery began within two minutes. Worker 51 saw feature_flag=false at 09:43:07; job 319 then retried with a 1,000 ms delay and succeeded at 09:43:10.
10:18	Backlog cleared.

Most likely root cause — inference, moderate confidence

The leading explanation is a defect in the flag-controlled tenant retry-policy path, potentially involving policy resolution or fallback when a policy is empty. That path appears associated with immediate retries and rapid budget exhaustion. Retry pressure likely contributed to DB contention and stalled queue processing; increasing worker concurrency may then have amplified contention.

This is a causal hypothesis, not an established implementation diagnosis. The evidence does not show whether an empty policy actually selects zero delay, whether it is merely a logging artifact, whether retries retain connections, or which failures initiated the retries. Three retries followed by budget exhaustion demonstrate a bounded sequence in the sample, not an infinite retry loop or a measured system-wide retry storm.

The flag intervention is more diagnostic than deployment timing alone: one worker logged a flag change followed by a delayed retry and success for the same job that had exhausted its budget earlier, and regional recovery followed. It is still not a controlled experiment; unrecorded changes or changing load could contribute. Rolling back the API did not establish that the worker-side policy path or its configuration had been reverted.

Competing hypotheses and evidence

Hypothesis	Supporting observations	Evidence that weakens or limits it	Assessment
Tenant retry-policy resolution/fallback defect caused excessive immediate retries and downstream contention.	Feature introduced shortly before degradation; job 81 has zero-delay retries and an empty policy; job 319 repeats empty-policy budget exhaustion after restart despite an exponential startup default. Flag disable is followed by a 1,000 ms retry, success, and recovery. Lower US flag exposure coincides with no alert.	No code, effective tenant settings, underlying retry errors, aggregate retry rate, or connection ownership traces. US traffic and eu-west exposure are unknown. Empty-policy logging alone does not prove bad execution.	Leading trigger; moderate confidence. Exact code defect and resource mechanism remain unproven.
Primary DB slowdown, lock contention, or connection leak caused retries and queue stalls.	Pool waits, low worker CPU, connection exhaustion, and temporary benefit from restart fit resource contention.	Immediate retries are logged before the first supplied pool-wait warning; recovery tracks flag disable. These weaken a DB-only explanation but cannot exclude earlier unlogged DB trouble. No query, lock, session-age, or DB resource data.	Plausible initial cause or co-cause; insufficient evidence to establish a leak or lock fault.
Worker scale-up independently caused connection exhaustion.	Raising workers from 40 to 90 preceded the global connection maximum; queue growth accelerated afterward.	Queue degradation and a local pool wait preceded scale-up. Only one pool's active count is supplied, so total demand cannot be calculated from the logs.	Cannot explain onset; likely amplifier if additional workers opened or competed for connections.
API release produced excess or malformed jobs, leaving a persistent backlog.	Release preceded onset; new job creation normalized after API rollback.	Backlog remained stuck after rollback; restarting a worker pool helped only briefly; flag disable preceded sustained recovery. No enqueue-rate or payload samples show excess or malformed jobs.	Possible upstream contributor, but an API-only explanation is incomplete.
Tenant-specific workload or a problematic job was the primary trigger.	acme job 81 retried while beta job 92 succeeded; job 319 later exhausted its budget.	Tiny sample; tenant for job 319 and per-tenant exposure/load are unknown. Job 319 succeeded after flag disable, which points toward an interaction with retry behavior.	Plausible reason retries began or impact varied; unsupported as a standalone root cause.

Contributing factors — inference unless stated otherwise

• Concurrency under DB pressure: scale-up occurred after a pool-wait warning and before global exhaustion (observed). Additional connection demand likely reduced headroom, but per-process pool sizes and actual connections are missing.
• Uneven mitigation coverage: API rollback normalized creation without clearing the stall; a worker subsequently reported the flag enabled (observed). The worker policy path likely remained active, but worker versions and the rollback's exact scope are unknown.
• Policy/default inconsistency: an exponential startup default and a later empty logged policy coexist (observed). A tenant override, dynamic config refresh, or fallback defect could explain this; none is confirmed.
• Retry pressure with limited progress: immediate retries, budget exhaustion, and DB waits coexist (observed). They may form a feedback loop that reduces useful throughput. The four-minute restart reprieve is compatible with resource accumulation or repeated workload exposure, not proof of either.
• Detection and rollout visibility: supplied alerts cover job age and checkout latency. The record does not establish whether retry-rate, config-validation, or DB-headroom alerts existed or whether this rollout was canaried.

Remaining unknowns

• Exact policy implementation, meaning of policy="", tenant overrides, flag evaluation history, worker versions, and configuration propagation across rollback/restart.
• Original retryable errors; aggregate attempts per job; delay distributions; requeue, budget-reset, and terminal-failure behavior; whether retries hold DB connections.
• DB connection ownership and limits by service, pool sizes, transaction/query duration, locks, resource pressure, and possible leaks before onset.
• Arrival versus completion rates, actual queue/job-age and checkout-latency values, eu-west flag exposure, and comparable US workload and alert sensitivity.
• Affected tenants/jobs/checkouts; final job outcomes, duplicates, data integrity, and whether service remained stable after 10:18.
• Any concurrent infrastructure changes or interventions omitted from the supplied timeline.

Task 09 — Incident analysis

Analyze TIMELINE.md and LOGS.txt. Produce:

• INCIDENT.md: impact, concise timeline, most likely root cause, contributing factors, evidence supporting or weakening each hypothesis, and what remains unknown.
• ACTIONS.md: immediate containment plus prioritized corrective actions with owners expressed as roles, verification signals, and rollback considerations.
• CUSTOMER_UPDATE.md: a plain-language update under 180 words. Do not claim certainty beyond the evidence or blame an individual.

Separate observed facts from inference. Avoid treating correlation as proof. Write RESPONSE.md with your confidence level and the next two pieces of data you would request. Work only in this directory.

Queue incident timeline (UTC)

• 09:02 deploy api 2026.09.23.1; change adds tenant-specific retry policy.
• 09:07 queue depth begins rising in eu-west; worker CPU falls from 65% to 18%.
• 09:11 alerts fire for job age and checkout latency.
• 09:14 on-call increases workers from 40 to 90; queue depth rises faster.
• 09:20 database connections reach configured maximum of 300.
• 09:24 API deploy rolled back; new job creation normalizes, backlog still stuck.
• 09:31 one worker pool restarted; it processes jobs normally for four minutes, then stalls.
• 09:43 feature flag tenant_retry_policy disabled; recovery begins within two minutes.
• 10:18 backlog cleared.
• US region uses the same release but had the flag enabled for only 2% of tenants; no alert fired there.

Confidence and next evidence

Confidence: moderate that the flagged tenant retry path contributed to queue stalls through retry amplification and database contention. Confidence is lower in the specific explanation that an empty tenant policy bypassed exponential fallback, and in whether connection exhaustion was caused by increased attempt volume, resource retention, or an independent database problem. The temporal sequence, repeated blank-policy failures, flag-disable recovery and regional exposure difference support the leading hypothesis; they do not prove its implementation or exclude other causes. Confidence is high only in the events explicitly recorded in the supplied timeline and sampled logs, subject to their completeness and timestamp accuracy.

The next two pieces of data I would request are:

1. An effective-policy and execution trace for jobs 81 and 319. Include the original retryable errors, policy resolution inputs/result, per-attempt delay and resource lifetime, corresponding code/build identifiers for API and workers, and timestamped flag/configuration changes for their tenants. Include the mapping of the logged blank policy to the actual internal value. This would distinguish faulty fallback from a logging artifact, identify the initiating failure, and explain why rollback/restart did not remove the behavior while flag disable did. If historical traces are unavailable, reproduce these configurations in an isolated environment and label that evidence separately.
2. A time-aligned database and workload record from 08:55–10:20 UTC. Request database session ownership/state, pool waits, locks/query latency and connection counts alongside new-job arrivals, retry attempts, completions, queue age and checkout latency/error rates; break down by region and service, and by tenant where retained. Align worker-count and intervention timestamps. This would test whether retries preceded and amplified saturation, distinguish load from held/leaked connections or blocking, compare regional exposure, and quantify impact. Historical gaps should remain explicit rather than being filled with current measurements.

All four deliverables were prepared from local TIMELINE.md and LOGS.txt. Proposed operational actions have not been executed as part of this analysis.

Containment and corrective actions

These are recommended actions, not claims that additional operational changes have been executed. The supplied timeline already records API rollback, flag disable, recovery, and backlog clearance. Role owners below are proposed assignments. P0 actions come first; P1 actions are required before broad re-enablement; P2 actions follow stabilization.

Immediate containment

Priority / action	Owner role	Verification signals	Rollback / tradeoffs
P0 — Keep tenant_retry_policy disabled and pause its rollout. Verify effective flag state and fallback policy on producers and workers in both regions, including cached or long-lived processes.	Incident commander + service on-call	Effective flag is false on sampled instances and affected tenants; observed retry delays follow the fallback; rapid retry bursts disappear. Confirm queue age and checkout health as well as flag state.	Preserve the prior flag/configuration snapshot. Re-enablement requires a validated fix and canary; if degradation returns, disable immediately. If flag refresh is unsupported, drain and replace instances in small batches while watching capacity.
P0 — Stop adding workers until database headroom is understood. Establish a fleet connection budget below 300 with an explicit reserve for API and operational access. If contention persists, gradually reduce worker concurrency or extra workers while measuring throughput; use 40 workers only as a comparison point, not a guaranteed safe setting.	SRE on-call + database on-call	Global connection count has agreed reserve; pool waits decline; completion rate exceeds total arrivals, including retries, while draining; oldest job age falls and checkout latency/errors return to baseline.	Reverse the last reduction if queue age worsens without a connection-health benefit. Drain workers gracefully; abrupt termination may cause retries or duplicate side effects. Increasing the database limit requires separate capacity evidence.
P0 — If rapid retries persist, use supported controls to bound retry attempts, elapsed time and concurrent work per tenant, with delayed backoff. Isolate repeatedly failing jobs in a recoverable holding/dead-letter queue if available.	Queue/service on-call	Attempts and retry arrival rate fall within configured budgets; healthy tenants keep completing; held-job counts and reasons are visible.	Snapshot controls and retain job identifiers/payloads. Restore settings gradually if healthy work is suppressed. Replay held jobs with idempotency checks after repair; do not delete the backlog or retry every held job at once.
P0 — Preserve logs, configuration/flag history and deployment identifiers; reconcile job and checkout outcomes before declaring closure. Restart only targeted instances that cannot refresh configuration, after evidence capture where practical.	Incident commander + service on-call	Backlog clearance is corroborated by queue age, sustained throughput and checkout health; sampled/reconciled outcomes identify any lost or duplicate work; records are retained for the two requests in RESPONSE.md.	Reconciliation is read-only initially. Any replay or compensation needs a checked list and idempotency protections. Repeated fleet restarts risk obscuring evidence and recreating load; transient restart success alone is not recovery.

Prioritized corrective actions

Priority / action	Owner role	Verification signals	Rollback / tradeoffs
P1 — Trace the effective retry policy and original errors for jobs 81 and 319. Reproduce the flagged path and confirm whether blank/invalid policies affect delay, attempts or resource lifetime. Inspect database evidence in parallel with this investigation.	Service engineering + database engineering	Reproduction identifies the responsible behavior or rules out empty-policy fallback; request/attempt traces explain resource use and the initiating error. Findings distinguish retry amplification from independent database blocking.	Investigation should use retained evidence and an isolated environment. Stop any production diagnostic that adds material load. Avoid deploying a guessed fix solely because policy="" appears in logs.
P1 — Fix the confirmed policy-resolution defect, if found. Validate missing, empty, malformed and valid tenant policies; apply a documented safe default and bounded delayed retries. Emit effective policy and source clearly enough to diagnose fallback.	Service engineering	Regression cases cover flag on/off, default selection, budget exhaustion and recurrent failures. Instrumented failing jobs produce bounded attempts and delays with no uncontrolled immediate retry loop. Valid tenant overrides retain intended behavior.	Keep the flag disabled until validation passes. Retain the previous release and flag snapshot; disable the feature first if regression appears, then roll back the relevant producer/worker build while verifying effective configuration.
P1 — Bound aggregate database demand and release resources between attempts. Inspect connection/transaction lifetime, fix demonstrated leaks or blocking, and cap worker concurrency/pools against the shared connection budget.	Database engineering + worker engineering	Representative failure and retry load at both 40 and 90 workers does not exhaust the fleet budget; waiting/held sessions are bounded and attributable; transactions/connections are released as intended. Queue throughput remains adequate.	Apply caps and resource-lifetime changes in stages. Revert the last change if throughput collapses or correctness changes. Pool-size reductions alone may increase queue delay; they must be evaluated with arrival and completion rates.
P1 — Validate configuration lifecycle and release the fix through a limited canary before broad flag rollout. Exercise worker restart, hot flag disable and API rollback independently; cover regional load and multiple tenant policies.	Release engineering + service engineering	Canary and controlled tests show stable queue age, checkout health, retry rate, pool waits and connection reserve. Disabling the flag reaches workers and stops the suspect behavior without requiring a fleet restart. Rollback tests demonstrate expected fallback.	Predefine stop thresholds against baseline and a named release/on-call owner. Disable the flag on threshold breach, halt expansion and roll back affected builds if needed. Lower US flag exposure is insufficient evidence of safety under EU load.
P2 — Add metrics and alerts for resolved retry policy, attempts/delay, budget exhaustion, retry versus new-job arrival rate, completion rate, oldest job age, pool wait and database session ownership. Use bounded-cardinality tenant diagnostics.	Observability engineering + SRE	A simulated tenant retry burst is visible before global saturation; alerts distinguish blocked workers from lack of work. Low CPU does not automatically trigger worker expansion without connection-headroom checks.	Stage dashboards and alerts in observation mode, then tune from baseline to avoid noise. Bound label cardinality and log volume; revert telemetry changes if overhead affects processing.
P2 — Update the incident/scaling runbook and review customer outcome reconciliation. Document flag propagation, safe draining, connection budgets, retry isolation and recovery/closure checks.	SRE lead + service owner + support lead	A tabletop using this incident selects flag containment and database-headroom checks before scaling; recovery requires sustained service health and checked job outcomes. Customer impact counts and any follow-up are documented.	Version the runbook and amend ineffective steps after review. Keep replays/compensations gated on verified outcomes to avoid duplicate customer effects.

Between 09:07 and 10:18 UTC, our eu-west service experienced delays in queued work, and monitoring detected increased checkout latency. The database also reached its connection limit. We are still determining how many customers were affected and whether any jobs failed or produced duplicate results.

Our current evidence suggests that the tenant-specific retry feature contributed to the disruption by retrying some work too quickly. We have not yet confirmed the exact cause. Rolling back the API release did not resolve the queued-work delays. We then disabled the retry feature at 09:43 UTC; recovery began within two minutes, and the backlog cleared at 10:18 UTC.

No alert fired in the US region, where the feature was enabled for a smaller share of tenants. Our next steps are to check customer outcomes and investigate retry behavior and database connection use. We plan to validate corrective changes before enabling the feature more broadly. We will share confirmed impact and findings as the investigation progresses.

Queue incident analysis

Impact and scope

Observed: In eu-west, queue depth began rising at 09:07 UTC. Alerts for job age and checkout latency fired at 09:11. Database connection capacity was reached around 09:20, and an API database-connection error was logged at 09:19:52. Recovery began within two minutes of the 09:43 flag disable; the backlog cleared at 10:18. The documented queue degradation lasted 71 minutes from first visible growth to clearance. This is not a measured duration of checkout unavailability.

Observed: No alert fired in the US, which ran the same API release with the flag enabled for 2% of tenants. That does not establish that every US request was unaffected.

Unknown: Affected customer and job counts, checkout latency distribution and error rate, individual job delays, and whether jobs or checkout side effects were lost or duplicated. The records do not establish a complete outage or data loss.

Concise timeline

All times are UTC. Sources are TIMELINE.md and LOGS.txt; events below are recorded observations.

Time	Event
09:02	API release 2026.09.23.1 deployed, adding tenant-specific retry policy.
09:06:58	Job 81 for tenant acme logged three retry attempts with delay_ms=0 in one second, then budget exhaustion with policy="", elapsed_ms=41.
09:07	eu-west queue depth began rising; worker CPU fell from 65% to 18%. At 09:07:01 one worker logged a 5-second pool wait, active=10, idle=0. Job 92 for beta succeeded at 09:07:03.
09:11–09:20	Job-age and checkout-latency alerts fired; workers increased from 40 to 90 at 09:14, followed by faster queue growth. Database connections reached the configured maximum of 300 around 09:20.
09:24	API release rolled back. New job creation normalized, but the backlog remained stuck.
09:31–09:35	One worker pool restarted and processed normally for four minutes before stalling. Startup logged default exponential and flag true; at 09:35:12 job 319 exhausted its retry budget with policy="", elapsed_ms=38.
09:43–09:45	tenant_retry_policy disabled; recovery began within two minutes. Worker 51 logged flag false at 09:43:07, a 1-second retry delay for job 319 at 09:43:09, and success at 09:43:10.
10:18	Backlog cleared.

Most likely root cause

Inference, moderate confidence: The enabled tenant-specific retry path likely allowed rapid retries for some jobs, increasing pressure on a shared database and starving workers of connections. This would explain growing queues alongside falling CPU, and why adding workers could worsen contention. The strongest evidence is the combination of zero-delay retries and blank logged policies before degradation, recurrence after restart despite an exponential startup default, and delayed retry plus success immediately after flag disable.

The precise defect is unconfirmed. A blank tenant policy overriding or bypassing the default is a plausible implementation explanation, not an observed fact. The logs do not establish what policy="" means internally, whether retries each open or retain connections, or how many jobs used this path. They also do not reveal the original failure that triggered retry. Flag disable is strong intervention evidence, but a single recovery sequence cannot exclude simultaneous changes or independent dependency recovery.

Competing hypotheses and evidence

Hypothesis	Supporting observations	Weakening evidence and limits	Assessment
H1: Flagged retry behavior amplified database load, possibly through an empty-policy fallback defect.	Job 81 had repeated zero-delay retries and a blank policy near onset. Job 319 had a blank policy after restart with the flag enabled, then a delayed retry and success after disable. The API rollback alone did not clear the backlog. US flag exposure was only 2%.	Few sampled jobs; no implementation, effective tenant configuration, retry-volume metrics, or controlled reproduction. EU flag exposure is unspecified. Regions may differ in load or dependencies. Blank policy logging alone does not prove faulty fallback.	Leading explanation; exact code/configuration cause remains unknown.
H2: An independent database leak, lock, slow-query problem, or capacity shortage caused the stalls; retries were a symptom.	A worker pool was full and waiting near onset; global capacity was later reached. Low CPU and transient restart relief are compatible with blocked work or accumulating resources.	Global saturation occurred after queue growth began. Recovery aligned with flag disable, and job 319 then succeeded. Neither observation excludes a database problem, but both favor a retry-related component. No session, lock, or query evidence identifies a leak or bottleneck.	Plausible underlying trigger or co-cause; unproven as the sole cause.
H3: The new API release increased job creation enough to overload the system.	Degradation followed deployment; new job creation normalized after API rollback.	Existing backlog stayed stuck after rollback. Flag disable aligned with recovery. The US shared the release without an alert. No arrival-rate measurements establish overload magnitude. Worker version and rollback propagation are unknown.	Possible contributor; insufficient explanation for persistent processing failure.
H4: A tenant-specific job or downstream failure triggered otherwise expected retries.	Acme retried while beta succeeded shortly afterward, consistent with differing inputs, configurations, or dependencies.	No original error, downstream-health record, or representative tenant sample. Zero-delay retries and blank policies still require explanation. Job 319's tenant is not given.	Possible initiating condition; does not by itself explain regional queue and database behavior.

Contributing factors

• Observed: The database had a shared maximum of 300 connections. One worker reported 10 active and zero idle connections. Inference: Connection contention constrained useful processing; those local counts do not establish every worker pool's size or ownership of all 300 connections.
• Observed: Worker count increased from 40 to 90 before faster queue growth and global saturation. Inference: More workers may have increased concurrent connection demand. Temporal association does not prove that scaling caused the worsening; existing retry load may also have grown.
• Observed: API rollback normalized new creation but did not free the backlog; restart relief lasted four minutes. Inference: The continuing flagged/configured worker path or accumulating database pressure was not removed by those interventions. The supplied records do not establish worker deployment versions or stale configuration.
• Observed: The restarted worker advertised an exponential default but later logged a blank job policy. Inference: Effective per-job policy validation, fallback handling, or logging needs investigation. An absent safeguard is not yet demonstrated.

Remaining unknowns

The effective policies and flag exposure by tenant and region; policy parsing and fallback behavior; original retryable errors; retry counts and database work per attempt; whether transactions or connections remained held during retry; database session ownership, locks and query latency; deployment/configuration propagation across producers and workers; other interventions around 09:43; and the extent and integrity of customer outcomes. The next evidence requests are specified in RESPONSE.md.

Task 09 — Incident analysis

Analyze TIMELINE.md and LOGS.txt. Produce:

• INCIDENT.md: impact, concise timeline, most likely root cause, contributing factors, evidence supporting or weakening each hypothesis, and what remains unknown.
• ACTIONS.md: immediate containment plus prioritized corrective actions with owners expressed as roles, verification signals, and rollback considerations.
• CUSTOMER_UPDATE.md: a plain-language update under 180 words. Do not claim certainty beyond the evidence or blame an individual.

Separate observed facts from inference. Avoid treating correlation as proof. Write RESPONSE.md with your confidence level and the next two pieces of data you would request. Work only in this directory.

Queue incident timeline (UTC)

• 09:02 deploy api 2026.09.23.1; change adds tenant-specific retry policy.
• 09:07 queue depth begins rising in eu-west; worker CPU falls from 65% to 18%.
• 09:11 alerts fire for job age and checkout latency.
• 09:14 on-call increases workers from 40 to 90; queue depth rises faster.
• 09:20 database connections reach configured maximum of 300.
• 09:24 API deploy rolled back; new job creation normalizes, backlog still stuck.
• 09:31 one worker pool restarted; it processes jobs normally for four minutes, then stalls.
• 09:43 feature flag tenant_retry_policy disabled; recovery begins within two minutes.
• 10:18 backlog cleared.
• US region uses the same release but had the flag enabled for only 2% of tenants; no alert fired there.

Response: incident analysis

Confidence

Medium-high that the retry-policy code behind tenant_retry_policy triggered the incident. Low-medium on exactly how it stalled the workers.

• Trigger (medium-high). The logs show policy="" and delay_ms=0 while the flag was on. Right after the flag was turned off, the same job on the same worker, with no restart, retried at a 1000 ms delay and succeeded. The API rollback and the worker restart did not fix the problem while the flag stayed on. Together, these point strongly to the code the flag controls. It is still an association in time, from a small sample of logs, and not a controlled test.
• Mechanism (low-medium). I infer that an empty policy skips the default and causes immediate retries, which saturate DB connections. Nothing in the data shows why workers sat blocked with low CPU instead of spinning. Nor does it show how retries that give up after about 40 ms kept connections tied up. The per-worker pool size of about 10 is inferred from a single log line.
• Contributing factors (medium). Scaling workers from 40 to 90 made the connection pressure worse. It could not have started the incident, because the queue was already rising. The US "no alert" point is weak and confounded evidence.

Facts are kept separate from inference throughout INCIDENT.md, and the unknowns are listed there.

Next two pieces of data I would request

1. Database connections broken down by client and state, 09:00–10:20 UTC, for eu-west. For example, pg_stat_activity snapshots: application or host, state (active / idle / idle in transaction), wait events and locks. Pair this with per-worker pool metrics (active, idle, waiting).

   • This tests the weakest link: whether workers were holding connections on the retry_budget_exhausted path (a leak or open transactions) or just competing for them.
   • It also shows why connections were released when the flag was turned off.

2. The code change to retry-policy resolution in 2026.09.23.1, the flag targeting rules, and the tenant retry-policy records for eu-west and the US. Specifically: which tenants had the flag on, and which resolved to an empty policy.

   • This confirms or refutes the inference that "" bypasses the default.
   • It shows how many tenants were exposed in each region.
   • It tests whether the US 2% cohort included any tenants with an empty policy, which is currently confounded evidence.

Deliverables

• INCIDENT.md: impact, timeline, root-cause hypotheses with evidence for and against, contributing factors, and unknowns.
• ACTIONS.md: containment steps and prioritized corrective actions, each with an owner role, a verification signal and rollback considerations.
• CUSTOMER_UPDATE.md: a plain-language update under 180 words, with no blame and no claims beyond the evidence.
• RESPONSE.md: this file.

Actions: eu-west queue stall (2026-09-23)

Owners are listed as roles. The priorities assume the leading hypothesis in INCIDENT.md (H1). Any action that depends on unconfirmed parts of H1 is marked.

Immediate containment (next 24 hours)

#	Action	Owner	Verification signal	Rollback consideration
C1	Keep tenant_retry_policy off in all regions, including the 2% in the US. Lock the flag so that changing it requires incident-commander approval.	Incident commander	The flag service shows 0% in every region. Workers log feature_flag=false. No policy="" log lines appear.	Low risk: the flag was already off in eu-west during recovery. Tenants that expected custom retry behavior fall back to the default. Tell their account owners.
C2	Check the US 2% cohort for the same pattern over the same window: policy="", delay_ms=0, retry_budget_exhausted, and DB pool waits.	On-call SRE (US)	A search of the US logs returns a count, even if it is zero.	Read-only.
C3	Find every job that logged retry_budget_exhausted in eu-west and the US between 09:00 and 09:45. Work out what happened to each one: completed, dead-lettered, dropped or duplicated.	Owning service engineer + Support lead	A reconciled list: each job ID with its final state.	Re-running jobs can cause double side effects, such as double charges. Confirm the jobs are idempotent before bulk replay. Replay in small batches.
C4	Return the worker count to a size whose total connection pools fit under the DB maximum, with room left for the API. Confirm no idle or leaked connections remain.	On-call SRE + DBA	DB connections are well below 300. Pool-wait warnings are at zero. The API has no remaining connection slots reserved errors.	Scale down gradually while watching queue depth. Keep the previous worker count on record.
C5	Hold further deploys of the retry-policy change until P0 items are fixed.	Release manager	The release calendar shows the hold.	None.

Corrective actions (prioritized)

P0: Fix the trigger

#	Action	Owner	Verification signal	Rollback consideration
P0-1	Fix policy resolution: when a tenant policy is empty, null or unknown, fall back to the configured default. Validate tenant policies when they are loaded and written, and reject "".	Owning service team (tech lead)	Unit tests for empty, null, unknown and valid policies. In staging, a tenant with no policy logs policy="exponential" (or whatever the default is) and delay_ms>0.	Ship behind the same flag, off by default. The flag stays the kill switch.
P0-2	Enforce a minimum retry delay with jitter in the retry library, whatever the policy says. This protects against any future bad policy.	Platform / retry-library owner	Tests: no retry has delay_ms below the minimum. Production logs have no delay_ms=0.	The change is a library default. If it slows recovery for latency-sensitive jobs, allow a per-job override that has to be reviewed.
P0-3	Find out what the retry_budget_exhausted path does with DB connections, transactions and job state. Confirm or rule out a connection leak or held lock. This tests the weakest link in H1.	Owning service engineer + DBA	A reproduction in staging with pool metrics. After the fix, active connections return to their baseline after exhaustion.	Investigation only. Any fix it leads to goes through normal review.

P1: Limit the blast radius and speed up mitigation

#	Action	Owner	Verification signal	Rollback consideration
P1-1	Connection budget: make sure max_workers × pool_size + API pool + headroom ≤ DB max. Enforce it in autoscaling. Give the API its own reserved connections or a separate pool or proxy.	Platform/SRE + DBA	A config check fails CI if the budget is exceeded. A load test at the maximum worker count keeps DB connections below 90%.	Lowering pool sizes can reduce throughput. Roll out one region at a time and compare job throughput.
P1-2	Runbook: when rolling back a release, list and turn off any feature flags changed or introduced with it. Warn against scaling workers when DB pool waits or connection limits are firing.	SRE lead	The runbook is updated, and a game day exercises it.	Documentation only.
P1-3	Alerts on the cause, not only on symptoms: the rate of retry_budget_exhausted, retries with delay_ms=0, DB pool-wait warnings, DB connections above 80%, and worker CPU falling while queue depth rises. Add equivalent alerts in the US.	Observability owner	The alerts fire in staging when a tenant with no policy is replayed.	Tune thresholds to avoid alert fatigue. Start with notification-only alerts for a week.
P1-4	Show flag changes and deploys together on the incident dashboards and timeline.	Observability owner	Flag changes appear as annotations on the queue and DB graphs.	None.

P2: Prevent similar incidents

#	Action	Owner	Verification signal	Rollback consideration
P2-1	Roll out flags gradually, per region, with automated safety checks: automatic rollback on queue age, retry exhaustion or DB connection signals. Start with 1–5% in each region.	Release engineering	The flag platform records the stages and automatic rollbacks. A test flag rolls back automatically in staging.	If the safety checks are too sensitive, fall back to a manual rollout.
P2-2	Audit tenant retry-policy data for empty or invalid values before turning the flag back on.	Owning service team + Data owner	Audit report: zero invalid records, or each one fixed.	Read-only, apart from data fixes, which should be backed up.
P2-3	Hold a blameless post-incident review once the data from RESPONSE.md is collected. Confirm or revise the root cause.	Incident commander	Review published, with root cause marked confirmed or revised.	None.

Re-enable criteria for tenant_retry_policy

Turn the flag back on only when all of the following are true:

• P0-1, P0-2 and P0-3 are done.
• P1-1 and P1-3 are in place.
• The tenant data audit (P2-2) is clean.
• The rollout follows P2-1, starting at 1% in one region.
• For 24 hours at each stage: no policy="" lines, no zero-delay retries, and DB connections below 70%.

Service update: delayed background jobs and slower checkout in EU (23 September 2026)

What happened: From about 09:07 to 10:18 UTC, background jobs in our EU region were delayed. For part of that time, checkout was slower than normal, and some requests may have failed.

Current status: The service has been stable since 10:18 UTC, and all delayed jobs have now been processed.

What we know so far: The problem began shortly after we released a change to how failed jobs are retried. It improved within minutes of our turning that change off. We have kept it turned off while we investigate. We have not yet confirmed the full cause.

What we are doing: We are checking whether any jobs did not complete correctly. We will contact affected customers directly. We are also adding safeguards and better alerting.

We are sorry for the disruption. We will share a fuller report once our review is complete.

Incident: eu-west job queue stall and checkout latency (2026-09-23)

Status: mitigated at 09:43 UTC, backlog cleared at 10:18 UTC. Root cause not yet confirmed.

Sources: TIMELINE.md (operator timeline) and LOGS.txt (12 sampled log lines from 5 hosts). The logs are a small sample. They are not a complete record.

Labels used below:

• [Fact]: stated directly in the timeline or logs.
• [Inference]: my interpretation. It could be wrong.

1. Impact

Area	What we know	Label
Region	eu-west. No alert fired in the US, but US impact was not measured.	Fact / Unknown
Duration	Queue depth began rising at 09:07. The backlog cleared at 10:18, about 71 minutes later.	Fact
Background jobs	Job age alert fired at 09:11. Jobs were delayed until the backlog cleared.	Fact
Checkout	A checkout latency alert fired at 09:11. How much latency rose, and for how many users, is unknown.	Fact / Unknown
API errors	At 09:19:52, api-03 logged remaining connection slots reserved, which means the API could not get database connections. Some API requests probably failed around 09:20.	Fact / Inference
Job creation	New job creation was abnormal until the 09:24 rollback, when it "normalized".	Fact
Jobs that gave up	Jobs 81 and 319 logged retry_budget_exhausted. We do not know what happens to a job after that: it may be dropped, dead-lettered or retried. There may be permanently failed or duplicated work that has not been found yet.	Fact / Unknown

Not yet quantified: the number of affected tenants, delayed jobs, failed checkouts and failed API requests, and the peak job age.

2. Timeline (UTC)

Time	Event	Source
09:02	API release 2026.09.23.1 deployed. It adds a tenant-specific retry policy behind flag tenant_retry_policy.	Timeline
09:06:58	worker-17, job 81, tenant acme: 3 attempts at delay_ms=0, then retry_budget_exhausted policy="" after 41 ms.	Log
09:07	Queue depth starts rising. Worker CPU falls from 65% to 18%.	Timeline
09:07:01	worker-22: DB pool wait exceeded 5000 ms, with active=10 idle=0.	Log
09:07:03	worker-08, job 92, tenant beta: succeeds on the first attempt.	Log
09:11	Alerts fire for job age and checkout latency. Detection took about 4 minutes.	Timeline
09:14	On-call raises workers from 40 to 90. Queue depth then rises faster.	Timeline
09:19:52 / 09:20	The API is refused DB connections. DB connections reach the configured maximum of 300.	Log / Timeline
09:24	API release rolled back. Job creation normalizes, but the backlog stays stuck.	Timeline
09:31	One worker pool is restarted. worker-51 starts with config_retry_default="exponential" feature_flag=true.	Timeline / Log
09:35	That pool stalls after about 4 normal minutes. worker-51 job 319 logs retry_budget_exhausted policy="" after 38 ms.	Timeline / Log
09:43:07	Flag tenant_retry_policy disabled. worker-51 logs config_changed feature_flag=false.	Timeline / Log
09:43:09–10	Job 319 retries with delay_ms=1000, then succeeds.	Log
~09:45	Recovery begins.	Timeline
10:18	Backlog cleared.	Timeline

3. Most likely root cause

Leading hypothesis (H1), medium-high confidence: the retry-policy code behind tenant_retry_policy misbehaves when a tenant has no specific policy.

With the flag on, some tenants resolved to an empty policy (policy=""). The code did not fall back to the configured default (exponential). Instead it retried immediately (delay_ms=0) and used up the retry budget in about 40 ms. This behavior, directly or indirectly, saturated the database connection pools. Workers then blocked while waiting for connections, and the queue stalled. When worker pools grew, the database hit its 300-connection limit, and the shared API was starved of connections. That caused the checkout latency and API errors.

Which parts of H1 are observed and which are inferred:

• [Fact] With the flag on, jobs logged policy="" and delay_ms=0 retries.
• [Fact] The same worker had config_retry_default="exponential" loaded.
• [Fact] After the flag was turned off, the same job on the same worker, with no restart, retried at delay_ms=1000 and succeeded.
• [Inference] An empty string is treated as a valid policy instead of "no policy". This fits the logs, but I have not read the code.
• [Inference, weakest link] How fast retries turn into connection exhaustion and a worker that stalls with low CPU. See the unknowns in section 6.

4. Hypotheses and evidence

H1: The flag-gated retry-policy path is the trigger (leading)

Supporting evidence:

• The logs show policy="" and zero-delay retries while the flag was on. They show a 1000 ms delay and a success right after it was turned off.
• This is the strongest single piece of evidence. It is a before-and-after on one job and one worker, with no restart, and the flag change is the only change we know of. It is still not a controlled experiment: other things may have changed at 09:43.
• Rolling back the API release did not clear the backlog. The flag stayed on, which fits the cause being in behavior the flag controls, not only in API code.
• Restarting a worker with the flag still on helped only briefly. The pool stalled again after a job logged policy="".
• Recovery started within about 2 minutes of turning the flag off. About 19 minutes had passed since the rollback without recovery.
• Job 92 for tenant beta succeeded while job 81 for acme failed. This fits a problem that affects only some tenants, for example those without their own policy.
• The US had the flag on for only 2% of tenants and no alert fired. This is consistent with an effect that grows with exposure.

Weakening evidence and caveats:

• The link from zero-delay retries to blocked, low-CPU workers is not shown. A tight retry loop would usually raise CPU, but CPU fell. Retries exhausting after about 40 ms do not by themselves explain workers staying stuck. Something else must be holding connections or blocking.
• Only 2 affected jobs appear in the logs. We do not know how common policy="" was, or which tenants had it.
• The US evidence is confounded. It may have different traffic, capacity or tenants. The 2% cohort may not include any tenants with empty policies. No alert does not mean no impact.
• Other changes around 09:43 have not been ruled out: backlog load falling after the rollback, connections timing out, or other operator actions.
• We do not know whether workers were also running 2026.09.23.1. The timeline says only "deploy api".

H2: Non-flag code in API release 2026.09.23.1

• For: the problem started 5 minutes after the deploy. The rollback normalized job creation, so the release did affect the API's job-intake side.
• Against: the backlog stayed stuck for 19 minutes after the rollback. Recovery followed the flag change, not the rollback.
• Assessment: probably contributed to the intake and API symptoms. Unlikely to be the main cause of the worker stall.

H3: A database problem unrelated to the change

• For: the pool wait at 09:07:01, the connection limit reached at 09:20, and the API refused connections.
• Against: saturation started within seconds of the first policy="" failure. It cleared after a flag change, with no database change. There was no evidence of a database fault.
• Assessment: most likely a symptom and amplifier, not the cause. Not fully ruled out, because we have no database-side metrics.

H4: The worker scale-up (40 to 90) caused the incident

• Against: queue depth was already rising at 09:07, 7 minutes before the scale-up.
• For it as an amplifier: the queue grew faster after the scale-up, and connections hit their maximum 6 minutes later.
• [Inference] active=10 idle=0 suggests each worker's pool holds about 10 connections. If so, 90 workers could ask for up to about 900 connections. That is far above the limit of 300, which the API also shares. Even 40 workers (about 400) could exceed it.
• Assessment: this was a contributing factor that made things worse. It was not the root cause.

H5: Bad data for one tenant only (for example, acme)

• A single bad tenant record is possible, but the flag-off recovery happened across the whole region. More likely, a tenant with no policy exposed the fallback bug in H1. Which tenants were involved is unknown.

5. Contributing factors

Each factor is labelled with how solid it is.

1. Missing fallback and validation (inference). An empty tenant policy was accepted instead of falling back to the default. There is also no minimum delay for retries.
2. Flag separate from the deploy (fact). Rolling back the release did not turn off the new behavior. This delayed mitigation by about 19 minutes, and about 32 minutes passed from the first alert to the flag change.
3. Scaling made it worse (fact plus inference). Adding workers added more connection pools against a database limit shared with the API. Nothing stopped workers × pool size from exceeding the database's maximum connections.
4. API and workers share one connection limit (fact plus inference). The worker backlog spread to checkout through connection starvation.
5. Detection was symptom-based (fact). Alerts fired on job age and latency. No alert covered retry-budget exhaustion, zero-delay retries or pool waits. In the US, no alert fired at all.
6. Rollout exposure (unknown). The flag was on for 2% of tenants in the US. The eu-west percentage is not recorded, and neither is whether the rollout had automatic safety checks.

6. What remains unknown

• How an exhausted retry leaves a worker stuck with low CPU. Candidates: connections not returned to the pool on the exhaustion path, open transactions or row locks held, or contention on a shared job table. Recovery without a restart argues against a permanent connection leak, but does not rule out leaks that time out.
• Why the restarted pool worked for exactly about 4 minutes. This could be the time needed to fill a 10-connection pool, or the time until it picked up a job with policy="".
• Which tenants resolved to policy="", and the eu-west and US flag percentages.
• Whether workers run code from 2026.09.23.1. If they do not, how the worker reads the tenant policy.
• Why the API rollback normalized job creation.
• What happens to jobs after retry_budget_exhausted, and whether any were lost, sent to a dead-letter queue or duplicated. This matters if checkout jobs are not idempotent.
• Why the attempt counter for job 319 reset to 1 after it had been exhausted.
• Database-side connection breakdown (by client and state) and US metrics for the same window.
• How many customers and requests were affected.

Task 09 — Incident analysis

Analyze TIMELINE.md and LOGS.txt. Produce:

• INCIDENT.md: impact, concise timeline, most likely root cause, contributing factors, evidence supporting or weakening each hypothesis, and what remains unknown.
• ACTIONS.md: immediate containment plus prioritized corrective actions with owners expressed as roles, verification signals, and rollback considerations.
• CUSTOMER_UPDATE.md: a plain-language update under 180 words. Do not claim certainty beyond the evidence or blame an individual.

Separate observed facts from inference. Avoid treating correlation as proof. Write RESPONSE.md with your confidence level and the next two pieces of data you would request. Work only in this directory.

Queue incident timeline (UTC)

• 09:02 deploy api 2026.09.23.1; change adds tenant-specific retry policy.
• 09:07 queue depth begins rising in eu-west; worker CPU falls from 65% to 18%.
• 09:11 alerts fire for job age and checkout latency.
• 09:14 on-call increases workers from 40 to 90; queue depth rises faster.
• 09:20 database connections reach configured maximum of 300.
• 09:24 API deploy rolled back; new job creation normalizes, backlog still stuck.
• 09:31 one worker pool restarted; it processes jobs normally for four minutes, then stalls.
• 09:43 feature flag tenant_retry_policy disabled; recovery begins within two minutes.
• 10:18 backlog cleared.
• US region uses the same release but had the flag enabled for only 2% of tenants; no alert fired there.

Response: confidence and next data requests

Confidence

• Trigger (flag-gated tenant retry policy resolving to an empty policy, producing zero-delay retries that exhaust the retry budget in milliseconds): moderate-high, roughly 75-80%. Four independent observations point the same way: the first policy="" delay_ms=0 line lands seconds before queue depth starts rising; the restarted pool stalls at its first policy="" event; the flag disable is followed within two seconds by the same job succeeding with a 1000 ms backoff and within two minutes by region-wide recovery; and the US region on the same release at 2% exposure had no alert. What holds this below "high": the log excerpt shows only two affected jobs and one affected tenant, the eu-west rollout percentage is not stated, and the "four minutes then stalls" match is a single coincidence on a single worker.
• Stall mechanism (why an exhausted retry budget left workers idle with saturated DB pools rather than simply failing the job): low, roughly 40-50%. Low CPU and active=10 idle=0 say the workers were blocked, and job 319 sat on worker-51 for ~8 minutes, but nothing in the data shows what they were blocked on.
• Role of the worker scale-up and DB connection cap as amplifiers rather than causes: moderate, roughly 65%. Queue depth was rising seven minutes before the scale-up, and the flag disable (not any DB change) produced recovery, but the arithmetic linking 90 workers x 10-connection pools to the 300 cap is inferred from a single active=10 log line.
• Underlying defect (why the policy was empty): unknown. Missing tenant config, a resolver bug, and eu-west-specific config propagation are all consistent with the evidence.

Next two pieces of data I would request

1. Complete worker logs for both regions, 08:55-10:30, joined with the flag rollout state and the tenant retry-policy configuration. Specifically: every retry_budget_exhausted, policy="", delay_ms=0 and db pool wait exceeded line with tenant and worker, plus the per-region flag percentage and the stored policy for each tenant that appears. This confirms or refutes the trigger at scale rather than from two jobs, reveals whether the empty policy came from missing config or a resolver defect, tells us whether the US 2% cohort had sub-threshold impact, and gives the list of jobs that need reconciliation.
2. Worker-51 diagnostics and DB connection accounting for 09:35-09:43. A thread or goroutine dump (or equivalent in-flight job state) from worker-51 during the stall, and DB connection counts broken down by client (each worker pool versus the API) across 09:05-09:45. This is the data that separates "the exhausted-retry path holds a connection or job lease" from "the pool was starved by other workers", explains why adding workers made the queue grow faster, and tells us whether a per-job timeout or a connection fix is the right corrective action.

If a third were allowed it would be the diff for the retry-policy resolver in 2026.09.23.1, which would likely answer the "why empty" question directly.

Actions: eu-west queue backlog (2026-09-23)

Owners are roles, not people. "Verification signal" is what we watch to know the action worked. "Rollback consideration" is what could go wrong with the action itself and how to undo it. Items are prioritized within each section.

A. Immediate containment (now, before anything else)

#	Action	Owner (role)	Verification signal	Rollback consideration
C1	Keep tenant_retry_policy disabled in every region, including the US 2% cohort, until A1-A3 ship.	Incident commander (on-call SRE)	Zero retry_budget_exhausted policy="" and zero result=retry delay_ms=0 lines in worker logs, both regions, for 24 h; eu-west queue depth, job age and worker CPU at pre-09:00 baseline.	Disabling in the US changes behaviour for the 2% tenants. Confirm they fall back to config_retry_default="exponential" (as worker-51 did at 09:43:09) and that no tenant was relying on the new policy. If one is, address it with an explicit per-tenant config review, not by re-enabling the flag.
C2	Put the flag under change control: no re-enable without an approved change ticket referencing this incident.	Release manager	Flag audit log shows no state changes; ticket link recorded.	None; this only adds friction to re-enabling.
C3	Return worker count from 90 to the pre-incident 40 (or whatever the DB can support with API headroom, see A5).	On-call SRE	DB connections stay below ~70% of the 300 cap at peak; no db pool wait exceeded warnings for 24 h; queue depth stable.	If the backlog re-forms, scale up in steps of 10 and only while DB connections have >20% headroom. Scaling up blindly made things worse at 09:14.
C4	Reconcile every job that logged retry_budget_exhausted between 09:06 and 09:45 in eu-west: determine whether each was requeued, dead-lettered, or dropped; replay idempotent ones; hand non-idempotent ones to manual review; list affected tenants for Support.	Queue platform engineer, with Support lead	Dead-letter count back to baseline; replayed-job success rate; reconciled list attached to the incident; no duplicate side-effect reports from tenants after 48 h.	Replay only jobs known to be idempotent. Keep the list of replayed job IDs so effects can be reversed if duplicates surface. If job creation was elevated 09:07-09:24 (unknown), also check for duplicate jobs before replaying.
C5	Preserve evidence before log retention rolls: full worker and API logs 08:55-10:30 in both regions, DB connection metrics by client, flag audit history, and any worker-51 diagnostics (thread dumps, in-flight job state).	SRE (incident scribe)	Artifacts attached to the incident record; checksums noted.	None.
C6	Query the US region for policy="" events and for job-age p99 among the 2% flagged tenants during 09:00-10:30 to confirm "no alert" also meant "no impact".	On-call SRE (US)	Query results documented in the incident record.	None.
C7	Establish exactly what the 09:24 rollback reverted (API only? workers?) and which build every worker pool is running now.	Release manager	Build IDs per pool documented; workers confirmed on a build whose retry path is either pre-change or fixed.	If workers are still on 2026.09.23.1 with the flag off, that is acceptable short-term only because the flag gates the path; do not treat it as fully rolled back.

B. Corrective actions

Priority 1: required before the flag is re-enabled anywhere

#	Action	Owner (role)	Verification signal	Rollback consideration
A1	Fix retry-policy resolution: an empty, null or malformed tenant policy must fall back to config_retry_default; enforce a minimum retry delay floor (never delay_ms=0); treat an empty policy as a configuration error that is logged and alerted, not applied.	Worker platform tech lead	Unit tests covering empty/null/malformed policies resolve to the default; staging canary with a tenant that has no policy entry shows exponential delays; metric retry.delay_ms == 0 count is zero over a full staging soak.	Standard deploy rollback. Because the flag stays off during rollout, reverting this change has no tenant-facing effect.
A2	Identify and fix the stall mechanism: why a job whose retry budget was exhausted left worker-51 stalled for ~8 minutes with low CPU and a saturated DB pool. Ensure the exhausted-budget path acks/nacks the job, returns DB connections, and is bounded by a per-job timeout.	Worker platform engineer	Fault-injection test (jobs that always return retry): throughput on healthy jobs stays >= 95% of baseline; per-worker pool idle > 0 throughout; no job held longer than the per-job timeout.	Code-level change; deploy rollback. If the per-job timeout is set too low it will abort legitimate long jobs, so start at 2x the observed p99 job duration and tighten later.
A3	Validate tenant retry-policy configuration: schema validation at load time and in CI; refuse to enable tenant_retry_policy if any tenant resolves to an empty policy; log policies_validated=N at worker startup.	Config / platform engineer	CI fails on a synthetic empty policy; worker startup line shows the validated count; enabling the flag with a known-bad config is rejected in staging.	Validation can be switched to warn-only if it blocks legitimate deploys, but not before A1 ships.
A4	Update the incident runbook: for any flag-gated change, disable the flag first, then consider a deploy rollback; document which services evaluate each flag (this one is evaluated by workers, not just the API, so an API rollback did not revert it).	SRE lead	Runbook reviewed in the postmortem; a game-day exercise reaches flag-off in under 10 minutes from first alert.	None.

Priority 2: within two weeks

#	Action	Owner (role)	Verification signal	Rollback consideration
A5	Database connection budgeting: (worker replicas x pool size) + API pools must fit under the 300 cap with a reserve for the API tier; add a connection reserve or a pooler in front of the database; cap worker autoscaling at the DB-safe maximum.	Database platform lead	Load test at the maximum worker count produces zero remaining connection slots reserved errors on the API and no change in checkout p99; the cap is enforced in the autoscaler config.	Pool-size and reserve changes are config and reversible. Watch for worker throughput regression if per-worker pools shrink; if it appears, raise the DB cap rather than the pool size.
A6	Scaling guardrail: runbook step and autoscaler rule that blocks adding workers while db pool wait warnings are firing or DB connections exceed 80%; scale in steps with a settle period.	SRE lead	Autoscaler dry-run logs show blocked scale-ups under simulated saturation; runbook step present.	The rule can be disabled if it blocks legitimate scaling; keep a documented manual override.
A7	Alert on cause signals, not only symptoms: retry_budget_exhausted rate, delay_ms=0 retries, policy="", db pool wait exceeded, and the signature "queue depth rising while worker CPU falls".	Observability engineer	Replaying the 09:06-09:08 log lines into staging triggers an alert within 2 minutes; alert noise reviewed after one week in production.	Thresholds can be tuned or individual alerts muted if noisy; do not remove the policy="" alert until A3 is in place.

Priority 3: before re-enabling the flag / within 30 days

#	Action	Owner (role)	Verification signal	Rollback consideration
A8	Staged re-enable plan for tenant_retry_policy: 1% -> 10% -> 50% -> 100% per region, each stage held 24 h with automatic halt on job-age or checkout SLO burn; begin in the region with the most DB headroom.	Product / platform release manager	Each stage completes with zero cause-signal alerts (A7) and no policy="" events; job success-on-retry rate unchanged or better.	Flag off at any stage. Workers pick up the flag at runtime (config_changed at 09:43:07), so the effect should be near-immediate; confirm this at the 1% stage before proceeding.
A9	Run the postmortem with the unknowns in INCIDENT.md section 7 as explicit questions to close; track every action here to completion.	Incident commander	Each action has a ticket, owner and due date; unknowns are either answered or accepted as residual risk.	None.
A10	Investigate the underlying transient failures that make jobs return retry in normal operation (job 319 needed a retry even with the flag off), and reduce reliance on retries where possible.	Worker platform engineer	Baseline attempt>1 rate measured; a reduction target set and tracked.	None; this is investigative.

C. Notes on ordering

• C1-C3 are the immediate stabilizers and are already partly in effect; C4 is the customer-facing cleanup and should start today.
• A1 and A2 are the actual fixes and both must ship before A8; A3 and A4 prevent a repeat by a different route (bad config, wrong rollback target).
• A5-A7 reduce blast radius and shorten detection for the next incident of any cause, so they are worth doing even if A1/A2 fully explain this one.

Service update: delayed background jobs and slower checkouts in the EU region

Date: 23 September 2026 (times in UTC)

Between about 09:07 and 10:18, customers served from our EU region experienced delayed background job processing and slower checkouts. Processing recovered from about 09:45, and the remaining backlog was fully cleared by 10:18. Our monitoring shows no impact in the US region.

Our current understanding is that a change released that morning to how failed jobs are retried for some accounts caused work to pile up faster than it could be processed. We switched that change off at 09:43 and recovery began within minutes. We are still confirming the exact chain of events and whether any individual jobs were missed or run twice. If we find that any of your jobs were affected, we will contact you directly.

We are sorry for the disruption. We are adding safeguards so that a problem of this kind is caught and switched off much sooner, and we will share a full report once our review is complete.

Incident report: eu-west queue backlog and checkout latency (2026-09-23, UTC)

Sources: TIMELINE.md (operator timeline) and LOGS.txt (12 log lines, apparently a filtered excerpt). Throughout this document, Observed means the statement is taken directly from one of those two sources; Inferred means it is our interpretation and could be wrong. Correlation in time is noted as such and is not treated as proof of causation.

1. Summary

Shortly after the 09:02 deploy of api 2026.09.23.1 (which adds a tenant-specific retry policy behind the feature flag tenant_retry_policy), eu-west workers stopped draining the job queue. Symptoms lasted from 09:07 until the backlog cleared at 10:18 (about 71 minutes). Scaling workers up and rolling back the API deploy did not restore processing; disabling the feature flag at 09:43 did, with recovery starting within two minutes. The most likely trigger is that, with the flag on, the retry policy resolved to an empty policy (policy="") for some tenants, producing zero-delay retries that exhausted the retry budget within tens of milliseconds and left workers stalled. The exact mechanism by which exhausted retries stalled workers, and why the policy was empty, are not established by the available data.

2. Impact

Observed

• Region: eu-west. The US region ran the same release with the flag enabled for only 2% of tenants and raised no alert.
• Queue depth rose from 09:07; the backlog was cleared at 10:18 (~71 minutes of degraded or absent job processing).
• Alerts for job age and checkout latency fired at 09:11. Recovery began within two minutes of the 09:43 flag disable (~09:45), so alert-level customer impact lasted at least ~34 minutes, with a tail until 10:18.
• Database connections hit the configured maximum of 300 at 09:20; api-03 logged db remaining connection slots reserved at 09:19:52, so the API tier was unable to obtain database connections for at least part of the window.
• New job creation was abnormal until the API rollback at 09:24 (the timeline says it "normalizes" then; whether it had been failing or elevated is not stated).
• Worker CPU dropped from 65% to 18% at 09:07 while queue depth rose.

Inferred / not yet quantified

• Customer-visible effect: slower checkouts from at least 09:11 to ~09:45, and possibly checkout errors while the API had no database connections (around 09:20-09:24, duration unknown).
• Jobs that hit retry_budget_exhausted during the window may have been dead-lettered, delayed, or lost; their final disposition is unknown. Job 319 is known to have succeeded at 09:43:10. Job 81's outcome is not in the logs.
• Number of affected tenants and jobs: unknown. The only tenant seen with the empty policy is acme; tenant beta processed successfully at 09:07:03.
• If the API tier also applied zero-delay retries when creating jobs, duplicate jobs are possible. No evidence either way.

3. Concise timeline (UTC)

Time	Event	Source	Status
09:02	Deploy api 2026.09.23.1; adds tenant-specific retry policy	TIMELINE	Observed
09:06:58	worker-17, job 81, tenant acme: attempts 1-3 all result=retry delay_ms=0 within the same second; then WARN retry_budget_exhausted policy="" elapsed_ms=41	LOGS	Observed
09:07:01	worker-22: db pool wait exceeded wait_ms=5000 active=10 idle=0	LOGS	Observed
09:07	eu-west queue depth starts rising; worker CPU 65% -> 18%	TIMELINE	Observed
09:07:03	worker-08, job 92, tenant beta: success in 830 ms	LOGS	Observed
09:11	Alerts fire: job age, checkout latency	TIMELINE	Observed
09:14	On-call scales workers 40 -> 90; queue depth rises faster	TIMELINE	Observed
09:19:52	api-03: ERROR db remaining connection slots reserved	LOGS	Observed
09:20	DB connections at configured max (300)	TIMELINE	Observed
09:24	API deploy rolled back; new job creation normalizes; backlog still stuck	TIMELINE	Observed
09:31:10	Restarted pool: worker-51 startup with config_retry_default="exponential" feature_flag=true	LOGS	Observed
09:31-09:35	Restarted pool processes jobs normally for ~4 minutes	TIMELINE	Observed
09:35:12	worker-51, job 319: retry_budget_exhausted policy="" elapsed_ms=38; the pool stalls at about this time	LOGS + TIMELINE	Both observed; that one caused the other is inferred
09:43:07	worker-51: config_changed feature_flag=false	LOGS	Observed
09:43:09-10	worker-51, job 319: attempt 1 result=retry delay_ms=1000; attempt 2 success in 190 ms	LOGS	Observed
~09:45	Recovery begins	TIMELINE	Observed
10:18	Backlog cleared	TIMELINE	Observed

Derived intervals (arithmetic on observed times): first diagnostic log line to symptom-based alert, 4 minutes; first symptom to flag disable, 36 minutes; API rollback to flag disable, 19 minutes; job 319 was associated with worker-51 for about 8 minutes (09:35:12 to 09:43:09) before being reset to attempt 1.

4. Most likely root cause

Inferred. With tenant_retry_policy enabled, the retry-policy lookup returned an empty policy (policy="") for at least some tenants instead of falling back to the configured default (config_retry_default="exponential"). An empty policy produced retries with delay_ms=0, so the retry budget was consumed in roughly 40 ms (three attempts for job 81), before whatever transient condition caused the first failure could clear. Jobs for affected tenants therefore failed outright instead of succeeding on a backed-off retry. The contrast is visible in the logs: with the flag off, job 319 got a 1000 ms delay on attempt 1 and succeeded on attempt 2.

Something in the exhausted-budget path then left workers stalled rather than merely failing the job: worker CPU fell to 18% (workers were waiting, not busy-looping), per-worker DB pools were fully in use (active=10 idle=0 with 5 s waits), and job 319 stayed attached to worker-51 for ~8 minutes. The stall mechanism (a held connection, an unreleased job lease, a blocking requeue, or something else) is not identifiable from the data.

Why we call this the trigger rather than the defect: the flag disable resolved the incident, but the underlying defect could be any of (a) tenants missing a policy entry in config, (b) a resolver bug that returns an empty policy, or (c) a config propagation problem specific to eu-west. Which one is unknown.

5. Hypotheses and evidence

H1 (primary): flag-gated retry policy resolves empty -> zero-delay retries -> instant budget exhaustion -> worker stall

Supporting (observed):

• First delay_ms=0 / policy="" line at 09:06:58, five minutes after deploy and seconds before queue depth began rising at 09:07.
• The restarted pool worked normally until its first policy="" event at 09:35:12, about four minutes after startup, matching the timeline's "four minutes, then stalls".
• Disabling the flag at 09:43:07 was followed within two seconds by the same job (319) retrying with delay_ms=1000 and succeeding, and by region-wide recovery within two minutes. No other change was made at that time.
• The US region, same release, 2% flag exposure, had no alert.
• worker-51's own startup line shows the default policy was exponential, so the zero-delay behaviour came from the tenant-specific path, not the default. Weakening / caveats:
• The stall mechanism is not shown. Zero-delay retries alone should make a job fail fast, not block a worker with low CPU.
• Only two affected jobs (81, 319) and one affected tenant (acme) appear in the excerpt; the fraction of tenants with an empty policy is unknown.
• "Four minutes then stalls" and the 09:35:12 event is a single coincidence on a single worker.
• We do not know the eu-west flag rollout percentage; the US comparison assumes it was substantially higher than 2%.

H2: database connection exhaustion was the primary cause

Supporting (observed): pool-wait warning at 09:07:01; API remaining connection slots reserved at 09:19:52; DB at max at 09:20; checkout latency alert. Weakening (observed): the pool-wait warning came 3 seconds after the first zero-delay retry storm; DB reached its maximum only after workers were scaled to 90 at 09:14; the API rollback restored job creation but not the backlog; the flag disable, which changed nothing about the database, triggered recovery. DB exhaustion is better explained as a consequence and amplifier than as the origin. Unknown: whether the exhausted-retry path leaks or holds connections, which would link H1 and H2 directly.

H3: the worker scale-up (40 -> 90) caused the incident

Weakening (observed): queue depth was already rising at 09:07, seven minutes before the scale-up. Supporting as an amplifier (observed): queue depth rose faster after 09:14, DB hit 300 connections six minutes later, and the API lost DB access at 09:19:52. Inferred: with per-worker pools of 10 (from active=10 idle=0), 90 workers can demand 900 connections against a cap of 300, so more workers meant more contention for the same database the API depends on.

H4: the API code change itself (independent of the flag) caused the incident

Supporting (observed): new job creation normalized after the API rollback at 09:24. Weakening (observed): the backlog stayed stuck after rollback; US ran the same release without alerting; the flag disable, not the rollback, produced recovery. Open question: "normalizes" could mean job creation had been failing (starved of DB connections) or had been elevated (e.g. API-side zero-delay retries creating duplicates). Either reading is consistent with the evidence.

H5: pre-existing transient failures are the real problem

Supporting (observed): even with the flag off, job 319's attempt 1 returned retry, so the underlying operation needs retries in normal operation. Assessment: this is a latent condition that the retry policy normally absorbs, not the cause of the outage; it does mean retry behaviour is on the critical path and any regression in it is high-impact.

6. Contributing factors

1. No safe fallback for an empty policy (inferred). The default policy was exponential, but an empty tenant policy was applied as-is, yielding delay_ms=0.
2. Worker scale-up under DB contention (observed effect, inferred mechanism). Increasing workers to 90 at 09:14 made queue depth rise faster and coincided with the DB reaching its 300-connection cap and the API losing DB access.
3. Shared database with no connection reserve for the API tier (inferred from the api-03 error). Worker-side saturation spilled over into checkout latency.
4. Rollback scope mismatch (observed timing, inferred lesson). Rolling back the API at 09:24 did not change worker behaviour; workers still evaluated feature_flag=true at 09:31. The flag was not disabled until 09:43, 19 minutes later. Whether workers were part of the rollback is unknown.
5. Uneven flag rollout (partly observed). US was at 2%; eu-west's percentage is unknown but evidently high enough for region-wide impact, with no staged ramp visible in the timeline.
6. Detection gap (observed). The diagnostic signal (policy="", delay_ms=0, retry_budget_exhausted) was logged at WARN at 09:06:58; alerts fired on downstream symptoms four minutes later, and identifying the flag took a further 32 minutes.
7. Retry-dependent workload (observed). Jobs return retry in normal operation, so retry policy defects convert transient failures into hard failures.

7. What remains unknown

• Why policy was empty: missing tenant config entries, resolver bug, or eu-west config propagation. Which tenants and what fraction were affected.
• The eu-west flag rollout percentage, and whether the US 2% tenants saw any policy="" events or sub-threshold degradation.
• The stall mechanism: what a worker does after retry_budget_exhausted (ack, nack, requeue, hold), whether DB connections are released, and why job 319 stayed on worker-51 for ~8 minutes with its attempt counter later reset to 1.
• What the 09:24 rollback actually reverted (API only, or workers too), and whether the retry logic lives in a shared library or is driven by config.
• Whether "new job creation normalizes" means creation had been failing or elevated, and whether duplicate jobs were created.
• The cause of the underlying transient retry results (job 81 failed three times in 41 ms, far faster than a 5 s pool wait, so those failures were not pool waits).
• Database connection accounting during the window: connections held by workers versus API, and whether any leaked.
• Final disposition of every job that exhausted its retry budget between 09:06 and 09:45, and the customer-facing error rate (not just latency) on checkout.
• Whether the log excerpt is complete for the workers shown; gaps (e.g. worker-51 between 09:35:12 and 09:43:07) may be filtering rather than silence.

Task 09 — Incident analysis

Analyze TIMELINE.md and LOGS.txt. Produce:

• INCIDENT.md: impact, concise timeline, most likely root cause, contributing factors, evidence supporting or weakening each hypothesis, and what remains unknown.
• ACTIONS.md: immediate containment plus prioritized corrective actions with owners expressed as roles, verification signals, and rollback considerations.
• CUSTOMER_UPDATE.md: a plain-language update under 180 words. Do not claim certainty beyond the evidence or blame an individual.

Separate observed facts from inference. Avoid treating correlation as proof. Write RESPONSE.md with your confidence level and the next two pieces of data you would request. Work only in this directory.

Queue incident timeline (UTC)

• 09:02 deploy api 2026.09.23.1; change adds tenant-specific retry policy.
• 09:07 queue depth begins rising in eu-west; worker CPU falls from 65% to 18%.
• 09:11 alerts fire for job age and checkout latency.
• 09:14 on-call increases workers from 40 to 90; queue depth rises faster.
• 09:20 database connections reach configured maximum of 300.
• 09:24 API deploy rolled back; new job creation normalizes, backlog still stuck.
• 09:31 one worker pool restarted; it processes jobs normally for four minutes, then stalls.
• 09:43 feature flag tenant_retry_policy disabled; recovery begins within two minutes.
• 10:18 backlog cleared.
• US region uses the same release but had the flag enabled for only 2% of tenants; no alert fired there.

Response note

Confidence: medium-high (not certain)

I judge the flag-gated empty retry policy (policy="" → delay_ms=0) causing retry-driven database saturation to be the most likely cause, but correlation is not proof. The timing (onset after deploy, recovery within ~2 minutes of flag disable), the same job switching from zero-delay exhaustion to 1,000 ms backoff plus success across the flag flip, the restart-then-restall pattern while the flag stayed on, and the US region running the same release at 2% flag exposure with no alert all point the same way. Against certainty: the log sample is tiny, policy="" is unexplained without code/config, aggregate retry and tenant distributions are missing, and DB topology is unknown, so an independent DB-side contribution cannot be fully excluded.

Next two pieces of data I would request

1. Full worker retry telemetry plus the policy-resolution code and configuration: aggregate delay_ms, retry_budget_exhausted, and success/failure counts by tenant and time (09:00–10:30, eu-west vs. US), with the code path that maps a tenant to policy="" and then to delay_ms=0, and the flag's rollout history per region/tenant. This would confirm or refute the empty-policy mechanism and show whether one tenant or all tenants drove the storm.
2. Database and queue resource time series with topology: DB connections, pool waits, active/idle time, queries and slow queries by source (API vs. workers) overlaid with queue depth, worker count, and worker CPU, plus whether eu-west and US share a database and the per-service connection budgets. This would establish whether DB saturation was the mediating bottleneck and why scaling workers made the queue rise faster.

Corrective actions — eu-west queue stall 2026-09-23

Owner convention: roles, not individuals.

Immediate containment (already taken — verify they hold)

1. Keep tenant_retry_policy disabled in all regions until the fix is validated.
   • Owner: Release Manager / Feature Flag Owner.
   • Verification: flag state reads false everywhere; worker logs show no policy="" and no delay_ms=0; retry_budget_exhausted rate at baseline.
   • Rollback: re-enabling is the rollback of this containment — do not re-enable except under the staged rollout below.
2. Confirm full recovery and job disposition.
   • Owner: On-call / SRE.
   • Verification: queue depth and job age at baseline since 10:18 with no rebound; worker CPU back to ~65%; DB connections well below 300 with no pool-wait warnings; audit dead-letter/dropped jobs from 09:07–10:18 and replay or notify as policy requires.
   • Rollback: n/a (observational).
3. Freeze related deploys and flag changes until reviewed.
   • Owner: Incident Commander.
   • Verification: deploy/flag audit log shows no retry-policy changes.
   • Rollback: lift freeze explicitly after P0 fixes land.

Prioritized corrective actions

P0 — Fix unsafe retry fallback (root-cause fix)

• Action: when tenant policy is missing/empty/unparseable, fall back to the configured default (exponential) instead of zero delay; reject delay_ms=0 with a validated minimum backoff plus jitter; add unit + integration tests for empty/missing/legacy tenant configs.
  • Owner: Backend / Worker Service Owner.
  • Verification: tests cover policy="", missing tenant, malformed policy; staging chaos test shows delay_ms >= floor always; canary shows zero policy="" / delay_ms=0 events.
  • Rollback: revert to prior worker build and keep flag off; rollback signal is any recurrence of delay_ms=0 or retry_budget_exhausted spike on canary.

P0 — Cap retry amplification

• Action: enforce per-job minimum delay, per-worker retry rate limit, and a time-based retry budget (not just attempt count); add a circuit breaker that pauses retries for a tenant/worker when exhaustion rate spikes.
  • Owner: Backend / Worker Service Owner.
  • Verification: load test with forced downstream errors shows bounded retry rate and stable DB connections; breaker trips and recovers in staging.
  • Rollback: disable breaker via flag if it false-trips; keep minimum-delay floor in place (do not roll back to zero-delay behavior).

P0 — Protect the database bottleneck

• Action: split and cap connection budgets (API vs. workers, per pool) so workers cannot starve the API and vice versa; set per-worker pool max consistent with DB max (e.g., total possible < 300 with headroom); add pool-wait and connection-use alerts.
  • Owner: Database Reliability Engineer with SRE.
  • Verification: worst-case connection math documented; saturation test shows graceful queueing with alerts before errors; remaining connection slots never recurs below declared capacity.
  • Rollback: restore prior pool settings if throughput regresses; change one pool at a time during low traffic.

P1 — Safe scaling and backpressure

• Action: block worker scale-up when pool waits or DB saturation are active; scale on job age only when downstream headroom exists; document "do not scale into saturation" in the runbook.
  • Owner: SRE.
  • Verification: game-day replay of 09:14 shows autoscaler/runbook refusing scale-up and paging instead.
  • Rollback: revert autoscaler policy change; manual scaling remains available.

P1 — Staged flag rollout with auto-halt

• Action: roll out tenant_retry_policy progressively (e.g., 1% → 10% → 50% → 100% per region) with automatic halt/rollback on guardrails: delay_ms=0 count, retry_budget_exhausted rate, queue age, worker CPU drop, DB waits.
  • Owner: Release Manager / Feature Flag Owner with SRE.
  • Verification: staged rollout in US first with halt drill; eu-west re-enable only after P0 fixes and soak period.
  • Rollback: one-click flag disable; pre-tested kill switch.

P1 — Detection and alerting gaps

• Action: alert on leading indicators, not just lagging job age: rate of retry_budget_exhausted, any delay_ms=0, worker CPU drop concurrent with queue rise, DB pool-wait p99, connection utilization >80%.
  • Owner: SRE / Observability Owner.
  • Verification: replay 09:06–09:11 logs against new rules — alert would have fired by ~09:07.
  • Rollback: tune thresholds if noisy; do not delete without replacement.

P2 — Post-incident hygiene

• Action: publish incident review; reconcile dead-lettered jobs; add a worker/DB saturation load test to CI; separate API-deploy rollback from worker/flag state in runbooks.
  • Owner: Incident Commander (review); Backend Owner (tests); SRE (runbooks).
  • Verification: review approved; job audit complete; new load test passing.
  • Rollback: n/a.

Customer update — checkout and job delays in eu-west

On September 23, between about 09:07 and 10:18 UTC, some customers in eu-west saw slow checkouts and delayed background jobs. Our systems alerted us at 09:11, and the backlog was fully cleared by 10:18. Customers in the US were not affected.

What happened, as far as we can tell so far: after a software change, some jobs began retrying too aggressively, which overloaded a shared database and slowed processing. Scaling up did not help at first. Service began recovering shortly after we turned off the related feature, and then returned to normal.

We are still investigating the full cause and checking whether any jobs need to be replayed. Next, we are adding safer retry defaults, better overload protection, and earlier alerts. We will share another update once our review is complete. Thank you for your patience.

Incident report — eu-west job queue stall — 2026-09-23

Convention in this file: Observed = directly stated in TIMELINE.md / LOGS.txt. Inference = interpretation; labeled as such.

Impact (observed + bounds)

Observed:

• Region affected: eu-west. US region had no alert (same release, flag at 2% of tenants).
• Queue depth began rising ~09:07 UTC; backlog cleared 10:18 UTC (~71 minutes of degraded queueing).
• Alerts fired 09:11 for job age and checkout latency — checkout path was user-impacted.
• Worker CPU fell 65% → 18% while queue rose: workers were stalled/blocked, not keeping up.
• Database connections reached configured maximum (300) at 09:20; API logged remaining connection slots reserved at 09:19:52; workers logged DB pool waits (5,000 ms, 0 idle).
• Some jobs still succeeded during the incident (e.g., job 92, tenant beta, 09:07:03, 830 ms), so impact was partial, not a full stop — exact success/failure split is unknown.

Unknown (not in evidence): number of delayed/failed jobs, delay distribution, number of affected tenants/customers, checkout latency magnitude or failed checkouts, whether exhausted jobs were lost or later succeeded, any data-integrity effects.

Concise timeline (observed, UTC)

• 09:02 — API 2026.09.23.1 deployed; change adds tenant-specific retry policy.
• 09:06:58 — worker-17 logs job 81 (tenant acme): 3 retry attempts with delay_ms=0 in the same second, then retry_budget_exhausted policy="" elapsed_ms=41.
• 09:07 — Queue depth rises (eu-west); worker CPU 65% → 18%.
• 09:07:01 — worker-22: DB pool wait exceeded (5 s, active=10, idle=0).
• 09:11 — Alerts: job age, checkout latency.
• 09:14 — Workers scaled 40 → 90 (inference: manual mitigation); queue then rises faster.
• 09:19–09:20 — DB contention: API connection-slot error; DB connections at max 300.
• 09:24 — API deploy rolled back; new job creation normalizes, backlog stays stuck.
• 09:31 — One worker pool restarted; processes normally ~4 minutes, then stalls. Restarted worker shows config_retry_default="exponential" feature_flag=true.
• 09:35:12 — Same restarted worker logs retry_budget_exhausted policy="" again.
• 09:43:07 — Feature flag tenant_retry_policy disabled; worker logs config_changed feature_flag=false.
• 09:43:09–09:43:10 — Same job 319: retry with delay_ms=1000, then success (190 ms). Recovery begins within ~2 minutes.
• 10:18 — Backlog cleared.
• US: same release, flag at 2% of tenants, no alert.

Most likely root cause (inference, medium-high confidence)

Enabling tenant_retry_policy caused affected workers to resolve an empty retry policy (policy="") with zero backoff (delay_ms=0), producing rapid retries that exhausted the per-job retry budget in ~40 ms. The resulting retry amplification plausibly saturated the database connection pool (300 max), causing workers to block on pool waits (CPU drop, queue growth). Scaling workers increased DB demand and worsened the stall. Disabling the flag restored the default exponential backoff (1,000 ms delay observed) and recovery followed within minutes.

This is the best-fit inference, not proven causation: timing is correlational, the log sample is small, and the code path resolving policy="" was not examined. See hypotheses and unknowns below.

Contributing factors (inference grounded in observations)

1. Unsafe fallback: empty/missing tenant policy appears to yield zero delay instead of the configured exponential default.
2. No retry floor: delay_ms=0 retries with no minimum backoff or jitter allowed a tight retry burst.
3. Shared, capped downstream: DB max 300 connections became the bottleneck; API and workers contend for the same slots (both logged saturation).
4. Scale-up without backpressure: adding workers (40 → 90) during pool-wait/connection saturation increased contention; queue rose faster afterward.
5. Flag scoping: eu-west appears broadly exposed while US was at 2% — no progressive rollout or auto-halt on queue/CPU/DB signals is evident.
6. Split mitigation surface: API rollback (09:24) did not change worker/flag behavior, so the backlog stayed stuck; worker restart helped only transiently (~4 min).

Hypotheses — evidence for and against

H1 — Flag-gated empty retry policy → retry storm → DB exhaustion → worker stall (favored)

Supporting (observed):

• delay_ms=0 × 3 + retry_budget_exhausted policy="" in <1 s, twice (09:06:58, 09:35:12), both while flag was on.
• After feature_flag=false, same job shows delay_ms=1000 then success — behavior change coincides with flag change on the same worker.
• Recovery begins within ~2 min of flag disable; backlog clears by 10:18.
• Worker restart with feature_flag=true stalls again after ~4 min — consistent with a flag-dependent load trigger, not a one-time poisoned state.
• US on same release with 2% flag exposure had no alert — weakens "release alone" and is consistent with "flag exposure dose matters."
• CPU fall + pool waits + connection max + worse-after-scale-up match downstream-saturation mechanics.

Weakening / missing:

• Small log sample (a handful of jobs); no aggregate retry-rate or per-tenant failure distribution.
• No code/config showing how policy="" arises or that zero-delay retries issue DB load per attempt.
• CPU drop is consistent with blocking but does not by itself prove DB was the first bottleneck.

H2 — The API release itself (bad job creation / poison jobs), independent of the flag

Supporting:

• Onset ~5 min after deploy (09:02 → 09:07) — temporal correlation.

Weakening:

• API rollback (09:24) normalized new job creation but did not unstick the backlog; recovery tracked flag disable (09:43), not rollback.
• US ran the same release with no alert.
• Retried-then-succeeded behavior on the same job across the flag flip points to worker retry handling, not permanently poisoned job payloads.

H3 — Independent database failure or capacity regression (not caused by retries)

Supporting:

• DB saturation is directly observed (pool waits, slot errors, 300/300).

Weakening:

• No DB change is in the timeline; saturation onset tracks the deploy/flag window, and recovery tracks flag disable with no recorded DB intervention.
• Restart-then-restall and US-no-alert patterns are unexplained by a purely DB-side fault unless DBs are regionally isolated (unknown — see below).

H4 — Worker leak / resource exhaustion unrelated to retry logic

Supporting:

• Restart produced ~4 min of normal processing — consistent with clearing leaked/accumulated state.

Weakening:

• Re-stall in 4 min is fast for a classic slow leak and matches re-saturation under continued bad retry load; permanent recovery required flag disable, not repeated restarts.

H5 — Single-tenant poison (e.g., only acme)

Supporting:

• The earliest failing job is tenant acme while tenant beta succeeds once at 09:07:03.

Weakening:

• Broad queue/DB/checkout impact and a second exhausted job (319) are inconsistent with one tenant unless acme dominates load (tenant mix unknown). policy="" on multiple jobs suggests a systemic fallback, not one bad tenant row.

What remains unknown

• What policy="" means (missing tenant row? parse failure? unmigrated default?) and the exact code path from policy to delay_ms=0.
• Aggregate blast radius: jobs delayed/failed vs. succeeded, per-tenant distribution, checkout latency/error counts, customer count.
• Final disposition of budget-exhausted jobs: dropped, dead-lettered, or retried to success after 09:43.
• DB topology: shared vs. regional; per-service connection split (API vs. workers); queries per retry attempt; what held connections (idle-in-transaction? slow queries?).
• Worker topology: pools, threads, and pool sizing (log shows active=10/worker — 40–90 workers × 10 would oversubscribe 300 even before retries).
• Flag rollout history: when and to whom tenant_retry_policy was enabled in eu-west vs. US; who/what set it to 2% in US.
• Why new job creation normalized after API rollback while backlog stayed stuck (separate API vs. worker flag reads? cached policy?).
• Whether 09:24–09:43 backlog processing was zero or merely slow; queue-depth numbers over time.

Task 09 — Incident analysis

Analyze TIMELINE.md and LOGS.txt. Produce:

• INCIDENT.md: impact, concise timeline, most likely root cause, contributing factors, evidence supporting or weakening each hypothesis, and what remains unknown.
• ACTIONS.md: immediate containment plus prioritized corrective actions with owners expressed as roles, verification signals, and rollback considerations.
• CUSTOMER_UPDATE.md: a plain-language update under 180 words. Do not claim certainty beyond the evidence or blame an individual.

Separate observed facts from inference. Avoid treating correlation as proof. Write RESPONSE.md with your confidence level and the next two pieces of data you would request. Work only in this directory.

Queue incident timeline (UTC)

• 09:02 deploy api 2026.09.23.1; change adds tenant-specific retry policy.
• 09:07 queue depth begins rising in eu-west; worker CPU falls from 65% to 18%.
• 09:11 alerts fire for job age and checkout latency.
• 09:14 on-call increases workers from 40 to 90; queue depth rises faster.
• 09:20 database connections reach configured maximum of 300.
• 09:24 API deploy rolled back; new job creation normalizes, backlog still stuck.
• 09:31 one worker pool restarted; it processes jobs normally for four minutes, then stalls.
• 09:43 feature flag tenant_retry_policy disabled; recovery begins within two minutes.
• 10:18 backlog cleared.
• US region uses the same release but had the flag enabled for only 2% of tenants; no alert fired there.

Response assessment

Confidence: Moderate-high that the tenant_retry_policy feature-flagged path caused or enabled the incident. The flag disablement was followed by recovery, and the logs show zero-delay retries while enabled followed by a delayed retry after disablement. Confidence is moderate in the full mechanism from rapid retries through database saturation to the queue stall: the supplied evidence is sparse and cannot exclude an independent or additional database/worker issue.

The next two data requests are:

1. The exact 2026.09.23.1 code/config diff and effective retry-policy resolution for affected eu-west tenants, including why the logged policy was empty and the feature-flag history by tenant and region.
2. Synchronized eu-west time-series or traces from 09:00–10:30 UTC for attempts/retry delays, queue depth/age and throughput, worker concurrency/state, database connections/pool waits, checkout latency, and flag exposure. This would show whether retry amplification preceded and tracked the saturation and recovery.

Incident actions

Immediate containment

1. Keep tenant_retry_policy disabled across regions until policy resolution and backoff are corrected. Owner: Incident Commander / on-call service owner. Verify: queue depth and oldest-job age fall; job throughput recovers; DB pool waits and active connections return below their limits; checkout latency returns to baseline. Rollback: If disabling the flag causes a separate, verified customer-impacting regression, use the last known-safe retry behavior and re-evaluate with the Incident Commander; do not restore the affected policy path without a canary.
2. Hold worker concurrency at a controlled level and avoid further blind scale-ups while database connections are saturated. Owner: SRE on-call. Verify: connection usage stays below the configured cap, pool wait time falls, and completed-job throughput exceeds incoming work. Rollback: Restore the previous worker count if the cap worsens backlog drain after DB headroom is confirmed; scale in measured increments with connection monitoring.

Prioritized corrective actions

Priority	Action	Owner role	Verification signal	Rollback consideration
P0	Fix retry-policy resolution so a missing or empty tenant policy cannot produce zero-delay retries. Apply a safe exponential backoff with a minimum delay and jitter; validate policy config before serving traffic.	Queue / API service engineer	Regression coverage for missing, empty, malformed, and valid policies; canary shows no zero-delay retry loops and retry-budget behavior matches the configured policy.	Keep the feature flag off and revert the code/config change if the canary shows unexpected retry volume, job latency, or errors.
P0	Add a guarded re-enable plan for the flag: start with a small tenant cohort, expand only after health checks pass, and stop automatically on retry, queue, checkout, or DB-pool thresholds.	Release engineer with SRE	Canary cohort remains within agreed error, latency, queue-age, retry-rate, and DB-pool thresholds at each rollout step.	Disable the flag immediately and return to the known-safe policy on any threshold breach.
P1	Bound retry amplification with per-job attempt limits, per-tenant rate/concurrency limits, and a circuit breaker or equivalent pause for persistent dependency failures.	Queue reliability engineer	Load/failure tests demonstrate bounded attempts and DB demand; production retry rates and connection use remain capped during dependency degradation.	Revert individual guardrails if they suppress valid work, while retaining the safe backoff and feature flag.
P1	Instrument retry behavior and worker/DB pressure together: effective policy, attempts and delay, queue age/depth, worker state/concurrency, DB pool active/idle/wait, and flag exposure. Alert on rapid retries plus rising queue age or pool waits.	Observability engineer / SRE	Dashboard correlates these signals by region and tenant; a controlled fault test fires the intended alerts before the connection cap is reached.	Revert noisy alert thresholds or dashboard changes; retain raw metrics/log fields needed to diagnose retry behavior.
P2	Update incident and release runbooks with this containment sequence, including when to disable a feature flag, how to cap workers against DB headroom, and recovery checks.	Service owner / SRE lead	Tabletop or game-day confirms responders can identify the retry pattern, contain it, and verify backlog drain without exceeding DB capacity.	Revert confusing runbook edits and retain the last reviewed procedure; no runtime rollback required.

Do not declare full recovery until queue depth and oldest-job age are normal, DB pool waits are clear, and checkout latency is stable. Confirm job completion and integrity before closing the incident.

Customer update

On September 23, some customers in our EU region experienced slower checkout responses and delays in queued work. We traced the issue to a recently enabled retry-policy path that caused some jobs to retry too quickly and contributed to pressure on our processing and database capacity. We disabled that path at 09:43 UTC, processing recovered, and the backlog was cleared by 10:18 UTC. We are reviewing the retry behavior and adding safeguards before enabling it again. We have not yet confirmed the full number of affected customers or whether any jobs were lost or duplicated. We will share an update if our review finds additional customer impact.

Queue incident analysis

Impact

Observed: The timeline records a growing eu-west queue from 09:07 UTC, alerts for job age and checkout latency at 09:11, and database connections reaching the configured maximum of 300 at 09:20. Rolling back the API deployment normalized new job creation, but the backlog remained stuck. Recovery began within two minutes of disabling tenant_retry_policy at 09:43; the backlog was cleared by 10:18. The US region ran the same release with the flag enabled for 2% of tenants and had no alert.

Inferred: EU customers likely experienced slower checkout and delayed queued work. The evidence does not establish the number of affected customers or jobs, whether any jobs were lost, or whether checkout requests failed. The incident window for the reported backlog was about 71 minutes (09:07–10:18 UTC); checkout impact may have started later or ended earlier.

Concise timeline (UTC)

• 09:02 — Observed: API 2026.09.23.1 deployed with a tenant-specific retry policy change.
• 09:06:58–09:07 — Observed: A worker logged three immediate retries for one job (delay_ms=0), then retry_budget_exhausted with an empty policy. Queue depth began rising and worker CPU fell from 65% to 18%.
• 09:11 — Observed: Job-age and checkout-latency alerts fired.
• 09:14 — Observed: Worker count increased from 40 to 90; the queue then rose faster.
• 09:20 — Observed: Database connections reached the configured maximum of 300; a worker log also shows a 5-second DB-pool wait.
• 09:24 — Observed: API rollback normalized new job creation, but the backlog remained stuck.
• 09:31–09:35 — Observed: A restarted worker pool processed normally for four minutes, then stalled. Startup logged the default retry mode as exponential while the feature flag was true; a later job again exhausted its retry budget with an empty policy.
• 09:43 — Observed: The flag was disabled. Within two minutes recovery began. The same job then retried after 1,000 ms and succeeded on its second attempt.
• 10:18 — Observed: Backlog cleared.

Most likely root cause

Inference — moderate-high confidence: The feature-flagged tenant retry-policy path most likely caused or enabled the eu-west failure. The strongest mechanism indicated by the logs is that an empty/unresolved tenant policy allowed rapid retries with zero delay, exhausting retry budgets within tens of milliseconds and adding pressure while workers were already waiting on the database. Disabling the flag changed the observed retry behavior to a 1,000 ms delay and was followed closely by recovery. The supplied evidence does not prove the precise code defect or establish that retries alone caused the database connection limit to be reached.

Hypotheses and evidence

H1: The flagged tenant retry-policy path triggered retry amplification and queue stalls — most likely

• Supports (observed): The release introduced the policy change; a worker logged three zero-delay attempts followed by an exhausted budget and policy="". After a restart, the flag remained true and the same empty-policy warning recurred. Following flag disablement, the logged retry used a 1,000 ms delay and succeeded. The timeline says recovery began within two minutes of disablement.
• Weakens / limits: Only a few log lines are provided, with no retry-rate totals, tenant-level policy resolution, or direct trace from retries to DB connections. The logs do not show why the policy was empty. The 1,000 ms retry and success are one job, not proof that all jobs followed this path.

H2: Database connection exhaustion was the primary cause

• Supports (observed): The connection limit of 300 was reached, and a worker reported no idle DB connections and a 5-second pool wait.
• Weakens / limits: Queue growth and worker CPU decline began before the reported connection maximum. API rollback did not clear the stuck backlog, whereas flag disablement was followed by recovery. Database saturation is strongly supported as a proximate condition or contributing factor, but a separate database fault is not established.

H3: The API release caused a job-creation defect independent of the flag

• Supports (observed): Queue growth began five minutes after the API deployment; rollback normalized new job creation.
• Weakens / limits: The backlog stayed stuck after rollback and recovery followed flag disablement. The US region ran the same release without an alert at 2% flag exposure. This points toward exposure or workload differences, but does not isolate the flag because regional conditions are not provided.

H4: Insufficient worker capacity or a worker-pool defect was the initiating cause

• Supports (observed): Worker CPU fell, scaling the pool coincided with faster queue growth, and a restart only helped temporarily.
• Weakens / limits: These changes may be responses to the retry/DB pressure rather than independent causes. No worker saturation, scheduling, or pool-health diagnostics are provided. Increasing workers may have added concurrent DB demand, but that is an inference.

What remains unknown

• The exact deployed code/configuration and how empty tenant policies were resolved.
• Retry volume, retry delay distribution, affected tenants/jobs, and per-region flag exposure over time.
• Whether database connection growth was caused by retry amplification, worker scaling, a leak, or another workload.
• The number of customers affected, checkout failure rate, job loss/duplication, and any data integrity impact.
• Why the worker pool stalled again after its restart and how much of the 10:18 recovery was due to backlog draining versus restored processing.

Task 09 — Incident analysis

Analyze TIMELINE.md and LOGS.txt. Produce:

• INCIDENT.md: impact, concise timeline, most likely root cause, contributing factors, evidence supporting or weakening each hypothesis, and what remains unknown.
• ACTIONS.md: immediate containment plus prioritized corrective actions with owners expressed as roles, verification signals, and rollback considerations.
• CUSTOMER_UPDATE.md: a plain-language update under 180 words. Do not claim certainty beyond the evidence or blame an individual.

Separate observed facts from inference. Avoid treating correlation as proof. Write RESPONSE.md with your confidence level and the next two pieces of data you would request. Work only in this directory.

Queue incident timeline (UTC)

• 09:02 deploy api 2026.09.23.1; change adds tenant-specific retry policy.
• 09:07 queue depth begins rising in eu-west; worker CPU falls from 65% to 18%.
• 09:11 alerts fire for job age and checkout latency.
• 09:14 on-call increases workers from 40 to 90; queue depth rises faster.
• 09:20 database connections reach configured maximum of 300.
• 09:24 API deploy rolled back; new job creation normalizes, backlog still stuck.
• 09:31 one worker pool restarted; it processes jobs normally for four minutes, then stalls.
• 09:43 feature flag tenant_retry_policy disabled; recovery begins within two minutes.
• 10:18 backlog cleared.
• US region uses the same release but had the flag enabled for only 2% of tenants; no alert fired there.

RESPONSE.md

Confidence

Overall: moderate-to-high for the causal chain, low for the specific defect.

The evidence base is two files totalling roughly 30 lines. That is enough to reconstruct what happened with reasonable confidence, and not enough to pin the specific code or config defect. I have split the confidence deliberately rather than giving a single number, because the two halves are genuinely different in strength.

High confidence (directly observed, or falsified by a failed test):

• An unresolved retry policy produced zero backoff. policy="" with delay_ms=0 on all three attempts, and the same worker/job pair flipping to delay_ms=1000 two seconds after the flag was disabled. This is the closest thing to a controlled comparison available.
• Disabling tenant_retry_policy ended the incident; the backlog drained in 35 minutes.
• The retry storm preceded the queue growth, not the other way round: 09:06:58 vs 09:07.
• The API binary was not the whole cause. Its 09:24 rollback failed to recover service for 19 minutes. This is a real falsification, not a correlation.
• Corrupted worker state is ruled out. A freshly restarted pool worked normally for four minutes, then reproduced the identical policy="" failure with a correct config_retry_default="exponential". Note what that simultaneously proves: the default was healthy, so the fault is in the tenant-specific override path.
• A runaway CPU loop is ruled out. CPU fell 65% → 18%; workers were blocked on I/O.
• Workers were starving before the API was (09:07:01 vs 09:19:52), and pure database capacity is ruled out as root cause — the 300-connection cap was not reached until 13 minutes after degradation began.

Moderate confidence (inference, plausible, not proven):

• That zero-delay retries consumed connections by rate rather than through a connection leak on the failure path. Both produce the same symptoms; the inputs contain no connection-lifecycle data at all. This is the weakest load-bearing link in the chain and the reason I request metric data below.
• That the 40 → 90 scale-up amplified rather than caused the incident. Cause is cleanly excluded on timing. Amplification is likely but confounded: queue depth is (arrival rate − service rate), so faster growth could partly reflect rising arrivals.
• That us-east saw no impact because exposure was 2%. I would only claim the weaker statement: no alert fired there, which is not evidence of no impact. 2% exposure may simply sit under the alert threshold.

Low confidence / unresolved:

• Whether the fault is a code defect or a bad config value (H5). The two readings predict identical evidence so far and I am explicitly not claiming the code is at fault over the configuration.
• The trigger condition. First failure came ~5 min after deploy, and ~4 min after a worker restart. Something recurring provokes it, and I do not know what.
• Customer impact magnitude, and whether any job failed permanently, was dropped, or ran twice. Entirely absent from the inputs. I have not assumed "no data loss" — absence of evidence here is not evidence of absence.
• Whether the "new job creation normalizes" at 09:24 was caused by the rollback or coincided with it. Single region, no control.

Bounded by data access: this analysis is based only on TIMELINE.md and LOGS.txt. No external systems, logs, metrics, or configuration were consulted, and the logs are a 13-line sample, not a complete record — absence of a signal in this sample is weak evidence about the system.

---

The next two pieces of data I would request

1. The tenant retry-policy resolver diff for 2026.09.23.1, together with the per-tenant policy map as it actually stood at 09:02.

Specifically: the diff of the policy resolution path, and the config store's version history showing the full map contents at deploy time — including the entry for tenant acme and at least one tenant with no entry, plus the beta tenant for contrast against the one logged success at 09:07:03.

Why this is first: it is the only request that can resolve H5, which is the difference between a code fix and a config fix. It distinguishes a missing map entry, a tenant-key mismatch, a parse failure, and an error swallowed into an empty string — and it tells me whether the defect is reachable for any tenant or only for tenants with partial configuration. It also directly tests whether the tenant boundary (acme failing while beta succeeded) is load-bearing or coincidental. Nothing else in my analysis can substitute for it, and most of the P0 actions in ACTIONS.md change shape depending on the answer.

2. A 09:00–10:20 UTC time series for both eu-west and us-east: per-tenant retry attempt rate, database connection acquisition rate and hold duration, per-worker pool size against the global connection gauge, and the pool-wait histogram.

Why this is second: it is the only way to close the weakest link in the chain — whether connections were exhausted by retry rate or by a leak. It resolves the puzzle that worker-22 reported pool wait exceeded at active=10 idle=0 while the global cap of 300 was still 13 minutes away, which implies a worker-local constraint that is currently undocumented. It quantifies what the 40 → 90 scale-up actually added on top of the existing load, and comparing against us-east at 2% exposure is the only control available to test whether eu-west behaved differently for reasons other than exposure. It also gives the missing impact numbers — affected tenant count, retry volume, and how much of the delay each job absorbed — which is what a defensible customer-facing impact statement requires.

Between them, these two would move the specific defect from low confidence to high, and would let me put a real number on the affected population instead of an unquantified region.

ACTIONS.md — Corrective actions

Owners are roles, not individuals. Priorities are by risk reduction per unit of effort, not by effort alone. Every action carries a verification signal that is observable without the author's judgement, and a rollback note — a corrective action that cannot be independently reverted is itself a new risk.

Derived from INCIDENT.md. Root cause accepted at the level of chain, not specific defect, so every fix below must hold regardless of which of H5's two readings is correct.

---

Part 1 — Immediate containment (state at end of incident)

The incident is closed out: backlog cleared at 10:18. These are the containment positions to hold while Part 2 lands.

#	Containment action	Role	Status
C1	Keep tenant_retry_policy disabled in all regions. Do not re-enable until P0-1 and P0-3 are verified in a canary.	Release engineering	In effect since 09:43 — hold
C2	Pin the queue service to release 2026.09.23.1-pre retry code. Verify no tenant anywhere still has the flag on, including eu-west stragglers.	Queue service backend	Verify, do not assume
C3	Freeze deploys and worker-count changes to the queue service until the P0s are complete.	SRE on-call lead	New
C4	Return eu-west worker count to the pre-incident 40 until the connection-budget change (P1-1) lands, and confirm the queue is stable at that count.	SRE on-call lead	New
C5	Publish the incident to the on-call channel with a pointer to INCIDENT.md, and state explicitly that the 09:24 rollback was not an effective mitigation.	Incident commander	New

C2 caveat: the 09:24 API rollback left the system broken for 19 minutes, so a rollback alone is not evidence of a reverted state. C2 requires a positive check (resolved policy non-empty, backoff non-zero on a synthetic retry), not a version number.

---

Part 2 — Prioritized corrective actions

P0 — before any re-enablement

---

P0-1 · Make an unresolved retry policy fall back to the safe default, never to zero backoff

• Action: in the tenant retry-policy path, treat "policy not found / empty / unparseable" as a resolution failure and apply config_retry_default (already exponential and demonstrably correct in production at 09:31:10). Never allow a resolved policy to yield delay_ms=0. Add a hard floor on first-retry backoff independent of any policy value.
• Owner: Queue service backend
• Verification signal: unit test asserting that a tenant with no policy entry resolves to exponential with delay_ms >= floor; staging run with the flag on for a synthetic tenant produces delay_ms > 0 on first retry; metric retry_resolve_fallback_total increments instead of the policy silently resolving empty.
• Rollback: single revertible commit; the feature flag remains an independent kill switch, so reverting the code while the flag is off is a no-op. Must be reverted alone, never bundled with another change.

---

P0-2 · Enforce a global retry ceiling and a circuit breaker on the DB path

• Action: cap retry attempts globally and per tenant so no tenant's failure pattern can monopolize shared connections. Add a breaker that trips on pool-wait pressure. Make retry_budget_exhausted actually stop the loop — dead-letter or park the job, and record the disposition.
• Owner: Platform reliability, with Queue service backend
• Verification signal: fault-injection in staging — a tenant whose jobs fail 100% of the time — leaves eu-wide retry rate under the configured ceiling, DB pool wait p99 below threshold, and healthy tenants' p99 checkout latency unchanged. Dead-letter counts are non-zero and match injected failures.
• Rollback: thresholds are config-driven behind a flag; revert to the previous threshold set without a deploy. Verify the previous values are still known-good — do not rely on "git revert the whole service".

---

P0-3 · Validate retry configuration at load time and refuse to start on an invalid value

• Action: schema-validate the tenant policy map at worker startup. Reject or explicitly fall back on empty values, and log loudly rather than accepting "". A worker should never start in a state where its own policy is empty.
• Owner: Configuration / config-management, with Queue service backend
• Verification signal: staging worker started with an empty policy entry either refuses startup or starts on the documented default and increments retry_policy_empty_total; no worker in staging or production ever logs policy="" again.
• Rollback: validation strictness is toggleable; keep the toggle and its current value documented. Note the failure mode to avoid — making validation so strict that a malformed unrelated entry blocks all workers, which converts a degraded queue into a stopped one.

---

P0-4 · Audit job outcomes from the incident window

• Action: determine the final disposition of every job processed 09:07–10:18: succeeded, permanently failed, dead-lettered, dropped, duplicated, or executed more than once. Reconcile against checkout and payment totals. Contact affected customers; remediate duplicate or failed charges.
• Owner: Support engineering, with Data engineering
• Verification signal: audit returns a count per disposition class for the window; checkout and payment totals reconcile against the job ledger; a named owner confirms customer contact was made for every discrepancy found.
• Rollback: none needed — read-only analysis plus customer remediation. If the audit is inconclusive, record that explicitly rather than assuming no impact.
• Note: this is P0 despite being non-technical, because the inputs contain no evidence about job disposition, and permanent job failure is the failure mode that would actually hurt a customer.

---

P1 — before restoring full traffic

P1-1 · Make worker count and DB connection budget one coordinated setting

• Worker pool sizing, per-worker pool size, and the global 300-connection cap must be derived from a single documented budget. worker-22 showed pool-wait exhaustion at active=10 idle=0 while global usage was far from 300, so the worker-local budget is the binding constraint and is currently undocumented.
• Owner: Database / platform
• Verification signal: load test at the 90-worker configuration keeps global connection usage below the cap with stated headroom and pool-wait p99 under 100 ms; a written "max safe workers" number exists in the runbook.
• Rollback: the ratio is config-driven; restore the prior ratio independently. Reduce rather than remove workers, and watch the queue while doing it.

P1-2 · Require a canary and an automated rollback gate for flag-gated releases

• Flags default off. No region or tenant cohort reaches full exposure without a canary step. Automated rollback triggers on retry rate, empty-policy resolution, and pool wait.
• Owner: Release engineering
• Verification signal: a deliberately faulty retry policy in staging triggers automatic rollback within the configured window; the us-east 2% pattern is documented as the standard rollout shape.
• Rollback: rollback the gate policy only if it blocks a legitimate deploy — never by disabling the gate and the canary simultaneously.

P1-3 · Fix the rollback surface so a deploy revert also reverts flag-gated worker config

• The 09:24 rollback consumed time without recovering service. Rollback procedures must enumerate every component touched by a release, including flag and worker config, and must not report "reverted" until a behavioural check passes.
• Owner: Release engineering, with Queue service backend
• Verification signal: a rollback drill in staging reverts both the binary and flag-gated config, and verification asserts non-zero backoff rather than a version string.
• Rollback: n/a — this changes the rollback procedure itself.

P1-4 · Fix the runbook's first move for queue/backlog incidents

• Remove "scale workers" as a default first action. Add a diagnostic gate: check retry rate and pool wait before scaling. Scaling is correct for a capacity-bound backlog and actively harmful against a shared-resource bottleneck, which is what happened at 09:14.
• Owner: SRE on-call lead / incident management
• Verification signal: runbook reviewed and signed off; the next DB-bound incident timeline shows a diagnostic step before any scaling decision.
• Rollback: n/a — documentation.

P1-5 · Add the missing signals and alert on the causal metric, not the symptom

• Instrument per-tenant retry attempt rate, resolved-policy value as a metric dimension, pool-wait histogram, connection-acquisition rate, and dead-letter counts. Alert on empty-policy resolution and on per-tenant retry rate.
• Owner: Observability / SRE
• Verification signal: a game day reproduces the empty-policy condition and the new alerts fire within 2 minutes — versus the observed 4-minute gap before a symptom alert, and a signal that was never alerted at all.
• Rollback: disable or re-threshold individual alerts; avoid a big-bang dashboard change that removes existing symptom alerts before the new ones exist.

---

P2 — durability

P2-1 · Enforce a maximum per-region flag exposure ceiling

• No region reaches 100% exposure on a retry or queue-affecting flag without an explicit, reviewed exception. Audit all regions' current percentages.
• Owner: Platform
• Verification signal: a config report shows no region above the ceiling without a recorded exception.
• Rollback: n/a — reporting.

P2-2 · Add a failure-path integration test for retry behaviour

• Simulate a failing job under the flag on and assert pacing, attempt cap, and disposition. Block release on this test.
• Owner: Queue service backend
• Verification signal: the test fails against 2026.09.23.1 and passes against the fixed build. (This is the regression test that would have caught the incident at 09:02.)
• Rollback: n/a — test-only.

P2-3 · Review and ratify the postmortem

• Walk INCIDENT.md with on-call, service, and database owners; capture corrections to the unknown list in Part 4 before the next release trains on it.
• Owner: Incident commander
• Verification signal: postmortem signed off by all three functions; the eight open unknowns are each marked resolved or explicitly accepted.
• Rollback: n/a.

---

Part 3 — Sequencing and interlock

• P0-1, P0-2, P0-3 must all land before the flag is re-enabled anywhere, even at the 2% canary level. P0-1 alone stops the storm; P0-2 alone bounds the blast radius if something else recurs; P0-3 alone prevents a recurrence at load time. Any one of them re-enabled alone leaves a known hole.
• P0-4 runs in parallel and is not gated by the engineering fixes. It should not wait behind them — it is the item with direct customer consequences.
• Re-enablement order: eu-west at 2% → hold through one peak → 25% → hold → 100%, with P1-5's alerts live and a named owner watching each step. Do not re-enable in us-east and eu-west simultaneously; the us-east cohort is what made the 2% pattern survivable the first time, and that evidence should not be spent twice in one step.
• Do not bundle corrective actions. Several of these touch the same worker config path; bundling them makes the 09:24 rollback failure mode — an unverifiable state after a partial revert — more likely to repeat.

Customer update — queued order processing, European region

Some customers in our European region saw slower processing of queued work, including longer checkout times, between approximately 09:07 and 10:18 UTC.

A software change released at 09:02 UTC affected how retry settings were resolved for individual accounts. For some accounts, no retry setting was found, and the system retried failed attempts immediately with no pause between them. That extra activity appears to have placed heavy demand on shared database connections, which slowed down work processing.

The effect was limited to our European region. Our US region, where the same change was active for only a small number of accounts, did not report this slowdown.

We rolled back part of the change at 09:24 UTC and fully disabled the affected setting at 09:43 UTC. Processing began recovering within two minutes, and the backlog was cleared by 10:18 UTC.

We are still reviewing whether any individual orders were delayed, repeated, or not completed. If we find any that were affected, we will contact those customers directly.

---

Body: 165 words (excluding this note and the heading).

INCIDENT.md — Queue processing degradation, eu-west

Analysis of TIMELINE.md and LOGS.txt only. No other data sources were consulted. Throughout, FACT marks something stated directly in the inputs and INFERENCE marks my interpretation of it. Hypotheses are stated with the evidence that would confirm or weaken them; no correlation below is treated as proof on its own.

---

1. Impact

Dimension	Assessment	Basis
Customer-visible duration	~71 minutes (09:07 → 10:18 UTC)	FACT (timeline)
Window with confirmed customer symptoms	~09:11 → 10:18 (alerts on job age + checkout latency)	FACT
Time to first alert after first anomaly	4 min (09:06:58 → 09:11)	FACT
Geographic blast radius	eu-west only; us-east used the same release with the flag on for 2% of tenants	FACT
Tenant-level blast radius	Not stated. Inferred to be "all tenants in eu-west" because the flag-off at 09:43 stopped the storm and eu-west saw full impact, vs 2% in us-east	INFERENCE
Symptoms	Growing queue depth, rising job age, elevated checkout latency, DB connection exhaustion	FACT
Throughput	Worker CPU fell 65% → 18% while queue depth rose	FACT
Data loss / duplication / double-charge	Unknown. No log line shows a job permanently failed, dead-lettered, or duplicated	—
Financial / customer-count impact	Unknown. No volume, tenant count, or order value in the inputs	—

Honest scoping of impact: the inputs let me state duration, region, and symptom type with confidence. They do not let me state how many customers or orders were affected, or whether any job was lost. retry_budget_exhausted (09:06:58, 09:35:12) means the retry budget was consumed; the inputs contain no evidence of what the system did with the job afterwards (dead-letter? drop? hold in queue?). That is the single largest customer-facing unknown and it drives the first corrective action in ACTIONS.md.

---

2. Concise timeline (UTC)

Time	Event	Type	Source
09:02	Deploy api 2026.09.23.1; change adds tenant-specific retry policy	FACT	timeline
09:06:58	worker-17: job 81 (tenant acme) attempt 1, 2 and 3 all result=retry, all delay_ms=0, within the same second. retry_budget_exhausted policy="" elapsed_ms=41	FACT	logs
09:07	Queue depth begins rising in eu-west; worker CPU falls 65% → 18%	FACT	timeline
09:07:01	worker-22: db pool wait exceeded wait_ms=5000 active=10 idle=0	FACT	logs
09:07:03	worker-08: job 92 (tenant beta) succeeds, elapsed_ms=830	FACT	logs
09:11	Alerts fire for job age and checkout latency	FACT	timeline
09:14	On-call increases workers 40 → 90; queue depth rises faster	FACT	timeline
09:19:52	api-03: db remaining connection slots reserved	FACT	logs
09:20	DB connections reach configured maximum of 300	FACT	timeline
09:24	API deploy rolled back; new job creation normalizes, backlog still stuck	FACT	timeline
09:31:10	A worker pool is restarted; worker-51 starts with config_retry_default="exponential" feature_flag=true	FACT	logs
~09:35:12	worker-51 (4 min after restart): job 319 retry_budget_exhausted policy="" elapsed_ms=38	FACT	logs
09:43:07	worker-51: config_changed feature_flag=false	FACT	logs
09:43:09	job 319 attempt 1 result=retry — now with delay_ms=1000	FACT	logs
09:43:10	job 319 attempt 2 result=success elapsed_ms=190	FACT	logs
09:43	Flag tenant_retry_policy disabled; recovery begins within 2 minutes	FACT	timeline
10:18	Backlog cleared	FACT	timeline
—	us-east: same release, flag on for 2% of tenants, no alert fired	FACT	timeline

Two ordering details carry most of the analytical weight:

1. 09:06:58 precedes 09:07. The first logged failure is before the timeline's first symptom. The retry storm started first; the queue merely grew afterwards.
2. 09:24 rollback did not clear the backlog while 09:43 flag-off did. These are two attempts to remove the change, one of which failed. That asymmetry is the most informative single fact in the dataset.

---

3. Most likely root cause

Confidence: high for the causal chain, low for the specific defect.

A defect in the tenant-specific retry-policy path introduced by 2026.09.23.1 resolved to an empty policy (policy="") for the affected tenants. An empty policy produced zero backoff (delay_ms=0), so every failed attempt was retried immediately, in a tight loop. Those retries consumed shared database connections faster than the pool could release them, starving workers of connections (pool wait exceeded wait_ms=5000), which throttled job throughput and let the queue grow. Disabling the feature flag at 09:43 restored the working exponential default backoff, retries became paced, and the queue drained.

The chain, step by step — each link and its support:

1. Tenant retry policy resolves to an empty string. FACT that policy="" is logged at 09:06:58 and 09:35:12. INFERENCE that this is the new code path misbehaving, because policy="" appears only in the window where the flag is on, and disappears after the flag is off.
2. Empty policy means no backoff. FACT that all three attempts of job 81 carry delay_ms=0 at 09:06:58, and that the whole budget burned in elapsed_ms=41. INFERENCE that delay_ms is sourced from the resolved policy, so an empty policy yields zero. The 1000 ms delay at 09:43:09 (same worker, same job, flag off) is the positive control.
3. Immediate retries generate connection pressure. INFERENCE, and it is the weakest link in the chain. It assumes each attempt acquires a DB connection and holds it through the failure path. The inputs do not show connection lifecycle. What the inputs do show is the downstream consequence — pool waits and the 300-connection cap — and CPU falling to 18%, which is the signature of threads blocked on I/O rather than spinning. This is a plausible mechanism, not a proven one.
4. Connection starvation throttled throughput. FACT (pool waits, reserved-slot error, cap reached, queue growth, CPU drop). This link is strong.
5. Removing the flag removed the storm. FACT, with a near-controlled comparison: same worker (51), same job (319), four minutes apart, behaviour flips from retry_budget_exhausted policy="" to delay_ms=1000 → success, two seconds after config_changed feature_flag=false.

What the root cause is not: not the API binary alone (its 09:24 rollback failed to recover the backlog while a flag flip succeeded), and not corrupted worker state (the 09:31 restart worked cleanly for four minutes, then reproduced the same policy="" failure). The defect lives in the flag-gated retry configuration path, which the API rollback did not revert.

---

4. Contributing factors

Ordered by how much they amplified impact. These are separate from the root cause: fixing the retry-policy defect would end the incident, but these are what made it large, slow to detect, and slow to fix.

1. No backoff on unresolved configuration (FACT). delay_ms=0 for every attempt. A configuration that resolves to "nothing" degraded into the most aggressive retry behaviour instead of falling back to the safe default that was demonstrably present (config_retry_default="exponential", logged at 09:31:10). Failing open into a thundering herd is the core design failure.
2. Retries continued past retry_budget_exhausted (FACT, INFERENCE on meaning). A retry budget was reached in 41 ms and the job kept cycling. A budget that does not stop the retry loop provides no protection.
3. Unbounded blast radius from flag configuration (FACT). 100% exposure in eu-west vs 2% in us-east on the identical binary. The severity of this incident was decided by configuration, not by code quality — the same release was survivable in us-east only because few tenants had the flag on. "No alert fired in us-east" is not evidence of "no impact in us-east": at 2% exposure the signal may simply have been below the alert threshold.
4. Scaling workers against a shared bottleneck (FACT + INFERENCE). 40 → 90 workers at 09:14 while the bottleneck was a shared, capped resource. The queue rose faster afterwards. INFERENCE: because more workers were blocked on the same constrained pool, the extra workers added connection demand without adding service capacity. Note the confounder: queue depth is (arrival rate − service rate), so faster growth could also reflect rising arrival rate rather than the scaling itself. Moderate confidence that this amplified the incident; it did not cause it, since degradation began 7 minutes earlier.
5. Incomplete rollback surface (FACT). The API rollback at 09:24 left the system in the incident state for another 19 minutes. A deploy rollback that does not revert flag-gated worker configuration gives a misleading "we have reverted" signal. The team lost ~19 minutes on a mitigation that could not work.
6. Restart as an ineffective mitigation (FACT). The 09:31 restart produced 4 good minutes and then reproduced the defect. It also created a false signal that the problem was transient. Useful as a falsification of the corrupted-worker-state hypothesis; wasted as a mitigation.
7. Detection was symptom-based (FACT). The first observable was queue depth at 09:07; the first alert at 09:11. The directly causal signal — a resolved policy that is empty, and retry rate per tenant — was never reported, at any point, by any system or person. It is present in the log data the whole time.
8. An unexplained 5-minute latency between deploy and first failure (FACT about the gap). 09:02 → 09:06:58. INFERENCE: the trigger is a recurring workload pattern rather than a deploy-time event, since a freshly booted worker also took ~4 minutes to reproduce (09:31:10 → 09:35:12). Unresolved: what the trigger condition actually is.

---

5. Hypotheses, with evidence for and against

H1 — Retry storm from unresolved tenant retry policy exhausted DB connections (accepted, high confidence in the chain)

Supporting

• policy="" at 09:06:58 and 09:35:12 — the signal is present in the data twice.
• delay_ms=0 on all three attempts of job 81 within one second, budget exhausted in 41 ms.
• Flag off at 09:43 → delay_ms=1000 on the same job and worker within 2 s → success. This is the closest thing to a controlled experiment in the dataset.
• CPU dropped 65% → 18%: workers were blocked, not spinning.
• 09:24 API rollback failed to recover; 09:43 flag-off did. The change that mattered was flag-gated, not binary-gated.
• A fresh worker (09:31) reproduced the failure with feature_flag=true and a correct config_retry_default — so the default was healthy and the tenant-specific override path was not.

Weakening / limits

• No connection-lifecycle data. The link "zero-delay retries consume connections" is inferred from downstream symptoms, not observed.
• The delay_ms → resolved-policy dependency is inferred from one before/after pair (job 319), not a documented config schema.
• Job 319 still needed two attempts after the flag was off. Backoff fixed the pacing, not an underlying per-job error. Some real failure condition remains unexplained; only the amplification was removed.
• Single-tenant evidence (acme failed at 09:06:58; beta succeeded at 09:07:03). Tenant attribution is thin.

H2 — Pure database capacity shortfall (rejected as root cause; contributory)

Weakening

• The 300-connection cap was not reached until 09:20, but degradation started at 09:07 — 13 minutes earlier.
• worker-22 at 09:07:01 reports pool wait exceeded with only active=10 idle=0, i.e. saturation while total usage was nowhere near 300. The binding constraint was already worker-local or hold-time driven, not global capacity.
• A pure capacity problem would not be resolved by disabling a retry-policy flag within 2 minutes.
• A pure capacity problem does not explain policy="".

Retained as contributory: low connection headroom plus worker scaling at 09:14 is why the situation escalated to a hard, system-wide failure instead of a slow degradation. Capacity was the victim, not the culprit.

H3 — Worker scaling (40 → 90) as root cause (rejected as root cause; contributory)

Weakening

• Onset at 09:07 precedes the 09:14 scale-up by 7 minutes.
• The flag-off at 09:43 restored throughput without reducing worker count.
• policy="" is unexplained by scaling.

Retained as contributory: see CF4 above, with the arrival-rate confounder stated.

H4 — Poison jobs / corrupt stuck backlog (mostly rejected, not fully excluded)

Weakening

• A restarted pool processed jobs normally for four minutes — inconsistent with a permanently wedged head-of-line job.
• The 09:24 rollback left the backlog stuck while new creation normalized, which fits workers continuing to churn existing jobs under the old flag, not a data defect.
• Recovery at 09:43 was immediate and complete, not gradual. A corrupt job would typically be skipped, not heal.
• The 09:43 success took one retried attempt, so the failure is not fully explained away.

Not excluded because: the inputs never show the final disposition of job 81, and no log line shows a dead-letter or drop path. A subset of jobs could have been individually poisoned while the bulk incident was H1. This is why the job-outcome audit in ACTIONS.md is P1 rather than P2.

H5 — Bad configuration value versus code defect in policy resolution (unresolved; both inside H1)

Supporting code-defect reading: config_retry_default="exponential" was present and correct at 09:31:10 (FACT) yet policy="" was still produced (FACT). A bad default therefore does not explain it; the tenant-specific lookup or the flag-gated code path did not resolve. Supporting config reading: a missing or malformed entry in the per-tenant policy map would produce an empty string without any code error, and would naturally be tenant-scoped — which would also explain why beta (09:07:03) behaved differently from acme. Status: the two readings predict the same evidence so far. I cannot distinguish them from these inputs, and I am explicitly not claiming the code is at fault rather than the config. This is why data request #1 in RESPONSE.md targets the resolver diff plus the live policy map.

H6 — Runaway CPU loop in the new retry path (rejected)

Weakening: worker CPU fell to 18%. A hot loop would raise CPU. The observed profile is blocking, not spinning.

---

6. What remains unknown

Ordered by how much it changes the conclusions above.

1. What the policy resolver actually did. Why did it return "" — missing map entry, tenant-key mismatch, parse failure, or an error swallowed into an empty string? Determines whether the fix is code, config, or both. (H5, data request #1.)
2. The connection lifecycle under retry. Are connections held through the failure path, leaked, or simply requested at excessive rate? Determines whether the fix must include connection budgeting, not just backoff. (H1 link 3, data request #2.)
3. Job dispositions, 09:07–10:18. How many jobs failed permanently, were dead-lettered, dropped, duplicated, or double-executed? This is the customer-facing unknown with the largest consequences and it is completely absent from the inputs.
4. The true exposure and customer impact. Tenant and order counts, affected checkout volume, whether us-east's 2% cohort had any impact that stayed below the alert threshold, and SLO/error-budget consumption.
5. Why the API rollback did not revert the behaviour. Was the flag independently managed, or did the rollback path omit worker config? Without this, future rollbacks carry the same 19-minute trap.
6. The 5-minute trigger delay (09:02 → 09:06:58, and 09:31:10 → 09:35:12). What workload condition provokes the first failure? Relevant to how quickly a fix can be validated.
7. Why policy="" and retry_budget_exhausted never alerted. Were these metrics unmonitored, monitored without thresholds, or simply invisible? Distinguishes "no telemetry" from "telemetry nobody read".
8. Whether workers were the first or second victim. 09:07:01 shows worker-22 pool-waiting 13 minutes before the API hit reserved-slot errors at 09:19:52 — workers failed first, but the inputs do not show why the API was spared for so long.

Task 09 — Incident analysis

Analyze TIMELINE.md and LOGS.txt. Produce:

• INCIDENT.md: impact, concise timeline, most likely root cause, contributing factors, evidence supporting or weakening each hypothesis, and what remains unknown.
• ACTIONS.md: immediate containment plus prioritized corrective actions with owners expressed as roles, verification signals, and rollback considerations.
• CUSTOMER_UPDATE.md: a plain-language update under 180 words. Do not claim certainty beyond the evidence or blame an individual.

Separate observed facts from inference. Avoid treating correlation as proof. Write RESPONSE.md with your confidence level and the next two pieces of data you would request. Work only in this directory.

Queue incident timeline (UTC)

• 09:02 deploy api 2026.09.23.1; change adds tenant-specific retry policy.
• 09:07 queue depth begins rising in eu-west; worker CPU falls from 65% to 18%.
• 09:11 alerts fire for job age and checkout latency.
• 09:14 on-call increases workers from 40 to 90; queue depth rises faster.
• 09:20 database connections reach configured maximum of 300.
• 09:24 API deploy rolled back; new job creation normalizes, backlog still stuck.
• 09:31 one worker pool restarted; it processes jobs normally for four minutes, then stalls.
• 09:43 feature flag tenant_retry_policy disabled; recovery begins within two minutes.
• 10:18 backlog cleared.
• US region uses the same release but had the flag enabled for only 2% of tenants; no alert fired there.

RESPONSE.md — analyst confidence and next data requests

Confidence level

Moderate (about 70%) that the root cause is the tenant_retry_policy path resolving an empty policy and issuing zero-delay retries that exhaust the retry budget and stall workers.

• High confidence on the failure mechanism shown in logs (delay_ms=0, policy="", retry_budget_exhausted in ~40 ms) and on containment (flag off → recovery in ~2 min).
• Moderate confidence on causation that this path started the outage (deploy/flag timing, US control group with 2% exposure and no alerts).
• Low confidence on why policy resolution returned empty — that is the main gap between a plausible story and a proven root cause.

Facts vs inference: timeline entries, log lines, CPU/connection numbers, and flag state are observed. "Empty policy caused the retry loop" and "DB saturation was downstream" are inferences from ordering and correlation, not proof.

Next two pieces of data I would request

1. Retry-policy resolution traces/config for tenant=acme (and 2–3 healthy tenants) around 09:00–09:45, including the raw policy store entries and code path that sets policy="". This would confirm or refute H1 directly.
2. US region logs/metrics for the same window with the flag at 2% exposure, especially any delay_ms=0 or policy="" occurrences and DB pool waits. If US shows the same log pattern without impact, exposure-only correlation weakens; if US is clean, H1 and the exposure hypothesis both strengthen.

Actions — queue incident 2026-09-23

Immediate containment (done or do now)

1. Keep tenant_retry_policy disabled globally until the empty-policy path is fixed. Owner: On-call SRE. Verify: no retry_budget_exhausted policy="" log lines; queue age falling. Rollback: re-enable flag only behind a 1% canary.
2. Do not scale workers further while retry thrash is unresolved. Owner: On-call SRE. Verify: DB active connections < 250. Rollback: N/A (safety hold).
3. Drain/inspect jobs that exhausted retry budget in 09:00–09:45 (e.g. job=81). Owner: Checkout Platform on-call. Verify: no silent drops; DLQ or retry status accounted for. Rollback: restore jobs from DLQ if dropped.

Prioritized corrective actions

P0 — Fix empty-policy retry fallback

• Add a hard default: when tenant policy resolves empty, use the global exponential backoff (config_retry_default), never delay_ms=0. Cap attempts and enforce minimum backoff > 0. Owner: Checkout Platform (API/retry library). Verify: unit/integration test — empty policy ⇒ delay_ms>=1000 exponential; staging log shows policy never empty in the retry path. Rollback: revert library version behind a config flag; default stays safe.

P0 — Fail loudly on policy resolution failure

• Emit ERROR + metric (retry_policy_resolution_failure) on empty resolution; alert on rate > 0. Owner: Checkout Platform + Observability. Verify: synthetic misconfigured tenant produces ERROR and alert within 5 min. Rollback: revert alert rule (keep log).

P1 — Guard retry load before it reaches the DB

• Per-tenant and per-worker retry rate limit / circuit breaker; shed or defer retries under DB pool pressure (pause retries when pool wait > threshold). Owner: Checkout Platform. Verify: load test — retry storm holds DB connections < 200 and queue age stable. Rollback: disable limiter via flag retry_guard.

P1 — DB connection budget and scale-out safety

• Reserve connection slots for non-retry paths (e.g. checkout writes); alert at 80% of 300; require DB headroom check before worker scale-out runbooks. Owner: DBA / SRE. Verify: alert fires in staging at 240 conns; scale-out runbook includes check. Rollback: restore previous pool limits.

P1 — Canary and flag exposure limits

• Require canary (≤2% tenants) for flag-gated policy changes before regional rollout; EU/US parity checks. Owner: Release Engineering. Verify: release checklist blocks full rollout without canary metrics; US-style low exposure can be reproduced in EU. Rollback: flag dial-down is the rollback path (already used).

P2 — Rollback runbook: flags ride with deploys

• Bundled rollback must disable related feature flags and restart worker pools that cached flag/config (see 09:31 stall). Owner: SRE / Platform. Verify: game-day — rollback clears backlog signals within 10 min. Rollback: restore previous runbook.

P2 — Retry observability

• Per-attempt metrics (delay, policy name, budget remaining) and dashboard panels for delay_ms=0 rate. Owner: Observability. Verify: dashboard shows the 09:06 pattern within 30 s of onset in replay test. Rollback: keep logs; remove dashboard.

Verification signals (summary)

• No retry_budget_exhausted policy="" lines.
• Worker CPU in normal band while queue drains (not collapsed to <20%).
• DB connections stable below 250 under peak.
• Queue age and checkout latency SLOs recovered and hold for 30 min after any flag change.
• retry_policy_resolution_failure metric rate is zero.

Rollback considerations

• Every fix above ships behind a flag or is version-revertible; P0 fallback default must be safe when the flag is off.
• Never roll back code without also flipping the associated feature flags and restarting worker pools (config caching caused the 09:31 false recovery).
• Roll back in this order: disable flag → revert code → scale workers back to baseline only after DB headroom is confirmed.

Customer update — September 23 checkout delays

Between roughly 09:07 and 10:18 UTC on September 23, some checkout and order-processing jobs in our EU region were delayed. A system that controls how failed jobs are retried appears to have been retrying them too quickly after a change we released that morning, which backed up our job queue and database connections. We rolled back the release and turned off the related setting around 09:43 UTC; processing recovered within minutes and the backlog cleared by 10:18 UTC.

We are still confirming details, including exactly which accounts saw delays and whether any orders need to be reprocessed. Our US customers were largely unaffected.

We are adding safeguards so retries slow down automatically if this kind of fault happens again, and we will notify you directly if we find that any of your orders need action on your side. We are sorry for the disruption.

Incident analysis — queue backlog, eu-west, 2026-09-23

Impact (observed)

• Queue depth rose in eu-west starting 09:07 UTC; job-age and checkout-latency alerts fired at 09:11.
• Checkout latency degraded for customers whose jobs entered the queue during the window.
• Database reached its configured maximum of 300 connections at 09:20 (API error: db remaining connection slots reserved).
• Backlog did not clear until 10:18 UTC. Approximate customer-facing duration: ~71 minutes of degraded checkout processing (09:07–10:18), worst between 09:11 and 09:45.
• US region: same release, tenant_retry_policy enabled for only 2% of tenants; no alert fired (no observed impact there).

Concise timeline (observed, UTC)

Time	Event
09:02	Deploy api 2026.09.23.1 (adds tenant-specific retry policy)
09:06:58	Logs: job=81 (tenant=acme) retried 3× with delay_ms=0; retry_budget_exhausted policy="" in 41 ms
09:07	Queue depth rises in eu-west; worker CPU falls 65% → 18%
09:07:01	Logs: worker-22 DB pool wait >5 s (active=10 idle=0)
09:11	Job-age and checkout-latency alerts fire
09:14	Workers scaled 40 → 90; queue depth then rises faster
09:19:52	Logs: api-03 DB connection slots exhausted
09:20	DB connections at max (300)
09:24	API deploy rolled back; new job creation normalizes; backlog remains stuck
09:31	One worker pool restarted; processes normally ~4 min, then stalls
09:35:12	Logs: worker-51 (started with feature_flag=true) hits retry_budget_exhausted policy=""
09:43	Feature flag tenant_retry_policy disabled; recovery begins within ~2 min
09:43:09–10	Logs: job=319 retries with delay_ms=1000, then succeeds
10:18	Backlog cleared

Most likely root cause (inference)

The tenant-specific retry policy path behind tenant_retry_policy fails to resolve a policy for some tenants and falls back to zero-delay retries with an empty policy (policy="", delay_ms=0). This burns the retry budget in tens of milliseconds and re-enters jobs in a tight loop, stalling workers (CPU drops) while flooding the DB pool — which then exhausts the 300-connection ceiling and blocks checkout processing.

This is an inference from the correlation of deploy/flag state with symptoms plus the direct log evidence of the zero-delay retry pattern; it is not proven by a root-cause code review.

Hypotheses and evidence

H1: Tenant retry policy resolves empty → zero-delay retry loop (most likely)

Supporting (observed):

• Deploy 09:02 introduced the policy; symptoms begin 09:07.
• Logs show attempts with delay_ms=0 and retry_budget_exhausted policy="" within ~40 ms.
• After the flag is off, the same job type retries with delay_ms=1000 (the configured default exponential) and succeeds.
• Flag off at 09:43 → recovery within 2 minutes.
• US: same binary, flag at 2% exposure → no alert (exposure tracks with impact).
• Worker CPU fell while queue grew — consistent with workers blocked in rapid retry/DB-wait loops, not doing useful work.

Weakening:

• We have not seen the policy-resolution code path; policy="" could be a logging defect rather than an empty policy.
• EU exposure percentage is not stated (inferred from US being 2% and unaffected).
• No direct proof that zero-delay retries (vs. another flag-gated behavior) caused the stall.

H2: Worker capacity shortage

Supporting:

• Queue depth grew, and scale-up was attempted. Weakening (stronger):
• Scaling 40→90 made queue depth rise faster and coincided with DB exhaustion — more workers amplified load rather than draining it.
• Worker CPU dropped to 18%; workers were not CPU-bound.

H3: Database as root cause

Supporting:

• DB hit max connections at 09:20; API errors show slot exhaustion. Weakening (stronger):
• DB pool waits appear at 09:07:01, before the 09:20 ceiling and concurrent with the first retry-loop logs — DB saturation is a downstream consequence of retry thrash plus scale-out, not the first symptom.

H4: API deploy binary alone is broken

Supporting:

• Rollback at 09:24 normalized new job creation. Weakening:
• After rollback the backlog stayed stuck, and a worker pool restarted at 09:31 with feature_flag=true stalled again at 09:35 — the flag-gated path kept failing independently of the API deploy.
• US ran the same release with low flag exposure and stayed healthy.

Contributing factors (observed or strongly inferred)

1. No backoff when policy is empty (delay_ms=0); retry budget exhausted in ~40 ms.
2. Silent policy-resolution failure (policy="") with only a WARN.
3. Scale-out at 09:14 increased concurrent DB demand and retry traffic during the fault.
4. DB connection ceiling (300) turned worker/API stalls into a broader outage.
5. Broad tenant_retry_policy exposure in eu-west (vs 2% in US) concentrated impact.
6. Deploy rollback alone did not disable the flag, prolonging the stuck backlog.

What remains unknown

• Why policy resolution produced policy="" for tenant=acme (missing override, bug, or fallback design).
• Exact EU flag exposure percentage at 09:02–09:43.
• Whether exhausted-budget jobs were dropped, dead-lettered, or re-queued (job=81 fate unknown).
• Why the 09:31 worker pool worked ~4 minutes before stalling (trigger for the 09:35 failure).
• Why rollback at 09:24 did not clear the backlog (stuck jobs, workers still on new code, or requeue loops).
• Whether impact extended beyond checkout (other job types) or beyond eu-west at low level.
• Whether policy="" is also present in healthy US traffic (would weaken H1).

Task 09 — Incident analysis

Analyze TIMELINE.md and LOGS.txt. Produce:

• INCIDENT.md: impact, concise timeline, most likely root cause, contributing factors, evidence supporting or weakening each hypothesis, and what remains unknown.
• ACTIONS.md: immediate containment plus prioritized corrective actions with owners expressed as roles, verification signals, and rollback considerations.
• CUSTOMER_UPDATE.md: a plain-language update under 180 words. Do not claim certainty beyond the evidence or blame an individual.

Separate observed facts from inference. Avoid treating correlation as proof. Write RESPONSE.md with your confidence level and the next two pieces of data you would request. Work only in this directory.

Queue incident timeline (UTC)

• 09:02 deploy api 2026.09.23.1; change adds tenant-specific retry policy.
• 09:07 queue depth begins rising in eu-west; worker CPU falls from 65% to 18%.
• 09:11 alerts fire for job age and checkout latency.
• 09:14 on-call increases workers from 40 to 90; queue depth rises faster.
• 09:20 database connections reach configured maximum of 300.
• 09:24 API deploy rolled back; new job creation normalizes, backlog still stuck.
• 09:31 one worker pool restarted; it processes jobs normally for four minutes, then stalls.
• 09:43 feature flag tenant_retry_policy disabled; recovery begins within two minutes.
• 10:18 backlog cleared.
• US region uses the same release but had the flag enabled for only 2% of tenants; no alert fired there.

Response — confidence and next data

Confidence

High that the tenant_retry_policy feature flag combined with an empty resolved policy (policy="") is the proximate trigger of the retry storm. The supporting evidence is three-fold:

• Log signature: delay_ms=0 retries inside a single second, followed by retry_budget_exhausted with policy="", preceding the queue rise and DB pool saturation.
• Differential blast radius: same release, same DB tier, but US (flag at ~2% of tenants) showed no alert, while eu-west (flag effectively at 100%) degraded.
• Recovery: disabling the flag at 09:43 produced recovery within ~2 minutes on a freshly observed worker pool (worker-51), including a job which had previously failed and now succeeded on attempt 2 with a proper delay.

Medium on the underlying mechanism (e.g., why policy="" resolves to delay_ms=0 rather than a safe default) — the behaviour is consistent with the logs but the worker source has not been read in this analysis.

Medium on the customer-visible impact envelope. We know latency and job age regressed and the backlog eventually drained, but we have not yet quantified: dropped jobs, failed user actions, or the 2% of US tenants that may have had silent degradation.

Next two pieces of data I would request

1. The worker source / config-resolution code path for the retry scheduler. Specifically the branch taken when policy="" or policy=null. This will either confirm the delay_ms=0 fallback hypothesis directly or surface a second mechanism that the logs hint at (the "4 minutes healthy then stalls" pattern on the restarted worker pool).
2. The retry-budget and outcome data for the affected window. For every job processed by eu-west workers between 09:00 and 09:45 UTC, the attempt, delay_ms, terminal_outcome (success / exhausted / dropped), and tenant_id. This will tell us how many jobs exceeded their retry budget and were dropped, and whether the 2% of US tenants with the flag on saw any silent degradation — both of which are needed to size customer impact and to size the safety budget in Action 7.

Actions — Queue Retry-Policy Incident

Immediate containment (already executed / in progress)

#	Action	Owner (role)	Verification signal	Rollback consideration
1	Disable tenant_retry_policy feature flag in eu-west (done 09:43).	SRE on-call	retry_budget_exhausted log rate drops; delay_ms becomes >0.	Re-enable only after the fix in Action 4 ships via canary.
2	Drain eu-west backlog and confirm queue depth returns to baseline.	Queue Ops	Queue depth matches 7-day baseline at 10:30 UTC.	Drain is one-way; no rollback needed.
3	Hold the API deploy 2026.09.23.1 as rolled back; do not re-promote.	Release Manager	Deploy board shows rollback as the active artifact.	Re-promotion requires all Actions 4–7 in this table.
4	Communicate externally (see CUSTOMER_UPDATE.md).	Comms / Support Lead	Update on status page.	Update is additive; revisions are fine.

Prioritized corrective actions

Priority key: P0 = block re-enabling the flag; P1 = must ship before the next tenant-policy rollout; P2 = ship within the quarter.

P0 — Required to re-enable the flag safely

#	Action	Owner (role)	Verification signal	Rollback consideration
5	Fix the policy="" fallback: any unresolved policy must inherit the safe default (exponential with min delay ≥ 1s), never delay_ms=0.	Tenant Service Engineering	Unit test covering policy="", policy=null, policy=unknown; integration test shows non-zero delay_ms for all unknown inputs.	Code change; rollback to previous commit is safe once flag is off.
6	Add config-load validation: workers must refuse to start (or must log FATAL and exit) if the resolved retry policy lacks a delay schedule.	Worker Platform Team	Worker boot fails closed in tests when policy is missing/invalid.	Validation is opt-in via env flag during rollout; rollback = revert env.
7	Add a per-tenant retry budget guard that caps attempts within a window and trips a circuit-breaker on the tenant when exhausted.	Queue Ops + Tenant Service	Tripped circuit-breaker count rises in load test; retry storm on one tenant no longer fans out to the pool.	Circuit breaker can be disabled per tenant; safer than current behaviour.

P1 — Required before the next policy rollout of this shape

#	Action	Owner (role)	Verification signal	Rollback consideration
8	Introduce a staged tenant rollout for tenant_retry_policy: 2% → 10% → 25% → 100%, with automated halt if job-age p99 regresses >2× for 5 min.	Release Manager + SRE	Canary dashboard shows progressive stages; auto-halt fires on synthetic fault.	Auto-halt is reversible; manual re-promotion after fix.
9	Separate the API-tier retry config from the worker-tier retry config so an API rollback actually stops worker behavior too.	Platform Architecture	API rollback flips worker config in the same change; test asserts both tiers observe new behavior within 1 minute.	Architecturally reversible by re-merging, but new code is strictly safer.
10	Worker auto-scale must back-pressure when DB pool wait exceeds a configurable threshold (e.g., 500 ms p95), instead of adding more workers into a saturated downstream.	SRE / Capacity	Synthetic load test: scale-up pauses when pool wait p95 > threshold.	Threshold is configurable; rollback = restore prior autoscaler policy.
11	Add structured retry telemetry: per-job delay_ms, policy, attempt, tenant, exposed as a metric and a dashboard.	Observability	Dashboard shows delay_ms distribution; alerting on delay_ms=0 rate > 0.	Telemetry is additive; no rollback risk.

P2 — Within the quarter

#	Action	Owner (role)	Verification signal	Rollback consideration
12	Move the DB connection pool so API and worker paths have separate budgets, with a shared overflow gate.	Database Team	Load test: worker storm no longer starves API path.	Pool split is reversible; traffic can be re-pooled behind a flag.
13	Add a chaos test that injects a delay_ms=0 retry storm on one tenant and asserts the system stays within SLOs.	SRE	Quarterly chaos drill passes; alert fires on injected storm.	Test is read-only against prod traffic shape; rollback = disable drill.
14	Post-incident review document and circulate to engineering; track Action items to closure in the incident tracker.	Incident Commander	Review held within 5 business days; action items have owners and dates.	N/A.

Service update — job processing delays in eu-west

Between approximately 09:00 and 09:45 UTC today, customers in our eu-west region may have experienced slower than usual checkouts and longer than usual delays for background jobs (such as order confirmations and notifications).

The issue was caused by a configuration change in a background-job processor that caused some jobs to be retried more aggressively than intended. This in turn saturated a shared database resource. Our team identified the change as the cause, disabled the affected setting at 09:43 UTC, and saw the system recover within minutes. The backlog of delayed jobs was fully cleared by 10:18 UTC.

Our other regions were not affected in any customer-visible way.

We are sorry for the disruption. No customer data was lost. We are following up with targeted fixes and additional safeguards before any similar change is re-enabled, and we will share a full post-incident summary internally.

Incident Analysis — Queue API (eu-west)

Impact

• Region affected: eu-west (primary). US was nominally exposed but the feature flag was enabled for only ~2% of tenants there, so no alert fired and no customer-visible impact has been confirmed.
• Customer-visible symptoms: elevated checkout latency and rising job age in eu-west starting ~09:07 UTC. New job creation returned to normal after the API rollback at 09:24, but existing backlog continued to stall until ~10:18 UTC.
• Blast radius: all eu-west tenants were processed by the new code path once the deploy reached steady state; however, the failure manifested as per-job retries rather than data loss. Whether any jobs exceeded their retry budget and were dropped permanently is not yet confirmed (see Unknowns).
• Duration of customer impact: approximately 09:07 → 09:43 UTC (queue stalled, latency degraded), with full backlog drain at 10:18 UTC. End-to-end ~71 minutes of degraded service, ~35 minutes of severe backlog.

Concise timeline (UTC)

Time	Event
09:02	Deploy of api 2026.09.23.1 adds tenant-specific retry policy (tenant_retry_policy flag).
09:07	Queue depth begins rising in eu-west; worker CPU falls 65% → 18% (workers stuck waiting).
09:07	Logs show delay_ms=0 retries on the same job inside a single second; retry_budget_exhausted logged with policy="".
09:07	DB pool wait exceeded 5,000 ms; 10 active connections, 0 idle.
09:11	Alerts fire: job age and checkout latency.
09:14	On-call scales workers 40 → 90; queue rises faster (negative feedback loop).
09:19	DB error: remaining connection slots reserved (pool saturated at 300).
09:24	API deploy rolled back; new-job creation normalizes, but existing backlog remains stuck.
09:31	One worker pool restarted; jobs process normally for ~4 minutes, then stall again.
09:35	retry_budget_exhausted logged again on the restarted pool (policy="").
09:43	Feature flag tenant_retry_policy disabled; recovery begins within ~2 minutes.
10:18	Backlog cleared.

Most likely root cause

A new tenant-specific retry policy shipped via the tenant_retry_policy feature flag was wired up such that when the flag was on and the resolved policy string was empty (policy=""), jobs retried immediately with delay_ms=0 rather than falling back to a safe default. Three retries of job 81 inside a single second (09:06:58) is the clearest smoking gun. This produced a retry storm in eu-west that saturated the shared database connection pool (300 max) and starved legitimate work, manifesting as a stuck queue with low worker CPU (the classic "blocked on downstream" pattern).

Two pieces of evidence strongly support this being the proximate cause:

1. Temporal alignment: the delay_ms=0 retries and the first retry_budget_exhausted log (09:06:58) precede the queue-depth rise (09:07) and the DB pool saturation (09:07:01).
2. Differential exposure: US ran the same release but had the flag enabled for ~2% of tenants. No alert fired there. If the deploy itself were the problem, both regions should have been affected equally.

Hypotheses ranked

H1 — Empty-policy retry storm (MOST LIKELY)

• Claim: when tenant_retry_policy is on but the resolved policy is "", the worker falls back to a no-delay, multi-attempt loop.
• Supporting evidence:
  • delay_ms=0 repeats in the same second for the same job (job=81, three lines, 09:06:58).
  • retry_budget_exhausted carries policy="" twice (09:06:58, 09:35:12).
  • Worker-51 starts up at 09:31 with feature_flag=true, config_retry_default="exponential", then stalls within ~4 minutes with the same policy="" exhaustion signature — and recovers within ~2 minutes of the flag being turned off (09:43).
  • Differential blast radius between regions matches the flag's rollout %.
• Weakening evidence / gaps:
  • We have not yet read the worker source for the policy="" code path, so "fallback to no-delay retry" is an inference from log behaviour, not a confirmed code-level finding.
  • A worker pool restart at 09:31 looked healthy for ~4 minutes before stalling again, which is consistent with the flag being re-applied to new jobs that arrived after restart — but it is also consistent with a secondary resource (e.g., DB pool) being gradually exhausted. Both stories fit the data.

H2 — DB connection pool exhaustion as primary cause

• Claim: something else caused DB pool saturation, and the retry storm is merely a downstream symptom.
• Supporting evidence: pool saturation (remaining connection slots reserved) is real and preceded by 5,000 ms waits at 09:07:01.
• Weakening evidence:
  • Pool saturation occurs after the delay_ms=0 retry burst, not before.
  • Scaling workers from 40 → 90 (09:14) made the queue worse, not better — consistent with each worker amplifying the storm, not with a static DB bottleneck being hit by extra connections.
  • US ran the same code and DB tier with the flag at 2% and did not alert.
• Verdict: unlikely to be the primary cause; the pool saturation is best explained as a consequence of H1.

H3 — Deploy artifact / build issue (less likely)

• Claim: the binary itself is broken independent of the flag.
• Weakening evidence: US ran the same binary without impact proportional to eu-west. The flag toggle at 09:43 produced rapid recovery, which would not be expected if the binary itself were faulty.

Contributing factors

1. No safe default for an unresolved policy. policy="" should have either inherited the existing default backoff or refused to start, instead of silently allowing delay_ms=0 retries.
2. Flag rollout was not gated by tenant canary in eu-west. Going to 100% of eu-west tenants simultaneously removed the safety net that protected US.
3. Worker auto-scale without a saturation circuit breaker. Scaling 40 → 90 workers amplified the retry storm and accelerated DB pool exhaustion instead of relieving it.
4. Shared DB connection pool across API and worker paths. A retry storm in workers was able to starve API request capacity.
5. API rollback at 09:24 did not affect workers. The retry config was read at worker startup and re-applied per job, so rolling back the API tier only stopped new jobs from being created — it did not change the workers' behaviour. This delayed real recovery by ~19 minutes.
6. No automated rollback trigger. The feature flag was the correct lever, but it was disabled manually ~50 minutes after first impact.

What remains unknown

• The exact code path that turns policy="" into delay_ms=0. Logs are consistent with a fallback bug but the worker source has not been read.
• Whether the "4 minutes healthy, then stalls" pattern on the restarted pool is purely a function of new jobs arriving and re-triggering the storm, or whether there is a secondary slow-burn effect (e.g., a queue/cache).
• Total number of jobs that exceeded their retry budget and were dropped vs retried successfully once the flag was disabled.
• Customer-visible failure modes beyond latency (e.g., failed checkouts, abandoned carts, notification delays).
• Whether US saw any impact at all on the ~2% of tenants with the flag on, even though no alert fired.
• Why US and eu-west had such different flag rollout percentages (intentional canary or configuration drift).

Task 09 — Incident analysis

Analyze TIMELINE.md and LOGS.txt. Produce:

• INCIDENT.md: impact, concise timeline, most likely root cause, contributing factors, evidence supporting or weakening each hypothesis, and what remains unknown.
• ACTIONS.md: immediate containment plus prioritized corrective actions with owners expressed as roles, verification signals, and rollback considerations.
• CUSTOMER_UPDATE.md: a plain-language update under 180 words. Do not claim certainty beyond the evidence or blame an individual.

Separate observed facts from inference. Avoid treating correlation as proof. Write RESPONSE.md with your confidence level and the next two pieces of data you would request. Work only in this directory.

Queue incident timeline (UTC)

• 09:02 deploy api 2026.09.23.1; change adds tenant-specific retry policy.
• 09:07 queue depth begins rising in eu-west; worker CPU falls from 65% to 18%.
• 09:11 alerts fire for job age and checkout latency.
• 09:14 on-call increases workers from 40 to 90; queue depth rises faster.
• 09:20 database connections reach configured maximum of 300.
• 09:24 API deploy rolled back; new job creation normalizes, backlog still stuck.
• 09:31 one worker pool restarted; it processes jobs normally for four minutes, then stalls.
• 09:43 feature flag tenant_retry_policy disabled; recovery begins within two minutes.
• 10:18 backlog cleared.
• US region uses the same release but had the flag enabled for only 2% of tenants; no alert fired there.