Cloudflare container cost incident

Bottom line

The incident was noticeable, not catastrophic: approximately $105.01 of Containers resource charges were observed from 12 June through early 13 July, of which approximately $95.99 is estimated above Tegy's pre-growth baseline. Left at the 100-instance cap, idle memory and disk alone would have approached $24/day or $720/month before active CPU.

$105.01 Estimated Containers charges

Monthly included usage applied to June and July.

$95.99 Estimated incident excess

Usage above the 12–27 June daily baseline.

100 / 100 Fleet at incident cap

No free instance remained for a new chat turn.

9 / 100 Post-fix verification

2 active, 7 healthy, 0 failed at the final rollout sample.

The cost curve

Daily allocated memory is the clearest proxy for leaked wall time because every Tegy container reserves 1 GiB while it is alive. The baseline averaged 27.2 GiB-hours/day through 27 June; growth begins sharply on 28 June and peaks at 1,723.5 GiB-hours on 12 July. The 13 July point is partial and includes the rollout recovery window.

* 13 July is a partial day queried at 04:49 UTC. Unit: provisioned GiB-hours, not application RSS.

Observed resource charges

These calculations use Cloudflare's containersUsageAdaptiveGroups data—the same billing-estimation dataset that populates the dashboard—and official Workers Paid rates. Figures are estimates, not a finalized invoice.

Period	CPU	Memory	Disk	Estimated charge
12–30 June	97,383.56 sec	1,300.28 GiB-hr	5,201.10 GB-hr	$14.24
1–13 July*	509,935.23 sec	8,123.27 GiB-hr	32,493.09 GB-hr	$90.77
Total	607,318.79 sec	9,423.55 GiB-hr	37,694.19 GB-hr	$105.01

The billing query returned 607,309.98 CPU-seconds and 9,423.38 GiB-hours in the initial raw aggregation. The month-split rerun shown above differed by less than one minute of live ingestion; dollar totals are unchanged at displayed precision.

June estimate

After June's included allocation: $1.50 CPU + $11.48 memory + $1.26 disk.

$1.50 + $11.48 + $1.26 = $14.24

July estimate through the query

After July's included allocation: $9.75 CPU + $72.88 memory + $8.14 disk.

$9.75 + $72.88 + $8.14 = $90.77

What is attributable to the incident?

The counterfactual uses Tegy's observed 12–27 June average as the pre-growth baseline, then subtracts that daily rate from the 28 June–12 July growth window. This avoids calling all container usage “waste,” while acknowledging that a perfect counterfactual invoice is impossible after the fact.

Resource	Excess above baseline	Rate	Gross excess cost
Active CPU	537,959.08 vCPU-sec	$0.000020/sec	$10.76
Provisioned memory	8,515.91 GiB-hr	$0.009/GiB-hr	$76.64
Provisioned disk	34,063.62 GB-hr	$0.000252/GB-hr	$8.58
Estimated incident excess			$95.99

Why the slope was dangerous

At the 100-instance cap, 100 basic instances reserve 100 GiB memory and 400 GB disk. Keeping that fleet alive for a full day costs about $21.60 memory + $2.42 disk before active CPU.

Why this was not a four-figure incident

CPU is billed on active usage rather than provisioned capacity, egress was only 1.73 GB and remained well inside the included allotment, and the fleet was fixed before a full month at the cap.

User impact and recovery evidence

The same failure that created the cost curve exhausted all available instances and returned a raw Cloudflare 503 to new chats. The shipped fix restored production turns and released idle capacity.

Tegy showing the Cloudflare no container instance available 503 during the incident

Before: 100/100 live A new turn could not acquire a Container instance and surfaced the provider's raw capacity response.

Successful authenticated Tegy production response after the container lifecycle deployment

After: production HTTP 200 The authenticated new-chat journey returned the requested assistant phrase on the deployed image.

Root cause, fix, and verification

Root cause

The new-chat route eagerly prewarmed a unique container. The Worker awaited the health response but did not consume or cancel its body, leaving the Containers helper's in-flight request counter above zero. Activity expiry therefore kept renewing instead of stopping the instance.

Long-term fix

Prewarm now starts only after user intent, its response body is always drained, idle timeout is five minutes, and activity expiry explicitly stops the process. Capacity failures are normalized to a retryable JSON 503 rather than raw provider copy.

✓Regression coverage: worker routing tests, eight targeted browser checks, full flow suite, and production build passed.
✓Deployment: PR #987 shipped through an immediate container rollout on Worker version 8b37f4b5….
✓Production journey: authenticated chat turn returned HTTP 200 and rendered the expected assistant response.
✓Fleet decay: live instances fell 100 → 99 → 97 → 19 → 9, with clean exit code 0 lifecycle events and no failures.

Methodology and caveats

Billing-grade source

Usage came from Cloudflare Analytics GraphQL containersUsageAdaptiveGroups, filtered to Tegy's Containers application and grouped daily from 12 June through 13 July.

Cloudflare container billing metrics documentation

Rates and included usage

Workers Paid includes 25 GiB-hours memory, 375 vCPU-minutes, and 200 GB-hours disk per month. Overage rates are $0.0000025/GiB-second, $0.000020/vCPU-second, and $0.00000007/GB-second.

Cloudflare Containers pricing

This is an engineering estimate, not a Cloudflare invoice. It excludes the ordinary $5 Workers Paid base plan, taxes, contractual credits or discounts, and small adjacent Worker, Durable Object, or log charges. Calendar-month allowances were applied separately; a different invoice-cycle boundary would change the result by less than approximately $1. Network egress remained below the included allotment. The baseline method may attribute some legitimate growth to the incident or miss legitimate activity embedded in the baseline.

Daily allocated-memory evidence

Date	GiB-hours	Date	GiB-hours
12 Jun	11.55	28 Jun	225.88
13 Jun	7.79	29 Jun	332.37
14 Jun	21.13	30 Jun	306.78
15 Jun	26.43	1 Jul	647.43
16 Jun	9.02	2 Jul	420.59
17 Jun	23.82	3 Jul	517.43
18 Jun	27.91	4 Jul	817.78
19 Jun	41.41	5 Jul	424.37
20 Jun	26.73	6 Jul	967.46
21 Jun	26.51	7 Jul	570.38
22 Jun	26.16	8 Jul	384.04
23 Jun	48.17	9 Jul	236.63
24 Jun	28.27	10 Jul	441.86
25 Jun	35.46	11 Jul	907.41
26 Jun	35.34	12 Jul	1,723.53
27 Jun	39.53	13 Jul*	64.20

Guardrails now live

The journey monitor detected user-visible failure only after the fleet was exhausted. On 13 July, Tegy shipped three independent guardrails: a hard blast-radius limit, an application-specific early warning, and an account-wide spend backstop.

1. Hard cap — live

Production max_instances is now 40, down from 100 while Tegy is pre-launch. That leaves headroom above the observed development fleet but bounds idle memory-and-disk exposure near $9.61/day. The live container application reported the 40-instance ceiling after rollout.

2. Capacity alerts — live

A scheduled Worker probe now reads billing-grade container usage every 15 minutes. It warns in Discord at 20 live-equivalent instances, alerts critically at 32 (80% of the cap), reports recovery, and alerts immediately on any container_capacity_unavailable response.

3. Budget alerts — live

Three account-wide Cloudflare budget alerts are active at $10 warning, $25 action, and $50 critical per billing cycle. Each routes to the shared Tegy billing recipient rather than one operator.

Cloudflare budget alert setup

4. Next diagnostic layer

The shipped container.started, container.activity_expired, and container.stopped events can support a later alert when starts materially outpace stops beyond the five-minute idle window. A growing unmatched-start count is the leak signal this incident lacked.

Operating rule now monitored

Yellow: investigate at 20 live-equivalent instances. Red: stop nonessential tests and inspect lifecycle logs at 32. Capacity: keep the hard cap at 40 until launch planning explicitly raises it. Budget alerts are a daily financial backstop, not a substitute for the 15-minute operational probe.

Production evidence: Worker version 6cfa4e36-b677-4e8b-8128-536273919e64 deployed with the */15 * * * * trigger; the live analytics/Discord probe succeeded at 21.1 live-equivalent instances; and the remote D1 alert state migration completed. Implementation: PR #1001.

Cloudflare budget alerts are informational, account-wide, and evaluated from billing-period spend; they do not pause usage. Archiving old chats remains optional product/data-retention work because history is durable in D1/R2 and does not require a live container.