Results

What a proxy in the path costs

GET with a 1 KB response, open-loop load, every proxy pinned to one CPU with 256 MB. Added latency is the median over repetitions minus the same request with no proxy at all. CPU per request is the proxy container's own, at 4k rps, the highest rate every proxy held. Max is the highest offered rate at which most repetitions kept p99 under 50 ms, failures under 0.1% and throughput at 95% of the offered rate. Run 20261008-163533.

Past their maximum, the Go proxies and Envoy keep accepting work: requests queue in memory until the container reaches 256 MB and starts swapping, and throughput falls to a few hundred rps, or the kernel kills it. nginx and HAProxy stay far below the limit. PortalGun ran here without its cap on what it holds in total, so it fails the same way. The cap is measured below. The lab itself tops out near 20k rps with no proxy at all, so the fastest maximums are partly the lab's.

Near the knee, on a quieter night

The same test again at 6k, 7k, 8k rps, with everything else on the laptop stopped. Every proxy held every rate in every repetition, so the gap in the max column above says more about the noisier evening of the sweep than about the proxies. What stays is the tail. Run 20261009-210322.

Past the knee

Each point starts from a freshly restarted proxy and goes straight to the offered rate. Each cell shows the 2xx responses served per second, the p99 of those responses only, and the highest memory any repetition reached. Anything above the cap gets a 503 at once and isn't counted as served. A run counts as collapsed when more than 1% of its requests time out or never get sent. PortalGun's default cap is 256, the largest that never collapsed here. Run 20261010-133903.

One limit across two gateways

1,000 requests per second allowed for one key, split over two gateways that share Redis. Added latency and Redis trips are measured at 800 rps, under the limit, against the same load with no limiter. Differences under about 0.1 ms are noise on this laptop. Admitted is measured at 1,500 rps and compares what got through with a perfect single bucket replayed over the same requests: plus means over the limit, minus means good requests turned away. Shifting sends 90% of the traffic to one gateway and swaps every 5 s, and burst is 800 rps with spikes at 5,000, also shifting. Without Redis counts the decisions a gateway made on its own at 1,500 rps and an even split, out of about 52,000 per run. Run 20261009-150154.

In this run, Redis on every request and the token leases still had their first breaker: after three calls in a row missed their 2 ms deadline they stopped asking Redis for a second and decided on their own share of the limit. Here that happened often enough to undo the exact design: a gateway deciding alone stops spending from the shared bucket, so the other one gets all of it, and nothing was charged back afterwards. The local buckets only talk to Redis between requests and never had to. The rebuilt breaker is measured below.

The Redis breaker, before and after

The rebuilt breaker opens only when at least half of the calls in the last 100 ms failed, and at least five, stays open for 100 ms, then lets one call through. What a gateway admitted on its own is charged back to the shared bucket once Redis answers again, unless the bucket expired meanwhile. Measured against the code before it on one evening, alternating, three runs each, at 1,500 rps. The 10,000 keys row is 3,000 rps over keys drawn from a Zipf distribution. Without Redis counts the decisions made on the static split at an even split, out of about 52,000. Runs before/20261012-merged and after/20261012-merged.

When Redis is slow or gone

1,500 rps with Redis 5 ms slower for the whole run, or stopped. Degraded counts the decisions a gateway made on its own because Redis did not answer in time.

When a dimension breaks

2,000 rps spread over three dimensions, one of them broken for 30 s. Each cell is the median number of requests that failed in those 30 s, and how long until less than 1% of the traffic still touched the broken one (100 ms bins). Probes are active health checks, passive means ejecting on observed errors, retries go to another dimension. Run 20261009-175756.

One setting at a time

Probes, passive ejection and retries on, everything else at the default except the one thing named.

Ten minutes with nothing broken but 1% errors and log-normal delays everywhere: 0 false ejections (median of 3 runs, at most 0).

How often a good version gets rolled back

Range over the 36 cells: base error 0.1 to 2%, canary share 1 to 25%, 200 and 2,000 rps, with and without a 30 s outage downstream that hits both versions.

How fast a bad version goes, and how often it is caught

Median seconds to roll back, share of trials rolled back in brackets. The fixed-horizon and threshold rules look at errors only, so their numbers on the latency rows are the false rollbacks they would have made anyway.

Dimensions that come and go

Dimensions register themselves with FiveFamilies, a registry on Raft, and the controller turns its list into a config version. This run has a stack of its own without fault injection, 500 rps across both gateways and two retries on the route: 0 of 443,047 requests were lost in 43 runs. Run 20261021-225536-laptop.

A lost lease costs nothing here: the dimension registers again with the same address and version, the list doesn't change and no config version is written.