Mohtashim HusnainFounder & CEO, Tech Solutions

3 min readby Muhammad Mohtashim Husnain

Two CDNs in a row: three days of 429 errors

Images, fonts and whole pages kept failing with 429 errors. The server was innocent: two CDNs in a row, the second rate-limiting the first.

For three days this site lost pieces at random. One visit the Events page was black. The next, every client logo was missing. Then the home page itself returned an error. Each time the response was the same: HTTP 429, Too Many Requests.

I fixed it four times before I fixed it once. This is the write-up of what was actually wrong and how I should have found it on day one.

The wrong theories

"The host rate-limits bursts." A page asks for twenty files at once, so I assumed the server refused the burst. I merged twenty-four client logos into one image and cut the number of scripts. Fewer requests, same errors.

"The cache is holding an error." Partly true. My server config marked every file under /assets/ as cacheable for a year, and it attached that header to error responses too. So when a 429 was produced, the CDN stored it as the permanent content of that address. I shortened the shared-cache lifetime to five minutes. That made it worse: now the CDN re-fetched every file on almost every visit, which produced more 429s, each one shown to everyone for five minutes.

"Retry on the client." I added a retry for every file type under a fresh address. That hid most failures for images and scripts. It could not help when the HTML document itself was refused.

All three changed symptoms. None asked the only question that mattered: who is sending the 429?

The test that settled it

I sent requests straight to the hosting server, skipping everything in front of it:

curl --resolve example.com:443:SERVER_IP https://example.com/page

Then I sent a hundred at once. Every one came back 200 from the web server, in about 0.4 seconds. The server had never refused anything.

So the 429 came from something in between. The response headers had been telling me that all along. A failing response carried two sets of fingerprints: one from Cloudflare, and a request id from the hosting company's own CDN.

Two proxies, one bucket

The domain's DNS was on Cloudflare with the proxy switched on. The DNS records there pointed at the hosting company's CDN addresses, because that is what the hosting panel tells you to use when its CDN is enabled. So every request travelled:

visitor → Cloudflare → hosting CDN → web server

The hosting CDN limits how many requests a single address may send. That is sensible when the address is a visitor. But behind Cloudflare, every visitor in the world arrives from a small set of Cloudflare addresses. One busy minute from all visitors combined looked like one abusive client, and the limiter did its job.

That also explains why it was random, why it hit ordinary pages at a slow pace, and why it happened from networks that had never visited the site.

The fix

Three DNS changes and one switch:

  1. Point the root A record at the web server's own address.
  2. Delete the other A and AAAA records that pointed at the hosting CDN.
  3. Point www at the root domain.
  4. Turn the hosting CDN off, since nothing uses it any more.

The order matters. Turning the hosting CDN off before changing DNS would have left Cloudflare sending traffic to an address that no longer served the site.

Afterwards twelve pages loaded in a real browser without a single failed request, and a typical page took about 1.2 seconds.

What I would do differently

  • Find out who produced the error before fixing it. Bypass every layer and test the origin directly. It takes one command.
  • Read all the response headers. Two CDN signatures on one response is a diagram of the problem.
  • Never attach long cache headers to error responses. Check what your server sends on a 404, a 429 and a 500, not only on a 200.
  • Do not test with a crawler. My own checker requested 250 files quickly and turned a rare fault into a total one for fifteen minutes.
  • Keep the retries anyway. A page that recovers from one refused file is better than one that does not, whatever the cause.

The earlier half of this story, a different black page with a different cause, is in Why this site went black at the last section. The service this feeds is company website development.