PyPI resolves intermittent 502 and 503 errors in file hosting after two-week outage

PyPI experienced intermittent 502 and 503 errors for approximately two weeks in August, affecting users downloading files from files.pythonhosted.org. The issue was traced to a single Fastly cache node misconfiguration triggered by a canary deployment. This misconfiguration caused Fastly's routing layer to return 502 responses for traffic reaching the affected node. The problem was compounded by pre-existing bugs in PyPI's Fastly configuration, particularly around origin fallback and range request behavior, which only became apparent during the incident

The root cause analysis revealed two distinct problems. First, a Fastly canary deployment left a misconfigured cache node in place, which consistently returned 502 errors. Second, PyPI's own Fastly configuration had latent bugs that failed to properly handle B2-to-S3 fallback scenarios and range requests. These issues surfaced when users reported persistent errors, with one report isolating the problem to a specific cache node (cache-pae2080020) that had been failing for over 19 hours

PyPI's file hosting infrastructure relies on a multi-origin CDN setup with Fastly as the front-end, Backblaze B2 as the primary cache, and Amazon S3 as the fallback origin. The zero-cost egress agreement between Fastly and Backblaze makes B2 the preferred origin, but S3 serves as a fallback for durability. The incident exposed weaknesses in this setup, particularly in how Fastly detected and handled B2 failures, leading to incomplete failover to S3. The fixes included removing the problematic routing override, improving logging, and addressing bugs in segmented caching and URL normalization

The timeline of the incident shows a progressive diagnosis and resolution process. Initial reports of persistent 502 errors led to investigations that uncovered the misconfigured cache node. Subsequent fixes addressed the immediate routing issue and then dug deeper into PyPI's configuration, revealing and patching bugs in origin fallback, range request handling, and segmented caching. The final fixes included exempting certain request types from segmented caching and normalizing URLs to prevent bad responses from being cached and served repeatedly

This incident underscores the complexity of maintaining reliable file hosting infrastructure, particularly when relying on multi-vendor CDN setups with canary deployments. The combination of external misconfigurations and internal bugs created a scenario where intermittent failures were difficult to diagnose. For engineers, the takeaway is the importance of robust failover mechanisms, comprehensive logging, and thorough testing of edge cases in CDN configurations, especially when dealing with range requests and segmented caching