DEV TOOLS Signal 464
GitHub outage caused by autoscaling misconfiguration and VS Code retry storm overloaded load balancers
GitHub attributed an 8-hour service disruption to a monitoring blind spot in autoscaling policies and excessive retries from Visual Studio Code.
This outage highlights risks in relying on tightly coupled SaaS infrastructure, where a single misconfiguration or client-side retry logic can cascade into prolonged downtime. For engineers, it underscores the need to validate autoscaling policies against all dependent components, not just primary services.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
A misconfigured autoscaling policy failed to monitor Istio sidecar concurrency limits, allowing traffic to overwhelm load balancers.
A latent retry bug in Visual Studio Code amplified Copilot token service traffic by 10x, delaying recovery.
GitHub mitigated the issue by reducing gateway retries and blocking Copilot Token Service requests with HTTP 403 responses.
THE READ
What the cluster adds up to.
The outage stemmed from a gap in GitHub’s autoscaling monitoring. The policy tracked host service metrics but ignored Istio sidecar concurrency limits, creating a blind spot. When sidecars hit their limits, the system failed to scale, and load balancers saturated under traffic. This demonstrates how distributed systems can fail at integration points, even when individual components appear healthy. Engineers should audit autoscaling policies to ensure they account for all dependent services, not just the primary workload.
The incident was compounded by a retry storm from Visual Studio Code. A latent bug in the client triggered excessive retries for Copilot token requests, amplifying traffic by an order of magnitude. This highlights the fragility of client-server interactions in SaaS ecosystems, where a single misbehaving client can degrade shared infrastructure. GitHub’s mitigation, blocking Copilot requests with HTTP 403s, was a blunt but effective stopgap. Teams should review retry logic in their own clients and consider circuit breakers to prevent similar amplification.
Recovery was further delayed by external factors, including scraping attacks on codeload endpoints. While GitHub’s post-mortem focuses on internal misconfigurations, the outage reveals how SaaS reliability can be undermined by both internal and external pressures. The incident also arrives as competitors like Cursor and OpenAI accelerate their own developer tooling, raising questions about GitHub’s long-term dominance. For engineers, this outage is a reminder to evaluate redundancy and failover strategies, especially for critical workflows like CI/CD and code hosting.
Written by elseif from the cluster below · checked for specifics the sources never containedTHE CLUSTER