ELSEIF
Your brief EB
353 stories from 111 feeds 411 clusters Refreshed 1 second ago next pull 13:52

DEV TOOLS Signal 464

GitHub outage caused by autoscaling misconfiguration and VS Code retry storm overloaded load balancers

GitHub attributed an 8-hour service disruption to a monitoring blind spot in autoscaling policies and excessive retries from Visual Studio Code.

WHY IT MATTERS

This outage highlights risks in relying on tightly coupled SaaS infrastructure, where a single misconfiguration or client-side retry logic can cascade into prolonged downtime. For engineers, it underscores the need to validate autoscaling policies against all dependent components, not just primary services.

Written by elseif from the cluster below · every claim links back to a source

The three things worth knowing

01

A misconfigured autoscaling policy failed to monitor Istio sidecar concurrency limits, allowing traffic to overwhelm load balancers.

02

A latent retry bug in Visual Studio Code amplified Copilot token service traffic by 10x, delaying recovery.

03

GitHub mitigated the issue by reducing gateway retries and blocking Copilot Token Service requests with HTTP 403 responses.

THE READ

What the cluster adds up to.

ORIGINAL ANALYSIS

The outage stemmed from a gap in GitHub’s autoscaling monitoring. The policy tracked host service metrics but ignored Istio sidecar concurrency limits, creating a blind spot. When sidecars hit their limits, the system failed to scale, and load balancers saturated under traffic. This demonstrates how distributed systems can fail at integration points, even when individual components appear healthy. Engineers should audit autoscaling policies to ensure they account for all dependent services, not just the primary workload.

The incident was compounded by a retry storm from Visual Studio Code. A latent bug in the client triggered excessive retries for Copilot token requests, amplifying traffic by an order of magnitude. This highlights the fragility of client-server interactions in SaaS ecosystems, where a single misbehaving client can degrade shared infrastructure. GitHub’s mitigation, blocking Copilot requests with HTTP 403s, was a blunt but effective stopgap. Teams should review retry logic in their own clients and consider circuit breakers to prevent similar amplification.

Recovery was further delayed by external factors, including scraping attacks on codeload endpoints. While GitHub’s post-mortem focuses on internal misconfigurations, the outage reveals how SaaS reliability can be undermined by both internal and external pressures. The incident also arrives as competitors like Cursor and OpenAI accelerate their own developer tooling, raising questions about GitHub’s long-term dominance. For engineers, this outage is a reminder to evaluate redundancy and failover strategies, especially for critical workflows like CI/CD and code hosting.

Written by elseif from the cluster below · checked for specifics the sources never contained

THE CLUSTER

Same story, 1 feed.

ORDERED BY FIRST SEEN
www.theregister.com - Articles GitHub blames 8-hour outage on autoscaling fail and VS Code retry storm Open ↗