DEV TOOLS Signal 503
GitHub outage autoscaling policy missed Istio sidecar limits, illustrating component substitution fallacy
↗
A GitHub outage was partly caused by an autoscaling policy that monitored host service metrics but not Istio sidecar concurrency limits, highlighting what David Woods calls the component substitution fallacy, the mistaken focus on fixing individual defective components rather than examining system interactions.
Engineers configuring autoscaling need to account for all components in the request path, including sidecars, not just the primary service. The deeper lesson is that incident analysis should examine interactions between multiple factors rather than fixating on a single misconfigured component, since systems contain many latent defects that only cause failures in combination.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
GitHub's autoscaling policy only monitored host service metrics, not Istio sidecar concurrency limits, causing the sidecar to saturate without triggering scale-up.
The component substitution fallacy describes the tendency to fixate on individual defective components rather than treating system interactions as first-class concerns.
The GitHub outage involved interactions between changing traffic patterns, autoscaling policy, sidecar saturation, retry logic, HAProxy saturation, and authentication traffic.
THE CLUSTER