Moniepoint replaces Kafka-on-VMs with Strimzi Kubernetes operator to eliminate manual scaling pain

The migration demonstrates how architectural decisions can be made proactively rather than reactively. The team identified that their Kafka-on-VMs setup was working but not scaling efficiently, with manual processes for configuration changes, upgrades, and monitoring. This systems-level observation led to a rebuild before any outage occurred, showing how discomfort with operational patterns can drive improvement without a crisis trigger.

The operational pain points were specific and measurable. Each Kafka instance required SSH access and port-forwarding for changes, version drift created inconsistent behavior across clusters, and Docker Compose restarts introduced unnecessary downtime risk. These manual processes didn't break the system but made it increasingly difficult to maintain as the number of pipelines grew. The team's documentation of these friction points provides a clear checklist for other engineers evaluating their own Kafka deployments.

Strimzi's Kubernetes operator model represents a fundamental shift in operational responsibility. Instead of engineers manually managing Kafka instances, the operator software continuously reconciles the desired state with the actual state. This automation eliminates the manual toil while introducing new requirements: Kubernetes expertise, declarative configuration management, and cluster resource planning. The team's decision to use a dedicated GKE cluster shows how they balanced Kafka's resource needs against other workloads.

The implementation choices reveal important tradeoffs. Namespace isolation per pipeline provides clear separation but requires careful network policy configuration. Moving to Kubernetes eliminates some operational tasks while introducing others, like cluster management and monitoring of the operator itself. The team's approach of maintaining the same data pipelines while changing the underlying infrastructure shows how architectural improvements can be made incrementally without disrupting business logic.