GitLab CI: end of runner crashes
Five Docker runner hosts, pipelines saturating CPU to 100% several times a week, and runner crashes requiring manual restarts. That was the starting point.
Before
- 5 hosts running Docker executor runners
- Pipeline saturates 100% CPU: 8–9 times a week
- Runner crash: ~3 times a week
- Job slowdowns felt every day
- cgroups limits not isolating workloads
After
- Kubernetes executor on RKE2 cluster
- ResourceQuota per project
- shell-operator assigns limits per service
- 0 runner crashes per month
- 40–60% faster pipelines
Project walkthrough
The problem was not compute capacity — 200 vCPU and 320 GB RAM is substantial — but the lack of isolation. CI jobs shared hosts without real limits, so one heavy pipeline could starve all the others and bring down a runner.
We moved job execution to Kubernetes executor: each job gets its own pod with explicitly declared CPU/RAM requests and limits. ResourceQuota per project guarantees no single team can consume the entire cluster, and shell-operator automatically assigns limits per service — without manually adding them to every repository.
The result: Kubernetes' scheduler packs jobs more densely and safely than a static host split, pipelines sped up by 40–60%, and since the rollout there has not been a single runner crash.
Technical deep-dive on the blog →
Is your CI choking during peak hours?
We'll show you how to fix it — first consultation is free.
Free consultation →