GitLab CI/CD on Kubernetes: Hard Resource Limits per Project

200 vCPU. 320 GB RAM. Five hosts running Docker runners. Still not enough.

One poorly designed pipeline could consume an entire node. The rest of the jobs? Waiting in queue or crashing with OOM. Sound familiar?

📋 TL;DR

• Problem: Docker runner doesn’t isolate resources — one pipeline can saturate the entire host
• Solution: Kubernetes executor + automatic ResourceQuota per project
• Bonus: Shell-operator for setting limits per service (PostgreSQL vs Redis)
• Result: 40-60% faster pipelines, zero runner crashes, full predictability

---

⚠️ Problem: Docker runner doesn’t protect against “noisy neighbors”

The client had a classic CI/CD infrastructure:

• 5 hosts with GitLab Docker runners
• 200 vCPU + 320 GB RAM total
• Dozens of projects competing for resources

Consequences of no isolation:

Symptom	Frequency
Pipeline saturates 100% CPU on host	8-9x / week
Runner crash requiring manual intervention	3x / week
Job slowdown due to lack of resources	daily

Docker limits based on cgroups are “soft”. Under heavy load, they don’t provide the isolation that Kubernetes offers with ResourceQuota and LimitRange.

🎯 Goal: Hard limits per project

Client requirements were clear:

1. Hard CPU/RAM limits at the project or project group level
2. Different resource pools for different teams:
   • Frontend: 10 vCPU / 10 GiB RAM
   • Backend: 20 vCPU / 20 GiB RAM
3. Flexibility — teams decide how to distribute budget between containers in a job
4. Zero manual work — namespaces and quotas are created automatically

🔧 Step 1: Kubernetes executor with dynamic namespaces

First step was spinning up an RKE2 cluster and installing GitLab Runner with Kubernetes executor.

Key runner configuration snippet:

[[runners]]
  [runners.kubernetes]
    namespace_overwrite_allowed = "ci-.*"

This allows overriding namespace per job using the KUBERNETES_NAMESPACE_OVERWRITE variable.

In the group/project template:

variables:
  KUBERNETES_NAMESPACE_OVERWRITE: "ci-${CI_PROJECT_NAMESPACE}-${CI_PROJECT_NAME}"

Mapping result:

GitLab Project	K8s Namespace
backend-projects/app1	ci-backend-projects-app1
frontend-projects/app2	ci-frontend-projects-app2

Each project = separate namespace = possibility of its own ResourceQuota.

🔧 Step 2: Automatic ResourceQuota per group

For automatic quota creation, we used nsinjector (today Kyverno would make more sense, but the principle is the same).

Configuration for frontend and backend groups:

- name: ci-quota-injector-front
  namespaces:
    - ci-frontend-projects-*
  excludedNamespaces:
    - ci-backend-projects-*
  resources:
    - |
      apiVersion: v1
      kind: ResourceQuota
      metadata:
        name: ci-resourcequota
      spec:
        hard:
          requests.cpu: "10"
          requests.memory: 10Gi
          limits.cpu: "10"
          limits.memory: 10Gi

- name: ci-quota-injector-back
  namespaces:
    - ci-backend-projects-*
  excludedNamespaces:
    - ci-frontend-projects-*
  resources:
    - |
      apiVersion: v1
      kind: ResourceQuota
      metadata:
        name: ci-resourcequota
      spec:
        hard:
          requests.cpu: "20"
          requests.memory: 20Gi
          limits.cpu: "20"
          limits.memory: 20Gi

In practice:

• Project backend-projects/app1 → limit 20 CPU / 20 GiB
• Project frontend-projects/app2 → limit 10 CPU / 10 GiB

Kubernetes enforces the sum of requests/limits in the namespace and doesn’t allow exceeding them.

💡 Pro tip: Need individual limits for a specific project? Add a separate namespace pattern + dedicated ResourceQuota.

⚠️ Problem #2: limits per service in a job

GitLab runner allows setting limits for services, but shared across all of them:

[runners.kubernetes]
  service_cpu_limit = "500m"
  service_cpu_limit_overwrite_max_allowed = "10000m"
  service_cpu_request = "128m"
  service_cpu_request_overwrite_max_allowed = "10000m"
  service_memory_limit = "512Mi"
  service_memory_limit_overwrite_max_allowed = "10Gi"
  service_memory_request = "128Mi"
  service_memory_request_overwrite_max_allowed = "10Gi"

You can override them per job in .gitlab-ci.yml, but they’re still shared for all services.

That’s not enough if PostgreSQL needs 2 GiB RAM, while Redis only needs 256 MiB.

🔧 Solution: Shell-operator for patching per service

We used shell-operator from Flant — a tool that runs hooks in response to K8s events.

How it works:

1. Runner assigns Pods the label shell-operator: "true"
2. Shell-operator watches for Pods with this label
3. When a Pod is created, the hook:
   • Reads spec from BINDING_CONTEXT_PATH
   • Identifies service containers
   • Generates a patch based on env from .gitlab-ci.yml or default values
   • Executes kubectl patch pod

Full hook:

#!/usr/bin/env bash

if [[ $1 == "--config" ]] ; then
  cat <<EOF
configVersion: v1
kubernetes:
- apiVersion: v1
  kind: Pod
  executeHookOnEvent:
  - Added
  labelSelector:
    matchLabels:
      shell-operator: "true"
  allowFailure: true
EOF
else
  type=$(jq -r '.[0].type' $BINDING_CONTEXT_PATH)
  if [[ $type == "Event" ]] ; then
    echo $BINDING_CONTEXT_PATH
    # Default limits per service type
    source ./limits.conf

    # Pod name and namespace
    podName=$(jq -r '.[0].object.metadata.name' $BINDING_CONTEXT_PATH)
    podNamespace=$(jq -r '.[0].object.metadata.namespace' $BINDING_CONTEXT_PATH)

    # All containers in the Pod
    podContainers=$(jq -r '.[0].object.spec.containers[].name' $BINDING_CONTEXT_PATH)

    # List of containers that are services (names containing "svc")
    svc_containers=$(echo $podContainers | awk -F" " '{for(i=1;i<=NF;i++){if ($i ~ /svc/){print $i}}}')

    if [[ ! -z $svc_containers ]] ; then
      for i in $svc_containers; do
        # Service container image
        SVC_IMAGE_NAME=$(jq --arg svc_container_name "$i" -r '.[0].object.spec.containers[] | select(.name==$svc_container_name).image' $BINDING_CONTEXT_PATH)

        # Try to read values from env set in .gitlab-ci
        CPU_LIMIT=$(jq --arg svc_container_name "$i" -r '.[0].object.spec.containers[] | select(.name==$svc_container_name).env | .[] | select(.name=="KUBERNETES_SVC_CPU_LIMIT").value' $BINDING_CONTEXT_PATH)

        # If no env in job – default values from limits.conf
        if [[ -z $CPU_LIMIT ]] ; then
          case "$SVC_IMAGE_NAME" in
            *postgresql*)
              CPU_LIMIT=$KUBERNETES_PSQL_CPU_LIMIT
              CPU_REQUEST=$KUBERNETES_PSQL_CPU_REQUEST
              MEM_LIMIT=$KUBERNETES_PSQL_MEMORY_LIMIT
              MEM_REQUEST=$KUBERNETES_PSQL_MEMORY_REQUEST
            ;;
            # Here you can add redis, elasticsearch, mysql etc.
          esac
        elif [[ ! -z $CPU_LIMIT ]] ; then
          CPU_REQUEST=$(jq --arg svc_container_name "$i" -r '.[0].object.spec.containers[] | select(.name==$svc_container_name).env | .[] | select(.name=="KUBERNETES_SVC_CPU_REQUEST").value' $BINDING_CONTEXT_PATH)
          MEM_LIMIT=$(jq --arg svc_container_name "$i" -r '.[0].object.spec.containers[] | select(.name==$svc_container_name).env | .[] | select(.name=="KUBERNETES_SVC_MEM_LIMIT").value' $BINDING_CONTEXT_PATH)
          MEM_REQUEST=$(jq --arg svc_container_name "$i" -r '.[0].object.spec.containers[] | select(.name==$svc_container_name).env | .[] | select(.name=="KUBERNETES_SVC_MEM_REQUEST").value' $BINDING_CONTEXT_PATH)
        else
          echo "Nothing to do! Exiting.."
          exit 0;
        fi

        # Substitute values into patch template
        sed -e "s/SVC_CONTAINER_NAME/$i/" \
            -e "s/CPU_LIMIT/$CPU_LIMIT/" \
            -e "s/CPU_REQUEST/$CPU_REQUEST/" \
            -e "s/MEM_LIMIT/$MEM_LIMIT/" \
            -e "s/MEM_REQUEST/$MEM_REQUEST/" svc-env.yaml > $i.yaml

        # Patch container resources in the Pod
        kubectl patch pod $podName -n $podNamespace --patch-file $i.yaml
      done
    else
      echo "Nothing to do! Exiting.."
      exit 0;
    fi
  fi
fi

Template svc-env.yaml:

spec:
  containers:
    - name: "SVC_CONTAINER_NAME"
      resources:
        limits:
          cpu: "CPU_LIMIT"
          memory: "MEM_LIMIT"
        requests:
          cpu: "CPU_REQUEST"
          memory: "MEM_REQUEST"

Usage in .gitlab-ci.yml:

services:
  - name: postgres:15
    alias: psql-svc
    variables:
      KUBERNETES_SVC_CPU_LIMIT: "1000m"
      KUBERNETES_SVC_CPU_REQUEST: "500m"
      KUBERNETES_SVC_MEM_LIMIT: "2Gi"
      KUBERNETES_SVC_MEM_REQUEST: "1Gi"

No variables? Default values from limits.conf kick in.

📊 Deployment results

Metric	Before	After
Average pipeline time	100% (baseline)	40-60% faster
Runner crashes / month	~12	0
Manual interventions / week	2-3	0
Time predictability	❌ random	✅ stable

What changed:

• ✅ Guaranteed resource pool — each project has its own, doesn’t compete with others
• ✅ End of “noisy neighbors” — one pipeline can’t ruin the day for the entire team
• ✅ Conscious management — teams know how much they have and how to distribute it
• ✅ Zero crashes — Kubernetes throttles instead of crashing

⚠️ Honest trade-off: Some pipelines (those that used to consume the entire node) now run longer. But they’re predictable and don’t disrupt others’ work.

🎯 When does this make sense?

This solution works well when:

• You have many teams / projects sharing CI/CD infrastructure
• You experience the “noisy neighbor” problem — one project hurts others
• You need auditability and limits per team
• Your pipelines use heavy services (databases, search, cache)

You don’t need this if:

• You have few projects and plenty of resources
• All pipelines are similar and lightweight
• You don’t have Kubernetes expertise on the team

---

Summary

Migration from Docker runners to Kubernetes executor isn’t just a technology change. It’s a paradigm shift in CI/CD resource management.

Instead of “first come, first served” you get hard guarantees.

Kubernetes with ResourceQuota + shell-operator for fine-grained control = full control over who can use what. And the predictability that Docker runner simply can’t provide.

---

Having a similar CI/CD problem? We’d be happy to discuss your case.