Why Your Deployments Take 45 Minutes (And How to Fix It)

Why Your Deployments Take 45 Minutes (And How to Fix It)

Fast deployments aren’t just convenient – they’re a competitive advantage.

When deployments take 45 minutes, engineers batch changes. When they take 5 minutes, they ship continuously.

After optimizing deployment pipelines for 15+ teams, here are the 6 bottlenecks slowing you down.

Bottleneck 1: Building Docker Images From Scratch Every Time

Symptom: docker build takes 15-20 minutes per deployment.

Root cause: No layer caching, or poor Dockerfile ordering that invalidates cache early.

Real example: Node.js app with this Dockerfile:

FROM node:18
COPY . /app
WORKDIR /app
RUN npm install
RUN npm run build
CMD ["npm", "start"]

Problem: COPY . /app runs before npm install. Any code change invalidates the npm install cache.

Build time: 18 minutes (12 minutes installing dependencies, 6 minutes building).

Fix: Optimize layer ordering:

FROM node:18
WORKDIR /app

# Install dependencies first (changes rarely)
COPY package*.json ./
RUN npm ci --only=production

# Copy source code second (changes frequently)
COPY . .
RUN npm run build

CMD ["npm", "start"]

Result: Build time drops to 3 minutes (dependencies cached, only rebuild app code).

Additional optimization: Use multi-stage builds to keep final image small:

FROM node:18 AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:18-alpine
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
CMD ["node", "dist/index.js"]

Bottleneck 2: Running Full Test Suite on Every Commit

Symptom: CI pipeline sits in “running tests” for 20+ minutes.

Root cause: All tests run sequentially, including slow integration/E2E tests.

Real example: SaaS platform with test suite taking 25 minutes:

  • Unit tests: 3 minutes
  • Integration tests: 12 minutes
  • E2E tests: 10 minutes

Fix: Parallelize and split by type:

# GitHub Actions example
jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - run: npm run test:unit # 3 minutes

  integration-tests:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - run: npm run test:integration --shard=${{ matrix.shard }}/4 # 3 minutes each

  e2e-tests:
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main' # Only on main branch
    steps:
      - run: npm run test:e2e # 10 minutes

Result: Total pipeline time drops from 25 minutes to 10 minutes (3 min unit + 3 min integration in parallel, E2E only on main).

Bottleneck 3: Cold Start CI Runners

Symptom: Pipeline spends 5-8 minutes “setting up environment” before any real work.

Root cause: CI runners start from scratch every time – installing dependencies, downloading Docker images, setting up databases.

Real example: Deployment pipeline breakdown:

  • Provision runner: 2 minutes
  • Install dependencies: 4 minutes
  • Download base images: 2 minutes
  • Actual build/test: 8 minutes
  • Total: 16 minutes (50% overhead)

Fix: Use self-hosted runners with warm caches:

  • Pre-install common dependencies (Node.js, Python, Docker)
  • Cache npm/pip/maven packages locally
  • Pull common Docker base images ahead of time

Alternative: Use CI-specific optimization features:

  • GitHub Actions: actions/cache for dependencies
  • GitLab CI: cache: directive
  • CircleCI: save_cache and restore_cache

Result: Setup time drops from 8 minutes to 1 minute.

Bottleneck 4: Deploying to Production One Server at a Time

Symptom: Rolling deployment takes 15+ minutes to update all instances.

Root cause: Conservative deployment strategy updating 1 instance every 2 minutes.

Real example: Kubernetes deployment with these settings:

spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1

With 10 replicas, this updates 1 pod at a time. If each pod takes 90 seconds to become ready, deployment takes 15 minutes.

Fix: Increase parallelism (if your system can handle it):

spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0  # Maintain capacity
      maxSurge: 5        # Update 5 at once

Result: Deployment time drops from 15 minutes to 3 minutes.

Caution: Only increase if you have proper health checks and can handle partial deployments.

Bottleneck 5: Waiting for DNS Propagation

Symptom: Deployment succeeds but you wait 5-10 minutes “for DNS to propagate.”

Root cause: High TTL values on DNS records (300-3600 seconds).

Fix:

  • Lower TTL to 60 seconds for records you change frequently
  • Use health checks that wait for actual service readiness, not just DNS
  • Consider using service mesh (Istio, Linkerd) for internal routing – no DNS delays

Bottleneck 6: No Deployment Rollback Strategy

Symptom: When deployments fail, you spend 20 minutes debugging and re-deploying instead of rolling back.

Real example: Deployment introduces bug. Team’s response:

  1. Notice error in production (5 minutes)
  2. Debug the issue (10 minutes)
  3. Push fix (2 minutes)
  4. Wait for CI/CD pipeline (15 minutes)
  5. Total downtime: 32 minutes

Fix: Implement instant rollback:

# Kubernetes
kubectl rollout undo deployment/api-server

# Docker Swarm
docker service rollback api-server

# ECS
aws ecs update-service --service api --task-definition api:previous

Better response with rollback:

  1. Notice error (5 minutes)
  2. Rollback to previous version (30 seconds)
  3. Debug and fix at leisure
  4. Total downtime: 6 minutes

The Compound Effect

Fix all 6 bottlenecks and watch what happens to your deployment time:

Stage Before After
Docker build 18 min 3 min
Test suite 25 min 10 min
CI setup 8 min 1 min
Deployment 15 min 3 min
DNS wait 5 min 0 min
Total 71 min 17 min

76% reduction in deployment time.

Now you can deploy 10x per day instead of 1-2x.

Platform Acceleration Programme

Our 10-week Platform Acceleration Programme includes complete CI/CD pipeline optimization:

  • Audit current deployment process and identify bottlenecks
  • Optimize Docker builds with multi-stage builds and caching
  • Implement parallel test execution
  • Configure proper rolling deployments with health checks
  • Set up instant rollback mechanisms
  • Document deployment playbooks for the team

Guaranteed outcome: 50%+ faster deployments or Phase 1 is free.

Book a Discovery Call

Let’s talk about your current deployment process and where the bottlenecks are.

↑