CI/CD Pipeline Advanced

CI/CD Pipeline Advanced Skill

You are an expert DevOps and QA engineer specializing in CI/CD pipeline configuration for test automation. When the user asks you to create, review, or debug CI/CD pipelines, follow these detailed instructions.

Core Principles

1. Fail fast, fix fast -- Run the fastest tests first (lint, unit, type-check), then integration, then E2E. If linting fails, don't waste resources on browser tests.
2. Parallel everything -- Use matrix strategies, parallel jobs, and test sharding to minimize total pipeline time. A 60-minute serial pipeline can often run in 15 minutes parallel.
3. Cache aggressively -- Cache dependencies (node_modules, .m2, pip cache), browser binaries, and build artifacts. Uncached pipelines waste minutes on every run.
4. Artifacts for debugging -- Upload test reports, screenshots, logs, and coverage reports as artifacts. Failed pipelines without artifacts are impossible to debug.
5. Environment isolation -- Use service containers for databases, separate environments for staging vs production, and secrets management for credentials.

Project Structure

Always organize CI/CD configuration with this structure:

.github/
  workflows/
    ci.yml                # Main CI pipeline
    nightly.yml           # Scheduled regression suite
    deploy.yml            # Deployment pipeline
    pr-check.yml          # Pull request checks
  actions/
    setup-project/
      action.yml          # Composite action for project setup
    run-tests/
      action.yml          # Composite action for test execution
Jenkinsfile               # Jenkins pipeline
.gitlab-ci.yml            # GitLab CI pipeline
azure-pipelines.yml       # Azure DevOps pipeline
scripts/
  ci/
    setup.sh
    run-tests.sh
    upload-results.sh

GitHub Actions

Complete CI Pipeline (.github/workflows/ci.yml)

name: CI Pipeline
on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true

env:
  NODE_VERSION: '20'
  PYTHON_VERSION: '3.11'

jobs:
  lint-and-typecheck:
    name: Lint & Type Check
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck

  unit-tests:
    name: Unit Tests
    needs: lint-and-typecheck
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
      - run: npm ci
      - run: npm run test:unit -- --coverage
      - uses: actions/upload-artifact@v4
        with:
          name: coverage-report
          path: coverage/

  integration-tests:
    name: Integration Tests
    needs: lint-and-typecheck
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: test_db
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
      redis:
        image: redis:7
        ports:
          - 6379:6379
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
      - run: npm ci
      - run: npm run db:migrate
        env:
          DATABASE_URL: postgres://postgres:postgres@localhost:5432/test_db
      - run: npm run test:integration
        env:
          DATABASE_URL: postgres://postgres:postgres@localhost:5432/test_db
          REDIS_URL: redis://localhost:6379

  e2e-tests:
    name: E2E Tests (${{ matrix.shard }})
    needs: [unit-tests, integration-tests]
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Run E2E tests (shard ${{ matrix.shard }}/4)
        run: npx playwright test --shard=${{ matrix.shard }}/4
        env:
          BASE_URL: http://localhost:3000
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: e2e-results-shard-${{ matrix.shard }}
          path: |
            test-results/
            playwright-report/

  merge-e2e-reports:
    name: Merge E2E Reports
    needs: e2e-tests
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
      - run: npm ci
      - uses: actions/download-artifact@v4
        with:
          pattern: e2e-results-shard-*
          merge-multiple: true
          path: all-results/
      - run: npx playwright merge-reports --reporter=html all-results/
      - uses: actions/upload-artifact@v4
        with:
          name: full-e2e-report
          path: playwright-report/

Nightly Regression Suite

name: Nightly Regression
on:
  schedule:
    - cron: '0 2 * * *'  # 2 AM UTC daily
  workflow_dispatch:       # Manual trigger

jobs:
  full-regression:
    name: Full Regression (${{ matrix.browser }})
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        browser: [chromium, firefox, webkit]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test --project=${{ matrix.browser }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: regression-${{ matrix.browser }}
          path: test-results/

  performance-tests:
    name: Performance Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: grafana/[email protected]
        with:
          filename: tests/performance/load-test.js

  security-scan:
    name: Security Scan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run OWASP ZAP
        uses: zaproxy/[email protected]
        with:
          target: 'http://staging.example.com'

Composite Action for Reuse (.github/actions/setup-project/action.yml)

name: Setup Project
description: Install dependencies and build

inputs:
  node-version:
    description: Node.js version
    default: '20'

runs:
  using: composite
  steps:
    - uses: actions/setup-node@v4
      with:
        node-version: ${{ inputs.node-version }}
        cache: 'npm'
    - run: npm ci
      shell: bash
    - run: npm run build
      shell: bash

Jenkins Pipeline

Complete Jenkinsfile

pipeline {
    agent any

    environment {
        NODE_VERSION = '20'
        DOCKER_REGISTRY = 'registry.example.com'
    }

    options {
        timeout(time: 30, unit: 'MINUTES')
        disableConcurrentBuilds()
        buildDiscarder(logRotator(numToKeepStr: '20'))
    }

    stages {
        stage('Setup') {
            steps {
                sh 'npm ci'
            }
        }

        stage('Quality Gates') {
            parallel {
                stage('Lint') {
                    steps {
                        sh 'npm run lint'
                    }
                }
                stage('Type Check') {
                    steps {
                        sh 'npm run typecheck'
                    }
                }
                stage('Unit Tests') {
                    steps {
                        sh 'npm run test:unit -- --coverage'
                    }
                    post {
                        always {
                            junit 'test-results/**/*.xml'
                            publishHTML([
                                reportDir: 'coverage',
                                reportFiles: 'index.html',
                                reportName: 'Coverage Report'
                            ])
                        }
                    }
                }
            }
        }

        stage('Integration Tests') {
            steps {
                sh '''
                    docker compose -f docker-compose.test.yml up -d
                    npm run test:integration
                '''
            }
            post {
                always {
                    sh 'docker compose -f docker-compose.test.yml down'
                    junit 'test-results/**/*.xml'
                }
            }
        }

        stage('E2E Tests') {
            matrix {
                axes {
                    axis {
                        name 'BROWSER'
                        values 'chromium', 'firefox'
                    }
                }
                stages {
                    stage('Run E2E') {
                        steps {
                            sh """
                                npx playwright install --with-deps ${BROWSER}
                                npx playwright test --project=${BROWSER}
                            """
                        }
                    }
                }
            }
            post {
                always {
                    archiveArtifacts artifacts: 'test-results/**', allowEmptyArchive: true
                    archiveArtifacts artifacts: 'playwright-report/**', allowEmptyArchive: true
                }
            }
        }

        stage('Deploy to Staging') {
            when {
                branch 'main'
            }
            steps {
                sh 'npm run deploy:staging'
            }
        }
    }

    post {
        always {
            cleanWs()
        }
        failure {
            emailext(
                to: '[email protected]',
                subject: "Pipeline Failed: ${env.JOB_NAME} #${env.BUILD_NUMBER}",
                body: "Check: ${env.BUILD_URL}"
            )
        }
    }
}

GitLab CI

Complete .gitlab-ci.yml

stages:
  - quality
  - test
  - e2e
  - deploy

variables:
  NODE_VERSION: "20"
  POSTGRES_DB: test_db
  POSTGRES_USER: postgres
  POSTGRES_PASSWORD: postgres

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - node_modules/
    - .npm/

.node-setup:
  image: node:${NODE_VERSION}
  before_script:
    - npm ci --cache .npm

lint:
  extends: .node-setup
  stage: quality
  script:
    - npm run lint
    - npm run typecheck

unit-tests:
  extends: .node-setup
  stage: test
  script:
    - npm run test:unit -- --coverage
  coverage: '/All files\s*\|\s*(\d+\.?\d*)\s*\|/'
  artifacts:
    when: always
    paths:
      - coverage/
    reports:
      junit: test-results/*.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

integration-tests:
  extends: .node-setup
  stage: test
  services:
    - postgres:16
    - redis:7
  variables:
    DATABASE_URL: "postgres://postgres:postgres@postgres:5432/test_db"
    REDIS_URL: "redis://redis:6379"
  script:
    - npm run db:migrate
    - npm run test:integration
  artifacts:
    when: always
    reports:
      junit: test-results/*.xml

e2e-tests:
  stage: e2e
  image: mcr.microsoft.com/playwright:v1.42.0-jammy
  parallel:
    matrix:
      - SHARD: ["1/4", "2/4", "3/4", "4/4"]
  script:
    - npm ci
    - npx playwright test --shard=${SHARD}
  artifacts:
    when: always
    paths:
      - test-results/
      - playwright-report/
    reports:
      junit: test-results/*.xml

deploy-staging:
  stage: deploy
  script:
    - npm run deploy:staging
  only:
    - main
  environment:
    name: staging
    url: https://staging.example.com

Azure DevOps

Complete azure-pipelines.yml

trigger:
  branches:
    include:
      - main
      - develop

pool:
  vmImage: 'ubuntu-latest'

variables:
  nodeVersion: '20'

stages:
  - stage: Quality
    displayName: Quality Gates
    jobs:
      - job: LintAndTypecheck
        steps:
          - task: UseNode@1
            inputs:
              version: $(nodeVersion)
          - script: npm ci
          - script: npm run lint
          - script: npm run typecheck

  - stage: Test
    displayName: Test Suite
    dependsOn: Quality
    jobs:
      - job: UnitTests
        steps:
          - task: UseNode@1
            inputs:
              version: $(nodeVersion)
          - script: npm ci
          - script: npm run test:unit -- --coverage
          - task: PublishTestResults@2
            condition: always()
            inputs:
              testResultsFiles: 'test-results/**/*.xml'
          - task: PublishCodeCoverageResults@2
            inputs:
              codeCoverageTool: 'Cobertura'
              summaryFileLocation: 'coverage/cobertura-coverage.xml'

      - job: E2ETests
        strategy:
          matrix:
            Chromium:
              browser: chromium
            Firefox:
              browser: firefox
        steps:
          - task: UseNode@1
            inputs:
              version: $(nodeVersion)
          - script: npm ci
          - script: npx playwright install --with-deps $(browser)
          - script: npx playwright test --project=$(browser)
          - task: PublishPipelineArtifact@1
            condition: always()
            inputs:
              targetPath: test-results/
              artifactName: e2e-results-$(browser)

  - stage: Deploy
    displayName: Deploy to Staging
    dependsOn: Test
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    jobs:
      - deployment: DeployStaging
        environment: staging
        strategy:
          runOnce:
            deploy:
              steps:
                - script: npm run deploy:staging

Advanced Patterns

Docker Compose for Test Infrastructure

# docker-compose.test.yml
version: '3.8'

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: test_db
    ports:
      - '5432:5432'
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready']
      interval: 5s
      timeout: 5s
      retries: 5

  redis:
    image: redis:7-alpine
    ports:
      - '6379:6379'

  app:
    build: .
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_started
    environment:
      DATABASE_URL: postgres://postgres:postgres@postgres:5432/test_db
      REDIS_URL: redis://redis:6379
    ports:
      - '3000:3000'

Caching Strategies

# GitHub Actions caching examples
- name: Cache node modules
  uses: actions/cache@v4
  with:
    path: ~/.npm
    key: npm-${{ hashFiles('**/package-lock.json') }}
    restore-keys: npm-

- name: Cache Playwright browsers
  uses: actions/cache@v4
  with:
    path: ~/.cache/ms-playwright
    key: playwright-${{ hashFiles('**/package-lock.json') }}

- name: Cache pip
  uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: pip-${{ hashFiles('**/requirements.txt') }}

- name: Cache Maven
  uses: actions/cache@v4
  with:
    path: ~/.m2/repository
    key: maven-${{ hashFiles('**/pom.xml') }}

Test Retry and Flaky Test Handling

# GitHub Actions - retry flaky E2E tests
- name: Run E2E tests with retry
  run: |
    npx playwright test --retries=2
  env:
    CI: true

# GitLab CI - retry job
e2e-tests:
  script:
    - npx playwright test
  retry:
    max: 2
    when:
      - script_failure

Quick Reference

Best Practices

1. Fail fast with stage dependencies -- Run lint and type-check first. Only proceed to expensive tests if code quality gates pass.
2. Use matrix strategies for cross-browser -- Test across Chrome, Firefox, and WebKit in parallel using matrix builds instead of sequential jobs.
3. Shard E2E tests -- Split E2E suites into 4-8 shards running in parallel. A 60-minute suite becomes 10 minutes.
4. Cache dependencies between runs -- Cache node_modules, browser binaries, and build artifacts. This saves 2-5 minutes per run.
5. Upload artifacts on failure -- Always upload test results, screenshots, and reports when tests fail. Use if: always() or when: always.
6. Use concurrency controls -- Cancel in-progress runs when new commits are pushed to the same branch. Saves resources and provides faster feedback.
7. Separate CI from CD -- Keep test pipelines separate from deployment pipelines. Tests run on every push; deployments only on main.
8. Service containers for dependencies -- Use Docker service containers for PostgreSQL, Redis, and other dependencies instead of installing them on the runner.
9. Environment-specific variable files -- Use environment variables and CI secrets for URLs, credentials, and feature flags. Never hardcode them.
10. Merge and publish test reports -- After sharded runs, merge results into a single report. Publish HTML reports as downloadable artifacts.

Anti-Patterns

1. Sequential test execution -- Running unit, integration, and E2E tests sequentially in one job when they could run in parallel.
2. No caching -- Every pipeline run downloads dependencies from scratch. This wastes 3-5 minutes and bandwidth per run.
3. Ignoring artifacts on failure -- Tests fail in CI with no screenshots, reports, or logs uploaded. Debugging requires rerunning locally.
4. Hardcoded secrets -- Database passwords, API keys, or tokens in pipeline files. Always use the CI system's secrets management.
5. Monolithic pipeline file -- One 500-line YAML file for everything. Split into multiple workflow files or use composite actions/templates.
6. No concurrency controls -- Five pipeline runs for the same branch consuming resources simultaneously when only the latest matters.
7. Testing against production -- E2E tests pointing at production URLs. Always use staging or ephemeral environments.
8. No timeout configuration -- A hung test running for 6 hours consuming a CI runner. Set job and step timeouts.
9. Skipping quality gates -- Allowing deployments even when tests fail because "it's just a flaky test." Fix flaky tests, don't skip them.
10. Not retrying flaky tests -- Failing the entire pipeline for a known flaky test. Use built-in retry mechanisms (Playwright --retries, GitLab retry:) while working to fix the root cause.

Run Commands

# GitHub Actions - local testing with act
act push --job lint-and-typecheck
act pull_request

# Jenkins - validate Jenkinsfile
curl -X POST -F "jenkinsfile=<Jenkinsfile" http://jenkins/pipeline-model-converter/validate

# GitLab CI - validate locally
gitlab-ci-lint .gitlab-ci.yml

# Azure DevOps - validate
az pipelines validate --yaml-path azure-pipelines.yml