Component Testing Strategies

This guide covers comprehensive testing strategies for GitLab CI/CD components.

Testing Levels

1. Syntax Validation

Test: YAML is valid Tool: yq, yamllint Speed: Seconds Frequency: Every commit

2. Unit Tests

Test: Component logic works in isolation Tool: GitLab CI jobs Speed: Minutes Frequency: Every commit

3. Integration Tests

Test: Component works with real pipelines Tool: Test projects Speed: Minutes to hours Frequency: Before release

4. End-to-End Tests

Test: Complete workflows with components Tool: Real projects in staging Speed: Hours Frequency: Before major releases

5. Acceptance Tests

Test: Component meets user requirements Tool: Beta testing with real teams Speed: Days to weeks Frequency: Major releases only

Syntax Validation

Basic YAML Validation

# .gitlab-ci.yml
validate-yaml:
  stage: validate
  image: alpine:latest
  before_script:
    - apk add --no-cache yq
  script:
    - |
      echo "Validating component YAML files..."
      for file in templates/*.yml templates/*/template.yml; do
        if [ -f "$file" ]; then
          echo "Checking $file"
          yq eval '.' "$file" > /dev/null || exit 1
          echo " $file is valid"
        fi
      done

Advanced YAML Validation

# .gitlab-ci.yml
validate-components:
  stage: validate
  image: alpine:latest
  before_script:
    - apk add --no-cache yq python3 py3-pip
    - pip3 install yamllint
  script:
    - |
      # Validate YAML syntax
      yamllint templates/

      # Validate component structure
      for file in templates/*.yml templates/*/template.yml; do
        if [ -f "$file" ]; then
          echo "Validating $file"

          # Check for required spec section
          if ! yq eval '.spec' "$file" > /dev/null 2>&1; then
            echo "ERROR: $file missing spec section"
            exit 1
          fi

          # Check for separator between spec and jobs
          if ! grep -q '^---$' "$file"; then
            echo "ERROR: $file missing --- separator"
            exit 1
          fi

          echo " $file structure valid"
        fi
      done

Schema Validation

# component-schema.json
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["spec"],
  "properties": {
    "spec": {
      "type": "object",
      "properties": {
        "inputs": {
          "type": "object",
          "patternProperties": {
            "^[a-z_]+$": {
              "type": "object",
              "properties": {
                "type": {
                  "enum": ["string", "number", "boolean", "array"]
                },
                "default": {},
                "options": {
                  "type": "array"
                },
                "regex": {
                  "type": "string"
                }
              }
            }
          }
        }
      }
    }
  }
}

# .gitlab-ci.yml
validate-schema:
  stage: validate
  image: node:20
  script:
    - npm install -g ajv-cli
    - |
      for file in templates/*.yml templates/*/template.yml; do
        if [ -f "$file" ]; then
          echo "Validating $file against schema"
          yq eval '.spec' "$file" -o json | ajv validate -s component-schema.json -d -
        fi
      done

Unit Testing

Testing Individual Components

# .gitlab-ci.yml
test-docker-build:
  stage: test
  needs: []
  variables:
    TEST_IMAGE_NAME: test-image:latest
  trigger:
    include:
      - local: templates/docker-build.yml
    strategy: depend
  script:
    - |
      # Set up inputs for component
      cat > .gitlab-ci.yml <<EOF
      include:
        - local: templates/docker-build.yml

      variables:
        INPUTS_IMAGE_NAME: ${TEST_IMAGE_NAME}
        INPUTS_DOCKERFILE: test/Dockerfile
      EOF

      # Trigger pipeline
      curl --request POST \
        --form "token=${CI_JOB_TOKEN}" \
        --form "ref=main" \
        "https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/trigger/pipeline"

Input Validation Testing

# tests/test-inputs.yml
test-valid-inputs:
  stage: test
  script:
    - |
      # Test with valid inputs
      include:
        - local: templates/k8s-deploy.yml

      # Verify component accepts valid inputs
      test-valid:
        stage: test
        script:
          - echo "Testing valid inputs"
          # Component should work
  artifacts:
    when: on_failure

test-invalid-inputs:
  stage: test
  script:
    - |
      # Test with invalid inputs (should fail)
      include:
        - local: templates/k8s-deploy.yml

      # This should fail validation
      test-invalid:
        stage: test
        script:
          - echo "Testing invalid inputs"
          # Component should reject invalid inputs
  allow_failure: true  # We expect this to fail

Mock Testing

# tests/test-docker-build.yml
# Mock external dependencies

test-docker-build-mock:
  stage: test
  image: docker:latest
  services:
    - docker:dind
  before_script:
    # Mock docker push (don't actually push)
    - |
      cat > /usr/local/bin/docker-push-mock.sh <<'EOF'
      #!/bin/sh
      echo "MOCK: docker push $1"
      exit 0
      EOF
      chmod +x /usr/local/bin/docker-push-mock.sh
      alias docker push='docker-push-mock.sh'
  script:
    # Test component
    - |
      include:
        - local: templates/docker-build.yml

      # Build should work, push is mocked

Integration Testing

Test Project Pattern

Create dedicated test projects:

gitlab_components/
 templates/
    docker-build.yml
 tests/
     test-docker-simple/
        .gitlab-ci.yml
        Dockerfile
        app.js
     test-docker-multi-stage/
        .gitlab-ci.yml
        Dockerfile.multi
        app.go
     test-node-pipeline/
         .gitlab-ci.yml
         package.json
         src/index.js

Test Project: Simple Docker Build

# tests/test-docker-simple/.gitlab-ci.yml
include:
  - component: $CI_PROJECT_PATH/docker-build@$CI_COMMIT_SHA
    inputs:
      image_name: test-simple:$CI_COMMIT_SHA
      dockerfile: Dockerfile

verify:
  stage: test
  image: docker:latest
  script:
    - docker pull test-simple:$CI_COMMIT_SHA
    - docker run test-simple:$CI_COMMIT_SHA node --version

Test Project: Complex Pipeline

# tests/test-node-pipeline/.gitlab-ci.yml
include:
  - component: $CI_PROJECT_PATH/node-pipeline@$CI_COMMIT_SHA
    inputs:
      node_version: '20'
      enable_lint: true
      enable_tests: true
      enable_build: true

verify-build:
  stage: deploy
  script:
    - test -d dist/
    - ls -la dist/

Multi-Project Test Pipeline

# .gitlab-ci.yml in component repository
stages:
  - validate
  - unit-test
  - integration-test
  - release

# Syntax validation
validate-yaml:
  stage: validate
  script:
    - yamllint templates/

# Unit tests
.unit-test-template:
  stage: unit-test
  needs: [validate-yaml]

test-docker-build:
  extends: .unit-test-template
  script:
    - test-component.sh docker-build

test-k8s-deploy:
  extends: .unit-test-template
  script:
    - test-component.sh k8s-deploy

# Integration tests
.integration-test-template:
  stage: integration-test
  needs: []
  trigger:
    strategy: depend

test-simple-docker:
  extends: .integration-test-template
  trigger:
    project: my-org/gitlab_components-tests/test-docker-simple
    branch: main

test-node-pipeline:
  extends: .integration-test-template
  trigger:
    project: my-org/gitlab_components-tests/test-node-pipeline
    branch: main

test-multi-stage:
  extends: .integration-test-template
  trigger:
    project: my-org/gitlab_components-tests/test-docker-multi-stage
    branch: main

# Release (only if all tests pass)
release:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli:latest
  rules:
    - if: $CI_COMMIT_TAG
  needs:
    - test-simple-docker
    - test-node-pipeline
    - test-multi-stage
  script:
    - echo "All tests passed, releasing $CI_COMMIT_TAG"
  release:
    tag_name: $CI_COMMIT_TAG
    description: 'Release $CI_COMMIT_TAG'

Test Coverage

Component Coverage Matrix

Track which scenarios are tested:

# tests/coverage-matrix.yml
coverage:
  docker-build:
    tested_scenarios:
      - simple_dockerfile
      - multi_stage_dockerfile
      - custom_build_args
      - different_contexts
    input_combinations:
      - image_name: required
      - dockerfile: [Dockerfile, custom.Dockerfile]
      - context: [., ./app, ./services/api]
      - build_args: [null, "--build-arg VERSION=1.0"]

  k8s-deploy:
    tested_scenarios:
      - rolling_update
      - blue_green_deployment
      - canary_deployment
      - rollback
    input_combinations:
      - strategy: [rolling, blue-green, canary]
      - replicas: [1, 3, 5]
      - environment: [dev, staging, production]

Automated Coverage Reporting

// tests/coverage-report.ts
interface TestCoverage {
  component: string;
  total_inputs: number;
  tested_inputs: number;
  coverage_percentage: number;
  untested_scenarios: string[];
}

function generateCoverageReport(): TestCoverage[] {
  const components = scanComponents('templates/');
  const tests = scanTests('tests/');

  return components.map(component => {
    const inputs = extractInputs(component);
    const testedInputs = extractTestedInputs(tests, component.name);

    return {
      component: component.name,
      total_inputs: inputs.length,
      tested_inputs: testedInputs.length,
      coverage_percentage: (testedInputs.length / inputs.length) * 100,
      untested_scenarios: inputs.filter(i => !testedInputs.includes(i))
    };
  });
}

// Generate report
const report = generateCoverageReport();
console.table(report);

// Fail if coverage < 80%
const avgCoverage = report.reduce((sum, r) => sum + r.coverage_percentage, 0) / report.length;
if (avgCoverage < 80) {
  console.error(`Test coverage ${avgCoverage.toFixed(1)}% below 80% threshold`);
  process.exit(1);
}

Beta Testing

Beta Testing Process

Phase 1: Alpha Testing (Internal)

# Select 2-3 internal test projects
alpha_test_projects:
  - test-agent-mesh        # Complex microservices
  - test-simple-api        # Simple REST API
  - test-static-site       # Static site

timeline: 1 week
criteria:
  - All pipelines pass
  - No regressions
  - Performance acceptable

Phase 2: Beta Testing (Early Adopters)

# Select 5-10 early adopter projects
beta_test_projects:
  - agent-mesh             # Production service
  - documentation-site     # Low risk
  - internal-tools         # Internal users
  - experimental-api       # Experimental
  - legacy-migration       # Edge case

timeline: 2 weeks
criteria:
  - 90% of pipelines pass
  - <5% rollback rate
  - Positive user feedback
  - Migration guide validated

Beta Feedback Template

# Beta Testing Feedback: Component v3.0.0-beta.1

## Project Information
- **Project:** agent-mesh
- **Team:** Platform Team
- **Tester:** @alice
- **Test Date:** 2026-01-15

## Test Scenario
Tested k8s-deploy component with blue-green deployment strategy.

## Results
- [x] Pipeline succeeded
- [ ] No issues encountered
- [x] Found 2 minor issues

## Issues Found

### Issue 1: Timeout Too Short
**Severity:** Medium
**Description:** Blue-green deployment times out after 5 minutes, needs 10.
**Workaround:** Manually set timeout in project config.
**Recommendation:** Add `timeout` input to component.

### Issue 2: Unclear Error Message
**Severity:** Low
**Description:** When namespace doesn't exist, error message is cryptic.
**Recommendation:** Add better error handling.

## Migration Experience
- **Difficulty:** Easy
- **Time:** 30 minutes
- **Documentation Quality:** 4/5
- **Issues:** None

## Overall Feedback
Component works well. Two minor issues need addressing. Migration guide was clear.

## Recommendation
- [ ] Approve for stable release
- [x] Approve after fixes
- [ ] Needs more work

Collecting Beta Metrics

// beta-metrics.ts
interface BetaMetrics {
  project: string;
  pipeline_success_rate: number;
  avg_pipeline_duration: number;
  errors_encountered: number;
  rollback_required: boolean;
  user_satisfaction: number;  // 1-5
}

async function collectBetaMetrics(projects: string[]): Promise<BetaMetrics[]> {
  return Promise.all(
    projects.map(async project => {
      const pipelines = await fetchPipelines(project, '2 weeks');
      const issues = await fetchIssues(project, 'label:component-beta');

      return {
        project,
        pipeline_success_rate: calculateSuccessRate(pipelines),
        avg_pipeline_duration: calculateAvgDuration(pipelines),
        errors_encountered: issues.length,
        rollback_required: checkRollback(project),
        user_satisfaction: await getUserRating(project)
      };
    })
  );
}

// Generate beta report
const metrics = await collectBetaMetrics(BETA_PROJECTS);
const summary = {
  total_projects: metrics.length,
  avg_success_rate: average(metrics.map(m => m.pipeline_success_rate)),
  projects_with_issues: metrics.filter(m => m.errors_encountered > 0).length,
  rollbacks: metrics.filter(m => m.rollback_required).length,
  avg_satisfaction: average(metrics.map(m => m.user_satisfaction))
};

console.log('Beta Testing Summary:', summary);

// Decide if ready for stable release
if (summary.avg_success_rate > 90 && summary.avg_satisfaction >= 4.0) {
  console.log(' Ready for stable release');
} else {
  console.log(' Not ready - needs more work');
}

Regression Testing

Automated Regression Suite

# .gitlab-ci.yml
regression-tests:
  stage: test
  rules:
    - if: $CI_COMMIT_TAG  # Run on every tag
  parallel:
    matrix:
      - COMPONENT: [docker-build, k8s-deploy, security-scan, node-pipeline]
        TEST_SUITE: [basic, advanced, edge-cases]
  script:
    - run-regression-suite.sh $COMPONENT $TEST_SUITE
  artifacts:
    reports:
      junit: regression-${COMPONENT}-${TEST_SUITE}.xml

Regression Test Suite Structure

tests/
 regression/
     docker-build/
        basic/
           simple-build.yml
           with-cache.yml
        advanced/
           multi-stage.yml
           build-args.yml
        edge-cases/
            no-dockerfile.yml
            large-context.yml
     k8s-deploy/
         basic/
         advanced/
         edge-cases/

Version Compatibility Testing

# Test new version with multiple GitLab versions
compatibility-test:
  stage: test
  parallel:
    matrix:
      - GITLAB_VERSION: [15.11, 16.0, 16.5, 17.0]
  image: gitlab/gitlab-ce:${GITLAB_VERSION}
  script:
    - test-component-compatibility.sh

Performance Testing

Pipeline Duration Testing

# tests/performance-test.yml
performance-test:
  stage: test
  script:
    - |
      echo "Testing pipeline duration..."
      START_TIME=$(date +%s)

      # Run component pipeline
      trigger-component-pipeline

      END_TIME=$(date +%s)
      DURATION=$((END_TIME - START_TIME))

      echo "Pipeline duration: ${DURATION}s"

      # Fail if too slow
      if [ $DURATION -gt 300 ]; then
        echo "ERROR: Pipeline took ${DURATION}s (max 300s)"
        exit 1
      fi

Resource Usage Testing

# Monitor resource usage during component execution
resource-test:
  stage: test
  script:
    - |
      # Monitor memory usage
      watch-memory.sh &
      MONITOR_PID=$!

      # Run component
      run-component

      # Check peak memory usage
      kill $MONITOR_PID
      PEAK_MEMORY=$(cat /tmp/peak-memory)

      if [ $PEAK_MEMORY -gt 2048 ]; then
        echo "ERROR: Peak memory ${PEAK_MEMORY}MB exceeds 2GB limit"
        exit 1
      fi

Load Testing

# Test component with high concurrency
load-test:
  stage: test
  parallel: 10  # 10 concurrent pipelines
  script:
    - stress-test-component.sh

Continuous Testing

Nightly Test Suite

# .gitlab-ci.yml
nightly-comprehensive-test:
  stage: test
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"  # Scheduled nightly
  script:
    - run-comprehensive-test-suite.sh
  artifacts:
    reports:
      junit: nightly-test-results.xml
    paths:
      - test-results/
    when: always

Component Health Monitoring

// monitor-component-health.ts
interface ComponentHealth {
  component: string;
  success_rate_24h: number;
  avg_duration_24h: number;
  error_count_24h: number;
  users_affected: number;
}

async function monitorComponentHealth(): Promise<ComponentHealth[]> {
  const components = ['docker-build', 'k8s-deploy', 'security-scan'];
  const health = [];

  for (const component of components) {
    const pipelines = await fetchPipelinesUsingComponent(component, '24h');

    health.push({
      component,
      success_rate_24h: calculateSuccessRate(pipelines),
      avg_duration_24h: calculateAvgDuration(pipelines),
      error_count_24h: pipelines.filter(p => p.status === 'failed').length,
      users_affected: new Set(pipelines.map(p => p.user)).size
    });
  }

  // Alert on degradation
  for (const h of health) {
    if (h.success_rate_24h < 95) {
      await sendAlert(
        `Component ${h.component} success rate dropped to ${h.success_rate_24h}%`
      );
    }
  }

  return health;
}

// Run every hour
setInterval(monitorComponentHealth, 60 * 60 * 1000);

Test Automation

Automated Test Generation

// generate-tests.ts
function generateComponentTests(component: Component) {
  const inputs = component.spec.inputs;

  // Generate test for each input
  for (const [name, input] of Object.entries(inputs)) {
    // Test with default value
    generateTest({
      name: `test-${component.name}-default-${name}`,
      input_name: name,
      input_value: input.default
    });

    // Test with each option
    if (input.options) {
      for (const option of input.options) {
        generateTest({
          name: `test-${component.name}-${name}-${option}`,
          input_name: name,
          input_value: option
        });
      }
    }

    // Test with invalid value (should fail)
    generateTest({
      name: `test-${component.name}-invalid-${name}`,
      input_name: name,
      input_value: 'INVALID_VALUE',
      expect_failure: true
    });
  }
}

Best Practices

DO:

1. Test every component before release
2. Use test matrices for input combinations
3. Create dedicated test projects for integration tests
4. Run regression tests on every tag
5. Collect beta feedback from real users
6. Monitor component health continuously
7. Automate everything possible
8. Track test coverage and aim for 80%+

DON'T:

1. Don't skip tests to save time
2. Don't test only happy path - test edge cases
3. Don't ignore beta feedback - users find real issues
4. Don't forget performance tests - watch duration and resources
5. Don't test in isolation only - integration tests are crucial
6. Don't release without regression tests
7. Don't ignore flaky tests - fix or remove them
8. Don't forget to test rollback scenarios

Testing Checklist

Before releasing a new component version:

• YAML syntax validation passes
• Schema validation passes
• Unit tests pass (all input combinations)
• Integration tests pass (with test projects)
• Regression tests pass (no breaking changes)
• Performance tests pass (duration, memory)
• Beta testing complete (5+ projects)
• User feedback collected and addressed
• Migration guide tested by beta users
• Documentation updated
• Test coverage 80%
• No blocking issues remain

Next Steps

• Creating Components: Build testable components
• Patterns: Use proven patterns
• Governance: Release approval process

References

• GitLab Testing Guide
• CI/CD Testing Best Practices
• Component Examples
• GitLab CI/CD Components Forum