GitLab Ultimate Secrets Management

This guide covers secrets management in GitLab Ultimate including CI/CD variables, OIDC authentication, external secrets integration (HashiCorp Vault, AWS Secrets Manager), and secret detection. Proper secrets management prevents credential exposure and security breaches.

Overview

GitLab Ultimate provides comprehensive secrets management:

• CI/CD Variables - Masked and protected variables for pipelines
• OIDC Authentication - Short-lived tokens instead of long-lived credentials
• External Secrets Integration - HashiCorp Vault, AWS Secrets Manager, Google Cloud Secret Manager
• Secret Detection - Prevent committed secrets (push protection and pipeline scanning)
• Secret Rotation - Automated secret rotation workflows
• Audit Trail - Complete secrets access logging

Secrets management follows the principle: Never commit secrets to Git, never use long-lived credentials.

CI/CD Variables

Overview

CI/CD variables store sensitive configuration values like API tokens, passwords, and certificates. They're available to pipelines without being committed to Git.

Variable Types

Regular Variables:

• Visible in job logs
• Accessible to all pipelines
• Use for non-sensitive configuration

Masked Variables:

• Hidden in job logs (shown as [masked])
• Cannot be less than 8 characters
• Must not contain special characters that break masking

Protected Variables:

• Only available to pipelines on protected branches/tags
• Use for production credentials
• Higher security for sensitive secrets

File Variables:

• Stored as temporary files instead of environment variables
• Use for certificates, keys, large secrets
• File path available in $VARIABLE_NAME

Configuration

Creating Variables:

1. Navigate to: Settings > CI/CD > Variables
2. Click "Add Variable"
3. Configure:
   • Key: Variable name (e.g., API_TOKEN)
   • Value: Secret value
   • Type: Variable or File
   • Protected: Yes/No
   • Masked: Yes/No
   • Environment scope: * or specific environment

Variable Scopes:

Variable: DATABASE_URL
Value: postgres://prod.example.com
Environment Scope: production
Protected: Yes
Masked: Yes

Variable: DATABASE_URL
Value: postgres://staging.example.com
Environment Scope: staging
Protected: No
Masked: Yes

Variable: DATABASE_URL
Value: postgres://localhost
Environment Scope: *
Protected: No
Masked: No

Best Practices

Security:

 Mask all secrets
 Protect production credentials
 Scope to specific environments
 Use file type for certificates/keys
 Rotate secrets regularly
 Limit variable access with roles

 Store secrets in .gitlab-ci.yml
 Echo secrets in scripts
 Use unmasked variables for credentials
 Use wildcard scope for production secrets
 Share secrets across projects unnecessarily

Naming Conventions:

Good:
- API_TOKEN
- DATABASE_PASSWORD
- AWS_ACCESS_KEY_ID
- PRIVATE_KEY_FILE
- STAGING_DB_URL

Avoid:
- apiToken (inconsistent casing)
- secret123 (unclear purpose)
- PASSWORD (too generic)
- prod_db (inconsistent naming)

Using Variables in Pipelines

Environment Variables:

deploy:
  stage: deploy
  script:
    - echo "Deploying to production..."
    - ./deploy.sh
  environment:
    name: production
  variables:
    DEPLOY_ENV: "production"
  # Variables automatically available:
  # - $API_TOKEN (from CI/CD settings)
  # - $DATABASE_URL (scoped to production)
  # - $DEPLOY_ENV (from job variables)

File Variables:

deploy:
  stage: deploy
  script:
    # File variable creates temporary file
    - cat $CERTIFICATE_FILE > /tmp/cert.pem
    - kubectl create secret tls my-cert --cert=/tmp/cert.pem --key=$PRIVATE_KEY_FILE
    - rm /tmp/cert.pem  # Clean up
  variables:
    # These are defined as File type in CI/CD settings
    # $CERTIFICATE_FILE contains path to temporary file
    # $PRIVATE_KEY_FILE contains path to private key file

Passing to Docker:

build:
  stage: build
  script:
    # Securely pass secrets to Docker build
    - echo "$NPM_TOKEN" | docker login -u $NPM_USER --password-stdin
    - docker build
        --build-arg NPM_TOKEN=$NPM_TOKEN
        --build-arg API_KEY=$API_KEY
        -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

Variable Precedence

Order (highest to lowest priority):

1. Trigger variables
2. Scheduled pipeline variables
3. Manual pipeline run variables
4. Job variables (defined in .gitlab-ci.yml)
5. Project variables
6. Group variables
7. Instance variables
8. Deployment variables
9. Predefined variables

OIDC Authentication

Overview

OIDC (OpenID Connect) authentication provides short-lived tokens for accessing external services without long-lived credentials. GitLab automatically generates JWT tokens that can be exchanged for service credentials.

Benefits:

• No long-lived credentials stored in GitLab
• Automatic token rotation
• Scoped access per job
• Audit trail of token usage
• Reduced security risk

Supported Services

Official Support:

• HashiCorp Vault
• AWS (via STS AssumeRoleWithWebIdentity)
• Google Cloud (via Workload Identity Federation)
• Azure (via Workload Identity Federation)

Custom Integration:

• Any service supporting OIDC/JWT authentication

GitLab OIDC Token

Token Structure:

{
  "jti": "c82eeb0c-5c6f-4a33-abf5-4c474b92b558",
  "iss": "https://gitlab.com",
  "aud": "https://gitlab.com",
  "sub": "project_path:mygroup/myproject:ref_type:branch:ref:main",
  "namespace_id": "72",
  "namespace_path": "mygroup",
  "project_id": "20",
  "project_path": "mygroup/myproject",
  "user_id": "42",
  "user_login": "alice",
  "user_email": "alice@example.com",
  "pipeline_id": "1234",
  "pipeline_source": "push",
  "job_id": "5678",
  "ref": "main",
  "ref_type": "branch",
  "ref_protected": "true",
  "environment": "production",
  "environment_protected": "true",
  "iat": 1516239022,
  "nbf": 1516239022,
  "exp": 1516239922
}

Key Fields:

• sub: Subject identifying the job context
• aud: Audience (customizable per ID token)
• iss: Issuer (GitLab instance URL)
• exp: Expiration (typically 1 hour)

Configuration

Defining ID Tokens (.gitlab-ci.yml):

job_with_oidc:
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com

  script:
    - echo $GITLAB_OIDC_TOKEN
    # Token automatically available in job

Multiple Audiences:

multi_cloud_deploy:
  id_tokens:
    AWS_TOKEN:
      aud: https://aws.amazon.com
    GCP_TOKEN:
      aud: https://gcp.example.com
    VAULT_TOKEN:
      aud: https://vault.example.com

  script:
    - echo "AWS token: $AWS_TOKEN"
    - echo "GCP token: $GCP_TOKEN"
    - echo "Vault token: $VAULT_TOKEN"

OIDC with HashiCorp Vault

Complete Integration:

1. Configure Vault JWT Auth:

# Enable JWT auth method
vault auth enable jwt

# Configure GitLab as OIDC provider
vault write auth/jwt/config \
  oidc_discovery_url="https://gitlab.com" \
  bound_issuer="https://gitlab.com"

2. Create Vault Role:

# Create role for GitLab project
vault write auth/jwt/role/myproject-role \
  role_type="jwt" \
  bound_audiences="https://gitlab.com" \
  bound_claims='{"project_id":"20","ref":"main","ref_type":"branch"}' \
  user_claim="user_email" \
  policies="myproject-policy" \
  ttl="1h"

3. Create Vault Policy:

# Create policy with required permissions
vault policy write myproject-policy - <<EOF
path "secret/data/myproject/*" {
  capabilities = ["read"]
}
EOF

4. Store Secrets in Vault:

# Store secrets
vault kv put secret/myproject/production \
  DATABASE_URL="postgres://prod.example.com" \
  API_TOKEN="secret-token-value"

5. Use in GitLab CI/CD:

deploy_production:
  stage: deploy
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://gitlab.com

  before_script:
    # Authenticate with Vault using OIDC token
    - export VAULT_ADDR="https://vault.example.com"
    - export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=myproject-role jwt=$VAULT_ID_TOKEN)"

  script:
    # Read secrets from Vault
    - export DATABASE_URL="$(vault kv get -field=DATABASE_URL secret/myproject/production)"
    - export API_TOKEN="$(vault kv get -field=API_TOKEN secret/myproject/production)"

    # Use secrets in deployment
    - ./deploy.sh

  environment:
    name: production

  only:
    - main

GitLab Secrets Keyword (Simplified):

deploy_production:
  stage: deploy
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://gitlab.com

  secrets:
    DATABASE_URL:
      vault: secret/myproject/production/DATABASE_URL@myproject-role
      token: $VAULT_ID_TOKEN
    API_TOKEN:
      vault: secret/myproject/production/API_TOKEN@myproject-role
      token: $VAULT_ID_TOKEN

  script:
    # Secrets automatically available as environment variables
    - echo "Database: $DATABASE_URL"
    - ./deploy.sh

  environment:
    name: production

OIDC with AWS

AWS STS AssumeRoleWithWebIdentity:

1. Create IAM OIDC Provider:

aws iam create-open-id-connect-provider \
  --url https://gitlab.com \
  --client-id-list https://gitlab.com \
  --thumbprint-list <thumbprint>

2. Create IAM Role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/gitlab.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "gitlab.com:aud": "https://gitlab.com",
          "gitlab.com:sub": "project_path:mygroup/myproject:ref_type:branch:ref:main"
        }
      }
    }
  ]
}

3. Use in GitLab CI/CD:

deploy_to_aws:
  stage: deploy
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com

  before_script:
    # Assume AWS role using OIDC token
    - export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s"
        $(aws sts assume-role-with-web-identity
          --role-arn arn:aws:iam::123456789012:role/gitlab-deploy-role
          --role-session-name "gitlab-${CI_PROJECT_ID}-${CI_PIPELINE_ID}"
          --web-identity-token $GITLAB_OIDC_TOKEN
          --duration-seconds 3600
          --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]'
          --output text))

  script:
    # AWS credentials automatically available
    - aws s3 cp ./build s3://my-bucket/ --recursive
    - aws ecs update-service --cluster prod --service my-app --force-new-deployment

OIDC with Google Cloud

Workload Identity Federation:

1. Create Workload Identity Pool:

gcloud iam workload-identity-pools create "gitlab-pool" \
  --project="my-project" \
  --location="global" \
  --display-name="GitLab CI/CD"

2. Create Workload Identity Provider:

gcloud iam workload-identity-pools providers create-oidc "gitlab-provider" \
  --project="my-project" \
  --location="global" \
  --workload-identity-pool="gitlab-pool" \
  --display-name="GitLab" \
  --attribute-mapping="google.subject=assertion.sub,attribute.project_path=assertion.project_path" \
  --issuer-uri="https://gitlab.com"

3. Grant Service Account Access:

gcloud iam service-accounts add-iam-policy-binding "my-service-account@my-project.iam.gserviceaccount.com" \
  --project="my-project" \
  --role="roles/iam.workloadIdentityUser" \
  --member="principalSet://iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/gitlab-pool/attribute.project_path/mygroup/myproject"

4. Use in GitLab CI/CD:

deploy_to_gcp:
  stage: deploy
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gcp.example.com

  before_script:
    # Authenticate with GCP using OIDC
    - echo $GITLAB_OIDC_TOKEN > /tmp/token.txt
    - gcloud iam workload-identity-pools create-cred-config \
        projects/123456789/locations/global/workloadIdentityPools/gitlab-pool/providers/gitlab-provider \
        --service-account="my-service-account@my-project.iam.gserviceaccount.com" \
        --credential-source-file=/tmp/token.txt \
        --output-file=/tmp/gcp-credentials.json
    - export GOOGLE_APPLICATION_CREDENTIALS=/tmp/gcp-credentials.json

  script:
    # GCP credentials automatically available
    - gcloud storage cp ./build gs://my-bucket/
    - gcloud run deploy my-service --image gcr.io/my-project/my-image:latest

External Secrets Integration

HashiCorp Vault

Advanced Configuration:

Dynamic Secrets:

deploy:
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://gitlab.com

  secrets:
    # Static secret
    API_TOKEN:
      vault: secret/data/myproject/api_token@myproject-role
      token: $VAULT_ID_TOKEN

    # Dynamic AWS credentials (generated on-demand)
    AWS_ACCESS_KEY_ID:
      vault: aws/creds/deploy-role/access_key@myproject-role
      token: $VAULT_ID_TOKEN

    AWS_SECRET_ACCESS_KEY:
      vault: aws/creds/deploy-role/secret_key@myproject-role
      token: $VAULT_ID_TOKEN

  script:
    - aws s3 cp ./build s3://my-bucket/
    # AWS credentials automatically revoked after job completes

Vault Namespaces:

secrets:
  DATABASE_PASSWORD:
    vault:
      engine:
        name: kv-v2
        path: secret
      path: myproject/database
      field: password
    vault_server:
      url: https://vault.example.com
      namespace: engineering/backend
      auth_path: jwt
    token: $VAULT_ID_TOKEN

AWS Secrets Manager

Using AWS CLI:

deploy:
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gitlab.com

  before_script:
    # Assume AWS role
    - export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s"
        $(aws sts assume-role-with-web-identity
          --role-arn $AWS_ROLE_ARN
          --role-session-name "gitlab-ci"
          --web-identity-token $GITLAB_OIDC_TOKEN
          --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]'
          --output text))

    # Retrieve secrets
    - export DATABASE_URL=$(aws secretsmanager get-secret-value
        --secret-id myproject/production/database_url
        --query SecretString --output text)
    - export API_TOKEN=$(aws secretsmanager get-secret-value
        --secret-id myproject/production/api_token
        --query SecretString --output text)

  script:
    - ./deploy.sh

Google Cloud Secret Manager

Using gcloud:

deploy:
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: https://gcp.example.com

  before_script:
    # Authenticate with GCP
    - gcloud auth login --cred-file=$GOOGLE_APPLICATION_CREDENTIALS

    # Retrieve secrets
    - export DATABASE_URL=$(gcloud secrets versions access latest --secret="database-url")
    - export API_TOKEN=$(gcloud secrets versions access latest --secret="api-token")

  script:
    - ./deploy.sh

Secret Rotation

Automated Rotation

Rotation Strategy:

1. Generate new secret
2. Deploy new secret to services (both old and new active)
3. Update secret in GitLab/Vault
4. Wait for rollout completion
5. Revoke old secret
6. Verify all services using new secret

Example: Database Password Rotation:

rotate_database_password:
  stage: maintenance
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://gitlab.com

  script:
    # 1. Generate new password
    - NEW_PASSWORD=$(openssl rand -base64 32)

    # 2. Update database user password
    - psql -h $DB_HOST -U $DB_ADMIN -c "ALTER USER appuser PASSWORD '$NEW_PASSWORD';"

    # 3. Update secret in Vault
    - vault kv put secret/myproject/production database_password=$NEW_PASSWORD

    # 4. Trigger application redeployment
    - curl -X POST $DEPLOYMENT_WEBHOOK

    # 5. Wait for deployment
    - sleep 300

    # 6. Verify connectivity
    - psql -h $DB_HOST -U appuser -c "SELECT 1;"

  when: manual
  only:
    - schedules

Scheduled Rotation:

# .gitlab-ci.yml
include:
  - local: .gitlab/ci/secret-rotation.yml

# Scheduled pipeline: Weekly on Sundays at 2 AM

Rotation Best Practices

1. Automate Rotation - Use scheduled pipelines
2. Grace Periods - Keep old secrets active during transition
3. Verification - Test new secrets before revoking old ones
4. Rollback Plan - Be prepared to revert to old secrets
5. Audit Rotation - Log all secret rotation events
6. Notify Teams - Inform teams of rotation schedules
7. Emergency Rotation - Have process for immediate rotation

Secret Detection

Overview

Secret Detection identifies committed credentials and prevents exposure. GitLab provides two complementary approaches:

1. Secret Push Protection - Blocks secrets at push time
2. Pipeline Secret Detection - Scans in CI/CD pipelines

Secret Push Protection

How It Works:

• Runs in pre-receive hook
• Scans diff for secrets (not entire files)
• Blocks push immediately if secrets detected
• Prevents secrets from entering repository

Enabling:

1. Navigate to: Settings > Security & Compliance
2. Enable "Secret push protection"
3. Configure detection rules (optional)

Detected Secret Types:

• AWS Access Keys
• GCP Service Account Keys
• Azure Storage Keys
• GitHub Personal Access Tokens
• GitLab Personal Access Tokens
• Private SSH Keys
• Database Connection Strings
• JWT Tokens
• API Keys (various services)
• OAuth Client Secrets
• Stripe API Keys
• PayPal credentials

Bypass (Emergency):

# Only for false positives or emergency situations
git commit --trailer "skip-secret-detection=true" -m "Your commit message"

Pipeline Secret Detection

Configuration:

include:
  - template: Security/Secret-Detection.gitlab-ci.yml

secret_detection:
  variables:
    SECRET_DETECTION_EXCLUDED_PATHS: "test/,spec/,docs/"
    SECRET_DETECTION_HISTORIC_SCAN: "false"  # Set true for full history scan

Custom Patterns:

# .gitlab/secret-detection-ruleset.yml
patterns:
  - name: "Company API Key"
    pattern: "company_api_[a-zA-Z0-9]{32}"
    severity: "critical"

  - name: "Internal Token"
    pattern: "internal_token_[a-zA-Z0-9]{40}"
    severity: "high"

Integration:

include:
  - template: Security/Secret-Detection.gitlab-ci.yml

secret_detection:
  variables:
    SECRET_DETECTION_CUSTOM_PATTERNS: ".gitlab/secret-detection-ruleset.yml"

Handling Detected Secrets

If Secret Detected:

1. Immediate Actions:

   • Stop deployment immediately
   • Rotate exposed secret
   • Audit access logs
   • Determine exposure scope

2. Remove from History:

# Using git filter-branch
git filter-branch --force --index-filter \
  "git rm --cached --ignore-unmatch config/secrets.yml" \
  --prune-empty --tag-name-filter cat -- --all

# Using BFG Repo-Cleaner (faster, recommended)
bfg --delete-files secrets.yml
bfg --replace-text passwords.txt

# Force push (WARNING: Rewrites history)
git push origin --force --all

1. Notify Team:

   • Inform all team members of history rewrite
   • Have them rebase/re-clone
   • Document incident for audit

2. Prevent Recurrence:

   • Enable Secret Push Protection
   • Add to custom detection patterns
   • Implement pre-commit hooks
   • Train developers

Best Practices

General Principles

Never:

• Commit secrets to Git
• Store secrets in .gitlab-ci.yml
• Echo secrets in CI/CD logs
• Share secrets via chat/email
• Use long-lived credentials
• Store production secrets unencrypted

Always:

• Use CI/CD variables (masked, protected)
• Prefer OIDC over long-lived credentials
• Store secrets in external systems (Vault, AWS Secrets Manager)
• Rotate secrets regularly
• Audit secret access
• Enable Secret Push Protection

Security Hierarchy

Secret Storage (Best to Worst):

1. OIDC with External Secrets (Best)

   • Short-lived tokens
   • No stored credentials
   • Automatic rotation
   • Full audit trail

2. External Secrets Manager (Good)

   • Centralized management
   • Rotation capabilities
   • Access controls
   • Audit logging

3. GitLab CI/CD Variables (Acceptable)

   • Masked and protected
   • Scoped to environments
   • GitLab-managed

4. Encrypted Secrets in Git (Poor)

   • Still in version control
   • Key management challenges
   • Rotation difficulties

5. Plain Text in Git (Never)

   • Complete exposure
   • Permanent history
   • Unacceptable risk

Development Workflow

Local Development:

# Use .env files (never commit)
echo ".env" >> .gitignore

# .env.example (commit this)
DATABASE_URL=postgres://localhost/myapp
API_TOKEN=your-api-token-here

# .env (never commit)
DATABASE_URL=postgres://localhost/myapp
API_TOKEN=actual-secret-token

CI/CD Pipeline:

# Reference secrets from CI/CD variables
deploy:
  script:
    - export DATABASE_URL=$DATABASE_URL  # From CI/CD variable
    - export API_TOKEN=$API_TOKEN        # From CI/CD variable
    - ./deploy.sh

Production:

# Use OIDC + External Secrets
deploy_production:
  id_tokens:
    VAULT_ID_TOKEN:
      aud: https://gitlab.com

  secrets:
    DATABASE_URL:
      vault: secret/myproject/production/database_url@prod-role
    API_TOKEN:
      vault: secret/myproject/production/api_token@prod-role

  script:
    - ./deploy.sh

Troubleshooting

Common Issues

Variable not available in job:

Checklist:
 Variable defined in CI/CD settings
 Variable not protected (or branch is protected)
 Environment scope matches job environment
 Variable key matches reference ($VARIABLE_NAME)
 Job has permission to access variable

OIDC authentication failing:

Checklist:
 ID token defined in job
 Audience matches external service configuration
 External service OIDC provider configured correctly
 Token not expired
 Claims match external service requirements

Secret detected despite masking:

Issue: Secret visible in logs despite masked variable

Causes:
- Secret less than 8 characters
- Secret contains special characters
- Secret echoed by command output
- Secret in error messages

Resolution:
- Ensure secrets meet masking requirements
- Avoid echoing secrets
- Redirect secret-containing output
- Use file variables for complex secrets

Vault authentication failing:

Error: Permission denied

Checklist:
 Vault auth method enabled
 Vault role created with correct claims
 Vault policy grants required permissions
 GitLab OIDC token claims match role bounds
 Vault URL accessible from GitLab Runner

References

• CI/CD Variables Documentation
• OIDC Authentication Documentation
• HashiCorp Vault Integration
• Secret Detection Documentation
• AWS Integration

Next Steps

• Security Scanning - Configure secret detection scanners
• Security Policies - Enforce secret detection policies
• Supply Chain Security - Secure dependencies and builds