Secrets Management

Centralized secrets management with HashiCorp Vault and Kubernetes Secrets.

Overview

All secrets are managed securely:

• Secrets Engine: HashiCorp Vault (primary)
• Kubernetes: Encrypted at rest with KMS
• Access Control: RBAC with audit logging
• Rotation: Automatic rotation every 90 days
• Compliance: NIST 800-53, FedRAMP, SOC 2

Architecture

graph TB
    A[Application] --> B[Vault Agent]
    B --> C[HashiCorp Vault]
    C --> D[AWS KMS/HSM]
    A --> E[Kubernetes Secrets]
    E --> F[etcd Encryption]

HashiCorp Vault

Vault Setup

Installation:

# Install Vault
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install vault

# Start Vault server
vault server -config=/etc/vault/config.hcl

config.hcl:

storage "raft" {
  path = "/opt/vault/data"
  node_id = "vault-1"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_cert_file = "/etc/certs/vault.crt"
  tls_key_file = "/etc/certs/vault.key"
}

seal "awskms" {
  region = "us-west-2"
  kms_key_id = "arn:aws:kms:us-west-2:123456789012:key/abc123"
}

api_addr = "https://vault.local.bluefly.io:8200"
cluster_addr = "https://vault.local.bluefly.io:8201"
ui = true

Secret Engines

KV Secrets Engine (v2)

# Enable KV secrets engine
vault secrets enable -path=secret kv-v2

# Write secret
vault kv put secret/database/postgres \
  username=postgres_user \
  password=secure-password-123

# Read secret
vault kv get secret/database/postgres

# List secrets
vault kv list secret/database

Database Dynamic Secrets

# Enable database secrets engine
vault secrets enable database

# Configure PostgreSQL connection
vault write database/config/postgresql \
  plugin_name=postgresql-database-plugin \
  allowed_roles="readonly,readwrite" \
  connection_url="postgresql://{{username}}:{{password}}@postgres.local:5432/llm_platform" \
  username="vault_admin" \
  password="vault-admin-password"

# Create role for dynamic credentials
vault write database/roles/readwrite \
  db_name=postgresql \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
    GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Generate dynamic credentials
vault read database/creds/readwrite

Output:

Key                Value
---                -----
lease_id           database/creds/readwrite/abc123
lease_duration     1h
lease_renewable    true
password           A1a-random-password-xyz
username           v-token-readwrite-abc123

Transit Secrets Engine

# Enable transit engine
vault secrets enable transit

# Create encryption key
vault write -f transit/keys/bluefly-platform

# Encrypt data
vault write transit/encrypt/bluefly-platform \
  plaintext=$(echo "sensitive-data" | base64)

# Decrypt data
vault write transit/decrypt/bluefly-platform \
  ciphertext="vault:v1:encrypted-data-here"

# Rotate key
vault write -f transit/keys/bluefly-platform/rotate

Access Policies

policies/developer.hcl:

# Read application secrets
path "secret/data/app/*" {
  capabilities = ["read", "list"]
}

# Read database credentials
path "database/creds/readwrite" {
  capabilities = ["read"]
}

# Encrypt/decrypt with transit
path "transit/encrypt/bluefly-platform" {
  capabilities = ["update"]
}

path "transit/decrypt/bluefly-platform" {
  capabilities = ["update"]
}

Apply Policy:

# Create policy
vault policy write developer policies/developer.hcl

# Assign policy to user
vault write auth/userpass/users/developer \
  password=dev-password \
  policies=developer

Authentication Methods

AppRole (for services)

# Enable AppRole
vault auth enable approle

# Create role
vault write auth/approle/role/agent-brain \
  secret_id_ttl=24h \
  token_ttl=1h \
  token_max_ttl=4h \
  policies=agent-brain

# Get role ID
vault read auth/approle/role/agent-brain/role-id

# Generate secret ID
vault write -f auth/approle/role/agent-brain/secret-id

Application Login:

import Vault from 'node-vault';

const vault = Vault({
  apiVersion: 'v1',
  endpoint: 'https://vault.local.bluefly.io:8200'
});

// Login with AppRole
const result = await vault.approleLogin({
  role_id: process.env.VAULT_ROLE_ID,
  secret_id: process.env.VAULT_SECRET_ID
});

// Set token
vault.token = result.auth.client_token;

// Read secret
const secret = await vault.read('secret/data/app/config');
console.log(secret.data.data);

Kubernetes Auth

# Enable Kubernetes auth
vault auth enable kubernetes

# Configure
vault write auth/kubernetes/config \
  kubernetes_host=https://kubernetes.default.svc:443 \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  token_reviewer_jwt=@/var/run/secrets/kubernetes.io/serviceaccount/token

# Create role
vault write auth/kubernetes/role/agent-brain \
  bound_service_account_names=agent-brain \
  bound_service_account_namespaces=llm-platform \
  policies=agent-brain \
  ttl=1h

Pod Annotation:

apiVersion: v1
kind: Pod
metadata:
  name: agent-brain
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "agent-brain"
    vault.hashicorp.com/agent-inject-secret-config: "secret/data/app/agent-brain"
spec:
  serviceAccountName: agent-brain
  containers:
    - name: agent-brain
      image: agent-brain:latest
      env:
        - name: DATABASE_URL
          value: "file:///vault/secrets/config"

Kubernetes Secrets

Encrypted at Rest

kube-apiserver:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

Apply Configuration:

# Create encryption config
kubectl create secret generic -n kube-system \
  encryption-config --from-file=encryption-config.yaml

# Update kube-apiserver
--encryption-provider-config=/etc/kubernetes/encryption-config.yaml

Sealed Secrets

Install Sealed Secrets Controller:

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/controller.yaml

Create Sealed Secret:

# Create regular secret
kubectl create secret generic my-secret \
  --from-literal=password=super-secret \
  --dry-run=client -o yaml > secret.yaml

# Seal it
kubeseal < secret.yaml > sealed-secret.yaml

# Apply sealed secret (safe to commit)
kubectl apply -f sealed-secret.yaml

External Secrets Operator

Install ESO:

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets-system \
  --create-namespace

SecretStore:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: llm-platform
spec:
  provider:
    vault:
      server: "https://vault.local.bluefly.io:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "external-secrets"
          serviceAccountRef:
            name: "external-secrets"

ExternalSecret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: llm-platform
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: postgres-credentials
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: secret/database/postgres
        property: username
    - secretKey: password
      remoteRef:
        key: secret/database/postgres
        property: password

Secret Types

API Keys

Storage in Vault:

vault kv put secret/api-keys/openai \
  api_key=sk-proj-abc123... \
  organization=org-xyz789

Usage:

const secret = await vault.read('secret/data/api-keys/openai');
const apiKey = secret.data.data.api_key;

const openai = new OpenAI({ apiKey });

Database Credentials

Dynamic Credentials (recommended):

// Get dynamic credentials
const creds = await vault.read('database/creds/readwrite');

const pool = new Pool({
  host: 'postgres.local.bluefly.io',
  database: 'llm_platform',
  user: creds.data.username,
  password: creds.data.password
});

// Credentials auto-expire after 1 hour
// Vault automatically revokes them

OAuth Client Secrets

vault kv put secret/oauth/gitlab \
  client_id=abc123 \
  client_secret=secret-xyz789 \
  redirect_uri=https://llm.bluefly.io/auth/callback

Encryption Keys

# Store in transit engine (never leaves Vault)
vault write -f transit/keys/data-encryption

# Encrypt
vault write transit/encrypt/data-encryption \
  plaintext=$(echo "data" | base64)

# Decrypt (key material never exposed)
vault write transit/decrypt/data-encryption \
  ciphertext="vault:v1:..."

Secret Rotation

Automatic Rotation

Rotation Schedule:

• API keys: 90 days
• Database passwords: 30 days
• OAuth secrets: 90 days
• Encryption keys: 90 days

Rotation Script:

class SecretRotationService {
  async rotateSecret(path: string): Promise<void> {
    // 1. Generate new secret
    const newSecret = crypto.randomBytes(32).toString('hex');

    // 2. Write to Vault with new version
    await this.vault.write(`secret/data/${path}`, {
      data: { value: newSecret }
    });

    // 3. Update applications (gradual rollout)
    await this.updateApplications(path, newSecret);

    // 4. Audit log
    await this.auditLog.log({
      event: 'secret_rotated',
      path,
      timestamp: new Date()
    });

    // 5. Delete old version after grace period
    setTimeout(async () => {
      await this.vault.delete(`secret/metadata/${path}`);
    }, 24 * 60 * 60 * 1000); // 24 hours
  }
}

Database Password Rotation

// Vault automatically handles rotation
vault write database/rotate-root/postgresql

// Application automatically gets new credentials
const creds = await vault.read('database/creds/readwrite');

Secret Injection

Environment Variables

Vault Agent:

vault {
  address = "https://vault.local.bluefly.io:8200"
}

auto_auth {
  method {
    type = "approle"
    config = {
      role_id_file_path = "/vault/role-id"
      secret_id_file_path = "/vault/secret-id"
    }
  }
}

template {
  source = "/vault/templates/config.env.tmpl"
  destination = "/app/.env"
}

config.env.tmpl:

{{ with secret "secret/data/app/config" }}
DATABASE_URL=postgresql://{{ .Data.data.db_user }}:{{ .Data.data.db_password }}@postgres.local:5432/llm_platform
OPENAI_API_KEY={{ .Data.data.openai_key }}
{{ end }}

File Injection

Sidecar Container:

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  initContainers:
    - name: vault-agent
      image: vault:1.15
      command: ["vault", "agent", "-config=/vault/config.hcl"]
      volumeMounts:
        - name: vault-config
          mountPath: /vault
        - name: secrets
          mountPath: /secrets
  containers:
    - name: app
      image: myapp:latest
      volumeMounts:
        - name: secrets
          mountPath: /etc/secrets
          readOnly: true

Audit & Compliance

Audit Logging

Enable Audit:

vault audit enable file file_path=/var/log/vault/audit.log

Audit Events:

{
  "time": "2025-01-15T10:00:00Z",
  "type": "response",
  "auth": {
    "client_token": "hmac-sha256:abc123",
    "accessor": "hmac-sha256:xyz789",
    "display_name": "approle",
    "policies": ["default", "agent-brain"]
  },
  "request": {
    "operation": "read",
    "path": "secret/data/app/config"
  },
  "response": {
    "secret": true,
    "data": {
      "metadata": {
        "created_time": "2025-01-15T09:00:00Z",
        "version": 1
      }
    }
  }
}

Compliance Reports

# Generate secrets audit report
vault audit list -detailed

# List all secrets
vault kv list -format=json secret/ > secrets-inventory.json

# Check secret age
vault kv metadata get secret/app/config

Best Practices

1. Never Commit Secrets

Good:

const apiKey = process.env.OPENAI_API_KEY;

Bad:

const apiKey = "sk-proj-abc123...";  // NEVER!

2. Use Short-Lived Credentials

// Dynamic credentials with 1-hour TTL
const creds = await vault.read('database/creds/readwrite');
// Auto-revoked after 1 hour

3. Rotate Regularly

# Scheduled rotation
0 0 * * 0 /usr/bin/rotate-secrets.sh  # Weekly

4. Least Privilege

# Only grant required permissions
path "secret/data/app/myapp/*" {
  capabilities = ["read"]
}

5. Monitor Access

// Alert on unusual access patterns
if (accessCount > 1000 && timeWindow < 60) {
  alert('Unusual secret access pattern detected');
}

Next Steps

• Encryption at Rest
• Encryption in Transit
• Security Architecture
• NIST 800-53 Controls