Skip to main content

GitLab Observability Integration

GitLab Observability Integration

Unified observability for the entire blueflyio group, aggregating traces, metrics, and logs from all services.

Overview

GitLab Observability provides a centralized view of all observability data across the platform:

  • Traces: Distributed tracing across all services
  • Metrics: Prometheus metrics from all services
  • Logs: Centralized log aggregation
  • Service Map: Visual representation of service dependencies

Prerequisites

  1. GitLab Ultimate or Premium subscription (Observability is a premium feature)
  2. Group admin access to blueflyio group
  3. Observability enabled in GitLab UI

Setup

Step 1: Enable Observability in GitLab

  1. Navigate to: https://gitlab.com/groups/blueflyio/-/observability/setup
  2. Click "Enable Observability" or follow the setup wizard
  3. GitLab will provide:
    • OTEL endpoint URL: http://87749026.otel.gitlab-o11y.com:4318
    • Observability token (different from your PAT)

Step 2: Configure Environment

Add to .env.local in the workspace root:

GITLAB_OBSERVABILITY_TOKEN=your_observability_token_from_setup_page

IMPORTANT: Use the observability token from the setup page, NOT a regular Personal Access Token (glpat-*).

Step 3: OTEL Collector Configuration

The OTEL collector is configured in common_npm/agent-tracer/infrastructure/configs/otel-collector-config.yaml:

exporters:
  otlphttp/gitlab:
    endpoint: http://87749026.otel.gitlab-o11y.com:4318
    headers:
      Authorization: Bearer ${GITLAB_OBSERVABILITY_TOKEN}
    compression: gzip
    timeout: 30s

service:
  pipelines:
    traces:
      exporters: [otlphttp/gitlab, ...]
    metrics:
      exporters: [otlphttp/gitlab, ...]
    logs:
      exporters: [otlphttp/gitlab, ...]

Step 4: Start OTEL Collector

cd /Users/flux423/Sites/LLM/common_npm/agent-tracer/infrastructure
docker-compose -f docker-compose.observability.yml up -d otel-collector

Step 5: Verify Setup

Run the verification script:

cd /Users/flux423/Sites/LLM/common_npm/agent-tracer/infrastructure
./verify-gitlab-observability.sh

Viewing Data

GitLab Observability Dashboards

Configuration Details

Current Setup

Services Sending Data

  • agent-mesh: Sends traces via OTEL collector
  • OTEL Collector: Aggregates and forwards all telemetry data
  • Prometheus: Scrapes metrics and forwards via OTEL collector

Troubleshooting

504 Gateway Timeout

Symptom: 504 Gateway Time-out when sending traces

Cause: GitLab Observability backend is not active

Solution:

  1. Verify observability is enabled in GitLab UI: https://gitlab.com/groups/blueflyio/-/observability/setup
  2. Complete the setup wizard if not done
  3. Wait 2-3 minutes for backend provisioning

Authentication Errors

Symptom: "Unable to authenticate with the observability service"

Cause: Using wrong token type (PAT instead of observability token)

Solution:

  1. Get the observability token from the setup page (NOT a glpat-* token)
  2. Update .env.local with the correct token
  3. Restart OTEL collector: docker-compose restart otel-collector

No Data Appearing

Symptom: Data not visible in GitLab Observability dashboards

Solutions:

  1. Verify OTEL collector is running: docker ps | grep otel
  2. Check collector logs: docker logs agent-tracker-otel --tail 50
  3. Verify agent-mesh is sending traces: curl http://localhost:3005/health
  4. Wait 2-3 minutes for data processing delay
  5. Check pipelines are configured correctly in otel-collector-config.yaml

Integration with Other Services

agent-mesh

agent-mesh automatically sends traces to GitLab Observability via the OTEL collector:

// agent-mesh/src/tracing.ts
const GITLAB_OTEL_ENDPOINT = 'http://87749026.otel.gitlab-o11y.com:4318';
// Traces automatically sent via OTEL collector

Prometheus Metrics

Prometheus metrics are scraped by the OTEL collector and forwarded to GitLab:

# otel-collector-config.yaml
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'agent-mesh'
          static_configs:
            - targets: ['host.docker.internal:3005']

Related Documentation

Next Steps

After enabling observability:

  1. Monitor services in GitLab Observability dashboard
  2. Set up alerts for critical metrics
  3. Use service map to understand dependencies
  4. Analyze traces for performance optimization

Back to Observability & Analytics | OpenTelemetry Configuration