QITS - Quality Intelligence Testing System

QITS (Quality Intelligence Testing System) is an AI-powered test generation and quality assurance platform integrated into BuildKit Agent Runtime (BAR).

---

Overview

QITS automates test creation, execution, and optimization using AI-driven analysis. It's a core component of the BuildKit platform's quality enforcement strategy.

Key Features

• AI Test Generation: LLM-powered test case creation
• Mutation Testing: Automated code mutation and validation
• Security Scanning: Vulnerability detection and prevention
• Quality Metrics: Comprehensive quality scoring
• TDD Enforcement: Automated TDD compliance checking
• Real-time Feedback: Instant quality insights during development

---

Architecture

graph TB
    subgraph "QITS Core"
        TG[Test Generator]
        MT[Mutation Tester]
        SS[Security Scanner]
        QE[Quality Engine]
    end

    subgraph "BAR Integration"
        ROE[ROE Orchestrator]
        VORTEX[VORTEX v3]
        SWARM[SWARM Manager]
    end

    subgraph "Data Sources"
        CODE[Codebase]
        SPECS[OpenAPI Specs]
        OSSA[OSSA Manifests]
    end

    CODE --> TG
    SPECS --> TG
    OSSA --> TG

    TG --> ROE
    MT --> ROE
    SS --> ROE
    QE --> ROE

    ROE --> VORTEX
    ROE --> SWARM

---

AI Test Generation

Automatic Test Creation

QITS analyzes code and specifications to generate comprehensive test suites:

# Generate tests for a file
buildkit qits generate-tests src/auth/AuthService.ts

# Generate from OpenAPI spec
buildkit qits generate-tests --spec openapi/auth.yaml

# Generate for entire module
buildkit qits generate-tests src/auth/ --recursive

Test Generation Process

sequenceDiagram
    participant Dev as Developer
    participant QITS as QITS
    participant LLM as Claude/GPT
    participant BAR as BAR Runtime
    participant Code as Codebase

    Dev->>QITS: Request test generation
    QITS->>Code: Analyze source code
    QITS->>Code: Parse OpenAPI specs
    QITS->>LLM: Generate test scenarios
    LLM->>QITS: Return test cases
    QITS->>BAR: Validate tests
    BAR->>Code: Execute tests
    BAR->>QITS: Report results
    QITS->>Dev: Provide test suite

Generated Test Example

Input: src/auth/AuthService.ts

export class AuthService {
  async login(email: string, password: string): Promise<LoginResult> {
    const user = await this.userRepository.findByEmail(email)
    if (!user) {
      throw new Error('User not found')
    }

    const isValid = await bcrypt.compare(password, user.passwordHash)
    if (!isValid) {
      throw new Error('Invalid credentials')
    }

    return {
      success: true,
      token: this.generateToken(user),
      user: { id: user.id, email: user.email }
    }
  }
}

QITS Output: tests/auth/AuthService.test.ts

import { describe, it, expect, beforeEach } from 'vitest'
import { AuthService } from '../../src/auth/AuthService'
import { UserRepository } from '../../src/repositories/UserRepository'
import bcrypt from 'bcrypt'

describe('AuthService', () => {
  let authService: AuthService
  let mockUserRepository: jest.Mocked<UserRepository>

  beforeEach(() => {
    mockUserRepository = {
      findByEmail: jest.fn()
    } as any

    authService = new AuthService(mockUserRepository)
  })

  describe('login', () => {
    it('should successfully login with valid credentials', async () => {
      // Arrange
      const user = {
        id: '123',
        email: 'user@example.com',
        passwordHash: await bcrypt.hash('password123', 10)
      }
      mockUserRepository.findByEmail.mockResolvedValue(user)

      // Act
      const result = await authService.login('user@example.com', 'password123')

      // Assert
      expect(result.success).toBe(true)
      expect(result.token).toBeDefined()
      expect(result.user.email).toBe('user@example.com')
    })

    it('should throw error when user not found', async () => {
      // Arrange
      mockUserRepository.findByEmail.mockResolvedValue(null)

      // Act & Assert
      await expect(
        authService.login('nonexistent@example.com', 'password123')
      ).rejects.toThrow('User not found')
    })

    it('should throw error with invalid password', async () => {
      // Arrange
      const user = {
        id: '123',
        email: 'user@example.com',
        passwordHash: await bcrypt.hash('correct-password', 10)
      }
      mockUserRepository.findByEmail.mockResolvedValue(user)

      // Act & Assert
      await expect(
        authService.login('user@example.com', 'wrong-password')
      ).rejects.toThrow('Invalid credentials')
    })

    it('should validate email format', async () => {
      // Act & Assert
      await expect(
        authService.login('invalid-email', 'password123')
      ).rejects.toThrow('Invalid email format')
    })

    it('should handle empty credentials', async () => {
      // Act & Assert
      await expect(
        authService.login('', '')
      ).rejects.toThrow('Credentials required')
    })
  })
})

Coverage: 95% statements, 92% branches, 100% functions

---

Mutation Testing

Concept

Mutation testing validates test quality by introducing bugs (mutations) and checking if tests catch them.

# Run mutation testing
buildkit qits mutate src/auth/AuthService.ts

# Run with specific mutations
buildkit qits mutate src/auth/ --mutations arithmetic,conditional

# Generate mutation report
buildkit qits mutate src/ --report mutation-report.html

Mutation Types

1. Arithmetic Mutations

// Original
const total = price + tax

// Mutation
const total = price - tax  // Changed + to -

// Test should FAIL (catch mutation)

2. Conditional Mutations

// Original
if (age >= 18) {
  return true
}

// Mutation
if (age > 18) {  // Changed >= to >

// Test should FAIL (catch edge case)

3. Logical Mutations

// Original
if (isActive && isVerified) {
  return true
}

// Mutation
if (isActive || isVerified) {  // Changed && to ||

// Test should FAIL (catch logic error)

Mutation Score

$ buildkit qits mutate src/auth/AuthService.ts

 Mutation Testing Report


File: src/auth/AuthService.ts

Mutations Generated: 25
  Killed: 23 (92%)
  Survived: 2 (8%)

Mutation Score: 92% (threshold: 80%)

Survived Mutations:
  Line 45: Changed >= to > (conditional)
  Line 67: Changed && to || (logical)

Recommendations:
  - Add test for exact boundary condition (age === 18)
  - Add test for partial boolean conditions

Mutation Score 80% required for quality gate.

---

Security Scanning

Automated Vulnerability Detection

# Scan for security issues
buildkit qits security-scan src/

# Scan specific vulnerabilities
buildkit qits security-scan src/ --check sql-injection,xss

# Generate security report
buildkit qits security-scan src/ --report security-report.json

Security Checks

1. SQL Injection

// VULNERABLE - QITS DETECTS
async function getUser(userId: string) {
  return db.query(`SELECT * FROM users WHERE id = '${userId}'`)
}

// SAFE - QITS APPROVES
async function getUser(userId: string) {
  return db.query('SELECT * FROM users WHERE id = ?', [userId])
}

2. XSS Prevention

// VULNERABLE - QITS DETECTS
function renderHTML(userInput: string) {
  return `<div>${userInput}</div>`
}

// SAFE - QITS APPROVES
import { escape } from 'html-escaper'

function renderHTML(userInput: string) {
  return `<div>${escape(userInput)}</div>`
}

3. Authentication Flaws

// VULNERABLE - QITS DETECTS
if (user.password === providedPassword) {
  return true
}

// SAFE - QITS APPROVES
import bcrypt from 'bcrypt'

if (await bcrypt.compare(providedPassword, user.passwordHash)) {
  return true
}

4. Secrets in Code

// VULNERABLE - QITS DETECTS
const apiKey = 'sk-1234567890abcdef'

// SAFE - QITS APPROVES
const apiKey = process.env.API_KEY!

---

Quality Metrics

Quality Scoring

QITS generates comprehensive quality scores:

$ buildkit qits analyze src/auth/

Quality Score: 88/100 

Component Scores:
  Code Coverage: 92% 
  Mutation Score: 85% 
  Security: 95% 
  Complexity: 78% 
  Documentation: 82% 
  Performance: 90% 

Recommendations:
  - Reduce cyclomatic complexity in AuthService.login (12  8)
  - Add JSDoc comments to 3 functions
  - Optimize database query in TokenManager (N+1 detected)

Metrics Tracked

---

QITS Configuration

[object Object]

{
  "qits": {
    "testGeneration": {
      "enabled": true,
      "llmProvider": "anthropic",
      "model": "claude-sonnet-4-5-20250929",
      "temperature": 0.3,
      "coverageTarget": 90
    },
    "mutationTesting": {
      "enabled": true,
      "mutations": [
        "arithmetic",
        "conditional",
        "logical",
        "assignment"
      ],
      "threshold": 80
    },
    "securityScanning": {
      "enabled": true,
      "checks": [
        "sql-injection",
        "xss",
        "csrf",
        "auth-bypass",
        "secrets-exposure"
      ],
      "failOnVulnerability": true
    },
    "qualityMetrics": {
      "enabled": true,
      "thresholds": {
        "coverage": 80,
        "mutation": 80,
        "security": 95,
        "complexity": 10,
        "documentation": 70
      }
    }
  }
}

---

BAR Integration

QITS in Voice Agent Workflow

sequenceDiagram
    participant Dev as Developer
    participant VA as Voice Agent
    participant ROE as ROE
    participant QITS as QITS
    participant Code as Codebase

    Dev->>VA: "Create user authentication API"
    VA->>ROE: Route request
    ROE->>QITS: Generate test specifications
    QITS->>Code: Create test files
    QITS->>VA: Stream "Tests generated"

    ROE->>Code: Implement feature
    ROE->>QITS: Validate implementation
    QITS->>QITS: Run tests
    QITS->>QITS: Mutation testing
    QITS->>QITS: Security scan
    QITS->>VA: Stream "Quality checks passed"

    VA->>Dev: "API created with 92% coverage, secure"

---

CLI Usage

Test Generation

# Generate tests for file
buildkit qits generate src/auth/AuthService.ts

# Generate with coverage target
buildkit qits generate src/auth/ --coverage 95

# Generate from spec
buildkit qits generate --spec openapi/auth.yaml

Mutation Testing

# Run mutations
buildkit qits mutate src/auth/

# Specific mutation types
buildkit qits mutate src/auth/ --mutations arithmetic,conditional

# With threshold
buildkit qits mutate src/auth/ --threshold 85

Security Scanning

# Scan directory
buildkit qits scan src/

# Specific checks
buildkit qits scan src/ --checks sql-injection,xss

# Generate report
buildkit qits scan src/ --report json > security-report.json

Quality Analysis

# Analyze quality
buildkit qits analyze src/

# Full report
buildkit qits analyze src/ --detailed

# JSON output
buildkit qits analyze src/ --format json

---

CI/CD Integration

GitLab CI

# .gitlab-ci.yml
qits-quality:
  stage: quality
  script:
    - buildkit qits analyze src/
    - buildkit qits mutate src/ --threshold 80
    - buildkit qits scan src/
  artifacts:
    reports:
      quality: qits-quality-report.json
      security: qits-security-report.json

---

Related Pages

• TDD Methodology
• TDD Enforcement
• Unit Testing Strategies
• Agent Tracer