TESTING CHECKLIST

Agent Generator - Testing Checklist

Pre-Flight Checks

Node.js 20+ installed (node --version)
npm installed (npm --version)
Config.json exists at /Volumes/AgentPlatform/repos/bare/blueflyio/config.json
separation-of-duties.md exists at /Volumes/AgentPlatform/wikis/blueflyio/separation-of-duties.md
Write permissions for output directory

Installation Tests

npm install completes without errors
node_modules/ directory created
All dependencies installed (check package-lock.json)
No security vulnerabilities (npm audit)

Build Tests

npm run build completes successfully
dist/ directory created
TypeScript compilation succeeds
No type errors
Source maps generated

Unit Tests

npm test runs all tests
Parser tests pass (separation-of-duties-parser.test.ts)
OSSA generator tests pass (ossa-manifest-generator.test.ts)
MCP generator tests pass (mcp-tool-generator.test.ts)
Test coverage ≥ 80%
No failing tests

CLI Tests

List Projects Command

npm run dev -- list-projects works
Shows all 66 projects
Each project has name and purpose
Responsibility counts shown
No errors in parsing

List Canonical Agents Command

npm run dev -- list-projects --canonical works
Shows 16 canonical agents
Agents match expected list:
- task-dispatcher
- merge-request-reviewer
- pipeline-remediation
- release-coordinator
- issue-lifecycle-manager
- manifest-validator
- code-quality-reviewer
- drupal-standards-enforcer
- module-generator
- recipe-publisher
- cluster-operator
- kagent-catalog-sync
- mcp-server-builder
- vulnerability-scanner
- cost-intelligence-monitor
- documentation-aggregator

Generate Agents Command

npm run dev -- generate-agents --output /tmp/test-agents --overwrite works
Creates output directory
Generates 66 YAML files
Files have .yaml extension
Progress shown during generation
Summary statistics displayed
No generation errors

Generate MCP Tools Command

npm run dev -- generate-mcp-tools --output /tmp/test-mcp --overwrite works
Creates output directory
Generates 66 JSON files
Files have -mcp.json suffix
Progress shown during generation
Summary statistics displayed
No generation errors

Validate Command

npm run dev -- validate-manifests --path /tmp/test-agents works
Validates all YAML files
Shows ✓ for valid manifests
Shows ✗ for invalid manifests
Summary statistics displayed
All 66 manifests valid

Generated Manifest Quality Checks

Pick Random Manifest (e.g., platform-agents.yaml)

YAML syntax valid
apiVersion: ossa/v0.4.9
kind: Agent|Task|Workflow
metadata.name present and valid (DNS-1123)
metadata.version: 1.0.0
metadata.description meaningful
metadata.author: BlueFly.io Platform Team
metadata.license: Apache-2.0
metadata.repository valid URL
metadata.tags array present
spec.role contains PURPOSE
spec.role contains RESPONSIBILITIES
spec.role contains NOT RESPONSIBLE FOR
spec.role contains CRITICAL RULES
spec.llm.provider: anthropic
spec.llm.model: claude-sonnet-4
spec.tools array present
spec.runtime.type: docker
spec.dependencies present
spec.extensions.skills_compatibility.enabled: true
spec.extensions.llms_txt.enabled: true
spec.extensions.agents_md.enabled: true

Pick Random MCP Tool (e.g., platform-agents-mcp.json)

JSON syntax valid
name matches project
description meaningful
version: 1.0.0
protocol: mcp/1.0
tools array present
Execute tool present
Status tool present
Input schemas valid
Output schemas valid
Security configuration present
Authentication type: oauth2
RBAC roles present

Integration Tests

Full Workflow Test

Run ./quick-start.sh
Dependencies install
Build succeeds
Lists projects
Lists canonical agents
Generates all manifests
Generates all MCP tools
Validates all manifests
Shows summary
Output directory created
66 manifests generated
66 MCP tools generated
All validations pass

Overwrite Protection Test

Generate manifests to directory
Run generate again without --overwrite
Existing files skipped
Skip count shown in summary
No files overwritten

Overwrite Test

Generate manifests to directory
Modify one manifest manually
Run generate again with --overwrite
All files regenerated
Manual changes lost (expected)
Generation count matches total

Error Handling Tests

Invalid input path shows clear error
Missing config.json shows clear error
Invalid output path shows clear error
Malformed separation-of-duties.md shows clear error
Permission denied shows clear error

Config Integration Tests

Config.json loaded successfully
Tokens resolved ($NAS_MOUNT, $WIKIS_BASE, etc.)
Default paths work
Custom paths work with --input and --output
Environment variables substituted (${HOME}, ${USER})

Parser Edge Cases

Projects with minimal data parsed
Projects with no dependencies handled
Projects with empty responsibilities handled
Special characters in names sanitized
Duplicate project names handled
Missing sections handled gracefully

Generator Edge Cases

Agent kind determination correct
Task kind for executor projects
Workflow kind for orchestrator projects
Tool name sanitization works
Tag deduplication works
Dependency normalization works

Performance Tests

Generate 66 agents in < 2 seconds
Validate 66 manifests in < 3 seconds
Memory usage reasonable (< 500MB)
No memory leaks during generation
Build time < 10 seconds

Cross-Platform Tests (if applicable)

Works on macOS
Works on Linux
Works on WSL
Works in code-server
Paths resolve correctly on all platforms

Documentation Tests

README.md clear and complete
USAGE.md has working examples
BUILD-SUMMARY.md accurate
Code comments helpful
Examples run without errors

Deployment Readiness

All tests pass
No linting errors
Code formatted with Prettier
No TypeScript errors
No security vulnerabilities
Git repo clean (if applicable)
Documentation complete
Examples work
Quick start script works

Final Verification

Run full generation one more time
Spot-check 5 random manifests manually
Validate all manifests pass
MCP tools have correct structure
No errors in console
Summary statistics correct
Output files readable and well-formatted

Sign-Off

Developer tested: ________________
Peer reviewed: ________________
Date: ________________
Version: 0.1.0
Status: Ready for production ✅

Notes

Record any issues or observations during testing:

[Date] - [Issue/Observation]

Next Steps After Testing

Copy generated manifests to platform-agents repository
Create GitLab issue for deployment tracking
Review and customize manifests as needed
Deploy to agent-mesh
Test with GitLab Duo
Monitor in production