Validate Documentation vs Implementation

Purpose: Systematically identify and fix mismatches between actual implementation code and documentation.

Load When:

• •User says: "validate docs", "check documentation", "docs vs implementation"
• •User mentions: "documentation mismatch", "docs don't match code"
• •After major code changes that may affect docs
• •During pre-release validation

Core Value: Ensures documentation accurately reflects actual implementation, preventing user confusion and wasted debugging time.

Quick Reference

What This Skill Does:

1. •✅ Analyzes implementation code (scripts, core logic)
2. •✅ Identifies what the code actually creates/does
3. •✅ Searches all documentation for references to those entities
4. •✅ Compares documented behavior vs actual behavior
5. •✅ Fixes all mismatches systematically
6. •✅ Provides summary of changes made

Typical Mismatches Found:

• •File names (e.g., docs say "design.md" but code creates "plan.md")
• •Folder structure (e.g., docs missing "02-resources/" directory)
• •Counts (e.g., docs say "3 files" but script creates "4 directories + 3 files")
• •Naming conventions (e.g., old vs new terminology)
• •References to deprecated features

Workflow: 5-Step Validation Process

Step 1: Analyze Implementation

Goal: Understand what the code actually does

Actions:

1. •Read the implementation file(s) identified by user
2. •Document exactly what it creates/generates/does:
   • •Files created (names, locations, count)
   • •Folders created (names, structure, count)
   • •Data structures used
   • •Behavior patterns
3. •Create a reference list of "ground truth" facts

Example Output:

## Implementation Analysis: init_project.py

**Creates**:
- 4 directories: 01-planning/, 02-resources/, 03-working/, 04-outputs/
- 3 files in 01-planning/: overview.md, plan.md, steps.md

**Does NOT create**:
- design.md (old name)
- tasks.md (old name)
- requirements.md (never existed)

Step 2: Search Documentation

Goal: Find all references to the entities/behavior

Actions:

1. •

   Use Grep to search for old/mismatched terms across documentation:

   bash

   # Search for old file names
   grep -r "design\.md\|tasks\.md\|requirements\.md" 00-system/
   
   # Search for old folder structure
   grep -r "02-working\|03-outputs" 00-system/
   

2. •

   Identify all files that contain references

3. •

   For each file, note line numbers and context

Output: List of files needing updates with specific locations

Step 3: Compare & Identify Mismatches

Goal: Categorize what needs fixing

Actions:

1. •

   For each documentation file found:

   • •Read the relevant sections
   • •Compare against implementation truth
   • •Categorize mismatch type:
     • •File name mismatch: Wrong file names referenced
     • •Structure mismatch: Wrong folder structure shown
     • •Count mismatch: Wrong counts ("3 files" vs "4 directories + 3 files")
     • •Missing info: Implementation creates X but docs don't mention it
2. •

   Prioritize fixes:

   • •Critical: User-facing docs, core system maps
   • •Important: Skill workflows, reference materials
   • •Nice-to-have: Examples, comments

Step 4: Fix All Mismatches

Goal: Update documentation to match implementation

Actions:

1. •

   For each file with mismatches:

   • •Read the file
   • •Use Edit tool to fix each mismatch
   • •Preserve context and formatting
   • •Update related references (e.g., if fixing folder name, update all mentions)
2. •

   Common fix patterns:

   markdown

   # File name fixes
   OLD: "design.md"  → NEW: "plan.md"
   OLD: "tasks.md"   → NEW: "steps.md"
   
   # Structure fixes
   OLD: "02-working/" → NEW: "03-working/"
   OLD: "03-outputs/" → NEW: "04-outputs/"
   ADD: "02-resources/" (was missing)
   
   # Count fixes
   OLD: "3 core files"
   NEW: "4 directories (01-planning/, 02-resources/, 03-working/, 04-outputs/) + 3 planning files"
   

3. •

   Track progress with TodoWrite:

   • •Mark each file as completed after fixing
   • •Maintains visibility for user

Step 5: Verify & Generate Report

Goal: Confirm all fixes are consistent and create permanent validation record

Actions:

5A. Verification:

1. •

   Re-search for old terms to verify they're gone:

   bash

   grep -r "design\.md" 00-system/  # Should return no results
   

2. •

   Spot-check fixed files to ensure consistency

5B. Determine Report Location:

First time or if location not set:

AI: "Where should I save validation reports? Options:
    1. 04-workspace/validation-reports/ (recommended - organized, searchable)
    2. 01-memory/validation-reports/ (persistent record)
    3. Custom location (you specify)

    Or say 'skip' to not save reports."

User: [Chooses option]

AI: "Got it! I'll save reports to [location]. This will be remembered for future validations."

If location already configured (from previous validation):

AI: "Saving report to configured location: [location]"

5C. Generate & Save Report:

1. •Create validation report using template (see references/report-template.md)
2. •Generate filename: validation-{implementation}-YYYY-MM-DD.md
   • •Example: validation-init-project-2025-11-24.md
3. •Save to configured location
4. •Display summary to user with link to full report

Report includes:

• •Implementation analyzed
• •Files updated (with line-by-line changes)
• •Mismatch breakdown by type
• •Verification results
• •Status and next steps

Example output:

✅ Validation complete!

**Summary**:
- 6 files updated
- 23 total fixes
- 0 grep results for old terms

📄 Full report saved: [04-workspace/validation-reports/validation-init-project-2025-11-24.md](04-workspace/validation-reports/validation-init-project-2025-11-24.md)

Common Validation Scenarios

Scenario 1: Script Creates Different Files Than Documented

Symptoms:

• •User reports "docs mention design.md but I don't see it"
• •Script creates files not mentioned in docs

Process:

1. •Read script to see what it actually creates
2. •Search docs for old file names
3. •Replace all references systematically

Example: The create-project validation we just did

Scenario 2: Folder Structure Changed

Symptoms:

• •Documentation shows old folder structure
• •New folders not mentioned in docs

Process:

1. •Check actual project structure (ls)
2. •Compare against documented structure
3. •Update all structure diagrams and references

Scenario 3: Counts/Descriptions Wrong

Symptoms:

• •"Creates 3 files" but actually creates more
• •Missing important details about what's created

Process:

1. •Count actual artifacts created
2. •Find all count references in docs
3. •Update with complete, accurate descriptions

Best Practices

Be Systematic

• •Don't skip steps or assume you've found everything
• •Use grep to search exhaustively
• •Check related files (if you fix system-map.md, check framework-overview.md too)

Preserve Context

• •Don't just fix the immediate error
• •Update related references nearby
• •Ensure examples/diagrams stay consistent

Use Parallel Searches

• •When searching for multiple patterns, use multiple Grep calls in parallel
• •Saves time and provides complete picture faster

Track Progress

• •Use TodoWrite to show user what's being fixed
• •Mark items complete as you go
• •Provides transparency and confidence

Verify Fixes

• •Re-search for old terms after fixing
• •Spot-check that new terminology is consistent
• •Run validation commands if available

Integration Points

• •After code changes: Run this validation before committing
• •Pre-release: Validate all docs match current implementation
• •User reports mismatch: Use this skill to fix comprehensively
• •Documentation updates: Verify changes didn't introduce new mismatches

Success Criteria

✅ Zero grep results for old/deprecated terms ✅ All structure diagrams match actual folder structure ✅ All counts match actual artifact counts ✅ All file references use current names ✅ User can follow docs without confusion

Remember: Documentation debt compounds quickly. Fix mismatches systematically and completely the first time!