Validation: From Test Plan to Report

Before you review the agent's code, Fabriq runs a validation pipeline. It generates test cases from your test plan, executes them, and produces a report — all without involving the LLM after the plan is written.

The pipeline is:

testability.md → Case Generation → Execution → Report
     (you)         (deterministic)    (runners)    (results)

Step 1: Fill in testability.md

This is the only manual step. After implementation completes, open the Tests tab in the Feature Workbench. You'll see a file called tests/testability.md with a template:

# Testability: add-dark-mode-toggle

## Entrypoints
- Settings page at `/settings`

## Scenarios
- Toggle on → dark mode applies immediately
- Toggle off → light mode restores immediately
- Refresh page with dark mode on → preference persists
- Refresh page with light mode on → preference persists
- Toggle is accessible via keyboard

## Scope Constraints
- Do not test: browser compatibility, performance
- Skip: mobile viewport testing for now

What to fill in

Field	What it's for	Example
Entrypoints	Where to start testing	URL paths, API endpoints, component selectors
Scenarios	What to test	"Toggle on → dark mode applies" — one scenario per line
Scope Constraints	What to skip	Prevents the validator from testing areas not yet ready

Tips for good testability docs

• Be specific about each scenario: "User clicks toggle → dark mode CSS applies" covers the core flow
• Cover edge cases: "Refresh with dark mode on" uncovered a missing localStorage read → the agent fixes it before review
• Set scope boundaries: If you're not ready for mobile testing, say so — the validator won't generate cases for it

Step 2: Fabriq generates test cases

When you click Generate Cases, Fabriq parses testability.md deterministically — no LLM involved. The output is a set of structured test cases in tests/cases/*.yaml.

Each case includes:

• The scenario description
• Entrypoint and navigation steps
• Expected outcome
• Tags (smoke, regression, edge-case)

This is predictable and repeatable. Same testability.md → same test cases, every time.

Step 3: Run validation

Click Run Validation. Fabriq dispatches the test cases to one or more executors:

Executor	What it tests	When to use
API executor	Backend endpoints, business logic	Server-side changes, API modifications
Browser executor	UI interactions via Playwright	Frontend changes, visual features
Repo-test executor	Your existing test suite	Ensure existing tests still pass

The agent can work on multiple executor types in parallel.

Step 4: Read the report

Results are written to:

• tests/runs/<run-id>.json — raw results
• reports/validation-report.md — human-readable summary

The report shows:

# Validation Report: add-dark-mode-toggle
Run: 2026-07-20 14:30:00
Executor: Browser

## Results: 4/5 passed

✅ Toggle on → dark mode applies
✅ Toggle off → light mode restores
✅ Refresh page with dark mode on → preference persists
✅ Toggle accessible via keyboard
❌ Refresh page with light mode on → preference lost
   Expected: light mode persists
   Actual: dark mode appears briefly, then switches to light

What happens next

If tests pass (or warnings only): You can advance to the Review stage. The validation badges will show green.

If tests fail: The agent receives the failure details automatically. It analyzes the error, fixes the code, and you re-run validation. This loop continues until tests pass or you accept known failures.

Agent fix → Re-run validation → Check report → Approve or fix again

If you need to accept a known failure: You can advance to Review with failing tests, but the review screen will show a red badge. The approver (you or your team lead) decides whether to accept the risk.

Pro tips

• Fill testability.md early: Even a rough test plan helps the agent code better — it knows what outcomes will be checked
• Run validation incrementally: Don't wait until all code is written. Run the API executor after backend changes, the browser executor after UI changes
• Good testability = better code: Agents produce better code when they know what test scenarios they'll be held to

Next steps

• Reviewing Changes — review diffs and validation badges
• Stage Gates — gates that check validation completeness
• Feature Lifecycle — where validation fits in the stages