AI Scanner, Engine Validator: Why We Don't Ask LLMs to Count Lines

TL;DR. The industry tries two extremes for code understanding. Pure static analysis (lint rules, regex scraping) breaks when your framework changes. Pure LLM analysis (point an AI at your codebase and hope) hallucinates physical coordinates — "it's on line 142" when line 142 is a blank line. Fabriq takes a third path: a deterministic engine owns structure, call graphs, and evidence validation; an AI scanner owns semantics, intent, and business rule extraction. The orchestrator feeds the AI prepared slices with stable physical anchors, then revalidates every claim before it enters the workspace.

The two extremes#

Pure static analysis#

SonarQube, ESLint, hand-written regex patterns, framework-specific AST queries. These are fast, deterministic, and repeatable. They also share a critical weakness: they don't understand semantics.

A static analysis rule can tell you "this method has 80 lines" but not "this method implements a business rule about refund windows that was established in the 2024 compliance review." It operates on structure, not meaning.

The brittleness shows when frameworks evolve. You wrote a custom rule for your Express.js middleware patterns. Then you migrate to Fastify. Half your rules stop matching. You invested heavily in tool-specific knowledge that doesn't transfer.

Pure LLM analysis#

Point a large language model at your codebase and ask questions. It's flexible, it adapts to different frameworks, and it can reason about intent.

But it also suffers from a critical flaw: hallucinated physical coordinates.

When we tested LLMs on the task "find where payment validation happens and return the file and line number," the error rate was ~30% — even on codebases the model could describe accurately. A line number is a discrete physical coordinate. LLMs are not trained to count lines; they're trained to predict tokens. Asking them for line numbers is asking for something their architecture fundamentally cannot do reliably.

The problem compounds when you ask for more: "list all callers of this function." The LLM might guess 5 callers when there are 12, or invent 8 plausible-looking function names that don't exist.

The third path#

Fabriq draws a clean line between the two approaches. We call it AI Scanner, Engine Validator:

The orchestrator follows a strict pipeline:

Step 1: Engine prepares the slice#

Instead of giving the AI raw file content and asking it to find things, the engine first runs deterministic analysis:

// Engine: find all callers of a function deterministically
const callers = codeGraph.findCallers(
  "OrderController", 
  "validateRefundWindow"
);
// Returns: [{ file: "src/OrderController.java", line: 42 }, 
//           { file: "src/RefundService.java", line: 18 }]
// 100% accurate, no hallucination possible

The engine produces a prepared slice — a bounded, structured view of the relevant code with stable physical anchors. The AI never sees the raw codebase; it sees the slice.

Step 2: AI enriches with semantics#

The AI receives the prepared slice and answers semantic questions:

// AI: enrich the callers with semantic context
{
  "sources": [
    {
      "file": "src/OrderController.java",
      "symbol_handle": "OrderController#validateRefundWindow",
      "kind": "fact",
      "semantic_context": "Called during POST /orders/refund. Extracts purchase date from request body and compares against current time minus 30 days."
    }
  ]
}

The AI operates on symbol handles (OrderController#validateRefundWindow), not line numbers. The engine resolves the handle to a physical location deterministically.

Step 3: Engine revalidates#

Before any artifact enters the workspace, the engine runs a validation pass:

1. Resolve every symbol_handle — does OrderController#validateRefundWindow still exist in the current source?
2. Check every fact-level claim — can the engine independently verify the assertion?
3. Demote unverifiable claims — fact → inference if the symbol moved or disappeared
4. Reject artifacts with unresolvable critical dependencies

If the code changed between when the AI analyzed it and when the artifact is committed, the engine catches it. The artifact that reaches the workspace is guaranteed to be grounded in current reality.

Symbol handles: the anchor that survives refactoring#

The critical innovation is the symbol handle — replacing absolute line numbers with semantic identifiers.

# Before: line-number anchoring (brittle)
sources:
  - file: src/OrderController.java
    line: 42
    kind: fact
 
# After: symbol handle anchoring (resilient)
sources:
  - file: src/OrderController.java
    symbol_handle: OrderController#validateRefundWindow
    kind: fact

A symbol handle survives:

• Line shifts. Add a comment above the method? The handle still resolves.
• File reorganization. Move the method to another file? Update one reference.
• Refactoring. Rename the method? The handle changes explicitly, not silently.

Tree-sitter resolves every handle at scan time. If a handle stops resolving, the engine demotes the claim automatically — from fact to inference — never silently to unknown.

Three-level evidence scale#

Every claim in a Fabriq artifact carries an evidence level:

The scale is designed so that unknown is never a terminal state. There's always a resolution path: the AI can re-examine the code with updated context, or a human can provide the missing evidence.

The result: auditable by construction#

The AI Scanner, Engine Validator split produces artifacts that are:

• Physically grounded. Every claim that matters is linked to a real, currently-existing symbol in the codebase.
• Resilient to change. Symbol handles survive refactoring. Evidence degrades gracefully (fact → inference) instead of breaking silently.
• Auditable. The evidence chain is a first-class data structure. You can trace any claim back to its source code, its validation result, and its evidence level.
• Deterministic where it matters. The engine never hallucinates. The AI enriches but never owns the truth about physical coordinates.

This split is the deepest architectural commitment in Fabriq. It's not an optimization — it's the core insight that makes cognitive artifacts trustworthy enough to put in version control.

Read the CDR technical spec for the full architecture, or check the Tree-sitter integration in the Fabriq repo.