For decision makers
How it would work for real
What this prototype simulates, what the production version would do, and what it would cost to run.
What you saw
And what was simulated.
Real
The workflow, the screens, the scoring rules, the reviewer controls, the criteria library (six Markdown files), and the way evidence links to a document section or a page.
Simulated
The three offices and their documents are fictional. Findings were written in advance. Quick-check results for other addresses are generated from the address itself so the same input always gives the same output. Nothing is uploaded or sent anywhere.
Deliberately absent
Logins, a database, and a working free-text chat. Each is a known quantity and none changes the workflow.
The pipeline
Five steps. The model runs in one of them.
- 1
Submission
An office uploads Word documents, PDFs, or screenshots through the same form you saw. Files are stored privately with the evaluation.
In production. Word and PDF text extraction (open-source libraries, no AI cost).
- 2
Context assembly
The six standard documents, the instructions from Settings, and the extracted text are combined into one request.
In production. The criteria library is cached, so it is paid for once per day, not once per evaluation.
- 3
First pass
A language model returns one finding per criterion as structured data: status, confidence, a one-line summary, and the quoted evidence with its document and section.
In production. Claude via Vercel AI Gateway, with a fixed output schema so the app can validate every finding.
- 4
Verification
The app checks that every quote actually appears in the cited document. Quotes that don't match are dropped to Low confidence and flagged.
In production. Plain string matching. Catches the most common failure of a first pass: a confident citation that isn't there.
- 5
Review
The report you saw. The reviewer confirms, changes, or annotates each finding; scores update live; the record keeps both the AI's call and the reviewer's.
In production. Stored per evaluation with a full history.
What it costs
Estimates at October 2026 list prices. Model cost is the only per-use cost; it is small next to reviewer time.
| Task | Most capable | Balanced | Fastest |
|---|---|---|---|
| Full document evaluation (about 40 pages) | $0.28 | $0.14 | $0.01 |
| Quick check (about 10 public pages) | $0.20 | $0.10 | $0.01 |
| 50 offices × 4 evaluations + 500 quick checks per year | about $160 | about $80 | about $8 |
- Hosting on Vercel: free tier covers a tool of this size; about $20 a month if a team plan is needed.
- Document parsing and quote verification: no per-use cost.
- Reading public sites: free for ordinary pages. A hosted crawler (about $20 a month) only if sites require a full browser. Gated or staging pages come in as PDFs or screenshots.
- Assumes a 40-page submission, the criteria library cached, and about 5,000 words of output per evaluation.
Keeping it maintainable
- The standard is text, not code. Each of the six standards is a Markdown file. Changing a requirement or a snippet means editing a document, and the evaluator, the library pages, and Ask all pick it up.
- Staff settings stay small. Instructions, weights, and which of the 24 criteria are in scope. Nothing else needs tending.
- The reviewer's record is the product. The AI's call and the reviewer's decision are both kept, so the first pass can be measured against human judgment and improved.
Caliper