Where this tool is known to be wrong.
Vaulytica runs the same checks the same way every time, and its report lists every one it ran. That is a claim about consistency. It is not a claim about accuracy, and this page is what stands behind the difference.
1. Not measured
There is no precision or recall figure for this tool on real contracts. Measuring one needs real documents annotated by attorneys, and that work has not been done. No rule has been signed off by an attorney.
What to conclude: treat every finding as a prompt to read the clause, and every silence as silence. A short report is not a clean bill.
Also not measured: how well scanned pages are read; defects no rule exists for; and whether the severities (critical, warning, info) are calibrated to what matters in practice.
2. What it reads
- English only, and United States law.
- PDF, Word (.docx) and plain text, up to 50 MB. Scanned pages are read by OCR, up to 500 pages, in the browser only: the command line and the MCP server do not run OCR, and say so.
- A document is matched to one of 265 document types when its score reaches 0.5. A document that matches none gets a small generic rule set, and a short report on an unmatched document is not a clean bill.
- Up to 4 more document types are detected from a document's vocabulary and checked as well. They are detected, not confirmed: an absence reported under one may be a clause the document was never supposed to carry.
3. Measured on documents we wrote
The test suite holds 327 specimen documents whose findings are pinned exactly, and rewrites each one many ways — different line breaks, spellings, clause numbering, capital letters — to check that the findings do not move. The project's authors wrote those documents. They are synthetic, and the numbers below are self-measurements, not an error rate on real contracts.
4. Known wrong, held open
Each row is a defect the tests have measured and hold open on purpose. A test asserts the count exactly, so a new case fails the build, and so does a fix that is not recorded.
| Cases | What goes wrong |
|---|---|
| 47 | On 47 specimen-and-layout pairs, rewriting a document's line structure (blank lines stripped, hard-wrapped, double-spaced) makes the defined-terms table gain or lose a term. |
| 27 | On 27 specimens, a correct document does not use every defined term its document type expects, so enforcing that list as a rule would accuse it of a drafting defect. The list is not enforced for that reason. |
| 27 | On 27 specimen-and-layout pairs, a party's name is read differently when the layout changes — most often a corporate suffix drifting in or out of the name. |
| 25 | On 25 specimens, when a long word is split across lines with a hyphen (as a PDF does), a finding appears or disappears because the broken word hides a phrase a rule looks for. |
| 15 | On 15 specimens, setting the limitation of liability in capital letters moves a finding, usually because a party name or a heading can no longer be told from the words around it. |
| 10 | On 10 specimens, calling the document a Contract instead of an Agreement moves a finding — most often because the document's title no longer contains the word it defines. |
| 9 | On 9 specimens, a late-fee rate written in words ("five percent") is not read, so the usury check stays silent. |
| 6 | On 6 specimens, setting the indemnity clause in capital letters moves a finding; with no lowercase letters, where a sentence starts — and so who indemnifies whom — can be misread. |
| 4 | On 4 specimens, setting the warranty disclaimer in capital letters (as drafters do) moves a finding. |
| 3 | On 3 specimens that already have Articles, renaming every Section to an Article makes two numbering schemes collide, and a cross-reference finding appears that the original does not draw. |
| 1 | On 1 specimen, double-spacing the document changes which deadline the critical-dates register derives. |
| 1 | On 1 specimen, rewriting an indemnity in the passive voice ("shall be indemnified by") draws a one-sided-indemnity finding the original does not. |
| 1 | On 1 specimen, renaming every Section to a Clause moves a finding. |
4 more relations of the same kind are held at zero: they were measured, repaired, and are now asserted not to regress.
5. Reported by users
None recorded. Every report and result now links to a report page, which shows exactly what a report would contain — a few identifiers and a short note, never contract text. It cannot send yet; until it can, open an issue on GitHub, describing the clause in your own words.