Skip to content

Architecture.md

Latest commit

 

History

History
793 lines (638 loc) · 56 KB

Architecture.md

File metadata and controls

793 lines (638 loc) · 56 KB
Raw

Cavil Architecture

Components

Cavil has three primary components:

  1. Web Application providing the UI, MCP and REST API
  2. Job queue for processing background jobs
  3. AI text classification server

The web application and the job queue communicate via PostgreSQL database. All other components communicate via HTTP.

         +---------+     +---------------------+     +----------------+
         |         |     |                     |     |                |
User --> |         |     |                     |     |                |
         |  Nginx  | --> |   Web Application   | --> |   PostgreSQL   |
Bots --> |         |     |                     |     |                |
         |         |     |                     |     |                |
         +---------+     +---------------------+     |                |
                                                     |                |
                         +---------------------+     |                |
                         |                     |     |                |
                         |                     | --> |                |
                         |                     |     |                |
OBS <------------------- |                     |     +----------------+
                         |      Job Queue      |
Git <------------------- |                     |     +----------------+
                         |                     |     |                |
                         |                     | --> |       AI       |
                         |                     |     |                |
                         +---------------------+     +----------------+

Data

Additionally to PostgreSQL, there is also a significant amount of data stored on the file system. The location is configurable, but the structure will be the same everywhere. These are the actual packages Cavil has been tasked with creating legal reviews for and metadata created for those packages.

checkouts                                # Configurable root directory
|- gcc                                   # Root for all "gcc" packages
|  |- 047f96de986898d51f855f99d475b9b6   # Checkout of "gcc" version with that checksum
|  |- 29c482ed6d887d9aff78dff785a940b5   # Another checksout of "gcc" with a different checksum
|  +- ...
|
|- perl-Mojolicious
|  |- 0297a570e088c24551da36a4d31d785e
|  +- 11f5ece47a018082eaaedf9f9d038148
|     |- Mojolicious-9.31.tar.gz         # Compressed archive in checkout
|     |- perl-Mojolicious.changes
|     |- perl-Mojolicious.spec           # Specfile in checkout
|     |- .postprocessed.json             # Cavil metadata
|     |- .report.spdx.json.gz            # Cached (gzip-compressed) SPDX SBOM generated by Cavil for this checkout
|     |- .unpacked.json                  # Cavil metadata
|     +- .unpacked                       # Recursively unpacked archives from this checkout
|        |- Mojolicious-9.31             # "Mojolicious-9.31.tar.gz" unpacked
|        |  |- Changes
|        |  |- LICENSE
|        |  |- Makefile.PL
|        |  |- Makefile.processed.PL
|        |  +- ...
|        |- perl-Mojolicious.changes
|        |- perl-Mojolicious.spec        # Copy of file that did not need to be unpacked
|        +- .unpacked.json               # Cavil metadata
|
+- ...

Both the web application and the job queue interact with these files. For a system whose purpose is to hold legal evidence, the integrity model of that shared storage matters as much as the topology, so it is worth stating as invariants rather than leaving implicit:

  • Evidence is content-addressed and is never overwritten in place. A checkout lives at <name>/<verifymd5>, where verifymd5 is a checksum of the source itself. Identical source always maps to the same directory (re-import is idempotent), and any different source maps to a different directory — so importing a new version can never mutate the evidence an existing report was built from.
  • One pipeline owns a package at a time; within it, indexing writers are parallel but controlled. The stages that process a package — import, unpack, index, analyze, SBOM generation — are ordered by job dependencies (unpack before index before analyze before the SBOM), and each takes the same per-package lock (processing_pkg_<id>) for its duration, so no two stages, and no second pipeline (a re-import racing a reindex, say), ever run against the package at once — concurrent processing is refused, not interleaved. Indexing is the one stage that is deliberately not a single writer: for throughput it fans out into many parallel index_batch jobs and holds the package lock across the whole fan-out. These are controlled parallel writers, not a free-for-all — each batch owns a disjoint partition of the files and writes their rows in its own transaction, and the few tables several batches share (harvested URLs and emails, detected components) use upserts that merge concurrent contributions deterministically. The checkout filesystem still has a single writer throughout: the batch jobs only read the unpacked tree, and only the owning stage writes it (unpack builds the tree, index clears the stale SBOM). The web application never writes an existing checkout — it only creates a new content-addressed one (upload/import, and only when the directory does not yet exist) or reads files for display.
  • The source is the evidence; everything else is a derived cache. The imported source is authoritative. The unpacked tree, the .processed variants, the pattern/snippet/resolution rows in PostgreSQL, and the cached .report.spdx.json.gz SBOM are all regenerable from it. PostgreSQL is the source of truth for review decisions and the curated pattern corpus; the filesystem holds the evidence and these caches.
  • Reindexing invalidates and rebuilds; it never patches. Re-running the pipeline recreates the derived rows from scratch and recomputes the stored resolutions wholesale, and the SBOM is regenerated by its own job — so a report always reflects the current derived state rather than an accumulation of in-place edits. This is the same disposability the automated-resolution section below relies on.

AI

Text classification with a machine learning model is one of Cavil's components. The pattern matching that identifies clusters of legal keywords (snippets) has a false-positive rate of about 80%, and the classifier is what separates genuine legal text from that noise. The automated snippet resolution described below — scoring, folding, clearing and covering — all builds on the classifier's judgement, so a deployment without it never resolves any snippets and falls back to raw keyword matching. Even a simple model catches almost all of the false positives.

The openSUSE HuggingFace org has a collection of models fine-tuned specifically for this task, such as Cavil-Qwen3.5-4B. These models are usually deployed with a llama.cpp server.

Review Workflow

Cavil is designed for a human driven review process of RPM package sources. Package sources are imported (usually from OBS), recursively unpacked, and a legal report is created for them. This report is then reviewed by a human expert or lawyer and accepted or rejected. Under certain conditions the report may also be automatically accepted or rejected by Cavil.

States

Legal reports can be in one of five states:

  • new: Initial state, report ready for review.

  • acceptable: Reviewed and accepted by a human expert or automated system, but not a lawyer.

  • acceptable_by_lawyer: Same as acceptable, but review was performed by a lawyer.

  • unacceptable: Review by a lawyer that resulted in rejecting the report.

  • obsolete: Report no longer exists.

UI

The entire human driven workflow happens via HTTP web UI. These are the most important menu points:

  • Open Reviews: Lists all reports that are currently being prepared or ready for review. The report can be reviewed once the link with the unique report checksum becomes visible. From the report new license patterns can be created for newly identified snippets of potential legal text.
  • Recent Reviewed: Lists all review results from the past 3 months.
  • Snippets: Lists recently identified snippets of potential legal text and the associated AI text classification results if available. These results can be validated here to create new training data for future AI models.
  • Products: Lists all product codestreams and their associated packages that have been synchronized from OBS into Cavil.
  • Licenses: Lists known licenses and the associated license patterns. License patterns without a name are considered keyword patterns, and are used to identify new snippets of potential legal text.

Access Levels

Authorization is capability-based: every action asks "does the user have capability X?" rather than naming a role, and roles are named bundles of capabilities.

Capability What it allows
view Read reports, review results, licenses, snippets and statistics.
classify Validate AI text-classification results, creating training data.
propose Propose patterns, ignores and snippets; a proposal needs a curator to take effect.
curate Curate the corpus and drive reviews: create/edit/remove patterns, manage ignores and comment templates, accept or reject proposals, apply snippet decisions, reindex, and accept or reject reviews.
infra Operate the instance: the job dashboard and package upload.
review Move a report from new to acceptable (a non-lawyer expert sign-off).
review_lawyer Move a report from new to acceptable_by_lawyer (the legal sign-off).

user and admin are the base roles; the rest add capabilities on top. admin and lawyer share the same curator core and differ by one capability each — admin also runs the instance (infra), lawyer also carries the legal sign-off (review_lawyer) — so neither is a superset of the other, and the lawyer role alone is enough to curate and sign off.

Role view classify propose curate review infra review_lawyer
user ✓
classifier ✓ ✓
contributor ✓ ✓
manager ✓ ✓
admin ✓ ✓ ✓ ✓ ✓ ✓
lawyer ✓ ✓ ✓ ✓ ✓ ✓

acceptable_by_lawyer means a lawyer signed off, so it is always derived from the review_lawyer capability and never taken from the request: a non-lawyer curator can accept a package but can never assert a lawyer sign-off, on either the web or automation path. A user's capabilities are the union of their assigned roles.

Automatic Acceptance

Reports may be automatically accepted by the system under these conditions:

  • Low Risk with Manual Review: There was a prior manual review of the package by a human expert or lawyer. The maximum risk of any given license in the report is not higher than 3. And there are no unresolved keyword matches with a risk higher than 3. The resulting state can only be acceptable.
  • Low Risk without Manual Review: The maximum risk of any given license in the report is not higher than 2. And there are no unresolved keyword matches with a risk higher than 2. The resulting state can only be acceptable.
  • Previous Result: A previous report with the same checksum (based on licenses and keyword matches) exists for the same package. In this case the previous result will be inherited.
  • No Differences: A previous report exists where the checksum does not match but there are no significant differences between licenses and unique keyword matches.
  • Package Name: The package has been configured to always be acceptable. For SUSE instances of OBS this is usually done for empty metadata packages like 000product.

One condition overrides all of the above and is a first-class safety property: an incomplete checkout is never auto-accepted. If the source could not be fully retrieved — a file the spec references is missing, so what was indexed is not the whole package — the analyze step detects it and stops before any of the acceptance paths run, leaving the report in new for a human, with a notice that manual review is required because the checkout might be incomplete. Acceptance decisions are only ever made against evidence known to be complete; a partial checkout can hide exactly the license that matters, so the system declines to rule on it rather than accept on incomplete grounds. (A previous result is still surfaced as context, but only as a notice — never as an automatic acceptance.)

Standard Risk Levels

These are the standard risk levels used for license patterns included with Cavil:

  • 1 - Public Domain: (e.g., Public-Domain, CC0, Unlicense). Code is safe for any use with zero compliance overhead.
  • 2 - Permissive: (e.g., MIT, Apache 2.0, BSD-3-Clause). Attribution is required, but there are minimal restrictions on modification or distribution.
  • 3 - Weak Copyleft: (e.g., LGPL, MPL, EPL). Reciprocity applies at the file level. Changes to the library itself must be shared, but linking to non-copyleft code is permitted.
  • 4 - Strong Copyleft: (e.g., GPL-2.0-only, GPL-3.0-or-later). Reciprocity applies at the component or derivative work level.
  • 5 - Managed Obligations: (e.g., legacy advertising clauses, AGPL). Requires specialized compliance workflows. Includes network source disclosure or burdensome advertising clauses.
  • 6 - Restrictive Obligations: (e.g.,SSPL). Extreme reciprocity or non-free terms. Licenses may trigger source disclosure for the entire stack.
  • 7 - Non-Commercial: field-of-use, or ethical restrictions (e.g., "JSON License - Good not Evil"). They limit how customers can use the software.
  • 9 - Unknown: Keywords and phrases used to identify potential candidates for new license patterns.

Report Creation

Report creation is triggered via the REST Bot API, usually by bots like legal-auto and cavil-gitea. This results in various background jobs being created that are then performed by the jobs queue. Jobs can often be processed in parallel to make the best use of all available resources.

These jobs are involved in report creation and usually run in the listed order:

  1. obs_import or git_import: Checks out the package sources from OBS or git repo.
  2. unpack: Recursively unpacks all archives contained in package sources, then preprocesses the unpacked text files into .processed variants that the indexer scans instead of the originals (see Preprocessing below).
  3. index: Creates file lists and splits them up into batches for parallel processing.
  4. index_batch: Performs two phase pattern matching on all files in the batch with license and keyword patterns. There can be thousands of index_batch jobs at the same time.
  5. indexed: Synchronizes all pattern matching results.
  6. analyze: Combines patterns matching results to create the license report.
  7. analyzed: Checks the report for reasons to automatically accept it.
  8. spdx_report: Creates report in SPDX format.

The text classifier (see AI Text Classification) adds one more background job. It is enqueued automatically whenever analysis produces snippets that still need classifying, and (unlike the others) is not specific to a single package checkout.

  1. classify: Sends all unclassified snippets of potential legal text to the text classification server, and if necessary updates reports.

Preprocessing

Before indexing, the unpack job runs each unpacked text file through a preprocessing step that writes a <name>.processed.<ext> variant; the indexer then scans that variant instead of the original (the original stays on disk, and structured-metadata detection still reads it — injected line breaks would corrupt JSON and the like). Two transformations happen here:

  • Line wrapping. Over-long lines are split at word boundaries, so machine-generated single-line files stay readable in the file viewer and produce sensible snippet windows.
  • Markup stripping. HTML and XML files — including the component XML unpacked from ODF/OOXML office documents — are reduced to their entity-decoded text content, with <script>/<style> content dropped. This is done in-process with a streaming tokenizer (HTML::Parser), so peak memory stays flat regardless of file size. The result is line-wrapped like any other file. Without this step the matcher and reviewers would only ever see tag soup, forcing brittle markup-specific patterns; stripping means license text embedded in markup is matched as ordinary text. A file is only stripped when its extension and content both look like markup (a deliberately conservative gate), and any parser failure falls back to plain line wrapping so a file is never lost.

Rolling out a change to this step means re-unpacking existing packages; the unpack --rebatch command does that in paced batches (see the Maintenance guide).

Software Bill of Materials (SBOM)

Alongside the human-facing legal report, Cavil produces a machine-readable Software Bill of Materials — an inventory of every component and file that makes up a package, together with the licenses found in them.

Background

The EU Cyber Resilience Act (CRA) requires manufacturers of products with digital elements to compile an SBOM for their software. The CRA does not itself prescribe a file format; the German Federal Office for Information Security (BSI) publishes the technical guideline BSI TR-03183-2, which describes what an SBOM should contain and in which formats. Cavil's SBOM generator follows the SPDX format and data model described by that guideline, as detailed in the rest of this section.

Whether a given SBOM meets a particular regulatory obligation is a determination for the manufacturer and their legal advisers; it is not something Cavil asserts. The following describes what Cavil actually produces.

Format

SBOMs are emitted as SPDX version 3.0.1 documents in JSON (BSI TR-03183-2 requires SPDX 3.0.1 or newer, in JSON or XML). SPDX 3.0 models a document as a graph of typed elements — the SBOM itself, the package, its files and subcomponents, the licenses, and the relationships between them — rather than the flat "tag: value" text of the older SPDX 2.x. Cavil writes this graph directly, streaming it element by element so that even packages with hundreds of thousands of files can be exported without exhausting memory. The output validates against the official SPDX 3.0.1 JSON schema (this is checked by the test suite).

What the SBOM contains

The document describes the package as the primary component and records, for it and for the files and subcomponents below it, the following data fields described by BSI TR-03183-2, as far as the available metadata allows:

  • Who created the SBOM and when — a configurable creator identity (organisation name plus an email address, or a URL if no email is available) and a timestamp.
  • A unique identity for the SBOM — its own URI, so it can be referenced from other SBOMs.
  • Component name and version — for the package (from its spec file) and for each bundled component (from the component's own metadata).
  • Supplier / originator — derived from the Open Build Service coordinates the package came from.
  • Download and home page locations — where the source can be obtained and the upstream project page.
  • A package identifier — a package URL (purl) for cross-referencing in vulnerability databases.
  • A cryptographic hash — a SHA-512 checksum of the delivered source archive (the "deployable component").
  • Licenses — the package's declared and concluded licenses, plus the licenses Cavil detected in individual files (see License identification below).
  • Dependencies — the bundled components (see Bundled components) and the files contained in the package, expressed as SPDX relationships with an explicit completeness marker.

Following BSI TR-03183-2, the SBOM deliberately contains no vulnerability information — SBOM data is static, whereas vulnerability data changes over time and is exchanged through separate channels (such as CSAF/VEX).

License identification

BSI TR-03183-2 (section 6.1) is strict about how licenses are named, and Cavil follows its order of preference for every license it records:

  1. If the license has an official SPDX identifier (for example MIT or GPL-2.0-or-later), that is used.
  2. Otherwise Cavil consults the ScanCode LicenseDB and uses its identifier with the required LicenseRef-scancode- prefix (for example LicenseRef-scancode-proprietary-license).
  3. If the license is unknown to both, Cavil mints a LicenseRef-<entity>-… identifier in its own namespace.

License text is never used as a substitute for an identifier, as the guideline requires.

Bundled components

Modern packages frequently vendor their dependencies — a single source package can carry hundreds or thousands of npm, Rust, Go, Java, Python, PHP, .NET or Ruby modules copied straight into the tree. These are shipped to the user, so under the CRA they are part of what must be inventoried, yet they appear in neither the RPM spec file nor a naive file listing. Cavil detects them and lists each as a component of the package.

Detection is content-based. Every ecosystem ships a self-describing metadata file inside each installed module (npm package.json, Rust Cargo.toml, Python *.dist-info/METADATA, Java pom.properties, Go vendor/modules.txt, PHP vendor/composer/installed.json, .NET *.nuspec, Ruby specifications/*.gemspec, …). Cavil reads that file's contents to learn the component's name, version and declared license (where a manifest is executable code, such as a Ruby gemspec, the name and version come from its standardized filename instead) — so detection works no matter how deeply the module is buried or how the packaging tools renamed the directory. That last point matters in practice: vendored trees routinely end up under obscured names such as node_modules.obscpio._/package._1/, and any approach that relied on directory names would miss them. Each component is identified by a package URL (purl), the standard identifier for cross-referencing against vulnerability databases.

Two principles keep the result trustworthy:

  • Presence, not declaration. A component is reported only when its files are actually present in the delivered tree. A dependency merely declared in a manifest — a lint tool listed under devDependencies, say — but not shipped is never invented. This keeps the inventory to what is genuinely delivered.
  • Cavil's own license scan fills the gaps. Where a component's metadata omits the license, Cavil falls back to the license it detected by scanning that component's own files.

Detected components are stored per package (the same way harvested emails and URLs are), shown in the report UI, and emitted in the SPDX report as dependencies of the primary component, each carrying its purl, version and license (both the distribution and the original license, as BSI TR-03183-2 requires).

Detection runs during indexing, alongside the normal file scan, and adds negligible cost — only files whose name matches a known metadata file are parsed. Adding a new ecosystem is a small, self-contained detector module, so the list of supported ecosystems can grow easily.

Two honest limitations remain, both permitted by BSI TR-03183-2 (which allows omitting data that is not available from the way a component was assembled):

  • Cavil lists the full set of bundled components but does not reconstruct the dependency graph among them (which bundled module depends on which); they are recorded as dependencies of the package as a whole.
  • A bundled component's creator is not recorded, because ecosystem metadata rarely provides it as the email address or URL the guideline asks for.

The package's own top-level manifest (an npm project's root package.json, say) is not reported as a bundled component, even though it is present — it describes the primary artifact under review, not a vendored dependency, so listing it would make the package a subcomponent of itself. Only manifests nested below the source root are treated as vendored.

How it is generated and retrieved

SBOM generation is the final step of report creation (the spdx_report job) and can also run on demand. The result is cached next to the package checkout as .report.spdx.json.gz — stored gzip-compressed, because the JSON is highly repetitive and compresses roughly ten-fold, which matters at the scale of many cached reports. It is served over the REST API untouched to clients that accept gzip (essentially all of them) and decompressed on the fly for the rest; if a request arrives before the SBOM has been generated, the API returns a 408 status until it is ready. By default SBOMs are produced on demand, but an instance can be configured to generate one automatically whenever a package is indexed. The creator identity, document namespace and LicenseRef namespace are all set in the instance configuration.

See the User API documentation for the exact endpoint and an example document.

Pattern Matching

For a non-technical, lawyer-facing overview of how a piece of text becomes a reusable pattern — the review queues, who proposes what, and what "missing license" means — see PatternLifecycle.md. This section is the technical detail behind that life cycle.

Pattern matching is the heart of Cavil. It is how the index_batch job turns raw source files into the license and keyword matches that every report is built from. The matching itself is done by a separate, performance-critical C++ library; Cavil provides the glue that loads patterns into it, runs files through it, and stores the results.

The engine is selectable through the matcher configuration value. The default is Cavil::Matcher, described below; it keeps its compiled index in a memory-mapped file that every worker on a host shares, which is what removes the memory ceiling discussed under Scaling Characteristics. The alternative spooky engine (Spooky::Patterns::XS) is the original implementation and remains fully supported. Both use the same tokenizer and the same token-hash algorithm, and therefore produce identical matches and identical stored hashes, so an instance can switch between them at any time without a reindex or a migration.

Patterns and Keywords

There are two kinds of patterns, and this is the "two phase" matching referred to under Report Creation:

  • License patterns are named after a license (for example MIT or GPL-2.0-or-later) and carry a risk level from 1 to 7. When one matches, that license is added to the report.
  • Keyword patterns have no license name and always carry risk level 9. Rather than identifying a license, they flag a region of a file as potential legal text that no license pattern has recognised. These regions become snippets (see AI Text Classification) — the candidates from which experts propose new license patterns. Keyword matching is deliberately broad and has a false-positive rate of about 80%, which is exactly why AI classification is so useful for sifting the snippets.

A pattern can also be either global, in which case it applies to every package, or scoped to a single package. Package-scoped patterns let an expert refine or silence matches for one troublesome package without affecting everyone else.

The Pattern Language

Patterns are neither regular expressions nor plain substrings. Before any comparison happens, patterns and the files being scanned are put through the same normalisation, which is what makes matching survive reformatting and reflowing of license texts:

  • Everything is lower-cased and split into words on whitespace and the usual punctuation and markup characters.
  • Words that carry no meaning are thrown away — punctuation-only fragments and a small fixed set of comment and markup noise words. Comment markers and layout therefore never have to be matched literally.
  • Each remaining word is reduced to a numeric hash; only those hashes are ever compared, never the original text.

The one wildcard is $SKIP<n>, which matches up to n arbitrary words (the limit is 99). It lets a single pattern absorb the parts of a license text that legitimately vary, such as copyright holders, years or addresses:

Permission is hereby granted $SKIP30 to deal in the Software without restriction

A pattern may not start or end with a $SKIP, because a match has to be anchored on real words at both ends.

The Matching Engine

All of a package's patterns are loaded into one shared prefix tree, keyed on the word hashes described above. Patterns that begin with the same words share the same branch, and only diverge where the words diverge:

(start)
  ├─ permission ─ is ─ hereby ─ granted ─ $SKIP30 ─ to ─ ...   →  (MIT-style pattern)
  ├─ gnu ─ general ─ public ─ license ─ ...                    →  (a GPL pattern)
  └─ copyright ─ ...

Scanning a file means walking its words through this tree and noting every place a complete pattern is reached. The important consequence of this design is that matching speed barely depends on how many patterns there are: 29,000 patterns are not meaningfully slower to scan against than 2,900, because shared beginnings are walked only once and only branches that actually occur in the file are explored. The exception is $SKIP, which forces the scanner to try several continuations at once, so patterns built from many skips are the ones that cost extra.

When two patterns match overlapping regions of a file, the longer match wins; an exact tie is resolved in favour of the more recently added pattern, on the assumption that newer patterns are the more specific ones. A couple of safety limits guard against pathological input: very long lines are truncated, and the scanner only keeps a sliding window of recent words in memory. That window is sized relative to the longest known pattern, so a single enormous pattern quietly makes scanning heavier for every file — something to keep in mind when adding patterns.

Indexing a File

Each index_batch job builds its matcher once: it loads all the global patterns (almost always from a cache, described below), then adds the handful that are scoped to the package being indexed. Every file in the batch is then scanned, and each match is recorded for the report.

Matches against named license patterns are finished at that point. A match against a keyword pattern instead kicks off snippet extraction: Cavil takes the lines immediately around the keyword, merges nearby keyword regions that are only a few lines apart into one block, and stores each resulting block as a snippet for later classification. If a block's content matches one an expert has previously marked as uninteresting, it is silently dropped instead — this is how known false positives stay out of the queue.

Caches

Re-reading and re-tokenising every pattern from the database for each of the thousands of index_batch jobs would be far too expensive, so two files are kept in Cavil's cache directory. Their names identify the engine that produced them, since the two formats are not interchangeable, so switching engines never reads a cache the other one wrote:

  • The compiled prefix tree of all global patterns, which is what indexing jobs load instead of rebuilding it. It is written once and then reused. The default engine maps the file into memory read-only, so however many indexing workers run on a host, they all share a single physical copy of it; the spooky engine instead gives each worker its own copy on the heap.
  • A serialised "bag of patterns" similarity model, used for the closest-match lookups described below.

Both caches are discarded whenever any pattern is created or edited, which also schedules a background job to rebuild the similarity model. That job only republishes the model artifacts; the snippets themselves are (re)scored lazily when their package is next analysed (see Closest Match below), or in bulk by cavil snippets --rescore after a corpus change. The prefix tree is simply rebuilt the next time an indexing job notices it is gone. This "rebuild everything on any change" behaviour is simple and reliable, but it is also the part most sensitive to growth (see below).

Closest Match

Alongside the prefix tree, Cavil keeps a separate similarity model that can answer "which existing pattern does this text most resemble?". It scores a piece of text against every known pattern and returns the best matches, which powers the "this snippet looks like pattern X" hints in the review UI. Unlike the prefix tree, this comparison looks at every pattern on every query, so its cost grows directly with the number of patterns.

On its own this raw score is a blunt instrument. It compares text against individual pattern fragments, treats every word as equally important, and does no clean-up first, so the ubiquitous boilerplate that appears in nearly every license ("software", "redistribution", "license") drowns out the handful of phrases that actually tell one license from another. Cavil refines the score in three ways:

  • It normalises the text first — stripping markup, comment markers and copyright, author and URL lines — much as the SPDX license matching guidelines prescribe, so only the legally meaningful wording is compared.
  • It compares a snippet against a license's combined fingerprint — the union of all that license's patterns, rather than one arbitrary fragment. These fingerprints live in the database and are kept up to date as individual patterns are added, edited or removed, so an ordinary pattern change never triggers a wholesale rebuild of the scoring data.
  • It weights rare, license-specific phrases far more heavily than common boilerplate, and measures how much of the snippet is contained in the license (a snippet is usually only a fragment of the full text).

The result — the best-matching license, the closest individual pattern within it, a confidence score, and the runner-up's score — is stored on the snippet and reused for risk estimation and the fold-in described next. The license is chosen from its combined fingerprint, but the pattern is then the closest member of that license, so the risk read from it is correct even when one license's patterns span different risk levels (a "grab-bag" license such as Any CLA). Scoring runs as part of analyze, just before the fold/clear resolution, but only for snippets that lack a current-version score; so a freshly indexed package's snippets are scored the first time it is analysed, and the score is always current when a report is built — never dependent on a separate job's timing. Because snippets persist across reindexing (they are keyed by content), scoping the work by score version also self-heals any rows left at an older version (for example, scored before the similarity model existed). A change to the patterns does not bump the version, so propagating a corpus change to already-scored snippets is the job of cavil snippets --rescore.

Automated resolution: the safety properties that make it defensible

The four resolution paths below (fold, boilerplate-clear, overlap-clear, covered) each automate a legal call, so they hold to a set of invariants any future change to scoring, folding, or clearing must preserve:

  • Never asserts an unrecognised license — but a clear can hide one. A fold only attributes a snippet to a license already in the corpus, and a clear asserts nothing, so nothing here can add a license that is not present. The risk runs the other way: a clear can remove a genuine license from view when the only snippet naming it is mistaken for boilerplate (the covered path below shows the concrete case). The per-path guards bound that, and clearing is fully reversible — cleared regions stay visible in the file browser, nothing is auto-accepted at the risk levels where this matters, and disabling the feature and re-resolving restores every report. Those are calibrated bounds, not proofs, so the more aggressively an instance clears, the more it should also run the review-note control (see Defense in depth).
  • The curated corpus stays the source of truth. Resolutions are a derived, disposable cache; nothing is written back to the pattern or match tables. Turning the feature off and re-resolving reverts the reports.
  • Precision-first, recall-preserving. Only confident, unambiguous, low-risk matches resolve automatically; coin-flips between plausible licenses, high-risk licenses, and snippets that resemble a different license than the one they overlap are all left for human review. A genuine license is never silently removed — including the real match an overlap-clear itself depends on.
  • Auditable and reproducible. Every call is attributable (which license, which pattern, what risk), decided in one place, and stamped with the scorer version, so a report can be reproduced and explained.
  • Reversible, never silent. Derived resolutions are marked as derived, carry a one-click correction, and revert wholesale when disabled — a wrong call costs a correction, never a baked-in mistake.
  • Off by default, calibrated, rolled out gradually. Each path ships disabled; thresholds are calibrated against the existing corpus before, and during, a gradual, risk-tiered rollout.

Folding High-Confidence Snippets into Reports

Hand-writing a pattern for every one of the hundreds of thousands of unresolved snippets will never happen, and the curated pattern corpus has to stay clean — it is the source of truth and cannot be diluted with mediocre auto-generated patterns. So instead of creating a pattern, Cavil can simply treat a snippet as resolved when its refined similarity to a known license is confident enough: the snippet folds into the report as if it had matched that license, while the corpus is left untouched.

The decision is computed once, when a package is analysed, and stored as a resolution on each snippet occurrence — fold, clear, overlap, covered, or unresolved. Every consumer (the report, the file browser, the SPDX export, the Classify Snippets filter) then simply reads that column, so the legal call is made in exactly one place and the same answer is shown everywhere. It never writes anything back to the pattern or match tables; the resolution is a cache of a pure function of the snippet's similarity, the file's existing matches, and the thresholds, recomputed wholesale on every reindex (which recreates the occurrences from scratch) and on demand via cavil snippets --resolve. A model or threshold change therefore takes effect the next time affected packages are reindexed or resolved, and turning the feature off and resolving again makes the reports revert.

Because this is making a legal call, the gate is deliberately conservative. A snippet only folds when the text classifier has already judged it to be legal text, its similarity clears the configured threshold, the best-matching license beats the runner-up by a clear margin (genuine coin-flips between two plausible licenses are left for a human), and the license is not high-risk — risk 4 is safe to fold and 5 is acceptable, but anything riskier is always left unresolved for review. A snippet is also only trusted if it was scored by the current scorer, so stale scores left over from before an upgrade can never fold by accident.

Thresholds are calibrated against the existing corpus before folding is enabled — the eval_fold command reports precision and recall at each — and progressively higher-risk licenses are allowed to fold only as confidence grows.

Clearing License Boilerplate

In practice almost none of the unresolved backlog is a genuinely new license — new licenses appear only a few times a year, while thousands of new snippets arrive every month. The overwhelming majority are awkwardly-formatted sentences from the middle of well-known license texts, in license files, documentation and translations. Cavil already identifies the licenses themselves from their title text, so this leftover body text is essentially noise: whether it is matched or ignored makes no difference to the report.

Folding does not help here, because middle-of-license boilerplate is shared word-for-word between sibling licenses (the GPL and LGPL disclaimers, for instance), so no single license wins by a margin — and guessing one would just be wrong. The right move is not to label such a snippet but to recognise it as known license boilerplate and clear it from the backlog without recording any license. The real license is already on the report from its title, so nothing is lost, and because no license is asserted there is no risk of inventing one that is not actually present.

The recognition reuses the same similarity score: a snippet that is highly similar to some known license, and that the classifier considers legal text, is treated as resolved noise rather than an unresolved match. Genuinely novel licenses score low and so are never cleared — they continue to surface for human review. It is governed by its own threshold.

Clearing Snippets a Real Match Already Covers

There is a second, even more reliable way to recognise backlog noise. When the keyword scan expands a snippet, it often swallows a line that is already a genuine license declaration — typically an SPDX identifier — followed by ordinary code or documentation comments that merely happened to use one of our keywords in a non-legal sense. The real license on that line is already on the report through its own pattern match, so the snippet carries no information the report does not already have: it is recognised as redundant and cleared, without consulting similarity at all. This is precisely the situation reviewers most often write ignore patterns for by hand, so recognising it automatically removes the single most common piece of manual busywork.

Because clearing here rests on an exact match rather than a guess, it is safe regardless of where in the snippet the match falls or how many there are. The one guard is for the rare case where the snippet's own text looks like a different license than the one it overlaps — that might be a genuinely missed license sitting next to a known one, so it is kept for review instead of cleared. Because overlap depends on a file's own matches, this resolution is stored per occurrence (the same snippet can overlap a match in one file but not in another).

Clearing Snippets the File's Own License Already Covers

Overlap-clear only fires when a real license match falls inside the snippet's own lines. But the same redundancy occurs at a wider scope. The keyword scan keeps re-finding awkward license fragments — a stray disclaimer sentence, the tail of an MIT permission notice, a "released under a permissive license" aside — in files whose actual license Cavil has already established from a proper match elsewhere in the file, or in a sibling file in the same directory (a minified .js and its source, a .map next to the code it came from). The fragment scores in the awkward 60–90% similarity band, so folding will never claim it, yet it carries nothing the report does not already know. Recognising this needs context the snippet itself does not have: what else the file, or its directory, is already known to be.

So a snippet is cleared as covered when a concrete license that Cavil is confident about — a real pattern match, at a risk at least as high as anything the fragment could plausibly be — is already present in the same file (or, at directory scope, anywhere in the same directory). The license is already on the report through its own match.

Three guards make this safe, and they are the whole point:

  • Only a concrete license counts as coverage. Cavil's vocabulary includes grab-bag "licenses" — Any Permissive, Any reference local, All Rights Reserved, the version-less GPL-Unspecified family — that record "there is something here" without identifying a real, distributable license. These are flagged catch_all and never count as coverage. This matters: a file whose only match is All Rights Reserved might actually carry a real license (say, the Ruby or JasPer license) that only the unresolved fragment names — clearing that fragment against the weak marker would hide the one piece of real information. So coverage requires a genuine, identifiable license. The overlap-clear path applies the same rule: a snippet that overlaps only grab-bag markers is not cleared, because those markers are not the "genuine license declaration already on the report" that overlap-clear is premised on.
  • Risk-monotonic. A fragment is only covered when the established license is at least as risky as the fragment's own closest license. A snippet that resembles a higher-risk license than anything already on the file is kept — it might be a genuinely new, riskier license (a GPL fragment in an otherwise-MIT file) and must not be swept away.
  • In a license file, a grab-bag closest match needs high similarity to clear. A catch_all grab-bag has no reliable risk (Any CLA spans risks 0–5; the scorer just picks the member the text most resembles), so risk-monotonicity cannot be trusted for it. Inside a LICENSE/COPYING file, then, a grab-bag-closest fragment is cleared only when its similarity is high enough (the cover_guard threshold) to be genuine boilerplate; a weak match is kept. This keeps a custom relicense — a standard BSD/MIT body with a bolted-on non-commercial clause, whose novel text scores only moderately against a grab-bag — from being swept as filler. It is scoped to license files because the same weak match elsewhere (a stray disclaimer, or vendored license-list text in code/docs) is exactly the filler this feature should clear, while genuine boilerplate in a license file still scores high and clears.

Directory scope, rather than whole-vendored-component scope, is deliberate: only a small fraction of packages have detected components, whereas every file has a directory, and measurement showed directory scope recovers essentially all of the same redundancy (the sibling .map/minified cases) while staying universal and simple. The scope is a config switch (off / file / dir) so the safer file-only form can be enabled first and directory scope added once its precision has been checked against the corpus.

Every folded or cleared region is marked as derived (set apart from curated matches) and links to the snippet editor, so a reviewer can overrule a wrong call in one step — write a pattern, ignore the text, or mark it non-legal — which reindexes the package and recomputes the resolution. The file browser is where this happens, since it also shows cleared text that never reaches the report. This is what lets the thresholds be generous: a wrong call costs a correction, not a hand-written pattern for every license up front.

Defense in depth: the review-note layer as an operational control

The resolutions above are bounded by heuristics, not proofs: as the first invariant notes, a clear can still hide a license no pattern names. The mechanical guards plus reversibility keep that residual small on their own, but it grows with how aggressively an instance clears (directory scope, generous thresholds, higher-risk folding). A second, independent layer covers it: an AI review-note assistant (an agent skill) triages the open-review queue and leaves a short advisory note for the human lawyer.

For a deployment that clears aggressively this is a compensating operational control — run and monitored like any other job, not an optional extra (a conservative instance may treat it as optional). It must not become a substitute for the guards, since it is advisory and model-dependent. What makes it a real control is that it reads the package's own LICENSE/COPYING text rather than trusting the report, so it catches terms the scanner never surfaced and the resolution engine cleared away — above all a relicense from open-source to non-free terms, the worst outcome a review can miss. It never accepts, rejects, patterns, or ignores; the note is its only output, and the lawyer still makes the call.

Notes are shared across every review of a package name, which makes them the natural home for context that outlives a single version — but only if the context survives the churn. A note ages out two ways: newer notes push it below the fold, and a note written on a review whose license report has since changed is de-emphasized as not relevant to the report in front of you. Both are right for routine commentary and wrong for a standing decision. Pinning is the reviewer's override for exactly that case: it hoists the note above the chronological list and exempts it from the relevance filter everywhere the note is rendered, on the reviewer's own assertion that it applies to every future version. It is deliberately mutable curation state, unlike the lawyer-only flag, because the notes worth pinning are usually the ones that only turned out to matter later. A per-package limit keeps the pinned block small enough that reviewers still read it.

Suppressing Noise: Ignored Lines and Ignored File Globs

Two mechanisms keep known false positives out of reports, at different granularities:

  • Ignored lines suppress an individual snippet region by content checksum, regardless of which file or package it turns up in. This is the snippet-level "I have seen this before and it is not a license" decision.
  • Ignored file globs suppress whole files by path pattern, system-wide. A glob such as somepkg-*/testdata/*.log excludes captured test fixtures and license-detection reference data that should never have been indexed in the first place — the * in the top-level segment lets one glob cover every version of a package. Globs are evaluated both while indexing (matched files are skipped) and while building a report (matching files are filtered out), so a newly added glob takes effect on a package's next reindex.

Data Model

For readers who want to trace this through the database, the tables involved are:

  • license_patterns — the patterns themselves, including their license name, risk, global-vs-package scope, a uniquely-indexed checksum that prevents duplicates from being stored, and a per-license catch_all flag marking grab-bag/marker pseudo-licenses (so only concrete licenses count as coverage for the covered resolution).
  • pattern_matches — one row per match found in a file, recording the pattern, the file, and the line range.
  • snippets and file_snippets — the extracted keyword regions and where they were found; file_snippets.resolution stores each occurrence's automated decision (fold / clear / overlap / covered, or empty for unresolved).
  • ignored_lines — checksums of snippet regions that experts have chosen to suppress.
  • ignored_files — file path globs that exclude whole files from indexing.

Scaling Characteristics and Limits

Because the prefix tree keeps scanning speed roughly independent of pattern count, Cavil has scaled comfortably to around 29,000 patterns (about 5.7 million words, a roughly 128 MB cache). The limits that matter are therefore about memory and a few count-sensitive side paths, not the core matching loop. Roughly in the order they will be felt:

  1. Memory per worker — no longer the practical ceiling. With the spooky engine every indexing worker loads its own full copy of the pattern tree, so the memory needed grows in step with the total amount of pattern text, multiplied by how many workers run at once. As a rough extrapolation from today's ~128 MB cache (~0.3 GB resident per worker): about 1 GB per worker at 100k patterns, ~3 GB at 250k, and 10–15 GB at 1M — somewhere near one million patterns a fleet of workers stops fitting on ordinary hardware. The default engine removes that multiplication by memory-mapping its index read-only, so all workers on a host share one physical copy and the cost is paid once per machine rather than once per worker.

  2. Rebuilding the caches on every change — felt first. Any pattern edit throws away both caches and triggers a full rebuild, so the cost of those rebuilds rises with the pattern count, and bulk imports can have several workers rebuilding at the same time. This is the first thing likely to become annoying, and it is a software issue rather than a fundamental limit. The default engine is built to fix it: its index format is a set of segments with a manifest, so a new pattern can be compiled into its own small segment and a removed one hidden behind a tombstone, with occasional compaction in the background. Cavil does not use that yet — it still compiles one artifact and rebuilds it whenever a pattern changes — so this limit stands until it does.

  3. Closest-match lookups. These already scan every pattern per query; fine into the hundreds of thousands, then increasingly sluggish in the review UI.

  4. Built-in numeric ceilings — very far off. Internal identifiers for tree nodes and patterns are 32-bit, which only becomes a problem in the range of tens of millions to a couple of billion patterns — roughly thirty times beyond the memory ceiling above, so memory gives out long before.

  5. Pattern shape, not just count. One pathologically long pattern enlarges the scanner's working window for every file, and skip-heavy patterns make the tree branch more. It is worth watching the shape of new patterns, not only how many there are.

In short: the design is comfortable to a few hundred thousand patterns with no real change, and sharing one memory-mapped index between workers has pushed the memory ceiling well out of sight. What is left to hurt first — long before any hard limit — is the rebuild-everything-on-any-change cache behaviour, and the engine already has what is needed to fix it.

AI Text Classification

Text classification via a machine learning model runs as an HTTP service that needs to be configured; the automated snippet resolution depends on it. It usually runs on port 5000.

classifier => {
  url.  => 'http://localhost:5000',
  token => 'API_TOKEN'
}

Cavil schedules this classification itself: whenever a package is analyzed and it brought in snippets that still need classifying, it enqueues the classify background job automatically, coalesced so at most one run is ever pending. No cron job or systemd timer is required. After configuring a classifier for the first time, the next reindex — or the next analysis of any package — enqueues a run that works through the snippets that were indexed before it existed.

The only information submitted is the raw snippet of potential legal text, and a JSON document is returned, containing a boolean value indicating if the model believes this to be legal text, with a percentage value indicating its confidence in the decision.

$ curl -X POST --data '# SPDX-License-Identifier: GPL-2.0-only' http://127.0.0.1:5000
{"license": true, "confidence": 87.8}

The confidence value is presented to users in the UI and meant to help select the best possible candidates for new training data for future versions of the model.