Support pre-provisioning spaCy models for Auto Classification

[stout-yeoman]

Is your feature request related to a problem? Please describe.

Auto Classification uses spaCy models through Presidio for PII detection. The model is selected from the configured classificationLanguage, for example:

• en → en_core_web_md
• es → es_core_news_md
• de → de_core_news_md
• any → en_core_web_md
• Supported languages without a dedicated mapping → xx_ent_wiki_sm

When the selected model is unavailable locally, ingestion attempts to download and install it during workflow execution.

This makes workflow execution dependent on outbound internet access, proxy and CA configuration, external repository availability, and permission to install packages into the running Python environment. These dependencies are particularly problematic in restricted-egress and air-gapped deployments.

Including every model in the standard ingestion images would avoid runtime downloads but unnecessarily increase image size for deployments that do not use Auto Classification or need only a few languages.

Describe the solution you'd like

Provide a supported, first-class mechanism for provisioning selected Auto Classification models before workflow execution.

For containerized deployments, provisioning during image build should be the recommended production approach. For non-containerized deployments, the same mechanism should run during Python environment setup.

Keep the existing runtime-download fallback in this change. When a required model is missing, ingestion should continue downloading it, but emit an actionable warning recommending pre-provisioning.

This establishes two supported paths:

Path	Behavior
Recommended: pre-provisioning	Install selected models during image build or environment setup; workflows load them locally.
Backward-compatible fallback	Download a missing model during workflow execution and warn that pre-provisioning is preferred.

No trained spaCy models should be provisioned by default in the standard ingestion images.

Reusable provisioning mechanism

Provide a reusable CLI command for provisioning Auto Classification language models, conceptually:

metadata install-classification-models --languages en,es

The final command name can be determined during implementation.

The mechanism should:

• Use the same language-to-model mapping as runtime Auto Classification.
• Accept valid ClassificationLanguage values and reject invalid identifiers.
• Preserve existing mapping semantics, including any selecting the English model and valid languages without a dedicated mapping selecting the multilingual model.
• Install each required model once when multiple language identifiers resolve to the same model.
• Be safe to run repeatedly when the required model versions are already installed.
• Use release-specific model versions compatible with the spaCy version shipped by OpenMetadata.
• Fail clearly if provisioning cannot complete, including installation or compatibility failures.

The initial implementation should select models through the existing language mapping. Arbitrary model names, package specifications, and configurable model variants are outside this issue’s scope.

Image-build integration

Relevant ingestion Dockerfiles should expose an optional build argument, conceptually:

ARG AUTO_CLASSIFICATION_LANGUAGES=""

When nonempty, the image build should invoke the reusable provisioning mechanism for the selected languages. An empty selection should install no models.

Support should cover:

• The Airflow ingestion image.
• The ingestion-base runner image used for Kubernetes and ephemeral ingestion jobs.

Documentation should explain both building the ingestion images with selected languages and extending a published image with a provisioning step. Supplying a build argument alone does not modify an already-published image.

Users running standard published images should retain the current runtime-download behavior without being required to build a custom image.

Keep provisioning and workflow configuration separate

Provisioning determines which models are available in an ingestion environment. classificationLanguage continues to determine which language a particular workflow uses.

For example:

Environment provisioned with: en,es

Pipeline A: classificationLanguage=en
Pipeline B: classificationLanguage=es

Do not introduce a second language-to-model mapping in Dockerfiles or deployment scripts. Provisioning and runtime selection must consume the same canonical registry.

The provisioning mechanism should also be usable in externally managed Airflow, direct Python installations, and customer-managed or Collate-managed ingestion runners.

Runtime behavior and backwards compatibility

This change must preserve the existing runtime-download fallback:

Resolve model for classificationLanguage
                  ↓
          Model available locally?
            ├── Yes → Load model
            └── No  → Warn, download, and load model

Extend the existing download warning with actionable provisioning guidance, for example:

spaCy model 'es_core_news_md' is not installed for classification
language 'es'. Downloading it at runtime.

For production deployments, provision this model during image build
or environment setup using:
metadata install-classification-models --languages es

Automatic runtime downloading is not deprecated or removed by this issue. Any future change to disable or remove it should be proposed separately.

Offline deployment scope

A correctly provisioned environment should load the selected models and perform Auto Classification without outbound access for model retrieval.

This includes building an image in a connected environment and transferring it into an air-gapped deployment.

Provisioning inside a fully air-gapped build environment requires locally available artifacts or an internal mirror. Adding a dedicated offline artifact-distribution mechanism is outside the initial scope.

Acceptance criteria

• A reusable provisioning command is available outside Docker.
• Single and multiple language selections resolve through the same mapping used at runtime.
• Tests cover any, a language using the multilingual fallback, invalid identifiers, and duplicate model selections.
• Model versions are explicitly selected for compatibility with the shipped spaCy version.
• Repeated provisioning preserves already-installed matching versions.
• Provisioning failures return a nonzero exit status with an actionable error.
• Airflow and ingestion-base image builds support optional language selection and provision no models by default.
• Provisioned images can initialize Auto Classification and analyze sample input with networking disabled, using the normal runtime user.
• Missing models retain the existing runtime-download fallback and log guidance recommending pre-provisioning.
• Documentation includes image-build, derived-image, and external Python/Airflow provisioning examples.

Alternatives considered

• Bundle every model: avoids runtime downloads but increases all ingestion images unnecessarily.
• Bundle only English: improves the default path but adds a model to deployments that never use Auto Classification and leaves other languages unresolved.
• Keep runtime downloading as the only path: preserves convenience but retains runtime network and installation dependencies.
• Document manual installation only: remains possible, but lacks a supported interface tied to OpenMetadata’s model mapping and compatible versions.

Additional context

Current model selection and runtime-download logic is primarily located in:

• ingestion/src/metadata/pii/constants.py
• ingestion/src/metadata/pii/algorithms/presidio_utils.py

classificationLanguage remains per-workflow configuration. This proposal adds a supported way to prepare the models available to that workflow before execution.