proposal: add model sync adapter framework

[chlins]

This pull request introduces a comprehensive proposal for a new Model Sync Adapter Framework in Harbor, aimed at enabling seamless synchronization of AI models from upstream model hubs (starting with Hugging Face) into Harbor as OCI artifacts. The framework is designed to be extensible, policy-driven, and deterministic, while reusing Harbor's proven architectural patterns without modifying the existing OCI replication system.

The most important changes are:

Framework Design and Architecture

• Proposes a dedicated Model Sync Adapter Framework that runs parallel to Harbor's replication system, introducing a new adapter contract tailored for model hubs with a fetch → convert → push pipeline.
• Details a modular architecture: adapters per hub, system-level sync policies, deterministic packaging, and reuse of Harbor's registry management and execution/task management.

Adapter Contract and Extensibility

• Defines a narrow, hub-agnostic Model Adapter contract (resolve + stream) to make adding new model sources straightforward and maintainable, with Hugging Face as the reference implementation.

Deterministic Packaging and Provenance

• Describes a deterministic packaging process that produces bit-identical OCI artifacts for the same upstream revision and file filters, ensuring idempotence, deduplication, and reliable provenance tracking via tags and annotations.

API, Data Model, and UI Integration

[elliott-davis]

Something I don't see called out in this proposal is secrets management. Some models require the acceptance of a EULA and today their access is gated behind a users token that has accepted the EULA. Is there any effort to support these models in this proposal?

[elliott-davis]

Will this proposal make any attempt to differentiate model artifacts from container artifacts in a repository? A confused developer may try and pull one of these converted models expecting an engine to be packaged with it like a NVIDIA NIM and will experience a failure.

[elliott-davis]

Overall this proposal aligns with what I would like to accomplish for model syncing in Harbor.

[chlins]

Something I don't see called out in this proposal is secrets management. Some models require the acceptance of a EULA and today their access is gated behind a users token that has accepted the EULA. Is there any effort to support these models in this proposal?

@elliott-davis Good question. Gated models are covered by the existing token mechanism rather than a new secrets subsystem: the user completes the EULA or license acceptance on the hub side with their own account beforehand, and the access token configured in Harbor's registry management (encrypted at rest) carries that entitlement. From Harbor's perspective this is plain token authentication when fetching; it implements no hub-specific consent flow. I've updated the Security Considerations section to make this explicit.

[chlins]

Will this proposal make any attempt to differentiate model artifacts from container artifacts in a repository? A confused developer may try and pull one of these converted models expecting an engine to be packaged with it like a NVIDIA NIM and will experience a failure.

@elliott-davis Yes. Converted models are model-spec OCI artifacts with their own artifactType and media types, so Harbor's UI can distinguish them from container images and will render them with a distinct icon, the same way charts, SBOMs, and signatures are differentiated today. On the pull path, however, both go through the standard Distribution v2 API, and the server should not intercept or reject pulls based on artifact type. So the safeguard is discoverability (distinct type and icon in the UI, plus the model-spec media types for tooling to inspect), not server-side gating; whether an artifact is runnable is up to the consuming tooling.