feat: Automated UFM Certificate Rotation Support in NCX Infra Controller (carbide-api)

[hasayesh]

Is this a new feature, an enhancement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

High

Please provide a clear description of problem this feature solves

The NCX Infra Controller site API (carbide-api) supports two authentication
methods for UFM communication: token-based and mTLS. mTLS is the preferred
method and often required by site security policies in production deployments. However, the UFM server certificates are
issued by Vault PKI (forgeca/roles/forge-cluster) and there is currently no
mechanism for automated certificate rotation on the UFM side.

Today, generating and delivering UFM certificates is an entirely manual process:

1. Operator runs carbide-admin-cli credential generate-ufm-cert --fabric=default
2. Carbide-API calls Vault PKI and writes cert files to its own pod's ephemeral
   filesystem (/var/run/secrets/)
3. Operator must kubectl cp the files out, transfer them to the UFM host, and
   restart UFM

These files are ephemeral and lost on pod restart. These files are not accessible by external
systems, and there is no HTTP interface to retrieve them programmatically.

This means any external system that needs fresh certificates from the Forge CA
(such as UFM's) has no way to obtain them.

Feature Description

As a fabric manager integration, carbide-api shall expose HTTP endpoints that
serve Vault PKI-issued certificates on demand, enabling external systems to
programmatically retrieve fresh certificates without manual intervention.

Specifically, add two GET endpoints to carbide-api:

Endpoint	Returns
GET /api/v1/ufm/{fabric}/certs/ca	CA/intermediate certificate (PEM)
GET /api/v1/ufm/{fabric}/certs/server	Server certificate + private key (PEM)

These endpoints:

• Call the existing CertificateProvider::get_certificate() (Vault PKI) —
  the same code path that write_ufm_certs() already uses.
• Return PEM data directly in the HTTP response instead of writing to local disk.
• Authenticated via the existing mTLS layer — no new authentication code needed.
  The caller must present a valid client cert signed by the Forge CA.
• Are mounted at the top-level router (outside /admin) so they bypass the web
  UI session auth, but are still protected by the TLS handshake.

Describe your ideal solution

Provide a supported interface that allows UFM (and similar systems) to programmatically obtain and periodically rotate their mTLS certificates from Vault PKI (Forge CA), eliminating the need for manual kubectl cp operations or host-level file transfer.

Describe any alternatives you have considered

1:Increase Vault max_ttl to reduce rotation frequency
Issues: Does not eliminate manual rotation, just makes it less frequent. Does not scale across many sites
2:K8s CronJob to generate + push certs via SSH/SCP
issues: Requires SSH access from K8s to UFM host. Adds operational complexity and security surface
3:Carbide-API pushes certs to UFM via REST API
issues: Carbide doesn't know when UFM's cert is expiring. Requires storing UFM admin credentials. Overloads existing code path

Additional context

This feature enables automated certificate rotation for UFM, making the
transition from token-based to mTLS authentication practical at scale.
Short-lived certificates (TTL 30 days) with automatic renewal are more secure than
long-lived ones, and this approach eliminates the manual rotation burden
that currently makes adoption impractical. In addition, same cert-serving pattern can be
reused for other fabric manager integrations as they adopt certificate-based
authentication.

Code of Conduct

• I agree to follow NCX Infra Controller's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request

[hasayesh]

@ajf @Matthias247 Please review when you get a chance. Thanks.

Note: while the problem description references Vault PKI and its TTL constraints
(which is the current backend), the implementation uses the CertificateProvider
trait abstraction and does not depend on Vault directly. It will continue to work
unchanged if the PKI backend is replaced as part of the Vault dependency removal
initiative.

[Matthias247]

I guess its debatable if its NICos job or not. In the discussions with network operations teams so far, they preferred NICo not to manage any of the software on the host and to just use well-defined APIs.

And server-side certificate management sounds like "managing UFM software and configurations".
But maybe it could be behind a config file flag.

and periodically rotate their mTLS certificates from Vault PKI (Forge CA),

Consider that we will remove Vault as a NICo dependency (see #195).
NICo would just get its respective client certs via files on disk that get written by secret agents.
With this in mind I would not even know how it could author certs for other services.

[hasayesh]

@Matthias247 @nick-vetro

Regardless of the backend, certificates issued for UFM need to be rotated
before expiry. Today this is entirely manual and does not scale. This feature
provides the mechanism for automated rotation — the backend (Vault, secret
agent, or otherwise) is pluggable via the CertificateProvider trait. The same
pattern is applicable to other fabric manager integrations as they adopt
certificate-based authentication — for instance, NMX-M currently uses
username/password authentication and would benefit from this infrastructure
when it transitions to mTLS.

To clarify the scope: these endpoints are a read-only API that serves PEM
data. NICo does not manage UFM's configuration or software — UFM initiates
the pull via its own built-in cert_auto_refresh and handles installation
itself. This is the "well-defined API" pattern.

Regarding #195 (Vault removal): the implementation codes against the
CertificateProvider trait, not Vault directly. If the post-Vault model uses
secret agents that write certs to disk, the trait implementation could serve
those files over HTTP instead of dynamically generating from Vault. The HTTP
endpoints remain the same — only the backend changes.

If the long-term vision is for an external certificate management system to
issue and deliver certs directly to all services (including UFM), this feature
serves as a practical bridge for the current architecture. It can be deprecated
once that infrastructure is in place.