[Question] How can I centralise the blob info cache across multiple workers to enable ECR layer mounting?

[heliobmartins]

Hi folks,

I'm looking for advice on how to better leverage caching for the BlobMounting feature, specifically with AWS ECR, and would appreciate any insights.

Context

We replicate artifacts (OCI images) to AWS ECR using the podman/skopeo SDK. To benefit from the new ECR cross-repository layer sharing feature, we need BlobInfoCacheDir configured — otherwise the blob info cache lives only in memory.

Why the in-memory cache hurts us

No blob mounting. For each copy operation, Skopeo can't tell whether it has seen an artifact before, so it can't mount existing layers — copy times don't improve.
No cost reduction. Every operation is a fresh copy rather than a mount, so we don't get the associated savings.

The core problem

We run multiple worker pods that all perform replication, and each pod has its own local BlobInfoCacheDir. As a result:

The cache is not shared. A layer already replicated by one pod isn't known to the others, so shared/previously-replicated layers get re-copied unnecessarily.

What I'm looking for

I'd like to centralise the blob info cache (currently the SQLite DB under BlobInfoCacheDir) so all pods can share it. Has anyone tackled this? I'm considering options like a shared store (Redis) or a proper database, but I'm keen to hear recommended best practices before committing to an approach.

Any advice is greatly appreciated

Thanks in advance for your help,

Helio

[mtrmac]

Thanks for reaching out.

I think the principal avenue I’d recommend is to move towards a more deterministic distribution design; the cache is not designed to provide guaranteed matches (e.g. any software update can cause pre-existing entries to be ignored or deleted; we already had one complete change of the file format of the default database). E.g. if the same image should exist in multiple repos, explicitly copy it. (That, admittedly, runs into podman-container-tools/container-libs#274, but that could be solved.)

In particular, I think it should be generally unnecessary to rely on the non-exact matching (uncompressed vs. compressed layer, differently-compressed layers), apart from the very first push of a built image. That does still leave cross-repo mounts of exact duplicates.

---

As for sharing … the cache is designed to be concurrently accessed by multiple processes, so sharing it on a volume accessed from separate containers within a single kernel should probably work. Sharing it on a network filesystem accessed from multiple machines … probably not, IIRC the locks don’t propagate across NFS, and https://sqlite.org/lockingv3.html says the same thing.

I can imagine a network-aware database client accessing the blob info cache for a truly cross-machine information sharing, but it sounds like way too much complexity and exposing internal details we’d like to keep hidden, for something that is supposed to be a low-cost opportunistic optimization opportunity. E.g. the useful BlobInfoCache API is already internal-only, and we plan fairly significant design changes to it in near future.

---

It’s quite possible that dedicated mirroring software like oc-mirror has designs that directly target and optimize blob reuse within destination registries, but I don’t know whether that’s actually the case.

[heliobmartins]

Hello @mtrmac ,

First of all, thank you so much for your prompt response and assistance, I truly appreciate it!

When it comes to the sharing, it makes plenty of sense and I agree that sharing a common volume where different nodes will be writing to it may lead to a locking problem, such as an EFS system... we definitely don't want to do it.

I'm not familiar with oc-mirror, but definitely something I would be happy to look at.

---

When it comes to BlobInfoCache we indeed noticed that's internal only, but one of our thoughts was that if that was exposed through SystemContext, we could perhaps consider injecting our own implementation of storage for those blobs, for example, in a ElastiCache? But at the same time, after doing some readings, we have noticed that's also not recommended to use, since it is "deprecated".

---

Do you mind elaborating on what do you mean with "I think the principal avenue I’d recommend is to move towards a more deterministic distribution design;"? I agree 100% that the cache it is not meant to provide guarantee matches, but if we are able to maximise the effectiveness of this "cache", especially on distributed systems, like ours it would be great.

As far as I understand, the cache works as a best-effort, so if it doesn't exist or even if the file is corrupted, it fallback to memory replication, which is absolutely fine.

---

If I could share, maybe what I'm trying to do is to maximise as much as I can the cache ratio to be able to benefit from Blob Mounting Feature of ECR where we can have cost reduction and more importantly swift distributions using "copy". At the same time, I'm also trying to reach out to ECR team on why they have designed that in a way that put a dependency on a client to make us benefit from this feature.

Thank you once again for your help,
Helio

[mtrmac]

Do you mind elaborating on what do you mean with "I think the principal avenue I’d recommend is to move towards a more deterministic distribution design;"?

I obviously don’t know your codebase, but, perhaps, if the goal is to optimize duplicate pushes of “the same” layer from some source / set of sources to the same destination registry, don’t trigger those pushes in the first place?

For newly locally-built images, instead of podman push to 3 different repos, conceptually triggering 3 compressions operations and relying on BIC to turn that into one compression + mount: push to one repo on the destination registry and then do intra-registry copies for mounts into different repositories.

For large-scale mirroring, instead of triggering an independent job per image or per destination, and having unrelated workers attempt copies of the same layer and relying on BIC to use mounts: collect the layer digests of all sources, and having jobs based on layer digests, truly copying each layer only once, and afterwards triggering mounts within the destination.

[heliobmartins]

I can provide a high-level overview of what we have.

We have a system that distributes artifacts to ECRs for every deployment. Skopeo's role is to place these artifacts in the correct ECR location for each deployment it happens across our organisation. We cannot control what deployments send because hundreds of teams manage them.

Many replicated artifacts share layers due to golden images, and some images are duplicated. For example, if five workers each receive a request to replicate the same image, the worst case is each replication goes to a different pod, so we gain no benefit from the Blob Mount feature. Only on the sixth replication would the BlobInfoCacheDir be populated across all those pods.

We run this on K8s using a deployment configuration. Moving to a statefulset and optimising volumes may help but won’t solve the core issue, so that’s the reason why I wanted to explore alternative options.

---

Everything you say makes sense. For this scenario, it is unfortunate that ECR relies on the client side to benefit from those features. The backend should handle this, so we are contacting them.

I’m not sure if my context adds value. From what you say, there may be nothing we can change in Skopeo to improve distribution efficiency with this client.

Again, I appreciate your help on this matter.

[mtrmac]

Thanks … that makes sense, and it is tough to support.

One alternative avenue to explore might be the registry-side: repos are a registry-side concept and the registry might know that it already has the blob in another repo. (But letting the client know that and use is a more difficult part … just because a client knows a digest doesn’t automatically mean the client should be provided the contents.) I imagine that might also require a significant registry architecture change or run into scaling issues.

---

Skopeo as a CLI program is very unlikely to get plugin support for BIC. At the c/image library level (we’ve been talking about SystemContext), I can imagine some sort of parametrization. The hard part would be designing a stable API we can commit to — this code will very soon be changing and we don’t know how, yet.

Maybe it might work to expose a restricted version suitable only for exact matches (with RecordKnownLocation + CandidateLocationsExact, as a database of (transport, scope, digest) → location. That would need prototyping to confirm that the code can trigger mounts with such a restricted BlobInfoCache, but might be useful enough and sidestep most of the semantics issues we will be grappling with.

[heliobmartins]

One alternative avenue to explore might be the registry-side: repos are a registry-side concept and the registry might know that it already has the blob in another repo.

That's exactly what our thoughts are and the reason why we are trying to reach out to ECR team on this matter.

---

Yes, I understand that it might not be easy and some brainstorming / validation it would be required and unfortunately, I don't have all the knowledge on the intrinsics on the container BIC to contribute on that. But if you ever need assistance with validating the idea, or to test potential candidate feature improvements in a way that we can manage the database, I would certainly be keen on helping out.

---

I think from what I can tell, there is nothing that you guys can help right now and currently no planned work to allow users managing their own "Database" for BlobInfoCache, so, I reckon this issue can be closed?

But I truly appreciate all your assistance and patience in this matter @mtrmac.

Thank you!

[mtrmac]

I think a possible next step would be to prototype adding a Go interface for “a restricted version suitable only for exact matches” to copy.Options, and internally wrapping that in a BlobInfoCache2 that does nothing else. If that allows making copies correctly at all, and works usefully for you (notably giving up on reuse of differently-compressed layers), I’d be open to adding such an option — but it’s not an experiment I can be making personally within the next year or so.

[heliobmartins]

No worries at all! I completely understand the time constraint, I guess everyone are busy these days. 😀

So, just to understand your proposal... Your idea is to implement a new interface BlobInfoCache2 where within the copy.Options this can be received as parameter, where users (like) us can implement the methods of the interfaceBlobInfoCache2 referencing our own storage type and with the copy.Options we inject our own implementation?

Also, do you mind elaborating on what “a restricted version suitable only for exact matches” I'm not fully sure I understood that.

[mtrmac]

Yes, basically (except that BlobInfoCache2 exists privately, and will likely change. This would be a new ExactMatchBlobInfoCache or something like that; the prototype would implement the private BlobInfoCache2 by using the provided ExactMatchBlobInfoCache).

---

Also, do you mind elaborating on what “a restricted version suitable only for exact matches” I'm not fully sure I understood that.

Maybe it might work to expose a restricted version suitable only for exact matches (with RecordKnownLocation + CandidateLocationsExact, as a database of (transport, scope, digest) → location.

[mtrmac]

… to expand on “the restricted version”, the BlobInfoCache2 implementation can also handle non-exact matches (e.g. using a zstd-compressed version that already exists on a registry when the source is gzip-compressed if the options allow such a substitution), and a lot of the matching/prioritization logic is currently placed inside the BlobInfoCache2 implementation, it’s not a “raw database”. I don’t think we can commit to making that full interface stable, at the very least until SHA-512 support lands. But exact matches are a fairly narrow problem space where a public API is probably supportable.