Docker Stack Compose Directories Contain All Files Stored in Git Repository

[Adagis-MC]

Is your feature request related to a problem? Please describe.
AS an Administrator
I WANT to Portainer to only pull down the files necessary from Git to support the Compose Stack
SO THAT only necessary files are within the Compose folder and the full repository is not downloaded for each Compose Stack

Describe the solution you'd like
GIVEN THAT a Compose Stack has been pulled from Git
WHEN I navigate to the Portainer Compose folder
AND I view a Compose Stack subfolder
THEN only the files required by the Compose Stack are stored

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
For each Compose Stack retrieved from Git, the entire repository is retrieved and stored in the Compose folder for that Stack. If you store multiple compose files (and other files, documentation etc) in that repository, these files are also pulled. This results in unnecessary replication of unused data, increasing diskspace and potentially opening security risks for data that should not be generally available.

[jamescarppe]

The issue here becomes how you determine what files are "necessary" in order to deploy your compose stack. If your compose file references a .env file, you'd need to pull that as well. If it contains an image build, it would need to retrieve the Dockerfile, then potentially parse that Dockerfile to see what files that it requires.

[Adagis-MC]

I think that if it needed to retrieve the Dockerfile and parse it, then that would be done when the compose file is run. Pulling an entire repository which might contain multiple files (potentially multiple images) impacts network, disk space and timing. If you are developing, having it pull the entire repository every time you just want it to update a small compose file isn't an intelligent use of resource.

[jamescarppe]

We only pull the repo when the stack is created (ie, the compose file is run), so this would happen at the same time. Are you proposing that we instead pull once to get just the compose file, then once again to pull dependency files?

Bear in mind the docker compose command which we leverage to create the stacks requires that all the necessary files already be available or it will fail. We'd essentially need to write (and maintain) a parser for the compose file format that would detect the need to pull additional files. There's a myriad of ways that additional files could be included, the examples I gave are only two possibilities, so it would be quite a complex implementation.

A better bet might be to instead think about the structure of your remote repositories in terms of what is actually needed for deployments and tune them to suit.

It's worth noting also that we did originally just pull the compose files, but we ran into issues with missing dependencies and the desires of our users for us to pull the whole repo instead, so we switched to pulling the whole thing.

[Adagis-MC]

It's interesting because my use case is primarily for running stacks that pull images from the Docker hub, rather than a private repository. Consequently, all my Compose files are in one branch of one repository, since it doesn't make any sense to create separate repos to hold one YAML file of a few kb.
What then happens is that the entire repo is pulled rather than just the branch containing the specific stack. So I then have ten different copies of the same repository pulled each time when all I want is a small file.

So at the least, it should really only pull the branch that has been requested, not the entire repo. At the most, it could allow a user to select which files from a branch to pull (or parse the compose file to identify which files are needed).

The difficulty is of course that everyone uses repositories in different ways, so there is no one size fits all solution to this!

[adolfousier]

Hey Alexey, good find on the hardcoded taxonomy. To clarify the architecture:

The 7 standard brain files (SOUL, TOOLS, USER, MEMORY, AGENTS, CODE, SECURITY) are the agent's actual brain. They're the only targets for RSI writes. This is intentional — these files hold identity, rules, knowledge, and tool docs. The taxonomy is hardcoded by design.

Custom .md files (VOICE.md, AGENTVERSE.md, etc.) are reference material loaded on demand via load_brain_file, not brain files. The RSI agent can be aware of them for context, but they're not write targets. They're user-curated.

So the split is:

• Brain files (7 defaults) = write targets for RSI, always in the taxonomy
• Custom .md files = inflight fetch, read-only context, not in the taxonomy

The root cause of TOOLS.md bloat was already addressed — the RSI opportunity generator no longer directs tool failure data to TOOLS.md, it points to feedback_record / feedback_analyze instead. The taxonomy staying fixed to 7 files is correct, not a gap.

That said, the token usage angle from your earlier comment is worth exploring separately. How and when the 7 brain files get injected is a separate concern from bloat.

[adolfousier]

Your section-level loading idea is interesting and we may explore it for token optimization later. But here's the broader vision for how I see the brain file system:

TOOLS.md needs a revamp, but not splitting or section-level loading. It needs to shrink back to what it's supposed to be: a concise index that tells the agent what, when, and how to call. Not a 660-line reference manual.

Example: CODE.md isn't loaded by default. But TOOLS.md has a one-liner that tells the agent "for coding standards, load_brain_file("CODE.md")." Same with skills: TOOLS.md says "use /skills to browse loaded skills." The agent doesn't need the full reference in context, it just needs to know it exists and how to fetch it.

Same way a brain doesn't store every detail, it knows where to go to find it. You add more skills, more .md files, more reference docs, but TOOLS.md just gets a one-liner pointing to each. Concise routing.

For the per-tool TOML idea: I'm considering a multi-choice widget that lets the user select which tools they want enabled. If you never use Trello, disable those tools entirely. Less noise in the tool list = better model selection. That's a separate feature from the brain file architecture though.

The architecture is:

• 6 brain files (SOUL, TOOLS, USER, MEMORY, AGENTS, SECURITY) = the agent's persistent brain, always injected
• Custom .md files = reference material, loaded on demand via load_brain_file
• TOOLS.md = concise router/index, not a storage file. One-liners pointing to where the detailed docs live

The bloat problem is already fixed (RSI no longer appends failure notes to TOOLS.md). The token optimization angle is real and worth exploring, but the solution is keeping TOOLS.md lean as an index, not splitting it into more files.

Open to discuss improving as always.