Skip to content

Thumbnail purge operation consumes large amount of RAM #648

Description

@syldrathecat

I am aware this project is not being actively worked on. This is information to help those who need it. I am not a Go developer.

I'm not 100% sure of the details here, but the effect is that after an hour, the MMR process increased by 2GB in size, for us. This compounds with the fact that MMR seems to leak memory and grow in size endlessly.

The thumbnail purge task seems to read a massive number of rows from the thumbnails table in to memory that it doesn't end up purging. I think this is something to do with how the thumbnails also exist in media table, but I'm not exactly sure what the existence of media table rows with no user_id is supposed to indicate.

I think the logic is that it wants to avoid deleting files from disk if a thumbnail sha256 hash matches an existing media upload's sha256 hash? Either way, its either not done correctly or is written in a way that causes thumbnail purging to become very inefficient as the database grows.


Anyway, I was not able to figure out this logic to determine a way to fix the issue, but a workaround is to run matrix-media-repo with a non-zero machine ID environment variables, i.e. MACHINE_ID=1 ./media-repo

or in systemd:

Environment="MACHINE_ID=1"
WorkingDirectory=/opt/mmr/
ExecStart=/opt/mmr/bin/media_repo

This causes the process to no longer be responsible for running hourly periodic cleanup tasks, meaning it can go far longer between scripted reboots.

You could then run a separate instance of matrix-media-repo with MACHINE_ID=0, but not have it handle any user requests, so that the process can be restarted more regularly, without interrupting client connections.


Another option is to experiment with removing specific problematic scheduled tasks in code:

--- a/tasks/all.go
+++ b/tasks/all.go
@@ -8,7 +8,7 @@ func StartAll() {
        executeEnable()
 
        scheduleHourly(RecurringTaskPurgeRemoteMedia, task_runner.PurgeRemoteMedia)
-       scheduleHourly(RecurringTaskPurgeThumbnails, task_runner.PurgeThumbnails)
+       //scheduleHourly(RecurringTaskPurgeThumbnails, task_runner.PurgeThumbnails)
        scheduleHourly(RecurringTaskPurgePreviews, task_runner.PurgePreviews)
        scheduleHourly(RecurringTaskPurgeHeldMediaIds, task_runner.PurgeHeldMediaIds)
 

This would allow for remote media purging to continue while removing the thumbnail purging.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions