I am aware this project is not being actively worked on. This is information to help those who need it. I am not a Go developer.
I'm not 100% sure of the details here, but the effect is that after an hour, the MMR process increased by 2GB in size, for us. This compounds with the fact that MMR seems to leak memory and grow in size endlessly.
The thumbnail purge task seems to read a massive number of rows from the thumbnails table in to memory that it doesn't end up purging. I think this is something to do with how the thumbnails also exist in media table, but I'm not exactly sure what the existence of media table rows with no user_id is supposed to indicate.
I think the logic is that it wants to avoid deleting files from disk if a thumbnail sha256 hash matches an existing media upload's sha256 hash? Either way, its either not done correctly or is written in a way that causes thumbnail purging to become very inefficient as the database grows.
Anyway, I was not able to figure out this logic to determine a way to fix the issue, but a workaround is to run matrix-media-repo with a non-zero machine ID environment variables, i.e. MACHINE_ID=1 ./media-repo
or in systemd:
Environment="MACHINE_ID=1"
WorkingDirectory=/opt/mmr/
ExecStart=/opt/mmr/bin/media_repo
This causes the process to no longer be responsible for running hourly periodic cleanup tasks, meaning it can go far longer between scripted reboots.
You could then run a separate instance of matrix-media-repo with MACHINE_ID=0, but not have it handle any user requests, so that the process can be restarted more regularly, without interrupting client connections.
Another option is to experiment with removing specific problematic scheduled tasks in code:
--- a/tasks/all.go
+++ b/tasks/all.go
@@ -8,7 +8,7 @@ func StartAll() {
executeEnable()
scheduleHourly(RecurringTaskPurgeRemoteMedia, task_runner.PurgeRemoteMedia)
- scheduleHourly(RecurringTaskPurgeThumbnails, task_runner.PurgeThumbnails)
+ //scheduleHourly(RecurringTaskPurgeThumbnails, task_runner.PurgeThumbnails)
scheduleHourly(RecurringTaskPurgePreviews, task_runner.PurgePreviews)
scheduleHourly(RecurringTaskPurgeHeldMediaIds, task_runner.PurgeHeldMediaIds)
This would allow for remote media purging to continue while removing the thumbnail purging.
I am aware this project is not being actively worked on. This is information to help those who need it. I am not a Go developer.
I'm not 100% sure of the details here, but the effect is that after an hour, the MMR process increased by 2GB in size, for us. This compounds with the fact that MMR seems to leak memory and grow in size endlessly.
The thumbnail purge task seems to read a massive number of rows from the thumbnails table in to memory that it doesn't end up purging. I think this is something to do with how the thumbnails also exist in media table, but I'm not exactly sure what the existence of media table rows with no user_id is supposed to indicate.
I think the logic is that it wants to avoid deleting files from disk if a thumbnail sha256 hash matches an existing media upload's sha256 hash? Either way, its either not done correctly or is written in a way that causes thumbnail purging to become very inefficient as the database grows.
Anyway, I was not able to figure out this logic to determine a way to fix the issue, but a workaround is to run matrix-media-repo with a non-zero machine ID environment variables, i.e.
MACHINE_ID=1 ./media-repoor in systemd:
This causes the process to no longer be responsible for running hourly periodic cleanup tasks, meaning it can go far longer between scripted reboots.
You could then run a separate instance of matrix-media-repo with
MACHINE_ID=0, but not have it handle any user requests, so that the process can be restarted more regularly, without interrupting client connections.Another option is to experiment with removing specific problematic scheduled tasks in code:
This would allow for remote media purging to continue while removing the thumbnail purging.