Runs out of memory during syncing and crashes

[HalFinneyIsMyHomeBoy]

Trying to sync on mainnet for a few days now, but it runs out of memory and the process kills itself.
Syncing from block 709632. Server has 64 GB of RAM.

Is more RAM needed or is this a bug?

[tsjk]

Did you manage to work around this issue?

[HalFinneyIsMyHomeBoy]

No, never got around this issue. Gave up on it a few months back.
Has there been any fix since then?

[tsjk]

No, never got around this issue. Gave up on it a few months back. Has there been any fix since then?

No, there hasn't. But I experienced the same issue. I think the problem is that computing the scalars goes much faster than updating the unspents, and so
https://github.com/louisinger/silentiumd/blob/master/internal/application/syncer_service.go#L217-L219
becomes a very long memory consuming queue. I don't know any go, really - just looked at the code quickly. The code base is small.
One can change this to become a blocking call by just

--- a/internal/application/syncer_service.go
+++ b/internal/application/syncer_service.go
@@ -214,11 +214,8 @@ func (s *syncer) computeBlockScalars(block *btcutil.Block) {
                logrus.Error(err)
        }
 
-       go func() {
-               s.updateUnspentsCh <- block
-       }()
-
        logrus.Debugf("[%d] compute scalars done", block.Height())
+       s.updateUnspentsCh <- block
 }
 
 func (s *syncer) updateUnspents(block *btcutil.Block) {

and this makes it not eat all your memory. But syncing is very slow.
I know there is at least one other implementation, also in go. Maybe worth to check out - depending on what you want to do.

[tsjk]

Perhaps there is a middle-ground somewhere, like limiting the queue size, but as I don't know golang I have no concrete ideas.

[setavenger]

I know there is at least one other implementation, also in go. Maybe worth to check out - depending on what you want to do.

I assume you are referencing my implementation with the BlindBit suite. https://github.com/setavenger/blindbit-oracle
I'm not aware of any memory issues with Oracle, so give it a try if you want to.

[tsjk]

I know there is at least one other implementation, also in go. Maybe worth to check out - depending on what you want to do.

I assume you are referencing my implementation with the BlindBit suite. https://github.com/setavenger/blindbit-oracle I'm not aware of any memory issues with Oracle, so give it a try if you want to.

Indeed I was. Will look at that more closely as well.