There is an old ticket for the same feature, support for virtual maps that store all data in-memory without flushes to disk: #15448 That ticket is closed, and the corresponding PR is marked as merged. However, the feature is still not available. There is no way to configure a virtual map to skip flushes.
Here is a different approach how to keep all data in memory. When a map copy is about to be flushed, it checks whether the map is small enough to skip the flush. A new config can be introduced for this purpose. When the map is small, no data is flushed to the data source. However, the flush may not be a completely no-op. Old mutations that would be a part of the flush should be purged from the node cache, otherwise node cache would grow infinitely, and the process would be killed with OOME sooner or later.
Dropping all old mutations completely wouldn't work, as some data would be lost. Assume a key was updated in version X, but not in later versions. In regular mode, when X becomes eligible to flush (usually, as a part of another merged copy), the key is flushed to the data source and written to disk. If a read request is made for the key, it isn't in the cache, so it gets loaded from disk. In in-memory mode, such key should be preserved in the cache. So the algorithm is like this. When a flush is requested, but the virtual map is too small, a special merge is done instead. This merge first removes all overriding mutations (multiple mutations for a single key / path / hash chunk ID) in all cache copies to flush, then merges to the newer cache copy as usual. This is somewhat similar to filtering mutations during flush, but without unnecessary streams and without writing anything to the data source.
There is an old ticket for the same feature, support for virtual maps that store all data in-memory without flushes to disk: #15448 That ticket is closed, and the corresponding PR is marked as merged. However, the feature is still not available. There is no way to configure a virtual map to skip flushes.
Here is a different approach how to keep all data in memory. When a map copy is about to be flushed, it checks whether the map is small enough to skip the flush. A new config can be introduced for this purpose. When the map is small, no data is flushed to the data source. However, the flush may not be a completely no-op. Old mutations that would be a part of the flush should be purged from the node cache, otherwise node cache would grow infinitely, and the process would be killed with OOME sooner or later.
Dropping all old mutations completely wouldn't work, as some data would be lost. Assume a key was updated in version X, but not in later versions. In regular mode, when X becomes eligible to flush (usually, as a part of another merged copy), the key is flushed to the data source and written to disk. If a read request is made for the key, it isn't in the cache, so it gets loaded from disk. In in-memory mode, such key should be preserved in the cache. So the algorithm is like this. When a flush is requested, but the virtual map is too small, a special merge is done instead. This merge first removes all overriding mutations (multiple mutations for a single key / path / hash chunk ID) in all cache copies to flush, then merges to the newer cache copy as usual. This is somewhat similar to filtering mutations during flush, but without unnecessary streams and without writing anything to the data source.