[Bug]: CouchDB 3.5.1 upgrade leads to elevated memory and instability (OOM on single node + full cluster drop during rolling upgrade)

[oisheeaa]

Version

3.5.1

Describe the problem you're encountering

After upgrading from CouchDB 3.4.2 -> 3.5.1, we are seeing stability regressions in both:

• Single node environments (OOM kill crash)
• 3 node clustered environments (full cluster outage during rolling upgrade)
  These environments were stable on 3.4.2 under same workloads
  The main symptoms include:
• Increased Erlang VM RSS memory usage (beam.smp)
• Very large Erlang process counts post-upgrade
• CPU spikes during compaction/rebalancing
• Cluster becoming unavailable during node cycling

Expected Behaviour

• CouchDB 3.5.1 should not significantly increase baseline memory/process usage on the same workload
• Single node instances should not be OOM killed under normal compaction load
• Rolling node upgrades should not result in full cluster outage

Steps to Reproduce

We have observed the issues under the following conditions:

• Upgrade CouchDB 3.4.2 → 3.5.1 (Erlang/OTP 26)
• Run with medium memory nodes (~8GB RAM)
• Large shard count / very high DB count
• Compaction or shard movement occurring
• Cycle one node during rolling upgrade (ASG refresh)

Result: memory/process growth and potential node loss -> cluster unavailability

Your Environment

Deployment

• AWS EC2 instances with persistent couch_data
• CouchDB nodes managed via ASG refresh (one node at a time)
• HAProxy + Fauxton access through fronted endpoint

Cluster configuration:

• 3-node cluster (n=3, q=1)
• Placement: primary / secondary / trinary
• Data size: ~319k databases
• Disk usage: ~640GB used of ~1TB per node

Instance resources:
~7.6GB RAM

Additional Context

No response

[nickva]

Do you see any crashes in the logs during that time?

Check the output of /_node/_local/_system for each node (also accessible as _node/$nodename/_system).
In that response what does "memory" look like?. Any elevated counts in "message_queues"?

It could be related to OTP 27. We are using it in production for quite a while and haven't observed anything unusual.

Are there any particular requests that are slow during that time. Say, document updates are slower, view reads, changes?

[nickva]

Looking at a few older issues related to high resource usage may help:

#4926 (comment)

#5585 (comment)

[oisheeaa]

Do you see any crashes in the logs during that time?

Check the output of /_node/_local/_system for each node (also accessible as _node/$nodename/_system). In that response what does "memory" look like?. Any elevated counts in "message_queues"?

It could be related to OTP 27. We are using it in production for quite a while and haven't observed anything unusual.

Are there any particular requests that are slow during that time. Say, document updates are slower, view reads, changes?

We do have crash evidence and it started the week we upgraded to 3.5.1

1. Crashes during the incident windows
   Staging2 (single node):
   CouchDB was down ~10 minutes and systemd shows it was OOM killed:
   journalctl:
   couchdb.service: A process of this unit has been killed by the OOM killer
Failed with result 'oom-kill'
   This did not happen on the previous CouchDB version, it started in the same week we moved staging2 to 3.5.1.

Preprod-east & west (3-node cluster)
During the 3.5.1 rolling upgrade attempt, the cluster fully dropped (both primary + secondary target groups went unhealthy). We hit the same os_mon failure pattern we’ve seen before when memory pressure spikes:

• When primary went down / cluster instability on secondary:
  - mem3_distribution : node couchdb@primary down, reason: net_tick_timeout
  - rexi_server_mon : cluster unstable -> later cluster stable
  - gen_server memsup terminated with reason: {port_died,normal}
  - gen_server disksup terminated with reason: {port_died,normal}
  Then after primary refresh we saw on trinary:
  - gen_server disksup terminated with reason: {port_died,normal}
  Operationally, the pattern was:
• scale down primary first (stable)
• take down secondary -> primary memory spikes -> target groups unhealthy / lost connectivity
• scale secondary back up -> could SSM, but ALB still unhealthy
• scale down trinary -> secondary went unhealthy briefly, then recovered
• primary got refreshed again -> secondary dropped again with memory spike / unreachable
  So we’re seeing memory/process pressure symptoms across both single node and cluster setups on 3.5.1.

2. /_node/*/_system + queues / process counts (what we have already)
   From preprod today (same 3.5.1 stack), on node:

• process_count: 121,753
• process_limit: 262,144
• and beam.smp RSS was ~3.1 GB on an 8 GB instance
• beam.smp RSS ~ 3106652 KB

3. We don’t have a single one endpoint always slow captured, what we do have is compaction pressure showing up during the same periods:

• /_active_tasks showed database_compaction on _global_changes (document_copy, large total_changes)

[janl]

The only substantial difference between the version numbers is the parallel preads feature.

But the only way they can have an effect is that the cluster before the upgrade was constrained by the single couch_file queue. With the bottleneck gone, maybe the load on the cluster can overload it.

It is a bit of a stretch and I would have imagined we’ve seen more reports on this already, but it might be worth ruling out.

[nickva]

Thanks for responding @oisheeaa.

From the logs evidence, it doesn't seems like there is any specific error in the logs, for example from a process repeatedly crashing? The oom logs and mem3_distribution : node couchdb@primary down, reason: net_tick_timeout come from nodes disconnecting or being killed by the OOM.

During the 3.5.1 rolling upgrade attempt, the cluster fully dropped (both primary + secondary target groups went unhealthy).

During the upgrade before taking the node down do you put the node in maintenance mode? That's what we usually do - use maintenance mode on the node, then wait some time (a few minutes) for all the connections to drain before upgrading (load balancer is automatically setup to exclude the node from routing based on the output of the /_up endpoint).

/_node/*/_system + queues / process counts

When it gets close too OOM-ing what does the memory breakdown look like (how many bytes used by processes, binary, etc)?

Wonder if you could share some of your vm.args setting or config settings? Especially if you have any custom settings (changing scheduling, or busy waiting, any allocator settings). For config setting any custom limits, what's your max_dbs_open set to?

In general, it's hard to tell what may be using the memory. A few way to try to control the memory might be to lower max_dbs_open a bit (but if you lower it too much you might see all_dbs_active errors in the logs).

I had never tried these settings but if nothing else works maybe it's something to experiment with (especially if you have a test/staging environment)

https://elixirforum.com/t/elixir-erlang-docker-containers-ram-usage-on-different-oss-kernels/57251/12

[dianabarsan]

Just a shot in the dark here: you don't have debug mode logging enabled, right?
#5474

[oisheeaa]

Thanks for responding @oisheeaa.

From the logs evidence, it doesn't seems like there is any specific error in the logs, for example from a process repeatedly crashing? The oom logs and mem3_distribution : node couchdb@primary down, reason: net_tick_timeout come from nodes disconnecting or being killed by the OOM.

During the 3.5.1 rolling upgrade attempt, the cluster fully dropped (both primary + secondary target groups went unhealthy).

During the upgrade before taking the node down do you put the node in maintenance mode? That's what we usually do - use maintenance mode on the node, then wait some time (a few minutes) for all the connections to drain before upgrading (load balancer is automatically setup to exclude the node from routing based on the output of the /_up endpoint).

/_node/*/_system + queues / process counts

When it gets close too OOM-ing what does the memory breakdown look like (how many bytes used by processes, binary, etc)?

Wonder if you could share some of your vm.args setting or config settings? Especially if you have any custom settings (changing scheduling, or busy waiting, any allocator settings). For config setting any custom limits, what's your max_dbs_open set to?

In general, it's hard to tell what may be using the memory. A few way to try to control the memory might be to lower max_dbs_open a bit (but if you lower it too much you might see all_dbs_active errors in the logs).

I had never tried these settings but if nothing else works maybe it's something to experiment with (especially if you have a test/staging environment)

https://elixirforum.com/t/elixir-erlang-docker-containers-ram-usage-on-different-oss-kernels/57251/12

Thanks for your response and to add more context from our side:

• there’s no single process repeatedly crashing in the logs. What we consistently see across staging2, preprod, and a separate DevOps test cluster is:
  • steady memory growth
  • eventual OOM kill by OS
  • followed by node disconnects
• During the 3.5.1 rolling upgrade, nodes were drained from the load balancer before replacement (AWS target group health checks + deregistration). We did not explicitly put the node into CouchDB maintenance mode first
• Stortly before OOM:
  - ~3 GB on 8GB instances
  - overall memory usage ~70% and then climbed under load
  - _node/_local/_system showed elevated process_count (100k+), with process_limit at 262144
• We are running with a largely stock vm.args, with a small number of common production tweaks. No custom allocator settings and no scheduler busy wait changes

-name couchdb@<node-fqdn>
-setcookie <shared-cookie>
-kernel error_logger silent
-sasl sasl_error_logger false
-kernel prevent_overlapping_partitions false
+SDio 16
+zdbbl 32768
+Bd -noinput
-ssl session_lifetime 300

• Additional reproduction under controlled load
  In addition to staging2 and preprod, we were able to reproduce the same behaviour in a separate 3 node DevOps test cluster running CouchDB 3.5.1 on smaller instances (t3.small).
  Under sustained concurrent workload involving:
  • large numbers of database creates / deletes
  • concurrent document writes
  • high worker counts
    both primary and secondary nodes were OOM killed once memory usage crossed ~80%, while the trinary node remained healthy. this failure mode was consistent with what we observed in staging2 and preprod
    For comparison, our production cluster (still on 3.4.2) runs with:
  • significantly fewer open databases during non-peak (~<1k vs ~8k+ on 3.5.1 test clusters)
  • the same max_dbs_open = 60000
  • similar ulimit settings
    and has not exhibited this behaviour

[nickva]

Thank you @oisheeaa for your patience and for the additional info. I think what's causing the increased memory usage is the max_dbs_open = 60000 value. That's entirely too high for an 8GB instance. Even on 128GB instances we keep it at our default of 5000 and only on some clusters raise to 15000 or 20000.

In 3.4.2 there was an additional close_on_idle mechanism to periodically close idle db handles (see https://github.com/apache/couchdb/blob/3.4.2/src/couch/src/couch_db_updater.erl#L235 for details), however that mechanism had race conditions and a few other issues with it, so it was removed.

The way the db handles cache works is once the handle is opened, it will stay opened until used again. If at some point, if there is more room in the cache, the oldest unused handle is closed and replaced with the new one. So size the max_dbs_open accordingly to how much memory you can allocate to it. For an 8GB maybe start with 1000-1500 and see if you get any all_dbs_active errors. (Try on a staging cluster first if you can generate similar load as the production cluster). There is a balance to how much memory should be used for the db handle cache vs having more for the page cache.

[oisheeaa]

Thank you @oisheeaa for your patience and for the additional info. I think what's causing the increased memory usage is the max_dbs_open = 60000 value. That's entirely too high for an 8GB instance. Even on 128GB instances we keep it at our default of 5000 and only on some clusters raise to 15000 or 20000.

In 3.4.2 there was an additional close_on_idle mechanism to periodically close idle db handles (see https://github.com/apache/couchdb/blob/3.4.2/src/couch/src/couch_db_updater.erl#L235 for details), however that mechanism had race conditions and a few other issues with it, so it was removed.

The way the db handles cache works is once the handle is opened, it will stay opened until used again. If at some point, if there is more room in the cache, the oldest unused handle is closed and replaced with the new one. So size the max_dbs_open accordingly to how much memory you can allocate to it. For an 8GB maybe start with 1000-1500 and see if you get any all_dbs_active errors. (Try on a staging cluster first if you can generate similar load as the production cluster). There is a balance to how much memory should be used for the db handle cache vs having more for the page cache.

Thanks for the detailed explanation , this helps a lot.
We reverted our preprod cluster from 3.5.1 back to 3.4.2 (same infra, same max_dbs_open = 60000) and observed that the sustained 60k open DB plateau dropped immediately. That seems consistent with what you described regarding the removal of the close_on_idle mechanism in 3.5 and the change in DB handle lifecycle behaviour.

Given that:

• Is the removal of close_on_idle considered permanent going forward, or is there any plan to reintroduce an idle handle eviction mechanism in future releases?
• Is there an officially recommended upper bound for max_dbs_open for 8GB instances under 3.5? We understand it must now be sized according to memory, but are there general guidelines or tested ranges the project considers safe?
• In your experience, when clusters legitimately require very high distinct DB counts (tens of thousands), is the expectation that users:
  • Scale memory proportionally, or
  • Rely strictly on lower max_dbs_open values and tolerate all_dbs_active errors under pressure?
    We're trying to determine whether our architecture should assume lower max_dbs_open permanently in 3.5+, or that future versions may introduce improved idle handle management again.

Appreciate any clarification on the long term direction here.

[nickva]

Is the removal of close_on_idle considered permanent going forward, or is there any plan to reintroduce an idle handle eviction mechanism in future releases?

There are no plans to bring it back. The way it was implemented had race conditions and didn't work well enough. In fact we disabled it production for years because of issues with it.

Is there an officially recommended upper bound for max_dbs_open for 8GB instances under 3.5? We understand it must now be sized according to memory, but are there general guidelines or tested ranges the project considers safe?

The default of 500 is fairly conservative to work on a smaller instances. I'd bump it up to 5000 - 10000 and see what it looks like under concurrent load. As I mentioned for us, often our internal default of 5000 is enough for larger clusters. I don't have exact bytes needed / db handle. You can try a few values with a large number of databases. Scan through them, getting their db info and see how much memory your node will use.

In your experience, when clusters legitimately require very high distinct DB counts (tens of thousands), is the expectation that users: * Scale` memory proportionally, or *Rely strictly on lower max_dbs_open values and tolerate all_dbs_active errors under pressure?

There is a difference between having db handles in the cache and having them active. Your memory should be enough to have them in the cache. It doesn't mean they will all be active. For instance in your case 60k value was clearly not enough to have that many active db handles as 8GB is not enough memory for it. The nodes would have crashed just like they did when you upgraded to the new version if you had that many active connection at once (active could mean a client connection, or building indexes and other background tasks).

Another way to look at it, is there is no difference in the capacity of the system before and after the change. In fact with close_on_idle the performance would be worse since the same set of dbs would be opened and closed constantly if they are used on a period just a bit longer than the idle time.