Summary
HikariCP 7.0.2 — which was supposed to fix the virtual-thread yield-spin first reported in #2329 — still causes complete carrier-thread saturation under moderate virtual-thread load. The Thread.yield() branch in ConcurrentBag.requite() (line 199) and unreserve() (line 323) is taken on the overwhelming majority of iterations because the parkNanos fallback only triggers once every 256 iterations (or i % waiting == 0, which doesn't help when there are many waiters).
We captured thread dumps showing all 6 carrier threads simultaneously running virtual threads that are all stuck in the exact same requite() yield-spin — triggered by the normal Spring transaction commit path. The result is pod-level CPU saturation (5600m+ on a 6000m limit) and eventual liveness probe failures, causing pod restarts.
Note: #2366 reported a similar death spiral on 7.0.2 but was self-closed after the reporter concluded it was an unrelated DB trigger issue. The yield-spin symptom they described is real and independent of that — we're hitting it consistently under load.
Environment
|
|
| HikariCP |
7.0.2 (verified: BOOT-INF/lib/HikariCP-7.0.2.jar in the fat jar) |
| Spring Boot |
3.5.7 (default ships 6.3.3; overridden via <hikaricp.version>7.0.2</hikaricp.version>) |
| Hibernate |
6.6.x (Spring Boot 3.5 managed) |
| JDK |
21.0.10 Corretto (-Xmx19660m) |
| Virtual threads |
spring.threads.virtual.enabled=true |
| ForkJoinPool carriers |
6 (default, matches CPU allocation of 6 cores) |
| DB |
Aurora PostgreSQL 17.4 via AWS Advanced JDBC Wrapper 2.6.5 (F0 profile) |
| Pool config |
maximum-pool-size: 50, minimum-idle: 50 (to avoid the unfixed add() path), connection-timeout: 5000, max-lifetime: 1800000, keepalive-time: 120000 |
| Kubernetes |
Pod with CPU limit 6 cores, memory limit 24 GB |
Load: moderate per-pod RPS across a 24-pod deployment. Requests are read-heavy, each going through @Transactional Spring MVC controllers that run short JPA queries and return. Every request therefore opens and closes a connection via Hibernate, so every request hits HikariPool.recycle → ConcurrentBag.requite() on the way out.
What we see
Under sustained virtual-thread load, periodic CPU spikes occur on individual pods. The procfs snapshot during a spike shows:
87.5% nid=134 ForkJoinPool-1-worker-4 ← carrier
87.0% nid=226 ForkJoinPool-1-worker-5 ← carrier
87.0% nid=133 ForkJoinPool-1-worker-3 ← carrier
87.0% nid=132 ForkJoinPool-1-worker-2 ← carrier
86.5% nid=370 ForkJoinPool-1-worker-6 ← carrier
86.0% nid=130 ForkJoinPool-1-worker-1 ← carrier
84.5% nid=71 HikariPool-1:housekeeper ← ALSO spinning in unreserve()
9.0% nid=131 VirtualThread-unparker
That's all 6 carriers plus the housekeeper consuming CPU, totaling ~611% out of the 600% (6 core) limit. The pod gets kernel-throttled and eventually the liveness probe at /actuator/health/liveness fails.
Root cause: every virtual thread returning a connection hits requite() yield-spin
We captured JSON thread dumps via jcmd Thread.dump_to_file -format=json. We then mapped each carrier's "Carrying virtual thread #X" marker (from jcmd Thread.print) back to the full virtual thread stack in the JSON dump.
All 6 carriers were running virtual threads executing the identical code path — varying only in which controller initiated the request. Here are 3 of the 6 (the others are identical except for the top controller method):
Application-layer class/method names below are redacted (app.service.* / app.rest.*) — they are just different read-path REST endpoints, each backed by a @Transactional JPA service method. What matters is the identical HikariCP/Hibernate/Spring frames.
Carrier worker-1 (nid 130, 86.0% CPU) — virtual thread tomcat-handler-1241680
[ 0] java.base/java.lang.VirtualThread.tryYield(VirtualThread.java:772)
[ 1] java.base/java.lang.Thread.yield(Thread.java:443)
[ 2] com.zaxxer.hikari.util.ConcurrentBag.requite(ConcurrentBag.java:199) ← THE SPIN
[ 3] com.zaxxer.hikari.pool.HikariPool.recycle(HikariPool.java:447)
[ 4] com.zaxxer.hikari.pool.PoolEntry.recycle(PoolEntry.java:81)
[ 5] com.zaxxer.hikari.pool.ProxyConnection.close(ProxyConnection.java:268)
[ 6] org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.closeConnection
[ 7] org.hibernate.internal.NonContextualJdbcConnectionAccess.releaseConnection
[ 8] org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.releaseConnection
[ 9] org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.close
[10] org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.close
[11] org.hibernate.internal.AbstractSharedSessionContract.close
[12] org.hibernate.internal.SessionImpl.closeWithoutOpenChecks
[13] org.hibernate.internal.SessionImpl.close
[14] org.springframework.orm.jpa.EntityManagerFactoryUtils.closeEntityManager
[15] org.springframework.orm.jpa.JpaTransactionManager.doCleanupAfterCompletion
[16] org.springframework.transaction.support.AbstractPlatformTransactionManager.cleanupAfterCompletion
[17] org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit
[18] org.springframework.transaction.support.AbstractPlatformTransactionManager.commit
[19] org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning
[20] org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction
[21] org.springframework.transaction.interceptor.TransactionInterceptor.invoke
[22] org.springframework.aop.framework.ReflectiveMethodInvocation.proceed
[23] org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept
[24] app.service.UserService$$SpringCGLIB$$0.<method1>
[25] app.rest.UserController.<endpoint1>
Carrier worker-2 (nid 132, 87.0% CPU) — virtual thread tomcat-handler-1241678
This one caught the other branch of the same spin loop:
[ 0] java.base/jdk.internal.misc.Unsafe.park(Native Method)
[ 1] java.base/java.lang.VirtualThread.parkOnCarrierThread(VirtualThread.java:677)
[ 2] java.base/java.lang.VirtualThread.parkNanos(VirtualThread.java:648)
[ 3] java.base/java.lang.System$2.parkVirtualThread(System.java:2652)
[ 4] java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)
[ 5] java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)
[ 6] com.zaxxer.hikari.util.ConcurrentBag.requite(ConcurrentBag.java:196) ← PARK branch
[ 7] com.zaxxer.hikari.pool.HikariPool.recycle(HikariPool.java:447)
[ 8] com.zaxxer.hikari.pool.PoolEntry.recycle(PoolEntry.java:81)
[ 9] com.zaxxer.hikari.pool.ProxyConnection.close(ProxyConnection.java:268)
... (identical Hibernate/Spring chain: frames 10–27 same as above) ...
[28] app.service.UserService$$SpringCGLIB$$0.<method2>
[29] app.rest.UserController.<endpoint2>
### Carrier worker-4 (nid 134, 87.5% CPU) — virtual thread `tomcat-handler-1241676`
[ 0] java.base/java.lang.VirtualThread.tryYield(VirtualThread.java:772)
[ 1] java.base/java.lang.Thread.yield(Thread.java:443)
[ 2] com.zaxxer.hikari.util.ConcurrentBag.requite(ConcurrentBag.java:199)
[ 3] com.zaxxer.hikari.pool.HikariPool.recycle(HikariPool.java:447)
... (identical Hibernate/Spring chain: frames 4–23 same as above) ...
[24] app.service.SectionService$$SpringCGLIB$$0.execute
[25] app.rest.SectionController.
The other 3 carriers (worker-3, worker-5, worker-6) follow the same pattern. At the moment of the snapshot: **5 of the 6 caught at line 199 (the `Thread.yield()` branch), 1 at line 196 (the `parkNanos` branch).** That ratio is consistent with the 255:1 ratio in the code (`(i & 0xff) == 0xff` fires only every 256 iterations).
The `HikariPool-1:housekeeper` thread is doing the same thing in `unreserve()` (line 323) because its `KeepaliveTask` cycles a connection through and also has to hand it off to a waiter.
## Relationship to prior issues
- #2329 - (High CPU on 6.3.0+VT, closed by the fix in 7.0.2) — we have 7.0.2, and we're still hitting the variant of this problem, confirming the partial fix is insufficient for our load profile.
- #2366 - (Death Spiral on 7.0.2, self-closed as DB trigger) — the reporter's stack trace does show the housekeeper thread busy on 7.0.2. Their specific problem may have been a DB deadlock, but the CPU symptom they showed looks like the same yield-spin we're hitting. Worth revisiting.
- #1463 - (Make HikariCP loom-friendly) — the "JDK 24 fixes this" comment refers to JEP 491 (synchronized pinning), which is a DIFFERENT problem than the one we're hitting here.
## workaround
- This is confirmed to be working without issues.
- Disable virtual threads -Dspring.threads.virtual.enabled=false
Summary
HikariCP 7.0.2 — which was supposed to fix the virtual-thread yield-spin first reported in #2329 — still causes complete carrier-thread saturation under moderate virtual-thread load. The
Thread.yield()branch inConcurrentBag.requite()(line 199) andunreserve()(line 323) is taken on the overwhelming majority of iterations because theparkNanosfallback only triggers once every 256 iterations (ori % waiting == 0, which doesn't help when there are many waiters).We captured thread dumps showing all 6 carrier threads simultaneously running virtual threads that are all stuck in the exact same
requite()yield-spin — triggered by the normal Spring transaction commit path. The result is pod-level CPU saturation (5600m+ on a 6000m limit) and eventual liveness probe failures, causing pod restarts.Note: #2366 reported a similar death spiral on 7.0.2 but was self-closed after the reporter concluded it was an unrelated DB trigger issue. The yield-spin symptom they described is real and independent of that — we're hitting it consistently under load.
Environment
BOOT-INF/lib/HikariCP-7.0.2.jarin the fat jar)<hikaricp.version>7.0.2</hikaricp.version>)-Xmx19660m)spring.threads.virtual.enabled=truemaximum-pool-size: 50,minimum-idle: 50(to avoid the unfixedadd()path),connection-timeout: 5000,max-lifetime: 1800000,keepalive-time: 120000Load: moderate per-pod RPS across a 24-pod deployment. Requests are read-heavy, each going through
@TransactionalSpring MVC controllers that run short JPA queries and return. Every request therefore opens and closes a connection via Hibernate, so every request hitsHikariPool.recycle→ConcurrentBag.requite()on the way out.What we see
Under sustained virtual-thread load, periodic CPU spikes occur on individual pods. The procfs snapshot during a spike shows:
That's all 6 carriers plus the housekeeper consuming CPU, totaling ~611% out of the 600% (6 core) limit. The pod gets kernel-throttled and eventually the liveness probe at
/actuator/health/livenessfails.Root cause: every virtual thread returning a connection hits
requite()yield-spinWe captured JSON thread dumps via
jcmd Thread.dump_to_file -format=json. We then mapped each carrier's "Carrying virtual thread #X" marker (fromjcmd Thread.print) back to the full virtual thread stack in the JSON dump.All 6 carriers were running virtual threads executing the identical code path — varying only in which controller initiated the request. Here are 3 of the 6 (the others are identical except for the top controller method):
Application-layer class/method names below are redacted (
app.service.*/app.rest.*) — they are just different read-path REST endpoints, each backed by a@TransactionalJPA service method. What matters is the identical HikariCP/Hibernate/Spring frames.Carrier worker-1 (nid 130, 86.0% CPU) — virtual thread
tomcat-handler-1241680Carrier worker-2 (nid 132, 87.0% CPU) — virtual thread
tomcat-handler-1241678This one caught the other branch of the same spin loop:
[ 0] java.base/java.lang.VirtualThread.tryYield(VirtualThread.java:772)
[ 1] java.base/java.lang.Thread.yield(Thread.java:443)
[ 2] com.zaxxer.hikari.util.ConcurrentBag.requite(ConcurrentBag.java:199)
[ 3] com.zaxxer.hikari.pool.HikariPool.recycle(HikariPool.java:447)
... (identical Hibernate/Spring chain: frames 4–23 same as above) ...
[24] app.service.SectionService$$SpringCGLIB$$0.execute
[25] app.rest.SectionController.