Stabilize Dubbo write tail latency

Author: esasmer-dou

README.md

@@ -12,7 +12,7 @@ Shared sample contracts come from `com.reactor.sample:rest-sample-utility:0.2.0`
 come transitively from `com.reactor.sample:rust-sample-model:0.2.0`. The Dubbo interface package name is
 kept as `com.reactor.rust.dubbo.sample` so the provider and consumer keep the same service identity.
 
-[Release notes for v0.3.1](docs/RELEASE_NOTES_v0.3.1.md)
+[Release notes for v0.3.2](docs/RELEASE_NOTES_v0.3.2.md)
 
 ## Contents
 
@@ -107,7 +107,7 @@ java "-Dreactor.dubbo.registry-enabled=false" `
   "-Dsample.db.password=reactor" `
   "-Dsample.db.schema-init=true" `
   "-Dsample.db.warmup=true" `
-  -jar target/rest-sample-dubbo-provider-0.1.1.jar
+  -jar target/rest-sample-dubbo-provider-0.3.2.jar
 ```
 
 Keep this terminal open.
@@ -128,7 +128,7 @@ java "-Dserver.port=8080" `
   "-Dreactor.dubbo.native-connections-per-endpoint=2" `
   "-Dreactor.dubbo.native-async-workers=1" `
   "-Dreactor.dubbo.max-inflight=8" `
-  -jar target/rest-sample-dubbo-consumer-0.1.1.jar
+  -jar target/rest-sample-dubbo-consumer-0.3.2.jar
 ```
 
 ### 3. Call The Endpoints
@@ -409,7 +409,7 @@ same behavior.
 | Recipe | Use when | Key settings | Trade-off |
 |--------|----------|--------------|-----------|
 | `micro-1x1` | Smallest Dubbo consumer pod, low or moderate traffic, DB-backed provider with a small pool. | <small><code>reactor.runtime.profile=micro-dubbo</code><br><code>reactor.dubbo.native-connections-per-endpoint=1</code><br><code>reactor.dubbo.native-async-workers=1</code><br><code>reactor.dubbo.native-async-queue-capacity=32</code><br><code>reactor.dubbo.max-inflight=16</code></small> | Lowest RSS. Spikes may get controlled `503`. |
-| `micro-2x2` | Full sample writes to DB, or Hikari is `2` and `1x1` serializes calls before the DB. | <small><code>sample.dubbo.capacity-profile=micro-2x2</code><br><code>reactor.dubbo.native-connections-per-endpoint=2</code><br><code>reactor.dubbo.native-async-workers=2</code><br><code>reactor.dubbo.native-async-queue-capacity=64</code><br><code>reactor.dubbo.max-inflight=64</code></small> | One extra native worker and connection. This is the full sample default; catalog-only stays `1x1`. |
+| `micro-2x2` | Full sample writes to DB, or Hikari is `2` and `1x1` serializes calls before the DB. | <small><code>sample.dubbo.capacity-profile=micro-2x2</code><br><code>reactor.dubbo.native-connections-per-endpoint=2</code><br><code>reactor.dubbo.native-async-workers=2</code><br><code>reactor.dubbo.native-async-queue-capacity=64</code><br><code>reactor.dubbo.max-inflight=64</code><br><code>POST /customers=4/250ms</code><br><code>POST /customers/typed=4/250ms</code></small> | Keeps the two hot create routes from building a 16-request queue in front of one shared `2x2` backend. This is the full sample default; catalog-only stays `1x1`. |
 | `balanced-stable-4x4` | Read-heavy service, provider returns ready JSON, and you want more 2xx without hiding overload. | <small><code>reactor.runtime.profile=balanced-dubbo</code><br><code>reactor.dubbo.native-connections-per-endpoint=4</code><br><code>reactor.dubbo.native-async-workers=4</code><br><code>reactor.dubbo.native-async-queue-capacity=512</code><br><code>reactor.dubbo.max-inflight=64</code></small> | Higher RPS. Needs provider CPU/DB proof. |
 | `balanced-mid-4x4` | Distributed commands need more accepted work and provider capacity is measured. | Same `4x4` native settings, with wider route budgets than `stable`. | More 2xx possible. p99/RSS must be watched. |
 | `balanced-wide-4x4` | Only for a measured high-capacity provider, not as a default. | Same `4x4` native settings, with the widest route budgets. | Can hide overload in queues. Use with caution. |
@@ -433,6 +433,11 @@ Use this rule for DB-backed endpoints: do not size the consumer for client concu
 from provider and Hikari capacity first. If the provider has only two DB connections, `c64` traffic
 must either wait briefly or get controlled `503`; deep queues only increase RSS and p99.
 
+Route limits are not isolated capacity. If raw and typed create routes both call the same command
+service, their accepted work is added together. With a two-worker/two-connection backend, `8 + 8`
+allows sixteen requests to wait behind two durable PostgreSQL commits. The `micro-2x2` recipe uses
+`4 + 4` and a `250 ms` wait budget so WAL/fsync variance does not amplify across a deep queue.
+
 Run the repeatable mixed raw/typed write gate after the consumer, provider, and PostgreSQL containers
 join the same Docker network:
 
@@ -447,6 +452,17 @@ docker run --rm --network YOUR_NETWORK `
 The normal gate requires zero failed writes and p99 below `250 ms`. A high-concurrency overload test
 may allow controlled `503`, but it must not report native Dubbo errors, DB rollbacks, or pool exhaustion.
 
+Use the repeat gate for the `c16` tail-latency check. It rejects a result when one run exceeds
+`200 ms` p99, the p99 spread exceeds `75 ms`, or non-2xx exceeds `0.10%`:
+
+```powershell
+.\benchmark\dubbo-write-repeat-gate.ps1 `
+  -Network YOUR_NETWORK `
+  -BaseUrl http://consumer:8080 `
+  -VirtualUsers 16 `
+  -Repeats 3
+```
+
 ## Production Recipes
 
 ### Recipe 1: Kubernetes Consumer With ZooKeeper Discovery
@@ -496,7 +512,7 @@ Use this when the provider address is known and your REST API only exposes read
 mvn -q -Pnative-static-consumer package
 java -Xms8m -Xmx48m -Xss256k -XX:ActiveProcessorCount=1 `
   -Dreactor.dubbo.providers=provider:20880 `
-  -jar target/rest-sample-dubbo-consumer-0.1.1.jar
+  -jar target/rest-sample-dubbo-consumer-0.3.2.jar
 ```
 
 Effect:
@@ -512,8 +528,10 @@ Effect:
 Use this for POST/PATCH/DELETE routes where each REST request becomes one provider command call.
 
 ```properties
-reactor.rust.route-admission.post.api.v1.customers.max-concurrent=8
-reactor.rust.route-admission.post.api.v1.customers.queue-timeout-ms=150
+reactor.rust.route-admission.post.api.v1.customers.max-concurrent=4
+reactor.rust.route-admission.post.api.v1.customers.queue-timeout-ms=250
+reactor.rust.route-admission.post.api.v1.customers.typed.max-concurrent=4
+reactor.rust.route-admission.post.api.v1.customers.typed.queue-timeout-ms=250
 reactor.rust.route-admission.patch.api.v1.customers.id.segment.max-concurrent=8
 reactor.rust.route-admission.delete.api.v1.customers.id.max-concurrent=8
 reactor.dubbo.timeout-ms=1000
@@ -1999,9 +2017,9 @@ Route admission:
 | `reactor.rust.route-admission.get.api.v1.customers.db.queue-timeout-ms` | `150` | Customer DB read queue wait. Lower for faster fail-fast under DB saturation. |
 | `reactor.rust.route-admission.get.api.v1.customers.db.stats/by-segment/id.*` | `4-8` / `150` | Typed DB stats, list, and record lookup caps. Keep list queries lower than raw byte pass-through routes. |
 | `reactor.rust.route-admission.get.api.v1.customers.id.exists/display-name.*` | `8` / `150` | Small scalar lookup caps. Tune with provider DB pool if these call the database. |
-| `reactor.rust.route-admission.post.api.v1.customers.max-concurrent` | `8` | Create command cap. Keep bounded to avoid duplicate-write pressure and DB queue buildup. |
-| `reactor.rust.route-admission.post.api.v1.customers.queue-timeout-ms` | `150` | Create command wait budget. Lower if write p99 matters more than absorbing bursts. |
-| `reactor.rust.route-admission.post.api.v1.customers.typed.*` | `8` / `250` | Typed create cap for `micro-2x2`. This removes false admission timeouts at the normal write gate. |
+| `reactor.rust.route-admission.post.api.v1.customers.max-concurrent` | `4` | Raw create cap for `micro-2x2`. Raw and typed limits add up because both use the same two RPC/DB permits. |
+| `reactor.rust.route-admission.post.api.v1.customers.queue-timeout-ms` | `250` | Raw create wait budget. It absorbs short WAL/fsync variance without allowing a deep queue. |
+| `reactor.rust.route-admission.post.api.v1.customers.typed.*` | `4` / `250` | Typed create cap for `micro-2x2`. Together with the raw route, at most eight create requests enter the shared backend path. |
 | `reactor.rust.route-admission.patch.api.v1.customers.id.segment.max-concurrent` | `8` | Segment patch cap. Keep aligned with command provider capacity. |
 | `reactor.rust.route-admission.patch.api.v1.customers.id.segment.queue-timeout-ms` | `150` | Segment patch queue wait. Raise only with idempotent caller behavior and measured p99. |
 | `reactor.rust.route-admission.patch.api.v1.customers.id.status.max-concurrent` | `8` | Status patch cap. Keep bounded for write-side stability. |

README.tr.md

@@ -15,7 +15,7 @@ record'ları transitif olarak `com.reactor.sample:rust-sample-model:0.2.0` paket
 interface package adı `com.reactor.rust.dubbo.sample` olarak korunur. Böylece provider ve consumer
 aynı service kimliğini kullanmaya devam eder.
 
-[v0.3.1 sürüm notları](docs/RELEASE_NOTES_v0.3.1.md)
+[v0.3.2 sürüm notları](docs/RELEASE_NOTES_v0.3.2.tr.md)
 
 ## İçindekiler
 
@@ -110,7 +110,7 @@ java "-Dreactor.dubbo.registry-enabled=false" `
   "-Dsample.db.password=reactor" `
   "-Dsample.db.schema-init=true" `
   "-Dsample.db.warmup=true" `
-  -jar target/rest-sample-dubbo-provider-0.1.1.jar
+  -jar target/rest-sample-dubbo-provider-0.3.2.jar
 ```
 
 Bu terminal açık kalmalıdır.
@@ -131,7 +131,7 @@ java "-Dserver.port=8080" `
   "-Dreactor.dubbo.native-connections-per-endpoint=2" `
   "-Dreactor.dubbo.native-async-workers=1" `
   "-Dreactor.dubbo.max-inflight=8" `
-  -jar target/rest-sample-dubbo-consumer-0.1.1.jar
+  -jar target/rest-sample-dubbo-consumer-0.3.2.jar
 ```
 
 ### 3. Endpoint'leri Deneyin
@@ -424,7 +424,7 @@ Benchmark raporu sadece raw RPS vermez; bilinçli olarak şu alanları da taşı
 | Reçete | Ne zaman kullanılır? | Ana ayarlar | Bedel |
 |--------|----------------------|-------------|-------|
 | `micro-1x1` | En küçük Dubbo consumer pod, düşük/orta trafik, küçük DB pool kullanan provider. | <small><code>reactor.runtime.profile=micro-dubbo</code><br><code>reactor.dubbo.native-connections-per-endpoint=1</code><br><code>reactor.dubbo.native-async-workers=1</code><br><code>reactor.dubbo.native-async-queue-capacity=32</code><br><code>reactor.dubbo.max-inflight=16</code></small> | En düşük RSS. Spike altında kontrollü `503` dönebilir. |
-| `micro-2x2` | Full sample DB'ye yazıyor veya Hikari `2` olduğu halde `1x1` çağrıları DB'ye ulaşmadan seri hale getiriyor. | <small><code>sample.dubbo.capacity-profile=micro-2x2</code><br><code>reactor.dubbo.native-connections-per-endpoint=2</code><br><code>reactor.dubbo.native-async-workers=2</code><br><code>reactor.dubbo.native-async-queue-capacity=64</code><br><code>reactor.dubbo.max-inflight=64</code></small> | Bir ek native worker ve connection kullanır. Full sample varsayılanıdır; catalog-only `1x1` kalır. |
+| `micro-2x2` | Full sample DB'ye yazıyor veya Hikari `2` olduğu halde `1x1` çağrıları DB'ye ulaşmadan seri hale getiriyor. | <small><code>sample.dubbo.capacity-profile=micro-2x2</code><br><code>reactor.dubbo.native-connections-per-endpoint=2</code><br><code>reactor.dubbo.native-async-workers=2</code><br><code>reactor.dubbo.native-async-queue-capacity=64</code><br><code>reactor.dubbo.max-inflight=64</code><br><code>POST /customers=4/250ms</code><br><code>POST /customers/typed=4/250ms</code></small> | İki yoğun create route'unun ortak `2x2` backend önünde 16 istek biriktirmesini engeller. Full sample varsayılanıdır; catalog-only `1x1` kalır. |
 | `balanced-stable-4x4` | Read-heavy servis, provider hazır JSON dönüyor ve overload saklanmadan daha çok 2xx isteniyor. | <small><code>reactor.runtime.profile=balanced-dubbo</code><br><code>reactor.dubbo.native-connections-per-endpoint=4</code><br><code>reactor.dubbo.native-async-workers=4</code><br><code>reactor.dubbo.native-async-queue-capacity=512</code><br><code>reactor.dubbo.max-inflight=64</code></small> | RPS artar. Provider CPU/DB ölçümü gerekir. |
 | `balanced-mid-4x4` | Distributed command endpoint'leri daha çok kabul edilen iş istiyor ve provider kapasitesi ölçüldü. | Aynı `4x4` native ayarları, `stable` değerinden daha geniş route budget. | Daha çok 2xx alınabilir. p99/RSS izlenmelidir. |
 | `balanced-wide-4x4` | Sadece ölçülmüş yüksek kapasiteli provider için. Default yapılmamalıdır. | Aynı `4x4` native ayarları, en geniş route budget. | Overload queue içinde saklanabilir. Dikkatli kullanın. |
@@ -448,6 +448,12 @@ DB-backed endpoint için kural basittir: consumer'ı önce client concurrency de
 provider ve Hikari kapasitesine göre boyutlandırın. Provider'da iki DB connection varsa `c64`
 trafik ya kısa süre beklemeli ya da kontrollü `503` almalıdır. Derin queue sadece RSS ve p99 artırır.
 
+Route limitleri birbirinden bağımsız backend kapasitesi değildir. Raw ve typed create route'ları aynı
+command service'e gidiyorsa kabul ettikleri işler toplanır. İki worker ve iki DB connection önünde
+`8 + 8` kullanmak, iki güvenli PostgreSQL commit işleminin arkasında 16 istek biriktirir. `micro-2x2`
+reçetesi bu nedenle `4 + 4` ve `250 ms` bekleme bütçesi kullanır. WAL/fsync gecikmesi derin queue
+boyunca büyüyerek p99 değerini bozmaz.
+
 Consumer, provider ve PostgreSQL container'ları aynı Docker network'üne bağlandıktan sonra tekrarlanabilir
 raw/typed karma write gate'i çalıştırın:
 
@@ -462,6 +468,17 @@ docker run --rm --network YOUR_NETWORK `
 Normal gate sıfır hatalı write ve `250 ms` altında p99 ister. Yüksek concurrency overload testinde
 kontrollü `503` kabul edilebilir. Ancak native Dubbo hatası, DB rollback veya pool exhaustion olmamalıdır.
 
+`c16` tail-latency kontrolü için repeat gate'i kullanın. Tek bir koşuda p99 `200 ms` değerini aşarsa,
+p99 yayılımı `75 ms` değerinden büyükse veya non-2xx oranı `%0,10` üzerine çıkarsa test başarısız olur:
+
+```powershell
+.\benchmark\dubbo-write-repeat-gate.ps1 `
+  -Network YOUR_NETWORK `
+  -BaseUrl http://consumer:8080 `
+  -VirtualUsers 16 `
+  -Repeats 3
+```
+
 ## Production Reçeteleri
 
 ### Reçete 1: ZooKeeper Discovery Kullanan Kubernetes Consumer
@@ -511,7 +528,7 @@ expose ediyorsa bunu kullanın.
 mvn -q -Pnative-static-consumer package
 java -Xms8m -Xmx48m -Xss256k -XX:ActiveProcessorCount=1 `
   -Dreactor.dubbo.providers=provider:20880 `
-  -jar target/rest-sample-dubbo-consumer-0.1.1.jar
+  -jar target/rest-sample-dubbo-consumer-0.3.2.jar
 ```
 
 Etkisi:
@@ -528,8 +545,10 @@ Her REST isteğinin bir provider command çağrısına dönüştüğü POST/PATC
 kullanın.
 
 ```properties
-reactor.rust.route-admission.post.api.v1.customers.max-concurrent=8
-reactor.rust.route-admission.post.api.v1.customers.queue-timeout-ms=150
+reactor.rust.route-admission.post.api.v1.customers.max-concurrent=4
+reactor.rust.route-admission.post.api.v1.customers.queue-timeout-ms=250
+reactor.rust.route-admission.post.api.v1.customers.typed.max-concurrent=4
+reactor.rust.route-admission.post.api.v1.customers.typed.queue-timeout-ms=250
 reactor.rust.route-admission.patch.api.v1.customers.id.segment.max-concurrent=8
 reactor.rust.route-admission.delete.api.v1.customers.id.max-concurrent=8
 reactor.dubbo.timeout-ms=1000
@@ -2012,9 +2031,9 @@ Route admission:
 | `reactor.rust.route-admission.get.api.v1.customers.db.queue-timeout-ms` | `150` | Customer DB read queue wait. DB saturation altında hızlı fail-fast için düşürün. |
 | `reactor.rust.route-admission.get.api.v1.customers.db.stats/by-segment/id.*` | `4-8` / `150` | Typed DB stats, list ve record lookup cap'leri. List query'leri raw byte pass-through route'larından düşük tutun. |
 | `reactor.rust.route-admission.get.api.v1.customers.id.exists/display-name.*` | `8` / `150` | Küçük scalar lookup cap'leri. Bu method'lar DB'ye gidiyorsa provider DB pool ile tune edin. |
-| `reactor.rust.route-admission.post.api.v1.customers.max-concurrent` | `8` | Create command cap. Duplicate write baskısı ve DB queue büyümesini önlemek için bounded kalmalı. |
-| `reactor.rust.route-admission.post.api.v1.customers.queue-timeout-ms` | `150` | Create command wait budget. Write p99 burst absorption'dan önemliyse düşürün. |
-| `reactor.rust.route-admission.post.api.v1.customers.typed.*` | `8` / `250` | `micro-2x2` typed create sınırıdır. Normal write gate'teki hatalı admission timeout'larını kaldırır. |
+| `reactor.rust.route-admission.post.api.v1.customers.max-concurrent` | `4` | `micro-2x2` raw create sınırıdır. Raw ve typed limitleri aynı iki RPC/DB iznini kullandıkları için toplanır. |
+| `reactor.rust.route-admission.post.api.v1.customers.queue-timeout-ms` | `250` | Raw create bekleme bütçesidir. Derin queue oluşturmadan kısa WAL/fsync dalgalanmasını karşılar. |
+| `reactor.rust.route-admission.post.api.v1.customers.typed.*` | `4` / `250` | Typed create sınırıdır. Raw route ile birlikte ortak backend yoluna en fazla sekiz create isteği girer. |
 | `reactor.rust.route-admission.patch.api.v1.customers.id.segment.max-concurrent` | `8` | Segment patch cap. Command provider kapasitesiyle hizalayın. |
 | `reactor.rust.route-admission.patch.api.v1.customers.id.segment.queue-timeout-ms` | `150` | Segment patch queue wait. Sadece idempotent caller ve p99 ölçümü varsa artırın. |
 | `reactor.rust.route-admission.patch.api.v1.customers.id.status.max-concurrent` | `8` | Status patch cap. Write-side stabilite için bounded kalmalı. |