Kong Ingress Controller (DB-less) not converging state across pods – one instance missing routes leading to intermittent 404s

[akshaysharama]

🔎 Summary

In a DB-less Kong deployment with multiple replicas, one Kong instance intermittently serves 404 responses due to incomplete route configuration, while other instances function correctly.

This results in inconsistent routing behavior depending on which pod receives the request.

---

📦 Environment

• Kong Version: 3.3
• Kong Ingress Controller Version: 2.11
• Kubernetes Distribution: k3s
• Deployment Mode: DB-less (KONG_DATABASE=off)
• Replicas: Multiple (behind LoadBalancer)

Controller Resources:

requests:
  cpu: 25m
  memory: 100Mi
limits:
  cpu: 100m
  memory: 250Mi

---

🚨 Observed Behavior

• Requests via LoadBalancer intermittently return:

  • ✅ 200 OK (healthy pods)
  • ❌ 404 Not Found (faulty pod)

• Kong Admin API comparison across pods:

  curl -sk https://localhost:8001/routes | jq '.data | length'

  • Healthy pods → full route set
  • Faulty pod → partial routes (e.g., only TCPIngress)

• Faulty pod:

  • Remains in partial state indefinitely
  • Does not self-recover
  • Appears Ready from Kubernetes perspective

---

🧪 Steps to Reproduce (approximate)

1. Deploy Kong (DB-less) with KIC v2.11 and multiple replicas
2. Apply multiple Ingress and TCPIngress resources
3. Send traffic via LoadBalancer
4. Observe intermittent 404 responses
5. Compare /routes across pods → one pod shows missing routes

---

📊 Expected Behavior

All Kong instances should independently converge to the same configuration derived from Kubernetes resources, resulting in consistent routing across all pods.

---

🔍 Additional Observations

• No critical errors in logs (log level = warn)
• Occasional network timeout logs (unclear if related)
• Only partial configuration present on affected pod (TCPIngress present, HTTP Ingress missing)
• https://artifacthub.io/packages/helm/kong/kong/2.27.0 (helm chart we are using to deploy)

---

💥 Impact

• Intermittent production 404 errors
• Non-deterministic routing behavior
• Difficult to detect via health checks (pods remain Ready)

---

🔧 Workarounds

• Restarting affected pod → resolves issue temporarily

---

🙏 Any guidance or pointers would be appreciated.

[akshaysharama]

area/ingress-controller

[Icarus9913]

Hi @akshaysharama , can you convert this issue into https://github.com/Kong/kubernetes-ingress-controller repo?

[gray63659-glitch]

I will move this issue over to the kubernetes-ingress-controller repository as requested.

…

On Thu, Jun 18, 2026, 3:24 AM Icarus Wu ***@***.***> wrote: *Icarus9913* left a comment (Kong/kong#14905) <#14905 (comment)> Hi @akshaysharama <https://github.com/akshaysharama> , can you convert this issue into https://github.com/Kong/kubernetes-ingress-controller repo? — Reply to this email directly, view it on GitHub <#14905?email_source=notifications&email_token=B72N4ZAMMD37O5FHWQ6JDOL5AORNDA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZTHE3DINJYGAZ2M4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSWGM33PORSXEX3DNRUWG2Y#issuecomment-4739645803>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/B72N4ZDIK3QUC35QGI34F635AORNDAVCNFSNUABEKJSXA33TNF2G64TZHMZDMNZYGMZDSNJ3JFZXG5LFHM2DMMRZGY3TKMJXGCQXMAQ> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/B72N4ZHZBQMPO4MVVZGU36L5AORNDA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZTHE3DINJYGAZ2M4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSVGM33PORSXEX3JN5ZQ> and Android <https://github.com/notifications/mobile/android/B72N4ZDYB5QCDSKIZCQO6WD5AORNDA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINZTHE3DINJYGAZ2M4TFMFZW63VKON2WE43DOJUWEZLEUVSXMZLOOSXGM33PORSXEX3BNZSHE33JMQ>. Download it today! You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>