Skip to content

Add comprehensive diagnostic logging to SingleNodeExecutor to try to understand where the deadlock comes from#1137

Draft
guizmaii wants to merge 1 commit intomainfrom
add-deadlock-diagnostic-logging
Draft

Add comprehensive diagnostic logging to SingleNodeExecutor to try to understand where the deadlock comes from#1137
guizmaii wants to merge 1 commit intomainfrom
add-deadlock-diagnostic-logging

Conversation

@guizmaii
Copy link
Copy Markdown
Member

This adds detailed logging to help identify where deadlocks occur in the Redis client. Each promise gets a unique ID to track its full lifecycle from creation to completion.

Logging added:

  • Promise lifecycle: creation, queueing, completion/failure with unique IDs
  • Queue state tracking: requests and responses queue sizes throughout operations
  • Send operations: request processing, connection writes, promise movement to response queue
  • Receive operations: response processing, promise completion
  • Error handling: detailed error scenarios and promise cleanup
  • Runner fiber: send/receive fiber lifecycle and failure/retry scenarios

Log prefixes:

  • [REDIS-INIT]: Executor initialization
  • [REDIS-EXEC-{id}]: Individual request/promise lifecycle
  • [REDIS-SEND]: Send fiber operations
  • [REDIS-RECV]: Receive fiber operations
  • [REDIS-ERROR]: Error handling scenarios
  • [REDIS-RUNNER]: Runner fiber lifecycle

This will help identify deadlock scenarios in CI environments by showing:

  1. If promises are getting stuck at promise.await (no completion logs)
  2. If requests are getting stuck in queues (queue size logs)
  3. If send/receive fibers are failing (runner logs)
  4. If error handling is working properly (error logs)

This adds detailed logging to help identify where deadlocks occur in the Redis client.
Each promise gets a unique ID to track its full lifecycle from creation to completion.

Logging added:
- Promise lifecycle: creation, queueing, completion/failure with unique IDs
- Queue state tracking: requests and responses queue sizes throughout operations
- Send operations: request processing, connection writes, promise movement to response queue
- Receive operations: response processing, promise completion
- Error handling: detailed error scenarios and promise cleanup
- Runner fiber: send/receive fiber lifecycle and failure/retry scenarios

Log prefixes:
- [REDIS-INIT]: Executor initialization
- [REDIS-EXEC-{id}]: Individual request/promise lifecycle
- [REDIS-SEND]: Send fiber operations
- [REDIS-RECV]: Receive fiber operations
- [REDIS-ERROR]: Error handling scenarios
- [REDIS-RUNNER]: Runner fiber lifecycle

This will help identify deadlock scenarios in CI environments by showing:
1. If promises are getting stuck at promise.await (no completion logs)
2. If requests are getting stuck in queues (queue size logs)
3. If send/receive fibers are failing (runner logs)
4. If error handling is working properly (error logs)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant