Skip to content

feat: logging for semaphore#1399

Open
phact wants to merge 2 commits intomainfrom
semaphore-logging
Open

feat: logging for semaphore#1399
phact wants to merge 2 commits intomainfrom
semaphore-logging

Conversation

@phact
Copy link
Copy Markdown
Collaborator

@phact phact commented Apr 14, 2026

This pull request introduces improved tracking and logging of worker slot contention in the TaskService, making it easier to diagnose and monitor concurrency bottlenecks during file processing. The main changes include adding internal accounting for active and queued workers, logging when tasks wait for a worker slot, and updating tests to verify this behavior.

Worker slot accounting and logging:

  • Added _worker_stats_lock, _active_workers, and _queued_workers to TaskService for accurate tracking of worker slot usage. Introduced _acquire_processing_slot and _release_processing_slot async methods to handle slot acquisition, release, and contention logging.
  • Updated the file processing flow to use the new slot acquisition/release methods, ensuring that tasks log when they are queued and when they wait for a worker slot. [1] [2] [3]

Testing improvements:

  • Added an async test test_logs_when_waiting_for_worker_slot to verify that contention is properly logged when tasks must wait for a worker slot.

Minor cleanup:

  • Removed an unnecessary blank line in the shutdown exception handler for consistency.

@github-actions github-actions bot added backend 🔷 Issues related to backend services (OpenSearch, Langflow, APIs) tests labels Apr 14, 2026
@edwinjosechittilappilly edwinjosechittilappilly changed the title logging for semaphore feat: logging for semaphore Apr 15, 2026
Copy link
Copy Markdown
Collaborator

@edwinjosechittilappilly edwinjosechittilappilly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added enhancement 🔵 New feature or request lgtm labels Apr 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend 🔷 Issues related to backend services (OpenSearch, Langflow, APIs) enhancement 🔵 New feature or request lgtm tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants