Add remote configuration for concurrency limits and executor count

[robacarp]

Summary

This PR introduces remote configuration capabilities to Mosquito, allowing dynamic adjustment of concurrency limits and executor counts without restarting workers. Two new adapter/config classes enable reading and writing these settings to the backend (e.g., Redis) with support for both global and per-overseer configurations.

Key Changes

• RemoteConfigDequeueAdapter: A new dequeue adapter that wraps ConcurrencyLimitedDequeueAdapter with remotely configurable concurrency limits. Features include:

  • Polling the backend at a configurable interval for limit updates
  • Merging remote limits on top of defaults
  • Per-overseer configuration with precedence: defaults → global remote → per-overseer remote
  • Graceful fallback to defaults when backend is unreachable

• ExecutorConfig API: New class for managing remotely stored executor count overrides:

  • Global and per-overseer executor count storage
  • Resolution logic that prefers per-overseer over global settings
  • Instance and class-level methods for read/write operations

• ConcurrencyConfig API: New class providing a clean interface to RemoteConfigDequeueAdapter storage:

  • Read/write global and per-overseer concurrency limits
  • Delegates to the adapter's backend storage methods

• Overseer integration: Updated Overseer#each_run to poll for and apply remote executor count overrides at most once per heartbeat interval

• Api convenience methods: Added to Mosquito::Api for easy access to:

  • concurrency_limits / set_concurrency_limits
  • executor_count / set_executor_count
  • Both support optional overseer_id parameter for per-overseer configuration

• Configuration: Added overseer_id property to Mosquito::Configuration for stable worker identification

Implementation Details

• Remote limits are merged with defaults rather than replacing them, allowing partial overrides
• Per-overseer configuration enables asymmetric hardware setups (e.g., GPU workers with different limits)
• Refresh intervals prevent excessive backend polling while still allowing timely updates
• Backend errors are caught gracefully, falling back to current limits rather than crashing
• Comprehensive test coverage for all configuration scenarios and merge precedence rules

https://claude.ai/code/session_01PrikEg4dJya2KYjLkkKAmL

[Copilot]

Pull request overview

Adds remote-configurable runtime controls to Mosquito so workers can adjust queue concurrency limits and executor pool size without restarts, using backend-stored overrides (global and per-overseer).

Changes:

• Introduces RemoteConfigDequeueAdapter to apply backend-stored concurrency limit overrides (with optional per-overseer precedence).
• Adds Mosquito::Api::ExecutorConfig / Mosquito::Api::ConcurrencyConfig plus Mosquito::Api convenience methods for reading/writing remote settings.
• Integrates remote executor-count polling into Runners::Overseer#each_run and adds Configuration#overseer_id for stable worker identification, with accompanying specs.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/mosquito/runners/overseer.cr	Polls backend for executor-count overrides (rate-limited by heartbeat interval) and applies changes.
src/mosquito/dequeue_adapters/remote_config_dequeue_adapter.cr	New dequeue adapter that refreshes/merges remote concurrency limits over defaults, plus backend storage helpers.
src/mosquito/configuration.cr	Adds overseer_id to support per-overseer remote config resolution.
src/mosquito/api/executor_config.cr	New API for storing/resolving executor-count overrides (global + per-overseer).
src/mosquito/api/concurrency_config.cr	New API wrapper around remote concurrency limit storage helpers.
src/mosquito/api.cr	Adds convenience methods for reading/writing executor count and concurrency limits via Mosquito::Api.
spec/mosquito/runners/overseer_spec.cr	Tests remote executor-count application and precedence behavior.
spec/mosquito/dequeue_adapters/remote_config_dequeue_adapter_spec.cr	Comprehensive tests for remote concurrency limits, merge precedence, and API integration.
spec/mosquito/api/executor_config_spec.cr	Tests executor-count storage helpers, resolution logic, and API convenience methods.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

refresh_limits replaces @inner with a new ConcurrencyLimitedDequeueAdapter when limits change. This resets the inner adapter’s @active in-flight counters, so any jobs already dequeued are no longer counted against concurrency limits (and subsequent finished_with calls will decrement the new, empty counter). Consider updating the inner adapter’s limits in-place (preserving active counts) or otherwise carrying over the in-flight counts when applying new limits.

[Copilot]

The rescue path in load_remote_limits returns an empty hash, which causes refresh_limits to revert to defaults (potentially discarding previously applied remote overrides). The comment says it falls back to “whatever limits are already in use”, which doesn’t match the current behavior. Please align the implementation and documentation (either keep the last known-good limits on errors, or update the comment to reflect reverting to defaults).

Suggested change

	# If the backend is unreachable or the data is corrupt, fall back
	# to whatever limits are already in use.
	# If the backend is unreachable or the data is corrupt, return an
	# empty set of remote limits so the adapter falls back to defaults.

[Copilot]

executor_count= clamps values to at least 1. If the remote value is 0 or negative, this code will (a) log a change to the invalid value and (b) re-apply it every heartbeat because remote_count will never equal the clamped executor_count, resulting in repeated log spam and unnecessary pool adjustments. Consider validating/clamping the remote value before comparing/logging, or ignoring invalid remote counts.

Suggested change

	if remote_count != executor_count
	log.info { "Remote executor count changed: #{executor_count} → #{remote_count}" }
	self.executor_count = remote_count
	clamped_remote_count = remote_count < 1 ? 1 : remote_count
	if clamped_remote_count != executor_count
	log.info { "Remote executor count changed: #{executor_count} → #{clamped_remote_count}" }
	self.executor_count = clamped_remote_count