Queues

Senna supports multiple queues with weighted priority distribution, allowing you to prioritize certain jobs over others.

Queue Configuration

Configure queues when creating a worker:

w, err := worker.New(&worker.Config{
    Redis:     redisConfig,
    Namespace: "myapp",
    Settings: senna.WorkerSettings{
        Queues: []senna.QueueConfig{
            {Name: "critical", Priority: 10},
            {Name: "default", Priority: 5},
            {Name: "low", Priority: 1},
        },
    },
})

Priority Modes

Senna supports two queue priority modes: weighted random (default) and strict priority.

Weighted Random (Default)

By default, Senna uses weighted random queue selection. The Priority value determines the probability of checking each queue:

Queue	Priority	Selection Probability
critical	10	~62.5%
default	5	~31.25%
low	1	~6.25%

This ensures all queues get processed while still favoring higher priority queues. Lower priority jobs won't starve—they'll still be picked up, just less frequently.

Note

With weighted random, a job from a lower-priority queue may run before a higher-priority queue's job. This is by design to prevent starvation.

Strict Priority

Enable strict priority to always process higher priority queues first:

senna.WorkerSettings{
    Queues: []senna.QueueConfig{
        {Name: "critical", Priority: 10},
        {Name: "default", Priority: 5},
        {Name: "low", Priority: 1},
    },
    StrictPriority: true,
}

With strict priority, jobs in the critical queue are always processed before default, and default before low. Workers will only check lower priority queues when all higher priority queues are empty.

Warning

Strict priority can lead to starvation of lower priority queues if higher priority queues always have work. Use dedicated workers or weighted random if you need guaranteed processing of all queues.

Sequential Queues

Mark a queue as sequential to process jobs one at a time, in order, across all workers:

Queues: []senna.QueueConfig{
    {Name: "default", Priority: 5},
    {Name: "imports", Priority: 3, Sequential: true},
}

With Sequential: true:

• Only one job from the queue is processed at a time globally
• Jobs are processed in FIFO order
• Works across multiple worker processes using a distributed lock

Use Cases

Sequential queues are useful when:

• Order matters - Processing events for a single entity in sequence
• External constraints - An API that can only handle one request at a time
• Data consistency - Operations that must not overlap (like file imports)

// Import jobs must run one at a time to avoid conflicts
c.Enqueue(ctx, "import_spreadsheet", map[string]any{
    "file_id": "abc123",
}, client.WithQueue("imports"))

How It Works

Sequential queues use a two-level locking mechanism:

1. Local semaphore - Ensures only one goroutine per worker process attempts to fetch from the queue
2. Distributed Redis lock - Coordinates across multiple worker processes

When a worker fetches a job from a sequential queue:

1. Acquires the local semaphore (or skips if another goroutine has it)
2. Atomically checks/acquires the Redis lock AND fetches the job in one operation
3. Holds the lock while processing
4. Releases both locks after job completion (success or failure)

The lock uses SequentialLockTTL (default 30 seconds) and is automatically renewed every SequentialLockRenewInterval (default 10 seconds) during long-running jobs. If a worker crashes, the lock expires and another worker can take over.

Note

Sequential queues cannot use blocking fetch (BLMOVE) since multiple workers blocking would break exclusivity. Instead, workers poll the queue, which may add slight latency compared to regular queues.

Combining with Other Features

Sequential queues work with all other Senna features:

Queues: []senna.QueueConfig{
    {Name: "critical", Priority: 10},
    {Name: "payments", Priority: 8, Sequential: true},  // One payment at a time
    {Name: "default", Priority: 5},
    {Name: "imports", Priority: 3, Sequential: true},   // One import at a time
}

Sending Jobs to Queues

Use WithQueue to send jobs to a specific queue:

// High priority job
c.Enqueue(ctx, "process_payment", args, client.WithQueue("critical"))

// Normal priority (default)
c.Enqueue(ctx, "send_email", args) // Uses default queue

// Low priority
c.Enqueue(ctx, "generate_report", args, client.WithQueue("low"))

Queue names must contain at least one non-whitespace character. Senna validates this when enqueueing single jobs, bulk jobs, and batches.

Senna does not verify that a worker is currently configured for every valid queue name. A job sent to a queue with no workers will remain queued until a worker starts consuming that queue.

Default Queue

If no queue is specified, jobs go to the default queue:

// Client configuration
c, err := client.New(&client.Config{
    Redis:     redisConfig,
    Namespace: "myapp",
    Settings: client.Settings{
        DefaultQueue: "default", // Jobs without WithQueue go here
    },
})

// These are equivalent:
c.Enqueue(ctx, "job_type", args)
c.Enqueue(ctx, "job_type", args, client.WithQueue("default"))

Queue Design Patterns

By Priority

The most common pattern - separate queues for different priority levels:

Queues: []senna.QueueConfig{
    {Name: "critical", Priority: 10},  // Payments, alerts
    {Name: "default", Priority: 5},    // Normal operations
    {Name: "low", Priority: 1},        // Reports, cleanup
}

By Job Type

Separate queues for different job categories:

Queues: []senna.QueueConfig{
    {Name: "emails", Priority: 5},
    {Name: "webhooks", Priority: 5},
    {Name: "reports", Priority: 3},
}

By Customer Tier

Different queues for different customer tiers:

Queues: []senna.QueueConfig{
    {Name: "enterprise", Priority: 10},
    {Name: "pro", Priority: 5},
    {Name: "free", Priority: 1},
}

Dedicated Workers

You can run specialized workers that only process specific queues:

// Worker 1: Only critical jobs
w1, _ := worker.New(&worker.Config{
    Settings: senna.WorkerSettings{
        Concurrency: 20,
        Queues: []senna.QueueConfig{
            {Name: "critical", Priority: 1},
        },
    },
})

// Worker 2: Everything else
w2, _ := worker.New(&worker.Config{
    Settings: senna.WorkerSettings{
        Concurrency: 10,
        Queues: []senna.QueueConfig{
            {Name: "default", Priority: 5},
            {Name: "low", Priority: 1},
        },
    },
})

This is useful for:

• Ensuring critical jobs always have dedicated capacity
• Scaling different job types independently
• Isolating slow jobs from fast jobs

Single Queue Setup

For simple applications, a single queue works fine:

Queues: []senna.QueueConfig{
    {Name: "default", Priority: 1},
}

Queue Storage

Queues are stored in Redis as lists:

namespace:queue:critical   (Redis list)
namespace:queue:default    (Redis list)
namespace:queue:low        (Redis list)
namespace:queues           (Set of active queue names)

Best Practices

Tip

Start with a simple queue setup and add complexity as needed. Most applications work fine with just 2-3 queues.

1. Start simple: Begin with a single queue and add more as needed

2. Use meaningful names: Queue names should describe their purpose

3. Ensure every queue has workers: A valid queue name only controls where the job is stored; at least one worker still needs to consume that queue

4. Don't over-segment: Too many queues can be hard to manage

5. Monitor queue depth: High queue depth indicates workers can't keep up

6. Consider dedicated workers: For critical jobs that need guaranteed capacity

Next Steps

• Learn about Batches for grouped job processing
• Add Rate Limiters to control throughput
• Configure Middleware for logging