Robust background job processing for Elysium.js applications
Heracles is a powerful background job processing system for Elysium.js applications. It provides a flexible and scalable solution for running tasks in the background, managing queues, and distributing workloads across multiple workers using Redis as the transport.
- Job queues: Organize jobs into different queues with custom settings
- Transports: Abstract communication layer between queues and workers
- Scheduling: Schedule jobs to run at specific times
- Retries: Automatically retry failed jobs with configurable backoff
- Prioritization: Assign priorities to jobs within a queue
bun add @elysiumjs/heraclesimport { Job } from '@elysiumjs/heracles';
@Job.register({
queue: 'emails',
maxRetries: 3,
retryDelay: 5000
})
export class SendEmailJob extends Job {
constructor(
private readonly email: string,
private readonly subject: string,
private readonly body: string
) {
super();
}
protected async execute(): Promise<void> {
// Implementation for sending an email
this.info(`Sending email to ${this.email}: ${this.subject}`);
// Simulate email sending
await Bun.sleep(1000);
this.success(`Email sent successfully to ${this.email}`);
}
}import { Heracles, Queue, RedisTransport } from '@elysiumjs/heracles';
import { SendEmailJob } from './jobs/send-email.job';
// Configure the queue with Redis transport
const emailQueue = Queue.get('emails', {
transport: RedisTransport,
transportOptions: {
connection: 'default'
}
});
// Dispatch a job
await Heracles.dispatch(SendEmailJob, ['user@example.com', 'Welcome!', 'Welcome to our platform!']);
// Dispatch with options
await Heracles.dispatch(
SendEmailJob,
['user@example.com', 'Welcome!', 'Welcome to our platform!'],
{
priority: 5, // Higher priority (lower number)
scheduledFor: new Date(Date.now() + 60000), // Run after 1 minute
maxRetries: 5 // Override max retries
}
);You can use styx to launch Heracles workers through the CLI:
# Start a Redis worker for the 'emails' queue with concurrency of 3 jobs
bun styx heracles:work --id worker-1 --queues emails --concurrency 3 --redis default
# Start workers for multiple queues
bun styx heracles:work --id worker-2 --queues emails,notifications --redis defaultOr programmatically:
import { RedisWorker } from '@elysiumjs/heracles';
// Create and start a Redis worker
const worker = new RedisWorker('default', { id: 'worker-1' });
await worker.createQueue({ name: 'emails', concurrency: 3 });
await worker.start();const queue = Queue.get('emails', {
// Maximum number of concurrent jobs to process in this queue
concurrency: 5,
// Maximum number of retries for failed jobs
maxRetries: 3,
// Delay in milliseconds between retries
retryDelay: 5000,
// Whether to pause processing when an error occurs
pauseOnError: false,
// Redis transport configuration
transport: RedisTransport,
// Transport-specific options
transportOptions: {
connection: 'default'
}
});Heracles has a bundled Redis transport layer for job distribution, enabling distributed processing across multiple servers or processes. It uses Redis Streams for reliable message delivery and job coordination.
First, configure a Redis connection in your Elysium.js application:
import { Redis } from '@elysiumjs/core';
// Register a Redis connection
Redis.registerConnection('default', {
url: 'redis://localhost:6379'
// Other Redis options
});Or during the Application registration:
import { Application } from '@elysiumjs/core';
@Application.register({
redis: {
default: 'jobs',
connections: {
jobs: { url: 'redis://localhost:6379' }
}
}
// ...other configurations...
})
export class App extends Application {}import { Queue, RedisTransport } from '@elysiumjs/heracles';
const queue = Queue.get('emails', {
transport: RedisTransport,
transportOptions: {
connection: 'default', // Redis connection name
keyPrefix: 'myapp:jobs', // Prefix for Redis keys (default: 'elysium:heracles')
consumerGroup: 'workers', // Redis consumer group name (default: 'workers')
statusTTL: 86400, // Job status TTL in seconds (default: 86400 - 24 hours)
cleanupCompletedJobs: true, // Auto-cleanup completed jobs (default: true)
completedJobRetention: 3600, // Time to retain completed jobs in seconds (default: 3600 - 1 hour)
maxStreamSize: 1000 // Maximum stream size (default: 1000)
}
});Configure workers to process jobs from Redis queues:
import { RedisWorker } from '@elysiumjs/heracles';
// Create a Redis worker with a unique ID
const worker = new RedisWorker('default', {
id: 'worker-1', // Unique worker ID
consumerGroup: 'workers', // Redis consumer group name (default: 'workers')
consumerName: 'worker-1', // Unique consumer name (optional)
keyPrefix: 'myapp:jobs', // Use same prefix as queue configuration
pollInterval: 1000, // Poll interval in ms (default: 1000)
batchSize: 10 // Batch size for processing jobs (default: 10)
});
// Configure queue settings
await worker.createQueue({
name: 'emails',
concurrency: 5, // Process 5 jobs concurrently
maxRetries: 3, // Retry failed jobs up to 3 times
retryDelay: 5000, // Wait 5 seconds between retries
pauseOnError: false // Don't pause queue on error
});
// Start the worker
await worker.start();Jobs in Heracles go through several statuses:
- PENDING: Job is queued but not yet started
- RUNNING: Job is currently being processed
- COMPLETED: Job has successfully completed
- FAILED: Job has failed (may be retried if retries are available)
- CANCELLED: Job was cancelled and won't be processed
- SCHEDULED_FOR_RETRY: Job has failed and is scheduled for retry
// Cancel a specific job
const jobId = await Heracles.dispatch(HelloWorldJob);
const cancelled = await queue.cancelJob(jobId);
// Cancel all jobs in a queue
const cancelledCount = await queue.cancelAllJobs();// Pause a queue (stop processing new jobs)
await queue.pause();
// Resume a paused queue
await queue.resume();
// Drain a queue (complete current jobs but don't accept new ones)
await queue.drain();
// Clear all jobs from a queue
await queue.clear();Heracles commands are available via the styx CLI tool:
Usage: styx heracles:work [options]
Description: Start an Heracles worker for specific queues.
Arguments:
--id <string> (required)
Unique identifier for this worker
--queues <array of strings> (required) [default: default]
List of queues to work on
--concurrency <number> (required) [default: 1]
Number of concurrent jobs to process
--redis <string> (required) [default: default]
Name of the Redis connection to use
--max-retries <number> (optional) [default: 3]
Maximum retries for failed jobs before giving up
--retry-delay <number> (optional) [default: 5000]
Delay between retries in milliseconds
--pause-on-error <boolean> (optional) [default: false]
Pause the worker when an error occurs
--verbose <boolean> (optional) [default: false]
Enable verbose logging
Usage: styx heracles:clean [options]
Description: Clean up Redis streams by removing completed and failed jobs.
Arguments:
--redis <string> (required) [default: default]
Name of the Redis connection to use
--queues <array of strings> (optional) [default: ]
List of queues to clean. If empty, all queues will be cleaned
--dry-run <boolean> (optional) [default: false]
Show what would be removed without actually removing
--verbose <boolean> (optional) [default: false]
Enable verbose logging
--all <boolean> (optional) [default: false]
Remove all entries, not only completed and failed jobs
--key-prefix <string> (optional) [default: elysium:heracles]
Redis key prefix to clean
--retention <number> (optional) [default: 3600]
Number of seconds to retain completed jobs
--max <number> (optional) [default: 1000]
Maximum number of jobs to clean
Apache-2.0