Dragonfly

Use Dragonfly as a high-performance Redis alternative for BullMQ

Dragonfly offers a drop-in replacement for Redis, boasting a much faster and more memory-efficient implementation of several data structures used by BullMQ. It also enables the utilization of all available cores in your CPUs.

Check out this performance article for detailed benchmark results comparing Dragonfly with Redis.

Performance Benefits

Dragonfly provides several advantages over standard Redis:

• Multi-core utilization - Uses all available CPU cores for better performance
• Memory efficiency - More efficient memory usage for data structures
• Faster operations - Optimized implementation of Redis commands
• Drop-in replacement - Compatible with existing Redis clients

Configuration for BullMQ

To fully leverage Dragonfly’s capabilities with BullMQ, you need to configure your queues properly.

Queue Naming Convention

1. Use curly braces in queue names

   Name your queues using curly braces to allow Dragonfly to assign a thread to each queue.

   // Instead of:
   const queue = new Queue('myqueue');
   
   // Use:
   const queue = new Queue('{myqueue}');
   
2. Distribute queues across cores

   For multiple queues, this approach enables you to allocate different CPU cores to each queue.

   const queue1 = new Queue('{orders}');
   const queue2 = new Queue('{notifications}');
   const queue3 = new Queue('{analytics}');
   
3. Split single queue for multi-core processing (optional)

   Even with a single queue, you can exploit multi-core advantages by splitting it into multiple queues.

   const queue1 = new Queue('{myqueue-1}');
   const queue2 = new Queue('{myqueue-2}');
   const queue3 = new Queue('{myqueue-3}');
   
   // Distribute jobs using round-robin or random distribution
   const queueIndex = Math.floor(Math.random() * 3) + 1;
   const selectedQueue = eval(`queue${queueIndex}`);
   await selectedQueue.add('job-name', data);
   

Important Limitations

Be aware that certain features might not function across multiple queues:

• Priorities - Job priorities don’t work across separate queues
• Rate limiting - Rate limits are per-queue, not global
• Job ordering - Global FIFO ordering is lost when splitting queues

Your specific requirements will determine whether you can divide a single queue in this manner.

Installation and Setup

For comprehensive instructions and the necessary flags to optimize your Dragonfly instance for BullMQ, please consult the official Dragonfly integration guide.

Example Configuration

import { Queue, Worker } from 'bullmq';
import { createClient } from 'redis';

// Create connection to Dragonfly
const connection = {
  host: 'localhost',
  port: 6379,
};

// Use curly braces in queue name
const queue = new Queue('{myqueue}', { connection });

const worker = new Worker(
  '{myqueue}',
  async (job) => {
    console.log('Processing job:', job.id);
    // Process job
  },
  { connection }
);

Related Resources

Redis Compatibility

Learn about Redis compatibility requirements

Going to Production

Production deployment best practices