AWS MemoryDB

Configure AWS MemoryDB for Redis as a managed database for BullMQ

AWS MemoryDB provides a Redis 7 compatible managed database that is fully compatible with BullMQ. It offers durability, high availability, and automatic failover.

Key Considerations

When using AWS MemoryDB with BullMQ, keep these important points in mind:

• Cluster mode only - MemoryDB only works in Cluster mode, so you need to use “hash tags” to ensure queues are attached to a specific cluster node
• VPC access only - MemoryDB can only be accessed within an AWS VPC; you cannot access the Redis cluster from outside AWS

Setup Guide

1. Install IORedis Cluster support

   Install the IORedis library which provides cluster support:

   npm install ioredis
   
2. Create a Cluster connection

   Instantiate an IORedis Cluster instance with your MemoryDB endpoint:

   import { Cluster } from 'ioredis';
   
   const connection = new Cluster(
     [
       {
         host: 'clustercfg.xxx.amazonaws.com',
         port: 6379,
       },
     ],
     {
       tls: {},
     },
   );
   
3. Use the connection with BullMQ

   Pass the cluster connection to your Queue and Worker instances:

   import { Queue, Worker, Job } from 'bullmq';
   
   // Create queue with cluster connection
   const queue = new Queue('myqueue', { connection });
   
   // Create worker with cluster connection
   const worker = new Worker(
     'myqueue',
     async (job: Job) => {
       // Process your job
       console.log('Processing job:', job.id);
     },
     { connection },
   );
   
4. Graceful shutdown

   Always close both the worker and connection when shutting down:

   // Graceful shutdown
   process.on('SIGTERM', async () => {
     await worker.close();
     await connection.quit();
     process.exit(0);
   });
   

Complete Example

import { Cluster } from 'ioredis';
import { Worker, Queue, Job } from 'bullmq';

// Create cluster connection
const connection = new Cluster(
  [
    {
      host: 'clustercfg.xxx.amazonaws.com',
      port: 6379,
    },
  ],
  {
    tls: {},
  },
);

// Create queue
const queue = new Queue('myqueue', { connection });

// Create worker
const worker = new Worker(
  'myqueue',
  async (job: Job) => {
    // Do some useful work
    console.log('Processing:', job.data);
  },
  { connection },
);

// Handle shutdown
const shutdown = async () => {
  await worker.close();
  await connection.quit();
  process.exit(0);
};

process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);

Hash Tags for Cluster Mode

MemoryDB uses cluster mode, which means you should use hash tags in your queue names to ensure all keys for a queue are on the same cluster node. Learn more about Redis Cluster and hash tags.

// Use hash tags in queue names
const queue = new Queue('{myqueue}', { connection });
const worker = new Worker('{myqueue}', processor, { connection });

Best Practices

Enable TLS

Always use TLS for secure connections to MemoryDB

Use VPC Peering

Set up VPC peering if accessing from different VPCs

Monitor Performance

Use CloudWatch metrics to monitor your MemoryDB cluster

Configure Security Groups

Properly configure security groups for access control

Related Resources

AWS ElastiCache

Alternative AWS managed Redis option

Going to Production

Production deployment best practices