Auto-Removal of Jobs

Configure automatic removal of completed and failed jobs in BullMQ

By default, when jobs complete or fail, BullMQ stores them in special sets (“completed” and “failed”). While this is useful for debugging and examining results, it can fill Redis with data over time. BullMQ provides several strategies to automatically remove finalized jobs.

Overview

Auto-removal is configured using the removeOnComplete and removeOnFail options on individual jobs. These options can be set:

• When adding a job via queue.add()
• As default options on the queue instance
• As default options on the worker instance

Auto-removal works lazily. Jobs are not removed immediately, but when a new job completes or fails, triggering the cleanup process.

Remove All Finalized Jobs

The simplest option is to set removeOnComplete or removeOnFail to true. Jobs will be removed automatically as soon as they finalize:

TypeScript

import { Queue } from 'bullmq';

const queue = new Queue('tasks');

await queue.add(
  'task',
  { data: 'test' },
  { 
    removeOnComplete: true, 
    removeOnFail: true 
  }
);

const { Queue } = require('bullmq');

const queue = new Queue('tasks');

await queue.add(
  'task',
  { data: 'test' },
  { 
    removeOnComplete: true, 
    removeOnFail: true 
  }
);

Jobs will be deleted regardless of their names when using true.

Keep a Certain Number of Jobs

Specify a maximum number of jobs to keep. A good practice is to keep a handful of completed jobs and a larger number of failed jobs for debugging:

TypeScript

await queue.add(
  'task',
  { data: 'test' },
  { 
    removeOnComplete: 1000,  // Keep last 1000 completed jobs
    removeOnFail: 5000       // Keep last 5000 failed jobs
  }
);

await queue.add(
  'task',
  { data: 'test' },
  { 
    removeOnComplete: 1000,  // Keep last 1000 completed jobs
    removeOnFail: 5000       // Keep last 5000 failed jobs
  }
);

Keep Jobs Based on Age

Use the KeepJobs object to specify how long to keep jobs. You can combine age limits with count limits:

KeepJobs Options

age · number

Maximum age in seconds for jobs to be kept

count · number

Maximum count of jobs to be kept (used as a safeguard)

limit · number

Maximum quantity of jobs to be removed per cleanup operation

Example: Age-Based Removal

TypeScript

await queue.add(
  'task',
  { data: 'test' },
  {
    removeOnComplete: {
      age: 3600,    // Keep for up to 1 hour
      count: 1000   // But never more than 1000 jobs
    },
    removeOnFail: {
      age: 86400    // Keep failed jobs for up to 24 hours
    }
  }
);

await queue.add(
  'task',
  { data: 'test' },
  {
    removeOnComplete: {
      age: 3600,    // Keep for up to 1 hour
      count: 1000   // But never more than 1000 jobs
    },
    removeOnFail: {
      age: 86400    // Keep failed jobs for up to 24 hours
    }
  }
);

The count field acts as a safeguard: if you receive an unexpected burst of jobs, the count limit prevents memory issues by capping the total number of jobs kept.

Default Options on Queue

Set default removal options for all jobs added to a queue:

import { Queue } from 'bullmq';

const queue = new Queue('tasks', {
  defaultJobOptions: {
    removeOnComplete: {
      age: 3600,
      count: 1000
    },
    removeOnFail: {
      age: 86400,
      count: 5000
    }
  }
});

// This job will inherit the default removal options
await queue.add('task', { data: 'test' });

// Override default options for specific jobs
await queue.add('important-task', { data: 'test' }, {
  removeOnComplete: false  // Keep this one
});

Default Options on Worker

Workers can also define default removal options that override job-level settings:

import { Worker } from 'bullmq';

const worker = new Worker(
  'tasks',
  async job => {
    // Process job
  },
  {
    removeOnComplete: {
      age: 3600,
      count: 100
    },
    removeOnFail: {
      age: 86400
    }
  }
);

Worker-level settings override job-level settings. This is useful for enforcing consistent retention policies across all jobs.

Idempotence Considerations

One strategy for implementing idempotence with BullMQ is using unique job IDs. When you add a job with an ID that already exists, the new job is ignored and a duplicated event is triggered.

When using auto-removal with unique job IDs, be aware that a removed job is no longer considered part of the queue. Future jobs with the same ID will not be treated as duplicates.

Example: Idempotent Jobs with Auto-Removal

// Add a job with a unique ID
await queue.add(
  'process-user',
  { userId: 123 },
  { 
    jobId: 'user-123-sync',
    removeOnComplete: false  // Keep to maintain idempotence
  }
);

// Later, adding the same job ID will be ignored
await queue.add(
  'process-user',
  { userId: 123 },
  { jobId: 'user-123-sync' }
);
// This will trigger a 'duplicated' event

Practical Examples

Example 1: High-Volume Queue

For queues processing millions of jobs, aggressive cleanup is important:

const queue = new Queue('high-volume', {
  defaultJobOptions: {
    removeOnComplete: true,     // Remove immediately
    removeOnFail: {
      age: 3600,                // Keep failures for 1 hour
      count: 10000              // Max 10k failed jobs
    }
  }
});

Example 2: Critical Jobs

For important jobs, keep results longer:

const queue = new Queue('payments');

await queue.add(
  'process-payment',
  { amount: 100, userId: 123 },
  {
    removeOnComplete: {
      age: 2592000,  // 30 days
      count: 100000
    },
    removeOnFail: false  // Never auto-remove failures
  }
);

Example 3: Development vs Production

const isProduction = process.env.NODE_ENV === 'production';

const queue = new Queue('tasks', {
  defaultJobOptions: {
    removeOnComplete: isProduction 
      ? { age: 3600, count: 1000 }
      : false,  // Keep all in development
    removeOnFail: isProduction
      ? { age: 86400, count: 5000 }
      : false   // Keep all in development
  }
});

Manual Cleanup

For manual cleanup operations, see the clean method.

Related

• Adding Jobs
• Removing Jobs
• Queue API Reference
• KeepJobs API Reference
• Duplicated Event Reference