Idempotent Jobs

Design jobs that can be safely retried without side effects

In order to take advantage of the ability to retry failed jobs, your jobs should be designed with failure in mind.

What is Idempotence?

Idempotence means that it should not make a difference to the final state of the system if a job successfully completes on its first attempt, or if it fails initially and succeeds when retried.

Design Principles

To achieve idempotent behavior, your jobs should be:

1. Atomic

   Jobs should be as atomic as possible, performing a single clear action

2. Simple

   Avoid performing many different actions (database updates, API calls, etc.) at once

3. Trackable

   Make it easy to track the process flow and rollback partial progress when an exception occurs

Benefits

Easier Debugging

Simpler jobs mean simpler debugging and troubleshooting

Better Performance

Easier to identify and optimize bottlenecks

Reliable Retries

Jobs can be safely retried without causing duplicate side effects

Better Monitoring

Clearer metrics and observability

Complex Workflows

If necessary, split complex jobs using the flow pattern to create parent-child job dependencies.

Examples of Idempotent Operations

// ✅ Good: Idempotent update
await db.users.update(
  { id: userId },
  { status: 'active', updatedAt: new Date() }
);

// ✅ Good: Idempotent API call with idempotency key
await stripe.charges.create({
  amount: 1000,
  currency: 'usd',
  source: 'tok_visa',
  idempotency_key: jobId, // Use job ID as idempotency key
});

Examples of Non-Idempotent Operations

// ❌ Bad: Non-idempotent increment
await db.users.update(
  { id: userId },
  { $inc: { loginCount: 1 } }
);
// If retried, this will increment multiple times!

// ❌ Bad: Appending to array without checking
await db.users.update(
  { id: userId },
  { $push: { notifications: notification } }
);
// If retried, this will add duplicate notifications!

Related Resources

Retrying Failed Jobs

Learn about retry strategies and backoff

Flows

Split complex work into smaller jobs