StackPilots | Background Jobs in a MEAN App: BullMQ, Redis and Express 5 on Node 24

Almost every MEAN application we inherit has the same scar: a route that does too much. A CSV import that parses 40,000 rows inside the request. A PDF export that holds a connection open for ninety seconds. A signup handler that awaits an SMTP server nobody controls. The symptom is a gateway timeout; the cause is that the work was never asynchronous in the first place.

This tutorial moves that work into a background worker using BullMQ on Redis, alongside an Express 5 API, Mongoose 8 and an Angular 22 client. Versions are current as of August 2026: Node.js 24 LTS, BullMQ 5.x, Redis 7.4 or Valkey 8, Express 5, MongoDB 8.0.

What you end up with

An Express 5 endpoint that validates input, enqueues a job and returns 202 Accepted in milliseconds.
A separate worker process that does the slow work, with retries, backoff and concurrency limits.
A job record in MongoDB so status survives Redis being flushed.
A repeatable (cron-style) job for nightly work.
Graceful shutdown so a deploy never kills a half-finished job.
An Angular 22 view that polls job status and shows progress.

Step 0: Redis

BullMQ needs Redis (or Valkey) — not MongoDB. Locally:

docker run -d --name redis -p 6379:6379 redis:7.4-alpine \
  redis-server --appendonly yes --maxmemory-policy noeviction

The eviction policy matters. If Redis is allowed to evict keys under memory pressure, BullMQ will silently lose jobs. noeviction is the only safe setting for a queue instance, and a queue should not share a Redis instance with your cache.

npm install bullmq ioredis

Step 1: one shared connection and one queue

api/src/queue.js:

import { Queue } from 'bullmq';
import IORedis from 'ioredis';

export const connection = new IORedis(process.env.REDIS_URL ?? 'redis://localhost:6379', {
  maxRetriesPerRequest: null, // required by BullMQ workers
});

export const reportsQueue = new Queue('reports', {
  connection,
  defaultJobOptions: {
    attempts: 5,
    backoff: { type: 'exponential', delay: 2000 },
    removeOnComplete: { age: 3600, count: 1000 },
    removeOnFail: { age: 24 * 3600 },
  },
});

Two details people skip. maxRetriesPerRequest: null is mandatory for the blocking commands BullMQ workers use. And without removeOnComplete/removeOnFail, finished jobs accumulate in Redis forever; we have seen a 12 GB queue database whose owner thought BullMQ cleaned up on its own.

Step 2: enqueue from Express 5, return immediately

import { Router } from 'express';
import { z } from 'zod';
import { randomUUID } from 'node:crypto';
import { reportsQueue } from './queue.js';
import { JobRecord } from './models.js';

export const reports = Router();

const Body = z.object({
  from: z.coerce.date(),
  to: z.coerce.date(),
  format: z.enum(['csv', 'pdf']).default('csv'),
});

reports.post('/reports', requireAuth, async (req, res) => {
  const input = Body.parse(req.body);
  const jobId = randomUUID();

  await JobRecord.create({
    _id: jobId,
    owner: req.user.sub,
    type: 'reports:build',
    state: 'queued',
    progress: 0,
  });

  await reportsQueue.add('build', { jobId, owner: req.user.sub, ...input }, { jobId });

  res.status(202)
     .location(`/api/jobs/${jobId}`)
     .json({ jobId, state: 'queued' });
});

The payload is small on purpose: identifiers and parameters, never a 20 MB buffer. Redis is a coordination layer, not a file store. Upload large inputs to object storage first and pass the key.

Because we pass an explicit jobId, a duplicate submit (double-clicked button, retried mobile request) is deduplicated by BullMQ instead of producing two exports. If the natural idempotency key is business data rather than a UUID, hash it: createHash('sha256').update(${owner}:${from}:${to}:${format}).digest('hex').

Step 3: the worker process

Run the worker as its own process — a separate container, not a thread inside the API. That way CPU-heavy jobs cannot starve the event loop that serves HTTP.

worker/src/index.js:

import { Worker, UnrecoverableError } from 'bullmq';
import { connection } from '../../api/src/queue.js';
import { connectDb } from '../../api/src/db.js';
import { JobRecord } from '../../api/src/models.js';
import { buildReport } from './build-report.js';

await connectDb();

const worker = new Worker('reports', async (job) => {
  await JobRecord.updateOne({ _id: job.data.jobId }, { state: 'running', startedAt: new Date() });

  const result = await buildReport(job.data, async (pct) => {
    await job.updateProgress(pct);
    await JobRecord.updateOne({ _id: job.data.jobId }, { progress: pct });
  });

  return result; // stored as job.returnvalue
}, {
  connection,
  concurrency: 5,
  limiter: { max: 20, duration: 1000 }, // protect downstream services
});

worker.on('completed', async (job, result) => {
  await JobRecord.updateOne({ _id: job.data.jobId },
    { state: 'succeeded', progress: 100, finishedAt: new Date(), result });
});

worker.on('failed', async (job, err) => {
  const exhausted = !job || job.attemptsMade >= (job.opts.attempts ?? 1);
  if (job) {
    await JobRecord.updateOne({ _id: job.data.jobId }, {
      state: exhausted ? 'failed' : 'retrying',
      error: err.message,
      attempts: job.attemptsMade,
    });
  }
  console.error({ jobId: job?.id, attempt: job?.attemptsMade, err: err.message }, 'job failed');
});

Retry only what is worth retrying

Exponential backoff is right for a timed-out HTTP call or a transient Mongo write. It is wrong for format: 'xlsx' when you only support CSV — that will fail five times and pollute your dashboards. Throw UnrecoverableError for anything that will never succeed:

if (!SUPPORTED.has(job.data.format)) {
  throw new UnrecoverableError(`unsupported format ${job.data.format}`);
}

Make handlers idempotent

A worker can be killed mid-job; BullMQ will hand the job to another worker after the lock expires. Assume every handler may run twice. Write results with an upsert keyed on jobId, use $inc rather than read-modify-write, and only send an email after recording that you are about to (findOneAndUpdate({ _id, emailSentAt: null }, { emailSentAt: new Date() }) returning null means someone else already sent it).

Long jobs and stalled locks

A job holds a lock for lockDuration (30 s by default) and the worker renews it while the handler runs. A handler that blocks the event loop — a synchronous 90-second PDF render — cannot renew, so the job is declared stalled and re-run. Either chunk the work with await points between batches, or move the CPU-bound part into a worker_threads pool. Node 24 makes that cheap: new Worker(new URL('./render.js', import.meta.url)) with a small pool in front.

Step 4: scheduled work without a cron container

await reportsQueue.upsertJobScheduler(
  'nightly-rollup',
  { pattern: '0 3 * * *', tz: 'Europe/London' },
  { name: 'rollup', data: { scope: 'all' } },
);

upsertJobScheduler (BullMQ 5.x, replacing the older repeatable-job API) is idempotent, so calling it on every boot of every API replica creates exactly one schedule. Always set tz explicitly — the container clock is UTC and "3am" almost never means UTC to the person who asked for it.

Step 5: exposing status to Angular

app.get('/api/jobs/:id', requireAuth, async (req, res) => {
  const record = await JobRecord.findOne({ _id: req.params.id, owner: req.user.sub }).lean();
  if (!record) return res.status(404).json({ error: 'not found' });
  res.set('Cache-Control', 'no-store');
  res.json(record);
});

Read status from MongoDB, not from Redis. Job data is trimmed by removeOnComplete, and you do not want an authenticated user's dashboard depending on a queue key that expired an hour ago. The owner filter in the query is the authorization check.

On the Angular 22 side, poll with a signal-friendly resource until the job settles:

@Component({ /* ... */ })
export class ReportStatusComponent {
  private http = inject(HttpClient);
  jobId = input.required<string>();
  private tick = signal(0);

  job = resource({
    params: () => ({ id: this.jobId(), t: this.tick() }),
    loader: ({ params }) =>
      firstValueFrom(this.http.get<JobRecord>(`/api/jobs/${params.id}`)),
  });

  constructor() {
    effect((onCleanup) => {
      const state = this.job.value()?.state;
      if (state === 'succeeded' || state === 'failed') return;
      const t = setTimeout(() => this.tick.update((n) => n + 1), 2000);
      onCleanup(() => clearTimeout(t));
    });
  }
}

Polling every two seconds is fine for a handful of jobs. If you need instant updates for many users, push instead — a MongoDB change stream on the job collection feeding an SSE endpoint, which is exactly the pattern in our real-time MEAN tutorial.

Step 6: graceful shutdown

This is the step that turns a deploy from a source of corrupt half-written exports into a non-event:

let closing = false;
for (const signal of ['SIGTERM', 'SIGINT']) {
  process.on(signal, async () => {
    if (closing) return;
    closing = true;
    await worker.close();      // stop taking new jobs, finish the current one
    await connection.quit();
    await mongoose.disconnect();
    process.exit(0);
  });
}

worker.close() waits for in-flight jobs. Give the container room to do it: in Kubernetes set terminationGracePeriodSeconds comfortably above your longest job, or cap job duration so the two numbers are compatible. A worker SIGKILLed at 30 seconds every deploy will look like random job failures for months.

Operating it

Dead-letter visibility. Failed jobs stay in the failed set until removeOnFail expires. Alert on failed-set depth, not just on error logs.
A UI. Mount bull-board behind admin auth, or query counts with await reportsQueue.getJobCounts() and export them to Prometheus.
Tracing. Propagate the incoming traceparent header in the job payload and start the worker span as its child, so one trace covers request plus background work — see our OpenTelemetry tracing tutorial for the instrumentation setup.
Backpressure. concurrency bounds parallel jobs per worker; limiter bounds throughput against a fragile downstream API. Scale by adding worker replicas, and size your Mongo connection pool accordingly (maxPoolSize per worker × replicas must stay under the server limit).
Separate queues per workload. One slow queue should not block a fast one. Nightly rollups and password-reset emails do not belong in the same queue.

Common failure modes we get called about

Symptom	Usual cause
Jobs vanish under load	Redis with an eviction policy set, or shared with the cache
Every job runs twice	Handler not idempotent, plus stalled locks from blocking code
Queue grows forever	No `removeOnComplete`, or workers scaled to zero by an autoscaler that only watches HTTP
Nightly job fires at the wrong hour	No `tz` on the scheduler pattern
Deploys corrupt exports	No `worker.close()` on SIGTERM

Moving work off the request path is usually the cheapest performance win available in a MEAN application — bigger than most query tuning, and it changes the user experience from "the browser hangs" to "we will email you when it is ready".

If your API is timing out on work that should have been a job, we can help. See Performance Tuning in MEAN Stack and Enterprise MEAN Stack App Development, or just get in touch with a description of what is slow.