Learning objectives
After reading this article you will be able to:
- Tell cron jobs, workers and queues apart and what each one solves
- Pick the right tool for a scheduled, event-driven or heavy task
- Build a minimal queue with a retrying worker in Node.js
Sooner or later some task is too slow to finish before the user gets a response: sending an email, resizing an image, calling a flaky third-party API. Other tasks have no request at all, like a nightly cleanup. Three tools cover this ground, and people often mix them up: cron jobs, background workers and queues.
They are not rivals. They answer different questions: when does work start, who does it, and where does it wait.
What is each one?#
A cron job starts a task at times set by a schedule. Nothing triggers it except the clock. See what a cron job is for the syntax.
A background worker is a long-running process that does work outside the request cycle. It is defined by where it runs, not by how it gets work. A worker can poll a database, listen to a queue or run a loop.
A queue is a buffer of jobs. A producer (your web app) adds a job and moves on. A worker takes it later. Queues usually add retries, delays, priorities and concurrency limits.
How do they differ side by side?#
| Cron job | Background worker | Queue | |
|---|---|---|---|
| Starts because of | The clock | Whatever you code | A job was added |
| Reacts to events | No | Yes, if it listens | Yes |
| Retries on failure | No, unless you code it | Only if you code it | Usually built in |
| Survives a restart mid-task | No | No | Pending jobs survive, since they are stored |
| Handles bursts | Poorly | Limited by one process | Well: jobs wait their turn |
| Extra infrastructure | None | A process | A broker (for example Redis) |
The last two rows matter most. A queue decouples producing work from doing it, so a burst of 1,000 jobs does not need 1,000 simultaneous processes.
When should you use cron?#
Use cron when the trigger is time and the task is self-contained: nightly cleanups, daily reports, token refreshes, syncing data every hour. It needs no broker and no extra moving parts.
Cron falls short when a run fails and must be retried, when a run lasts longer than the interval, or when each run has to fan out into many pieces. An in-app scheduler also fires once per instance, so if you run two copies of your app, both fire the job.
When should you use a queue and a worker?#
Use a queue when something happened and work must follow: a user signed up (send a welcome email), a file was uploaded (generate thumbnails), a payment was confirmed (issue the receipt). The request returns quickly, and the worker finishes the job.
Choose a queue when you need any of these:
- Retries with backoff for flaky dependencies.
- Rate control, for example no more than 5 calls per second to an API.
- Scaling the work, by starting more worker processes.
- No lost work when a deploy or crash interrupts the process, because pending jobs live in the broker.
A job can run more than once, for example when a worker crashes mid-job (BullMQ describes at-least-once as its worst case). That is why jobs should be idempotent: running the same job twice must not charge a customer twice.
What does a minimal queue look like in Node.js?#
BullMQ is a Node.js queue library built on Redis. Install it with npm install bullmq. The producer adds jobs:
// producer.js (needs "type": "module" in package.json)
import { Queue } from "bullmq";
const connection = {
host: process.env.REDIS_HOST,
port: Number(process.env.REDIS_PORT),
password: process.env.REDIS_PASSWORD,
};
export const emailQueue = new Queue("emails", {
connection,
defaultJobOptions: {
attempts: 3,
backoff: { type: "exponential", delay: 1000 },
},
});
await emailQueue.add("welcome", { to: "ana@example.com" });The worker is a separate process that consumes them:
// worker.js
import { Worker } from "bullmq";
const connection = {
host: process.env.REDIS_HOST,
port: Number(process.env.REDIS_PORT),
password: process.env.REDIS_PASSWORD,
};
const worker = new Worker(
"emails",
async (job) => {
// throw an error here and BullMQ retries the job
console.log("sending welcome email to", job.data.to);
},
{ connection, concurrency: 5 },
);
worker.on("failed", (job, err) => {
console.error(`job ${job?.id} failed:`, err.message);
});With attempts: 3 and an exponential backoff of 1,000 ms, a failing job is retried after about 1 s and 2 s before it is marked failed. concurrency: 5 lets one worker process five jobs at once.
Two Redis details matter in production, according to the BullMQ docs: set maxmemory-policy to noeviction so Redis never silently drops queue keys, and keep maxRetriesPerRequest as null on worker connections (the default for Worker).
Can you combine them?#
Yes, and most mature apps do. Cron is a good trigger, a queue is a good buffer, and a worker is the doer. BullMQ's job schedulers add jobs to a queue on a cron pattern:
await emailQueue.upsertJobScheduler(
"daily-digest",
{ pattern: "0 0 8 * * *", tz: "America/New_York" },
{ name: "digest", data: {} },
);The pattern has six fields, with seconds first, as in the BullMQ docs examples (this one means 08:00:00 every day). Now the schedule lives in Redis, not in one process's memory, and the work gets retries and concurrency control like any other job.
How do you choose?#
- A short, harmless task on a clock: cron, inside your app or on the system.
- Work caused by an event, or work that can fail and must retry: queue plus worker.
- A scheduled task that is heavy or must not be lost: cron trigger feeding a queue.
- A task that is simply a long loop, such as a consumer for a stream: a worker on its own.
Start with the smallest option. Moving from cron to a queue later is easy if the job logic lives in a function that does not care who calls it. For the Redis side, see what Redis is and how to host a Node.js application.
How to do this on Square Cloud#
Workers and schedulers need processes that stay up. On Square Cloud, containers are always on, with no sleep mode, so a worker or an in-app scheduler keeps running. With automatic restart on (the default for dashboard deploys), the app comes back by itself when the process exits with an error.
For a queue, deploy the worker as its own application next to your web app. The plan's RAM is a pool shared across your applications, with a minimum of 256 MB per background application and 512 MB per web application. Redis hosting is available from the Standard plan, in the same datacenter as your apps, with TLS connections and persistence. The dashboard hands you the connection URL and the database certificate. Put the connection data in environment variables. Since TLS is mandatory and uses its own certificate, add a tls option with the downloaded CA to the connection object in the examples above (for example tls: { ca: fs.readFileSync("ca.crt") }).
Deploy from the dashboard, the CLI or GitHub. The squarecloud.app configuration file is optional: the dashboard generates one on deploy, with runtime, memory and main file. See the options for hosting APIs to get started.
API hosting
APIs in always-on containers, with no cold starts
FAQs
Is a queue the same thing as a worker?
No. The queue stores jobs waiting to be done. The worker is the process that takes jobs from it and runs them. You need both for a queue to be useful.
Can I use cron instead of a queue?
Only for work that depends on time. If a task starts because something happened, such as a payment or an upload, a queue reacts at once and retries failures, which cron does not.
Do queues need Redis?
Not necessarily: brokers like RabbitMQ and Amazon SQS work without Redis. Libraries such as BullMQ for Node.js, on the other hand, store their jobs in Redis and so need a Redis server.
How do I run a recurring job with a queue?
BullMQ has job schedulers that add a job to the queue on a cron pattern or fixed interval. The schedule lives in Redis, and any worker can process the job.
More in APIs
See all 15 articles- How to host an APIHosting an API means running its HTTP server around the clock at a public address with HTTPS. Step-by-step with Node.js and Python, testing and common errors.
- How to host a Node.js appTo host a Node.js app, prepare package.json, read the port and secrets from the environment, pick an LTS version and run it with node. Step-by-step guide.
- How to host a FastAPI, Flask or Django appTo host a FastAPI, Flask or Django app, run it with a production server (Uvicorn or Gunicorn) on 0.0.0.0. Commands, Django settings and common errors included.
- How to host a Python app or scriptTo host a Python app or script, pin the version, isolate dependencies in a venv, read secrets from the environment and handle SIGTERM. Step-by-step guide.