On AnyTeam, an AI-native B2B sales platform where I worked as a backend engineer, hundreds of tenants each needed recurring work done for them: webhook renewals, account research refreshes, email digests. These often ran on the user's local timezone or on a schedule the user could change.
The first approach: a cron job per user
At first I created individual cron jobs for each user, 20+ of them per user. That ran smoothly until the number of users grew:
- Thousands of individual jobs ran into the cloud scheduler's service limits.
- Every trigger is billed, so cost grew with the number of users, not with the amount of work.
- Debugging meant going through hundreds of schedules, each one a separate piece of infrastructure.
The fix: one tick, and the database decides
The schedule moved into the database. Each job is a row with a next_run_at, a job_type, a payload and a status. A single trigger, GCP Cloud Scheduler, fires once a minute and asks one question: what is due now?
Claiming due jobs safely
Several dispatchers can run at the same time, so two of them must never claim the same job. PostgreSQL's FOR UPDATE SKIP LOCKED takes care of that. Each dispatcher locks the rows it selects, and every other dispatcher skips those locked rows instead of waiting for them.
UPDATE scheduled_job
SET status = 'QUEUED'
WHERE id IN (
SELECT id
FROM scheduled_job
WHERE status = 'PENDING'
AND next_run_at <= now()
ORDER BY next_run_at
LIMIT 500
FOR UPDATE SKIP LOCKED
)
RETURNING id, tenant_id, job_type, payload;The dispatcher doesn't do the work
If the tick ran the jobs itself, the whole schedule would only move as fast as its slowest research refresh. Instead, the dispatcher publishes each claimed job to the RabbitMQ queue for its type and returns. Workers run the job, calculate the next next_run_at (in the user's timezone where that matters) and set the row back to PENDING. The AI and research workers scale horizontally without slowing the schedule down.
What changed
- One scheduled trigger instead of one per user, so service limits and per-trigger billing no longer grow with the number of users.
- Schedules are data: changing when a user gets their digest is an
UPDATE, not an infrastructure change. - Dispatchers can run in parallel without running a job twice.
The trade-off
This adds a polling and locking layer that an in-memory cron like node-cron doesn't have. But an in-memory cron lives inside one process, and with hundreds of tenants whose schedules keep changing, that wasn't an option. The jobs table also becomes something to monitor: a row that never leaves QUEUED means a job was lost between the dispatcher and the workers.
This started as a shorter LinkedIn post.