Background Jobs

Background work runs in a separate process built from the same codebase as the API, on its own machine, so a slow job can never slow down a request.

Scheduler

sequenceDiagram
  participant T as Tick (every minute)
  participant J as job_runs
  participant W as Job
  T->>J: which jobs are due?
  T->>J: update job_runs set locked_until<br/>where due and lease free
  alt lease taken
    T->>W: run with a deadline
    W-->>J: finish: status, duration, counts
  else another process holds it
    T-->>T: skip
  end

A job runs only while it holds a lease row with an expiry, so even two jobs processes can never run the same job at once.
A crashed run's lease simply expires; the next tick picks it up.
Every run is recorded in job_runs with its start, finish, outcome and last error. /statusz returns 503 when a job hasn't finished within twice its interval plus ten minutes, so the uptime check catches stuck jobs.

Jobs

Job	Every	What it does
`nudges`	5 minutes	At most one nudge a day, at the learner's chosen time: skipped if they were already active today or nothing is due. The nudge is recorded before sending, so a crash can never double-send.
`learner-state`	Hour (incremental)	Recomputes card summaries and mastery for learners with new learning data, plus those marked dirty
`content-sync`	Hour	Mirrors the live catalog into `content_boxes`, so analytics and reports can join on box metadata
`behaviour-interests`	Day	Derives interests from what learners open, keep and finish, next to the interests they chose
`retention`	Day	The data lifecycle: drops whole monthly partitions of events, decisions and exposures past the longest retention; deletes short-retention events (search text, web vitals) by age; prunes sync ids after 30 days and the change feed after 180
`upkeep`	Day	Creates the coming months' analytics partitions; erases accounts past their 7-day undo window; drops expired codes and old sessions
`lake-export`	Day	Writes each day's pseudonymous analytics to Parquet on R2, catching up missed days (see Analytics and Data Lake)

Push delivery

flowchart LR
  N[nudges job] --> Q{device kind}
  Q -- iOS --> APNS[APNs<br/>token auth]
  Q -- Android --> FCM[FCM v1]
  Q -- browser --> WP[Web Push<br/>VAPID]
  APNS & FCM & WP --> R{response}
  R -- gone / unregistered --> DEL[delete token]
  R -- ok --> LOG[done · nudge already recorded]

Dead tokens are removed on the first "gone" answer, so the device lists stay clean without a separate sweep.

scale Jobs are written to work in batches with a cursor and to be safe to re-run. Splitting a job across several workers later only needs a partition key on the lease, not a rewrite.

${sch.nav()}