Sync Engine

Every learner action is written to the device first. The sync engine moves those writes to the server and brings back what other devices wrote. It is designed so that retries are always safe, nothing is lost offline, and merges never need a human.

Moving parts

flowchart LR
  subgraph DEVICE["Device"]
    UI[UI action] --> TX["one IndexedDB transaction<br/>table write + outbox row"]
    TX --> OB[("outbox<br/>lane 1: user data<br/>lane 2: analytics")]
    SE["sync engine<br/>backoff · batching"] --> OB
    SE --> CUR[("cursor in kv")]
  end
  subgraph SERVER["API"]
    PUSH["POST /v1/sync/push"]
    PULL["GET /v1/sync/pull"]
  end
  subgraph PG["Postgres"]
    SM[("sync_mutations<br/>id = client ULID")]
    DATA[("keeps · review_logs · attempts<br/>decks · user_settings …")]
    UC[("user_changes<br/>per-user sequence")]
  end
  SE -->|batches of 100<br/>server accepts ≤200| PUSH
  PUSH --> SM --> DATA --> UC
  SE -->|cursor| PULL
  PULL --> UC

Push

The engine runs every 20 seconds while the app is visible, when the network returns and when the app is hidden.

  1. It takes up to 100 mutations, the user-data lane first, analytics only when user data is empty (the server accepts up to 200).
  2. The API checks a per-person limit of 600 mutations a minute; above it, it answers 429 and the device simply tries again later.
  3. Each mutation is validated against the shared protocol package. User-data mutations are applied each in its own transaction: the id is claimed in sync_mutations first, and an id that already exists means it was applied before, so it is acknowledged and skipped.
  4. Analytics events are applied together: one transaction, one statement to claim all ids, one to insert the rows (with repeat views collapsed). If that fails, they fall back to one at a time.
  5. Changes other devices need are appended to user_changes with the next per-user sequence number.
  6. The response lists accepted and rejected ids and the head: the newest change written by any other device.

The device deletes both accepted and rejected ids from its outbox. A rejected mutation (invalid, or one the server could not apply) is logged and dropped, not retried.

warning Dropping rejected mutations keeps the outbox from jamming, but a mutation from a newer build that an older server can't apply yet is lost. A safer design is on the Roadmap.

Pull

The device pulls only when there is something to fetch:

flowchart TD
  A{head from push > local cursor?} -- yes --> P[pull]
  B[app opened or resumed] --> P
  C[5 minutes since last pull] --> P
  A -- no --> W[wait]
  P --> APPLY["apply changes from other devices<br/>(own device id skipped)"] --> CUR[advance cursor]

Pull pages through user_changes after the cursor, 500 changes at a time, skipping the device's own writes, so a device that was offline for a month catches up in a few requests. Deck changes are caught up separately through the decks routes.

warning The change feed is pruned after 180 days. A device that stays offline longer than that would resume from a cursor whose earlier changes are gone, and nothing tells it so. See Roadmap.

Why merges are conflict-free

DataShapeMerge rule
Keeps, attempts, review logsAppend-only logs with ULIDsUnion; duplicates collapse on id
Card stateNot stored; replayed from the logsSame logs → same state on every device
Decks, settingsSmall records with updatedAtNewest change wins, per record
DeletionsTombstones (removedAt, deleted)A tombstone is just a newer change

decision Logs, not state. Syncing logs and deriving state means two devices that reviewed the same card offline end up with the same schedule once both logs are merged, with nothing to reconcile.

Reviews made at the same moment on two devices are replayed in a fixed order (by time, then id), so every device computes exactly the same schedule.

Accounts and merging

A learner starts anonymous: the app gets a session at first launch and syncs immediately. Signing up upgrades that anonymous user in place. Signing in to an existing account from a device with its own progress asks "Put them together?", and nothing is merged or thrown away without that answer. On merge, POST /v1/auth/merge moves the device's progress into the account (de-duplicating keeps by box), copies it into the account's change feed so every device catches up, marks the learner for a learner-state refresh, and deletes the anonymous user. Discard just deletes it.

Failure handling

  • Exponential backoff up to five minutes on network errors, 5xx and 429; the app keeps working offline meanwhile.
  • Expired session? A 401 signs the device in again and retries once.
  • Batches are idempotent, so a timeout after the server committed is harmless: the retry is acknowledged as already applied.
  • The analytics lane can't starve user data. It is capped at 5,000 queued per type, and an event guard drops any event firing more than 10 times a second.
  • Schema drift is handled by additive mutation types: old servers reject unknown types cleanly and new servers accept old ones.

${sch.nav()}