Observability

flowchart LR
  subgraph CLIENT["Clients"]
    E1[errors] --> S[(Sentry)]
    H[app_started health event<br/>Core Web Vitals] --> EV[(events)]
    G[event guard drops] --> EV
  end
  subgraph SERVER["API and jobs"]
    E2[errors] --> S
    M[Prometheus metrics] --> FLY[Fly metrics]
    J[job_runs] --> SZ["/statusz"]
  end
  UP[uptime checks] --> HZ["/healthz · /readyz"]
  S & FLY & UP --> AL[alerts]
  EV --> VIEWS["build_health · build_adoption · jobs"]

Signals

SignalSourceAnswers
Errors with readable stack tracesSentry (web, API), source maps uploaded on deployWhat broke, in which build, for how many
Request rate, latency, status codesPrometheus metrics on an internal portIs the API healthy and fast?
Cold start and content timeapp_started health eventIs startup getting slower? Did content come from the network or the device?
Core Web VitalsFirst-party eventsIs the web app fast for real users?
Event guard dropsHealth eventIs something stuck in a render loop?
Job runsjob_runs, /statusz, jobs viewDid every job run on time?
Build adoption and healthbuild_adoption, build_health viewsAre learners on new builds, and are those builds healthy?
UptimeExternal checks on health endpointsIs the site up from outside?

Alerts

  • A new or regressed error issue in Sentry.
  • Uptime check failures on the website or API health endpoints.
  • Jobs that stop finishing show on /statusz and in the jobs view.

note Client errors and health events are tagged with the build number, so a bad build shows up as a step change in build_health, and the fix can be a content switch (raise min, change a remote switch) instead of waiting for store review.

Runbook pointers

SymptomFirst look
App won't load new contentManifest signature and key id; edge cache of the manifest
Sync errors spikeSentry for /v1/sync/push; rejected mutation types from an old or new build
Reminders stop/statusz for the nudges job; push provider responses
Slow APIDatabase pool saturation; slow queries; Fly machine health

${sch.nav()}