Analytics and Data Lake

Analytics are first-party: events travel through the same outbox as user data (on a lower-priority lane), land in Postgres, are summarised by SQL views, and are exported daily as Parquet for notebooks and model training.

flowchart LR
  CAT["events package<br/>name · props · PII class · retention"] --> APP[client]
  APP -->|analytics lane| PUSH[sync push]
  PUSH --> EV[("events<br/>monthly partitions")]
  PUSH --> EXP[("exposures")]
  PUSH --> DEC[("decision_logs")]
  EV & EXP & DEC --> VIEWS["analysis views"]
  EV & EXP & DEC --> LAKE["lake-export job"] --> PQ[("Parquet on R2<br/>by day")]
  PQ --> NB[notebooks · model training]

The event catalogue

Every event is declared once in the events package with its properties, a PII class and a retention period. The client can only send declared events, and the server drops unknown ones. Typed search queries are scrubbed before they are stored.

Storage

  • events, decision_logs and exposures are partitioned by month. Upkeep creates partitions ahead of time; the retention job detaches and drops whole partitions past the longest retention (no row scans) and deletes only short-lived events by age. /statusz flags rows landing in the default partition.
  • Events carry a pseudonym (an HMAC of the user id and a per-user salt) rather than the account id. The salt is erased with the account, which unlinks exported history.

Analysis views

QuestionViews
Are people coming back?daily_active, retention, layer_reach, layer_time
Where do boxes lose people?box_dropoff, lesson_steps, screen_flow
What is hard?confusing_words, try_quality, topic_mastery, review_recall
What do people want?search_quality, content_requests
Is the planner any good?decision_performance
Are builds healthy?build_adoption, build_health, jobs
Daily puzzlesgames

The lake

  • The lake-export job writes events, decisions, exposures and pseudonymised learning tables (attempts and the like) as Parquet, hive-partitioned by day, and records row counts in lake_exports. Missed days are caught up, and days whose rows changed through late syncs are exported again.
  • A research_consent table in the lake holds each learner's latest choice. Anything that trains on learner data joins on the pseudonym and keeps only research_granted = true.

decision Parquet on object storage, not a warehouse. At this size a warehouse would cost more than the whole platform. Columnar files on R2 are read directly by DuckDB or Python, cost cents, and can be loaded into a warehouse later without changing the export.

scale When daily volume outgrows Postgres, events can be written straight to the lake by the edge (with Postgres keeping only recent days), without changing the client or the catalogue.

${sch.nav()}