Final results
Turn 60 million messy click-stream events into two exact daily Parquet tables. Fastest correct pipeline wins.
An online shop logs page views, add-to-carts, purchases, and ad clicks as CSV files. The logs are dirty, with duplicates, malformed lines, bad timestamps, and late events. One shopper can have many user IDs, and a third file links them across long chains and shared devices.
Clean the logs in /data, join each person's IDs, build sessions, credit each session to the ad click that brought the shopper in, count assisting campaigns from the week before, and write two daily Parquet tables (/output/user_daily/ and /output/campaign_daily/), one per person and one per campaign. The rules are exact so there is one right answer, and a fast answer that is almost right scores nothing.
Your score is container run time in seconds, and lower is better. You may submit 2 times an hour on hidden practice data (about 20 million events). The final runs on about 60 million events and caps each run at 3 minutes: if any final run goes over, you get no final score, so aim for under about 60 seconds in practice.
Download rules.md and a starter below (PySpark, DuckDB, or Polars). Any engine, any language, any amount of AI: Spark, DuckDB, Polars, Rust, C++, plain Python.
Each starter writes both tables with the right columns but no rows, so implementing the rules is up to you.
Submit a public GitHub repository with a Dockerfile at its root and the commit to score. Keep the repository at most 50 MB with no data checked in, because the dataset is mounted at run time.
We build your Dockerfile, check it on the dev set, and time it once in a fresh container on hidden practice data (about 20 million events). Runs under 20 seconds are timed 3 times and scored on the median.
At the cutoff, we rebuild your chosen commit (your best, unless you pick another) and time it 3 times on about 60 million new events made after submissions close, each run capped at 3 minutes. If any run goes over 3 minutes, you get no final score; otherwise your score is the median. At least the 30 fastest entries are then retimed 9 times in interleaved rounds across 3 fresh final datasets alone on one idle machine, and the mean of their per-dataset medians replaces their first score.
The cutoff is .