Final results
Given a code change, pick the tests that can catch it. Run less and you win; miss a failure and it costs you.
Every fast CI system has to decide which tests a commit needs. Run them all and you wait; skip the wrong one and a bug ships.
You get a batch of code changes to sqlglot, a Python SQL parser. For each change, say which of its tests to run and how to split them across up to 8 workers.
Most changes break at least one test. Some break nothing at all, and the right answer for those is to run nothing.
Download blast-radius-sample.tar.gz below and unpack it into an empty folder (it has no top-level folder). You need Docker, make, and Python 3.9 or newer.
Run make sample to build the starter image, run it under the limits below on the 30 sample commits, and score what it printed.
Submit a public GitHub repository with a Dockerfile at its top, and the commit to score. Any language, any libraries, and any amount of AI are allowed.
Every change your program receives is a patch against sqlglot at commit eb11aec4be8cb2122afe713a4816662c3f0fb2fa. Each one stands alone.
A test fails on a commit when pytest reports a failure, an error, or a timeout for it, or when its file cannot be imported. A test with subtests is one test, and one failing subtest fails it.
A change that only adds a comment is not promised to be harmless.
For each commit, you pay the busiest worker's recorded test seconds plus 2 seconds per worker, and 300 seconds if you leave out any failing test; your program's CPU seconds are added across the batch. With your score you are told each run's simulated seconds, CPU seconds, and how many commits missed, never which.
At the cutoff, your chosen commit (your best, unless you pick another) is scored on unseen commits, including kinds of change not in practice. At least the top 5 entries are then scored again on those same commits, 5 runs each instead of 3, one at a time on one idle machine; that middle total replaces their first score.
The cutoff is .