When the expensive complete pass cannot get faster, the transformation is to give a cheaper, more frequent pass the right to choose what to run. Google moved postsubmit ordering from 'all affected tests when capacity allows' to a ranked speculative slice, and median novel-breakage detection fell from 107 minutes to 37.
AI value · Median (p50) time to detect novel breaking targets
Verified
37 minutes with Speculative Cycles (approximately 65% / 70-minute reduction)
Comparison is to the prior comprehensive scheduler on empirically observed production timings at a 25% target budget, not a randomized controlled experiment.
Before
Submits a change after limited presubmit testing. → Waits for capacity, then runs all tests affected since the last definitive repository status. → Triage the first detected failure, identify the culprit change, and roll back when evidence thresholds are met.
After
Submits a change after limited presubmit testing. → Every ~20 minutes, after static affected-target filtering, scores targets for pass-to-fail transition risk and ranks. → Schedules the top-risk targets within a 25% test budget to find novel breakages sooner. → Consume earlier failure signals for investigation and…
Human boundary
Transition Prediction decides speculative scheduling order within a human-set budget. Humans retain rollback policy, feature/training configuration, and the continued existence of comprehensive cycles. The model does not skip required comprehensive coverage by itself.
Why it matters
Novel breakages need not wait for a comprehensive post-submit test cycle.
How the work changed
Before
How the work ran before the change.
Step 1 of 3
Developer
Submits a change after limited presubmit testing.
ControlPresubmit checks and code review; comprehensive testing is deferred
Step 2 of 3
TAP Postsubmit comprehensive cycle
Waits for capacity, then runs all tests affected since the last definitive repository status.
ControlStatic build-graph dependence; no ML ranking of which tests to run first
Step 3 of 3
Culprit finding / autorollback / developers
Triage the first detected failure, identify the culprit change, and roll back when evidence thresholds are met.
ControlAutorollback currently requires at least 10 distinct failing targets
What changed
Novel breakages need not wait for a comprehensive post-submit test cycle.
Decision rightSelection moves from a fixed rule to the model
After
How the same work runs now.
Step 1 of 4
Developer
Submits a change after limited presubmit testing.
ControlSame presubmit and review gates
Step 2 of 4
Transition Prediction (TRANSPRED)
Every ~20 minutes, after static affected-target filtering, scores targets for pass-to-fail transition risk and ranks.
ControlShallow gradient-boosted trees on coarse metadata; scores are ranks, not calibrated probabilities
Step 3 of 4
Speculative Cycle scheduler
Schedules the top-risk targets within a 25% test budget to find novel breakages sooner.
ControlFixed top-k cost cap; comprehensive cycles still run
Step 4 of 4
Culprit finding / autorollback / developers
Consume earlier failure signals for investigation and rollback.
Process model built from the published workflow evidence for Google. Every step, actor, and control appears in full below.Every step, actor, and control
Exception path
Missed breakages wait for the next comprehensive cycle. Flaky or low-evidence culprits are filtered by culprit verification. Late speculative detections that do not beat comprehensive timing still fall back to the existing gardening workflow.
Decision authority
Transition Prediction decides speculative scheduling order within a human-set budget. Humans retain rollback policy, feature/training configuration, and the continued existence of comprehensive cycles. The model does not skip required comprehensive coverage by itself.
Before
#
Actor
Action
Control
01
Developer
Submits a change after limited presubmit testing.
Presubmit checks and code review; comprehensive testing is deferred
02
TAP Postsubmit comprehensive cycle
Waits for capacity, then runs all tests affected since the last definitive repository status.
Static build-graph dependence; no ML ranking of which tests to run first
03
Culprit finding / autorollback / developers
Triage the first detected failure, identify the culprit change, and roll back when evidence thresholds are met.
Autorollback currently requires at least 10 distinct failing targets
After
#
Actor
Action
Control
01
Developer
Submits a change after limited presubmit testing.
Same presubmit and review gates
02
Transition Prediction (TRANSPRED)
Every ~20 minutes, after static affected-target filtering, scores targets for pass-to-fail transition risk and ranks them.
Shallow gradient-boosted trees on coarse metadata; scores are ranks, not calibrated probabilities
03
Speculative Cycle scheduler
Schedules the top-risk targets within a 25% test budget to find novel breakages sooner.
Fixed top-k cost cap; comprehensive cycles still run
04
Culprit finding / autorollback / developers
Consume earlier failure signals for investigation and rollback.
Same autorollback evidence threshold (AtLeast 10 targets)
Work that left the path
Waiting solely on the next capacity-gated comprehensive cycle to surface a novel breakage
Treating all affected tests as equal priority in the first postsubmit slice
Human role before
TAP Postsubmit used capacity-gated comprehensive batching of all affected tests. Developers did not set per-change test execution order; they waited on hour-scale postsubmit signal.
Human role after
Humans set scheduler policy (top-k budget, rollback thresholds, model refresh). The ML scheduler owns which postsubmit tests run in the fast speculative slice.
AI role
Predict which targets are newly likely to fail and feed a ranked top-k speculative cycle so novel breakages are found before the next comprehensive cycle.
Outcomes
Median (p50) time to detect novel breaking targets
Verified
107 minutes with comprehensive cycles only→37 minutes with Speculative Cycles (approximately 65% / 70-minute reduction)
Three months of production TAP Postsubmit data contemporaneous with the 2025 study · 120 billion test×cycle pairs, 7.7 million breaking targets, ~20,000 unique breakages
Comparison is to the prior comprehensive scheduler on empirically observed production timings at a 25% target budget, not a randomized controlled experiment. The paper says Google 'is implementing' Speculative Cycles; the evaluated production-data window does not state the fleet fraction live during every day of the three months. The Google Research abstract reports an approximately 70-minute median reduction; the ICST paper states 65% from 107 to 37 minutes, which is the same difference.
What leaders can reuse
Anti-pattern
Claiming that developers used to pick test order by hand, or reporting a 70-minute win without stating the 107-to-37-minute comprehensive-cycle baseline and the 25% budget.
Questions
01What is the first pass that is allowed to be incomplete in order to be faster?
02If the model is wrong, does a slower complete cycle still run, or did we just create silent coverage gaps?
03Is the outcome median time-to-detect on production traffic, or a replay against a synthetic scheduler?
Portability conditions
A high-accuracy history of which commits actually broke which targets, not just which tests failed
Enough volume that skipping 75% of a cycle is still cheaper than buying more machines
A remaining comprehensive cycle so missed breakages still surface
Human-set rollback evidence thresholds the speculative slice can actually meet
Reputation risk
low
Evidence and authority
What the public record supports.
Current · updated
1 peer reviewed, 1 primary; publication outcomes are verified.
Bundle 1.0.0 · reviewed 2026-09-06 · stable ID dd6d683bf4ef1237
